L7-Aware Traffic Management
Cilium provides a way to control L7 traffic via CRDs (e.g. CiliumEnvoyConfig and CiliumClusterwideEnvoyConfig).
Prerequisites
Cilium must be configured with NodePort enabled, using
nodePort.enabled=true
or by enabling the kube-proxy replacement withkubeProxyReplacement=true
. For more information, see kube-proxy replacement.
Caveats
CiliumEnvoyConfig
resources have only minimal validation performed, and do not have a defined conflict resolution behavior. This means that if you create multiple CECs that modify the same parts of Envoy’s config, the results may be unpredictable.In addition to this minimal validation,
CiliumEnvoyConfig
has minimal feedback to the user about the correctness of the configuration. So in the event a CEC does produce an undesirable outcome, troubleshooting will require inspecting the Envoy config and logs, rather than being able to look at theCiliumEnvoyConfig
in question.CiliumEnvoyConfig
is used by Cilium’s Ingress and Gateway API support to direct traffic through the per-node Envoy proxies. If you create CECs that conflict with or modify the autogenerated config, results may be unpredictable. Be very careful using CECs for these use cases. The above risks are managed by ensuring that all config generated by Cilium is semantically valid, as far as possible.
Installation
Cilium Ingress Controller can be enabled with helm flag ingressController.enabled
set as true. Please refer to Installation using Helm for a fresh installation.
$ helm upgrade cilium cilium/cilium --version 1.14.2 \ --namespace kube-system \ --reuse-values \ --set ingressController.enabled=true \ --set ingressController.loadbalancerMode=dedicated $ kubectl -n kube-system rollout restart deployment/cilium-operator $ kubectl -n kube-system rollout restart ds/cilium
Cilium can become the default ingress controller by setting the
--set ingressController.default=true
flag. This will create ingress entries even when the ingressClass
is not set.
If you only want to use envoy traffic management feature without Ingress support, you should only
enable --enable-envoy-config
flag.
$ helm upgrade cilium cilium/cilium --version 1.14.2 \ --namespace kube-system \ --reuse-values \ --set-string extraConfig.enable-envoy-config=true $ kubectl -n kube-system rollout restart deployment/cilium-operator $ kubectl -n kube-system rollout restart ds/cilium
Additionally, the proxy load-balancing feature can be configured with the loadBalancer.l7.backend=envoy
flag.
$ helm upgrade cilium cilium/cilium --version 1.14.2 \ --namespace kube-system \ --reuse-values \ --set loadBalancer.l7.backend=envoy $ kubectl -n kube-system rollout restart deployment/cilium-operator $ kubectl -n kube-system rollout restart ds/cilium
Next you can check the status of the Cilium agent and operator:
$ cilium status
Warning
Make sure you install cilium-cli v0.15.0 or later. The rest of instructions do not work with older versions of cilium-cli. To confirm the cilium-cli version that’s installed in your system, run:
cilium version --client
See Cilium CLI upgrade notes for more details.
Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "arm64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
shasum -a 256 -c cilium-darwin-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-darwin-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
See the full page of releases.
Warning
Make sure you install cilium-cli v0.15.0 or later. The rest of instructions do not work with older versions of cilium-cli. To confirm the cilium-cli version that’s installed in your system, run:
cilium version --client
See Cilium CLI upgrade notes for more details.
Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "arm64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
shasum -a 256 -c cilium-darwin-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-darwin-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
See the full page of releases.
Cilium Ingress Controller can be enabled with the below command
$ cilium install --version 1.14.2 --set kubeProxyReplacement=true --set ingressController.enabled=true --set ingressController.loadbalancerMode=dedicated
Cilium can become the default ingress controller by setting the
--set ingressController.default=true
flag. This will create ingress entries even when the ingressClass
is not set.
If you only want to use envoy traffic management feature without Ingress support, you should only
enable --enable-envoy-config
flag.
$ cilium install --version 1.14.2 --set kubeProxyReplacement=true --set-string extraConfig.enable-envoy-config=true
Additionally, the proxy load-balancing feature can be configured with the loadBalancer.l7.backend=envoy
flag.
$ cilium install --version 1.14.2 --set kubeProxyReplacement=true --set-string extraConfig.enable-envoy-config=true --set loadBalancer.l7.backend=envoy
Next you can check the status of the Cilium agent and operator:
$ cilium status
Hubble CLI is also used to observe the traffic in later steps.
Download the latest hubble release:
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
Download the latest hubble release:
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "arm64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
shasum -a 256 -c hubble-darwin-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-darwin-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-darwin-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
Download the latest hubble release:
curl -LO "https://raw.githubusercontent.com/cilium/hubble/master/stable.txt"
set /p HUBBLE_VERSION=<stable.txt
curl -L --fail -O "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz"
curl -L --fail -O "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz.sha256sum"
certutil -hashfile hubble-windows-amd64.tar.gz SHA256
type hubble-windows-amd64.tar.gz.sha256sum
:: verify that the checksum from the two commands above match
tar zxf hubble-windows-amd64.tar.gz
and move the hubble.exe
CLI to a directory listed in the %PATH%
environment variable after
extracting it from the tarball.
Supported Envoy API Versions
As of now only the Envoy API v3 is supported.
Supported Envoy Extension Resource Types
Envoy extensions are resource types that may or may not be built in to
an Envoy build. The standard types referred to in Envoy documentation,
such as type.googleapis.com/envoy.config.listener.v3.Listener
, and
type.googleapis.com/envoy.config.route.v3.RouteConfiguration
, are
always available.
Cilium nodes deploy an Envoy image to support Cilium HTTP policy enforcement and observability. This build of Envoy has been optimized for the needs of the Cilium Agent and does not contain many of the Envoy extensions available in the Envoy code base.
To see which Envoy extensions are available, please have a look at
the Envoy extensions configuration
file.
Only the extensions that have not been commented out with #
are
built in to the Cilium Envoy image. Currently this contains the
following extensions:
envoy.clusters.dynamic_forward_proxy
envoy.filters.http.dynamic_forward_proxy
envoy.filters.http.ext_authz
envoy.filters.http.jwt_authn
envoy.filters.http.local_ratelimit
envoy.filters.http.oauth2
envoy.filters.http.ratelimit
envoy.filters.http.router
envoy.filters.http.set_metadata
envoy.filters.listener.tls_inspector
envoy.filters.network.connection_limit
envoy.filters.network.ext_authz
envoy.filters.network.http_connection_manager
envoy.filters.network.local_ratelimit
envoy.filters.network.mongo_proxy
envoy.filters.network.mysql_proxy
envoy.filters.network.ratelimit
envoy.filters.network.tcp_proxy
envoy.filters.network.sni_cluster
envoy.filters.network.sni_dynamic_forward_proxy
envoy.stat_sinks.metrics_service
envoy.transport_sockets.raw_buffer
envoy.upstreams.http.http
envoy.upstreams.http.tcp
We will evolve the list of built-in extensions based on user feedback.
Examples
Please refer to one of the below examples on how to use and leverage Cilium’s Ingress features: