L7-Aware Traffic Management

Cilium provides a way to control L7 traffic via CRDs (e.g. CiliumEnvoyConfig and CiliumClusterwideEnvoyConfig).

Prerequisites

• Cilium must be configured with the kube-proxy replacement, using kubeProxyReplacement=true. For more information, see kube-proxy replacement.

Caveats

• CiliumEnvoyConfig resources have only minimal validation performed, and do not have a defined conflict resolution behavior. This means that if you create multiple CECs that modify the same parts of Envoy’s config, the results may be unpredictable.

• In addition to this minimal validation, CiliumEnvoyConfig has minimal feedback to the user about the correctness of the configuration. So in the event a CEC does produce an undesirable outcome, troubleshooting will require inspecting the Envoy config and logs, rather than being able to look at the CiliumEnvoyConfig in question.

• CiliumEnvoyConfig and CiliumClusterwideEnvoyConfig resources are implementation details. Both should be treated as cluster admin resources and permissions for their configuration should not be delegated to other users.

• CiliumEnvoyConfig is used by Cilium’s Ingress and Gateway API support to direct traffic through the per-node Envoy proxies. If you create CECs that conflict with or modify the autogenerated config, results may be unpredictable. Be very careful using CECs for these use cases. The above risks are managed by ensuring that all config generated by Cilium is semantically valid, as far as possible.

• If you create a CiliumEnvoyConfig resource directly (ie, not via the Cilium Ingress or Gateway API controllers), if the CEC is intended to manage E/W traffic, set the annotation cec.cilium.io/use-original-source-address: "false". Otherwise, Envoy will bind the sockets for the upstream connection pools to the original source address/port. This may cause 5-tuple collisions when pods send multiple requests over the same pipelined HTTP/1.1 or HTTP/2 connection. (The Cilium agent assumes all CECs with parentRefs pointing to the Cilium Ingress or Gateway API controllers have annotation cec.cilium.io/use-original-source-address set to "false", but all other CECs are assumed to have this annotation set to "true".)

Installation

HelmCilium CLI

Cilium Ingress Controller can be enabled with helm flag ingressController.enabled set as true. Please refer to Installation using Helm for a fresh installation.

helm upgrade cilium ./cilium \
   --namespace kube-system \
   --reuse-values \
   --set ingressController.enabled=true \
   --set ingressController.loadbalancerMode=dedicated
kubectl -n kube-system rollout restart deployment/cilium-operator
kubectl -n kube-system rollout restart ds/cilium

Cilium can become the default ingress controller by setting the --set ingressController.default=true flag. This will create ingress entries even when the ingressClass is not set.

If you only want to use envoy traffic management feature without Ingress support, you should only enable --enable-envoy-config flag.

helm upgrade cilium ./cilium \
   --namespace kube-system \
   --reuse-values \
   --set envoyConfig.enabled=true
kubectl -n kube-system rollout restart deployment/cilium-operator
kubectl -n kube-system rollout restart ds/cilium

Additionally, the proxy load-balancing feature can be configured with the loadBalancer.l7.backend=envoy flag.

helm upgrade cilium ./cilium \
   --namespace kube-system \
   --reuse-values \
   --set loadBalancer.l7.backend=envoy
kubectl -n kube-system rollout restart deployment/cilium-operator
kubectl -n kube-system rollout restart ds/cilium

Next you can check the status of the Cilium agent and operator:

$ cilium status

Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).

LinuxmacOSOther

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

Clone the Cilium GitHub repository so that the Cilium CLI can access the latest unreleased Helm chart from the main branch:

git clone git@github.com:cilium/cilium.git
cd cilium

It is also recommended that you install Hubble CLI which will be used used to observe the traffic in later steps.

Supported Envoy API Versions

As of now only the Envoy API v3 is supported.

Supported Envoy Extension Resource Types

Envoy extensions are resource types that may or may not be built in to an Envoy build. The standard types referred to in Envoy documentation, such as type.googleapis.com/envoy.config.listener.v3.Listener, and type.googleapis.com/envoy.config.route.v3.RouteConfiguration, are always available.

Cilium nodes deploy an Envoy image to support Cilium HTTP policy enforcement and observability. This build of Envoy has been optimized for the needs of the Cilium Agent and does not contain many of the Envoy extensions available in the Envoy code base.

To see which Envoy extensions are available, please have a look at the Envoy extensions configuration file. Only the extensions that have not been commented out with # are built in to the Cilium Envoy image. We will evolve the list of built-in extensions based on user feedback.

Examples

Please refer to one of the below examples on how to use and leverage Cilium’s Ingress features:

• L7 Path Translation
• L7 Load Balancing and URL re-writing
• L7 Circuit Breaking
• Proxy Load Balancing for Kubernetes Services (beta)
• L7 Traffic Shifting