L7-Aware Traffic Management

Cilium provides a way to control L7 traffic via CRDs (e.g. CiliumEnvoyConfig and CiliumClusterwideEnvoyConfig).

Prerequisites

• Cilium must be configured with NodePort enabled, using nodePort.enabled=true or by enabling the kube-proxy replacement with kubeProxyReplacement=true. For more information, see kube-proxy replacement.

Caveats

• CiliumEnvoyConfig resources have only minimal validation performed, and do not have a defined conflict resolution behavior. This means that if you create multiple CECs that modify the same parts of Envoy’s config, the results may be unpredictable.

• In addition to this minimal validation, CiliumEnvoyConfig has minimal feedback to the user about the correctness of the configuration. So in the event a CEC does produce an undesirable outcome, troubleshooting will require inspecting the Envoy config and logs, rather than being able to look at the CiliumEnvoyConfig in question.

• CiliumEnvoyConfig is used by Cilium’s Ingress and Gateway API support to direct traffic through the per-node Envoy proxies. If you create CECs that conflict with or modify the autogenerated config, results may be unpredictable. Be very careful using CECs for these use cases. The above risks are managed by ensuring that all config generated by Cilium is semantically valid, as far as possible.

Installation

HelmCilium CLI

Cilium Ingress Controller can be enabled with helm flag ingressController.enabled set as true. Please refer to Installation using Helm for a fresh installation.

$ helm upgrade cilium cilium/cilium --version 1.14.2 \
    --namespace kube-system \
    --reuse-values \
    --set ingressController.enabled=true \
    --set ingressController.loadbalancerMode=dedicated
$ kubectl -n kube-system rollout restart deployment/cilium-operator
$ kubectl -n kube-system rollout restart ds/cilium

Cilium can become the default ingress controller by setting the --set ingressController.default=true flag. This will create ingress entries even when the ingressClass is not set.

If you only want to use envoy traffic management feature without Ingress support, you should only enable --enable-envoy-config flag.

$ helm upgrade cilium cilium/cilium --version 1.14.2 \
    --namespace kube-system \
    --reuse-values \
    --set-string extraConfig.enable-envoy-config=true
$ kubectl -n kube-system rollout restart deployment/cilium-operator
$ kubectl -n kube-system rollout restart ds/cilium

Additionally, the proxy load-balancing feature can be configured with the loadBalancer.l7.backend=envoy flag.

$ helm upgrade cilium cilium/cilium --version 1.14.2 \
    --namespace kube-system \
    --reuse-values \
    --set loadBalancer.l7.backend=envoy
$ kubectl -n kube-system rollout restart deployment/cilium-operator
$ kubectl -n kube-system rollout restart ds/cilium

Next you can check the status of the Cilium agent and operator:

$ cilium status

Warning

Make sure you install cilium-cli v0.15.0 or later. The rest of instructions do not work with older versions of cilium-cli. To confirm the cilium-cli version that’s installed in your system, run:

cilium version --client

See Cilium CLI upgrade notes for more details.

Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).

LinuxmacOSOther

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

Hubble CLI is also used to observe the traffic in later steps.

LinuxMacOSWindows

Download the latest hubble release:

HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}

Supported Envoy API Versions

As of now only the Envoy API v3 is supported.

Supported Envoy Extension Resource Types

Envoy extensions are resource types that may or may not be built in to an Envoy build. The standard types referred to in Envoy documentation, such as type.googleapis.com/envoy.config.listener.v3.Listener, and type.googleapis.com/envoy.config.route.v3.RouteConfiguration, are always available.

Cilium nodes deploy an Envoy image to support Cilium HTTP policy enforcement and observability. This build of Envoy has been optimized for the needs of the Cilium Agent and does not contain many of the Envoy extensions available in the Envoy code base.

To see which Envoy extensions are available, please have a look at the Envoy extensions configuration file. Only the extensions that have not been commented out with # are built in to the Cilium Envoy image. Currently this contains the following extensions:

• envoy.clusters.dynamic_forward_proxy

• envoy.filters.http.dynamic_forward_proxy

• envoy.filters.http.ext_authz

• envoy.filters.http.jwt_authn

• envoy.filters.http.local_ratelimit

• envoy.filters.http.oauth2

• envoy.filters.http.ratelimit

• envoy.filters.http.router

• envoy.filters.http.set_metadata

• envoy.filters.listener.tls_inspector

• envoy.filters.network.connection_limit

• envoy.filters.network.ext_authz

• envoy.filters.network.http_connection_manager

• envoy.filters.network.local_ratelimit

• envoy.filters.network.mongo_proxy

• envoy.filters.network.mysql_proxy

• envoy.filters.network.ratelimit

• envoy.filters.network.tcp_proxy

• envoy.filters.network.sni_cluster

• envoy.filters.network.sni_dynamic_forward_proxy

• envoy.stat_sinks.metrics_service

• envoy.transport_sockets.raw_buffer

• envoy.upstreams.http.http

• envoy.upstreams.http.tcp

We will evolve the list of built-in extensions based on user feedback.

Examples

Please refer to one of the below examples on how to use and leverage Cilium’s Ingress features:

• L7 Path Translation
• L7 Load Balancing and URL re-writing
• Proxy Load Balancing for Kubernetes Services (beta)