Kubernetes Ingress Support

Cilium uses the standard Kubernetes Ingress resource definition, with an ingressClassName of cilium. This can be used for path-based routing and for TLS termination. For backwards compatibility, the kubernetes.io/ingress.class annotation with value of cilium is also supported.

Note

The ingress controller creates a service of LoadBalancer type, so your environment will need to support this.

Cilium allows you to specify load balancer mode for the Ingress resource:

  • dedicated: The Ingress controller will create a dedicated loadbalancer for the Ingress.

  • shared: The Ingress controller will use a shared loadbalancer for all Ingress resources.

Each load balancer mode has its own benefits and drawbacks. The shared mode saves resources by sharing a single LoadBalancer config across all Ingress resources in the cluster, while the dedicated mode can help to avoid potential conflicts (e.g. path prefix) between resources.

Note

It is possible to change the load balancer mode for an Ingress resource. When the mode is changed, active connections to backends of the Ingress may be terminated during the reconfiguration due to a new load balancer IP address being assigned to the Ingress resource.

This is a step-by-step guide on how to enable the Ingress Controller in an existing K8s cluster with Cilium installed.

Prerequisites

  • Cilium must be configured with NodePort enabled, using nodePort.enabled=true or by enabling the kube-proxy replacement with kubeProxyReplacement=true. For more information, see kube-proxy replacement.

  • Cilium must be configured with the L7 proxy enabled using l7Proxy=true (enabled by default).

Installation

Cilium Ingress Controller can be enabled with helm flag ingressController.enabled set as true. Please refer to Installation using Helm for a fresh installation.

$ helm upgrade cilium ./cilium \
    --namespace kube-system \
    --reuse-values \
    --set ingressController.enabled=true \
    --set ingressController.loadbalancerMode=dedicated
$ kubectl -n kube-system rollout restart deployment/cilium-operator
$ kubectl -n kube-system rollout restart ds/cilium

Cilium can become the default ingress controller by setting the --set ingressController.default=true flag. This will create ingress entries even when the ingressClass is not set.

If you only want to use envoy traffic management feature without Ingress support, you should only enable --enable-envoy-config flag.

$ helm upgrade cilium ./cilium \
    --namespace kube-system \
    --reuse-values \
    --set envoyConfig.enabled=true
$ kubectl -n kube-system rollout restart deployment/cilium-operator
$ kubectl -n kube-system rollout restart ds/cilium

Additionally, the proxy load-balancing feature can be configured with the loadBalancer.l7.backend=envoy flag.

$ helm upgrade cilium ./cilium \
    --namespace kube-system \
    --reuse-values \
    --set loadBalancer.l7.backend=envoy
$ kubectl -n kube-system rollout restart deployment/cilium-operator
$ kubectl -n kube-system rollout restart ds/cilium

Next you can check the status of the Cilium agent and operator:

$ cilium status

Warning

Make sure you install cilium-cli v0.15.0 or later. The rest of instructions do not work with older versions of cilium-cli. To confirm the cilium-cli version that’s installed in your system, run:

cilium version --client

See Cilium CLI upgrade notes for more details.

Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

Clone the Cilium GitHub repository so that the Cilium CLI can access the latest unreleased Helm chart from the main branch:

git clone git@github.com:cilium/cilium.git
cd cilium

Hubble CLI is also used to observe the traffic in later steps.

Download the latest hubble release:

HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}

Supported Ingress Annotations

Name

Description

Default Value

ingress.cilium.io/loadbalancer-mode
The loadbalancer mode for the ingress.
Allows a per ingress override
of the default set in the Helm value
ingressController.loadbalancerMode.
Applicable values are dedicated and
shared.
dedicated
(from Helm chart)

ingress.cilium.io/service-type
The Service type for dedicated Ingress.
Applicable values are LoadBalancer
and NodePort.

LoadBalancer

ingress.cilium.io/insecure-node-port
The NodePort to use for the HTTP Ingress.
Applicable only if ingress.cilium.io/service-type is NodePort. If unspecified, a random
NodePort will be allocated by kubernetes.

unspecified

ingress.cilium.io/secure-node-port
The NodePort to use for the HTTPS Ingress.
Applicable only if ingress.cilium.io/service-type is NodePort. If unspecified, a random
NodePort will be allocated by kubernetes.

unspecified

ingress.cilium.io/tcp-keep-alive
Enable TCP keep-alive. Applicable values
are enabled and disabled.

enabled

ingress.cilium.io/tcp-keep-alive-idle

TCP keep-alive idle time (in seconds)

10

ingress.cilium.io/tcp-keep-alive-probe-interval

TCP keep-alive probe intervals (in seconds)

5

ingress.cilium.io/tcp-keep-alive-probe-max-failures

TCP keep-alive probe max failures

10

ingress.cilium.io/websocket
Enable websocket passthrough support.
Applicable values are enabled and disabled.

disabled

ingress.cilium.io/tls-passthrough
Enable TLS Passthrough mode for this Ingress.
Applicable values are enabled and disabled,
although boolean-style values will also be
accepted.

Note that some conditions apply to TLS
Passthrough Ingresses, due to how
TLS Passthrough works:
* A host field must be set in the Ingress
* Default backends are ignored
* Rules with paths other than / are ignored
If all the rules in an Ingress are ignored for
these reasons, no Envoy config will be generated
and the Ingress will have no effect.

Note that this annotation is analogous to
the ssl-passthrough on other Ingress
controllers.

disabled

Additionally, cloud-provider specific annotations for the LoadBalancer service are supported.

By default, annotations with values beginning with:

  • service.beta.kubernetes.io

  • service.kubernetes.io

  • cloud.google.com

will be copied from an Ingress object to the generated LoadBalancer service objects.

This setting is controlled by the Cilium Operator’s ingress-lb-annotation-prefixes config flag, and can be configured in Cilium’s Helm values.yaml using the ingressController.ingressLBAnnotationPrefixes setting.

Please refer to the Kubernetes documentation for more details.

Examples

Please refer to one of the below examples on how to use and leverage Cilium’s Ingress features: