Kubernetes Ingress Support
Cilium uses the standard Kubernetes Ingress resource definition, with
an ingressClassName
of cilium
. This can be used for path-based
routing and for TLS termination. For backwards compatibility, the
kubernetes.io/ingress.class
annotation with value of cilium
is also supported.
Note
The ingress controller creates a Service of LoadBalancer type, so your environment will need to support this.
Cilium allows you to specify load balancer mode for the Ingress resource:
dedicated
: The Ingress controller will create a dedicated loadbalancer for the Ingress.shared
: The Ingress controller will use a shared loadbalancer for all Ingress resources.
Each load balancer mode has its own benefits and drawbacks. The shared mode saves resources by sharing a single LoadBalancer config across all Ingress resources in the cluster, while the dedicated mode can help to avoid potential conflicts (e.g. path prefix) between resources.
Note
It is possible to change the load balancer mode for an Ingress resource. When the mode is changed, active connections to backends of the Ingress may be terminated during the reconfiguration due to a new load balancer IP address being assigned to the Ingress resource.
This is a step-by-step guide on how to enable the Ingress Controller in an existing K8s cluster with Cilium installed.
Prerequisites
Cilium must be configured with NodePort enabled, using
nodePort.enabled=true
or by enabling the kube-proxy replacement withkubeProxyReplacement=true
. For more information, see kube-proxy replacement.Cilium must be configured with the L7 proxy enabled using
l7Proxy=true
(enabled by default).By default, the Ingress controller creates a Service of LoadBalancer type, so your environment will need to support this. Alternatively, you can change this to NodePort or, since Cilium 1.16+, directly expose the Cilium L7 proxy on the host network.
Installation
Cilium Ingress Controller can be enabled with helm flag ingressController.enabled
set as true. Please refer to Installation using Helm for a fresh installation.
$ helm upgrade cilium ./cilium \ --namespace kube-system \ --reuse-values \ --set ingressController.enabled=true \ --set ingressController.loadbalancerMode=dedicated $ kubectl -n kube-system rollout restart deployment/cilium-operator $ kubectl -n kube-system rollout restart ds/cilium
Cilium can become the default ingress controller by setting the
--set ingressController.default=true
flag. This will create ingress entries even when the ingressClass
is not set.
If you only want to use envoy traffic management feature without Ingress support, you should only
enable --enable-envoy-config
flag.
$ helm upgrade cilium ./cilium \ --namespace kube-system \ --reuse-values \ --set envoyConfig.enabled=true $ kubectl -n kube-system rollout restart deployment/cilium-operator $ kubectl -n kube-system rollout restart ds/cilium
Additionally, the proxy load-balancing feature can be configured with the loadBalancer.l7.backend=envoy
flag.
$ helm upgrade cilium ./cilium \ --namespace kube-system \ --reuse-values \ --set loadBalancer.l7.backend=envoy $ kubectl -n kube-system rollout restart deployment/cilium-operator $ kubectl -n kube-system rollout restart ds/cilium
Next you can check the status of the Cilium agent and operator:
$ cilium status
Warning
Make sure you install cilium-cli v0.15.0 or later. The rest of instructions do not work with older versions of cilium-cli. To confirm the cilium-cli version that’s installed in your system, run:
cilium version --client
See Cilium CLI upgrade notes for more details.
Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "arm64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
shasum -a 256 -c cilium-darwin-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-darwin-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
See the full page of releases.
Clone the Cilium GitHub repository so that the Cilium CLI can access the latest unreleased Helm chart from the main branch:
git clone git@github.com:cilium/cilium.git
cd cilium
Warning
Make sure you install cilium-cli v0.15.0 or later. The rest of instructions do not work with older versions of cilium-cli. To confirm the cilium-cli version that’s installed in your system, run:
cilium version --client
See Cilium CLI upgrade notes for more details.
Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "arm64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
shasum -a 256 -c cilium-darwin-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-darwin-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
See the full page of releases.
Clone the Cilium GitHub repository so that the Cilium CLI can access the latest unreleased Helm chart from the main branch:
git clone git@github.com:cilium/cilium.git
cd cilium
Cilium Ingress Controller can be enabled with the below command
$ cilium install --chart-directory ./install/kubernetes/cilium --set kubeProxyReplacement=true --set ingressController.enabled=true --set ingressController.loadbalancerMode=dedicated
Cilium can become the default ingress controller by setting the
--set ingressController.default=true
flag. This will create ingress entries even when the ingressClass
is not set.
If you only want to use envoy traffic management feature without Ingress support, you should only
enable --enable-envoy-config
flag.
$ cilium install --chart-directory ./install/kubernetes/cilium --set kubeProxyReplacement=true --set envoyConfig.enabled=true
Additionally, the proxy load-balancing feature can be configured with the loadBalancer.l7.backend=envoy
flag.
$ cilium install --chart-directory ./install/kubernetes/cilium --set kubeProxyReplacement=true --set envoyConfig.enabled=true --set loadBalancer.l7.backend=envoy
Next you can check the status of the Cilium agent and operator:
$ cilium status
Hubble CLI is also used to observe the traffic in later steps.
Download the latest hubble release:
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
Download the latest hubble release:
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "arm64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
shasum -a 256 -c hubble-darwin-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-darwin-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-darwin-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
Download the latest hubble release:
curl -LO "https://raw.githubusercontent.com/cilium/hubble/master/stable.txt"
set /p HUBBLE_VERSION=<stable.txt
curl -L --fail -O "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz"
curl -L --fail -O "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz.sha256sum"
certutil -hashfile hubble-windows-amd64.tar.gz SHA256
type hubble-windows-amd64.tar.gz.sha256sum
:: verify that the checksum from the two commands above match
tar zxf hubble-windows-amd64.tar.gz
and move the hubble.exe
CLI to a directory listed in the %PATH%
environment variable after
extracting it from the tarball.
Supported Ingress Annotations
Name |
Description |
Default Value |
---|---|---|
|
The loadbalancer mode for the ingress.
Allows a per ingress override
of the default set in the Helm value
ingressController.loadbalancerMode .Applicable values are
dedicated andshared . |
dedicated (from Helm chart)
|
|
The Service type for dedicated Ingress.
Applicable values are
LoadBalancer and
NodePort . |
|
|
The NodePort to use for the HTTP Ingress.
Applicable only if
ingress.cilium.io/service-type is
NodePort . If unspecified, a randomNodePort will be allocated by kubernetes.
|
unspecified |
|
The NodePort to use for the HTTPS Ingress.
Applicable only if
ingress.cilium.io/service-type is
NodePort . If unspecified, a randomNodePort will be allocated by kubernetes.
|
unspecified |
|
The port to use for the Envoy listener on the host
network. Applicable and mandatory only for
dedicated Ingress and if host network mode is
enabled.
|
|
|
Enable TLS Passthrough mode for this Ingress.
Applicable values are
enabled and disabled ,although boolean-style values will also be
accepted.
Note that some conditions apply to TLS
Passthrough Ingresses, due to how
TLS Passthrough works:
* A
host field must be set in the Ingress* Default backends are ignored
* Rules with paths other than
/ are ignoredIf all the rules in an Ingress are ignored for
these reasons, no Envoy config will be generated
and the Ingress will have no effect.
Note that this annotation is analogous to
the
ssl-passthrough on other Ingresscontrollers.
|
|
|
Enable enforced HTTPS redirects for this Ingress.
Applicable values are
enabled and disabled ,although boolean-style values will also be
accepted.
Note that if the annotation is not present, this
behavior will be controlled by the
enforce-ingress-https configurationfile setting (or
ingressController.enforceHttps in Helm).
Any host with TLS config will have redirects to
HTTPS configured for each match specified in the
Ingress.
|
unspecified |
Additionally, cloud-provider specific annotations for the LoadBalancer Service are supported.
By default, annotations with values beginning with:
lbipam.cilium.io
nodeipam.cilium.io
service.beta.kubernetes.io
service.kubernetes.io
cloud.google.com
will be copied from an Ingress object to the generated LoadBalancer Service objects.
This setting is controlled by the Cilium Operator’s ingress-lb-annotation-prefixes
config flag, and can be configured in Cilium’s Helm values.yaml
using the ingressController.ingressLBAnnotationPrefixes
setting.
Please refer to the Kubernetes documentation for more details.
Host network mode
Note
Supported since Cilium 1.16+
Host network mode allows you to expose the Cilium ingress controller (Envoy listener) directly on the host network. This is useful in cases where a LoadBalancer Service is unavailable, such as in development environments or environments with cluster-external loadbalancers.
Note
Enabling the Cilium ingress controller host network mode automatically disables the LoadBalancer/NodePort type Service mode. They are mutually exclusive.
The listener is exposed on all interfaces (
0.0.0.0
for IPv4 and/or::
for IPv6).
Host network mode can be enabled via Helm:
ingressController:
enabled: true
hostNetwork:
enabled: true
Once enabled, host network ports can be specified with the following methods:
- Shared Ingress: Globally via Helm flags
ingressController.hostNetwork.sharedListenerPort
: Host network port to expose the Cilium ingress controller Envoy listener. The default port is8080
. If you change it, you should choose a port number higher than1023
(see Bind to privileged port).
- Dedicated Ingress: Per
Ingress
resource via annotations ingress.cilium.io/host-listener-port
: Host network port to expose the Cilium ingress controller Envoy listener. The default port is8080
but it can only be used for a singleIngress
resource as it needs to be unique perIngress
resource. You should choose a port higher than1023
(see Bind to privileged port). This annotation is mandatory if the global Cilium ingress controller mode is configured todedicated
(ingressController.loadbalancerMode
) or the ingress resource sets theingress.cilium.io/loadbalancer-mode
annotation todedicated
and multipleIngress
resources are deployed.
- Dedicated Ingress: Per
The default behavior regarding shared or dedicated ingress can be configured via
ingressController.loadbalancerMode
.
Warning
Be aware that misconfiguration might result in port clashes. Configure unique ports that are still available on all Cilium Nodes where Cilium ingress controller Envoy listeners are exposed.
Bind to privileged port
By default, the Cilium L7 Envoy process does not have any Linux capabilities out-of-the-box and is therefore not allowed to listen on privileged ports.
If you choose a port equal to or lower than 1023
, ensure that the Helm value
envoy.securityContext.capabilities.keepCapNetBindService=true
is configured
and to add the capability NET_BIND_SERVICE
to the respective
Cilium Envoy container via Helm values:
Standalone DaemonSet mode:
envoy.securityContext.capabilities.envoy
Embedded mode:
securityContext.capabilities.ciliumAgent
Configure the following Helm values to allow privileged port bindings in host network mode:
ingressController:
enabled: true
hostNetwork:
enabled: true
envoy:
enabled: true
securityContext:
capabilities:
keepCapNetBindService: true
envoy:
# Add NET_BIND_SERVICE to the list (keep the others!)
- NET_BIND_SERVICE
ingressController:
enabled: true
hostNetwork:
enabled: true
envoy:
securityContext:
capabilities:
keepCapNetBindService: true
securityContext:
capabilities:
ciliumAgent:
# Add NET_BIND_SERVICE to the list (keep the others!)
- NET_BIND_SERVICE
Deploy Gateway API listeners on subset of nodes
The Cilium ingress controller Envoy listener can be exposed on a specific subset of nodes. This only works in combination with the host network mode and can be configured via a node label selector in the Helm values:
ingressController:
enabled: true
hostNetwork:
enabled: true
nodes:
matchLabels:
role: infra
component: ingress
This will deploy the Ingress Controller Envoy listener only on the Cilium Nodes matching the configured labels. An empty selector selects all nodes and continues to expose the functionality on all Cilium nodes.
Examples
Please refer to one of the below examples on how to use and leverage Cilium’s Ingress features: