Configuration
ConfigMap Options
In the ConfigMap there are several options that can be configured according to your preferences:
debug
- Sets to run Cilium in full debug mode, which enables verbose logging and configures eBPF programs to emit more visibility events into the output ofcilium monitor
.enable-ipv4
- Enable IPv4 addressing supportenable-ipv6
- Enable IPv6 addressing supportclean-cilium-bpf-state
- Removes all eBPF state from the filesystem on startup. Endpoints will be restored with the same IP addresses, but ongoing connections may be briefly disrupted and loadbalancing decisions will be lost, so active connections via the loadbalancer will break. All eBPF state will be reconstructed from their original sources (for example, from Kubernetes or the kvstore). This may be used to mitigate serious issues regarding eBPF maps. This option should be turned off again after restarting the daemon.clean-cilium-state
- Removes all Cilium state, including unrecoverable information such as all endpoint state, as well as recoverable state such as eBPF state pinned to the filesystem, CNI configuration files, library code, links, routes, and other information. This operation is irreversible. Existing endpoints currently managed by Cilium may continue to operate as before, but Cilium will no longer manage them and they may stop working without warning. After using this operation, endpoints must be deleted and reconnected to allow the new instance of Cilium to manage them.monitor-aggregation
- This option enables coalescing of tracing events incilium monitor
to only include periodic updates from active flows, or any packets that involve an L4 connection state change. Valid options arenone
,low
,medium
,maximum
.none
- Generate a tracing event on every receive and send packet.low
- Generate a tracing event on every send packet.medium
- Generate a tracing event on every new connection, any time a packet contains TCP flags that have not been previously seen for the packet direction, and on average once permonitor-aggregation-interval
(assuming that a packet is seen during the interval). Each direction tracks TCP flags and report interval separately. If Cilium drops a packet, it will emit one event per packet dropped.maximum
- An alias for the most aggressive aggregation level. Currently this is equivalent to settingmonitor-aggregation
tomedium
.
monitor-aggregation-interval
- Defines the interval to report tracing events. Only applicable formonitor-aggregation
levelsmedium
or higher. Assuming new packets are sent at least once per interval, this ensures that on average one event is sent during the interval.preallocate-bpf-maps
- Pre-allocation of map entries allows per-packet latency to be reduced, at the expense of up-front memory allocation for the entries in the maps. Set totrue
to optimize for latency. If this value is modified, then during the next Cilium startup connectivity may be temporarily disrupted for endpoints with active connections.
Any changes that you perform in the Cilium ConfigMap and in
cilium-etcd-secrets
Secret
will require you to restart any existing
Cilium pods in order for them to pick the latest configuration.
Attention
When updating keys or values in the ConfigMap, the changes might take up to 2 minutes to be propagated to all nodes running in the cluster. For more information see the official Kubernetes docs: Mounted ConfigMaps are updated automatically
The following ConfigMap is an example where the etcd cluster is running in 2
nodes, node-1
and node-2
with TLS, and client to server authentication
enabled.
apiVersion: v1
kind: ConfigMap
metadata:
name: cilium-config
namespace: kube-system
data:
# The kvstore configuration is used to enable use of a kvstore for state
# storage.
kvstore: etcd
kvstore-opt: '{"etcd.config": "/var/lib/etcd-config/etcd.config"}'
# This etcd-config contains the etcd endpoints of your cluster. If you use
# TLS please make sure you follow the tutorial in https://cilium.link/etcd-config
etcd-config: |-
---
endpoints:
- https://node-1:31079
- https://node-2:31079
#
# In case you want to use TLS in etcd, uncomment the 'trusted-ca-file' line
# and create a kubernetes secret by following the tutorial in
# https://cilium.link/etcd-config
trusted-ca-file: '/var/lib/etcd-secrets/etcd-client-ca.crt'
#
# In case you want client to server authentication, uncomment the following
# lines and create a kubernetes secret by following the tutorial in
# https://cilium.link/etcd-config
key-file: '/var/lib/etcd-secrets/etcd-client.key'
cert-file: '/var/lib/etcd-secrets/etcd-client.crt'
# If you want to run cilium in debug mode change this value to true
debug: "false"
enable-ipv4: "true"
# If you want to clean cilium state; change this value to true
clean-cilium-state: "false"
CNI
CNI - Container Network Interface is the plugin layer used by Kubernetes to delegate networking configuration. You can find additional information on the CNI project website.
CNI configuration is automatically taken care of when deploying Cilium via the provided
DaemonSet. The cilium
pod will generate an appropriate CNI configuration
file and write it to disk on startup.
Note
In order for CNI installation to work properly, the
kubelet
task must either be running on the host filesystem of the
worker node, or the /etc/cni/net.d
and /opt/cni/bin
directories must be mounted into the container where kubelet
is
running. This can be achieved with Volumes mounts.
The CNI auto installation is performed as follows:
The
/etc/cni/net.d
and/opt/cni/bin
directories are mounted from the host filesystem into the pod where Cilium is running.The binary
cilium-cni
is installed to/opt/cni/bin
. Any existing binary with the namecilium-cni
is overwritten.The file
/etc/cni/net.d/05-cilium.conflist
is written.
Adjusting CNI configuration
The CNI configuration file is automatically written and maintained by the cilium pod. It is written after the agent has finished initialization and is ready to handle pod sandbox creation. In addition, the agent will remove any other CNI configuration files by default.
There are a number of Helm variables that adjust CNI configuration management. For a full description, see the helm documentation. A brief summary:
Helm variable |
Description |
Default |
---|---|---|
|
Disable CNI configuration management |
false |
|
Remove other CNI configuration files |
true |
|
Install CNI configuration and binaries |
true |
If you want to provide your own custom CNI configuration file, you can do so by passing a path to a cni template file, either on disk or provided via a configMap. The Helm options that configure this are:
Helm variable |
Description |
---|---|
|
Path (inside the agent) to a source CNI configuration file |
|
Name of a ConfigMap containing a source CNI configuration file |
|
Install CNI configuration and binaries |
These Helm variables are converted to a smaller set of cilium ConfigMap keys:
ConfigMap key |
Description |
---|---|
|
Path to write the CNI configuration file |
|
Path to read the source CNI configuration file |
|
Whether or not to remove other CNI configuration files |
CRD Validation
Custom Resource Validation was introduced in Kubernetes since version 1.8.0
.
This is still considered an alpha feature in Kubernetes 1.8.0
and beta in
Kubernetes 1.9.0
.
Since Cilium v1.0.0-rc3
, Cilium will create, or update in case it exists,
the Cilium Network Policy (CNP) Resource Definition with the embedded
validation schema. This allows the validation of CiliumNetworkPolicy to be done
on the kube-apiserver when the policy is imported with an ability to provide
direct feedback when importing the resource.
To enable this feature, the flag --feature-gates=CustomResourceValidation=true
must be set when starting kube-apiserver. Cilium itself will automatically make
use of this feature and no additional flag is required.
Note
In case there is an invalid CNP before updating to Cilium
v1.0.0-rc3
, which contains the validator, the kube-apiserver
validator will prevent Cilium from updating that invalid CNP with
Cilium node status. By checking Cilium logs for unable to update
CNP, retrying...
, it is possible to determine which Cilium Network
Policies are considered invalid after updating to Cilium
v1.0.0-rc3
.
To verify that the CNP resource definition contains the validation schema, run the following command:
$ kubectl get crd ciliumnetworkpolicies.cilium.io -o json | grep -A 12 openAPIV3Schema
"openAPIV3Schema": {
"oneOf": [
{
"required": [
"spec"
]
},
{
"required": [
"specs"
]
}
],
In case the user writes a policy that does not conform to the schema, Kubernetes will return an error, e.g.:
cat <<EOF > ./bad-cnp.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: my-new-cilium-object
spec:
description: "Policy to test multiple rules in a single file"
endpointSelector:
matchLabels:
app: details
track: stable
version: v1
ingress:
- fromEndpoints:
- matchLabels:
app: reviews
track: stable
version: v1
toPorts:
- ports:
- port: '65536'
protocol: TCP
rules:
http:
- method: GET
path: "/health"
EOF
kubectl create -f ./bad-cnp.yaml
...
spec.ingress.toPorts.ports.port in body should match '^(6553[0-5]|655[0-2][0-9]|65[0-4][0-9]{2}|6[0-4][0-9]{3}|[1-5][0-9]{4}|[0-9]{1,4})$'
In this case, the policy has a port out of the 0-65535 range.
Mounting BPFFS with systemd
Due to how systemd mounts filesystems, the mount point path must be reflected in the unit filename.
cat <<EOF | sudo tee /etc/systemd/system/sys-fs-bpf.mount
[Unit]
Description=Cilium BPF mounts
Documentation=https://docs.cilium.io/
DefaultDependencies=no
Before=local-fs.target umount.target
After=swap.target
[Mount]
What=bpffs
Where=/sys/fs/bpf
Type=bpf
Options=rw,nosuid,nodev,noexec,relatime,mode=700
[Install]
WantedBy=multi-user.target
EOF
Container Runtimes
CRIO
If you want to use CRIO, use the instructions below.
Note
Make sure you have Helm 3 installed. Helm 2 is no longer supported.
Setup Helm repository:
helm repo add cilium https://helm.cilium.io/
Note
The Helm flag --set bpf.autoMount.enabled=false
might not be
required for your setup. For more info see Common CRIO issues.
helm install cilium cilium/cilium --version 1.14.2 \ --namespace kube-system \ --set containerRuntime.integration=crio
Since CRI-O does not automatically detect that a new CNI plugin has been installed, you will need to restart the CRI-O daemon for it to pick up the Cilium CNI configuration.
First make sure Cilium is running:
$ kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE
cilium-mqtdz 1/1 Running 0 3m 10.0.2.15 minikube
After that you can restart CRI-O:
minikube ssh -- sudo systemctl restart crio
Common CRIO issues
Some CRI-O environments automatically mount the bpf filesystem in the pods,
which is something that Cilium avoids doing when
--set bpf.autoMount.enabled=false
is set. However, some
CRI-O environments do not mount the bpf filesystem automatically which causes
Cilium to print the following message:
level=warning msg="BPF system config check: NOT OK." error="CONFIG_BPF kernel parameter is required" subsys=linux-datapath
level=warning msg="================================= WARNING ==========================================" subsys=bpf
level=warning msg="BPF filesystem is not mounted. This will lead to network disruption when Cilium pods" subsys=bpf
level=warning msg="are restarted. Ensure that the BPF filesystem is mounted in the host." subsys=bpf
level=warning msg="https://docs.cilium.io/en/stable/operations/system_requirements/#mounted-ebpf-filesystem" subsys=bpf
level=warning msg="====================================================================================" subsys=bpf
level=info msg="Mounting BPF filesystem at /sys/fs/bpf" subsys=bpf
If you see this warning in the Cilium pod logs with your CRI-O environment,
please remove the flag --set bpf.autoMount.enabled=false
from
your Helm setup and redeploy Cilium.