Installation Guide

Note

This is the detailed installation guide aimed at production installations. If you are looking to get started quickly, the Getting Started Using Minikube or the Quick Start guide may be better options.

This section describes how to install and run Cilium on Kubernetes. The deployment method we are using is called DaemonSet which is the easiest way to deploy Cilium in a Kubernetes environment. It will request Kubernetes to automatically deploy and run a cilium/cilium container image as a pod on all Kubernetes worker nodes.

Should you encounter any issues during the installation, please refer to the Troubleshooting section and / or seek help on Slack channel. See the Kubernetes Compatibility section for kubernetes API version compatibility.

Kubernetes Requirements

Enable automatic node CIDR allocation (Recommended)

Kubernetes has the capability to automatically allocate and assign per node IP allocation CIDR. Cilium automatically uses this feature if enabled. This is the easiest method to handle IP allocation in a Kubernetes cluster. To enable this feature, simply add the following flag when starting kube-controller-manager:

--allocate-node-cidrs

This option is not required but highly recommended.

Running Kubernetes with CRD Validation (Recommended)

Custom Resource Validation was introduced in Kubernetes since version 1.8.0. This is still considered an alpha feature in Kubernetes 1.8.0 and beta in Kubernetes 1.9.0.

Since Cilium v1.0.0-rc3, Cilium will create, or update in case it exists, the Cilium Network Policy (CNP) Resource Definition with the embedded validation schema. This allows the validation of CiliumNetworkPolicy to be done on the kube-apiserver when the policy is imported with an ability to provide direct feedback when importing the resource.

To enable this feature, the flag --feature-gates=CustomResourceValidation=true must be set when starting kube-apiserver. Cilium itself will automatically make use of this feature and no additional flag is required.

Note

In case there is an invalid CNP before updating to Cilium v1.0.0-rc3, which contains the validator, the kube-apiserver validator will prevent Cilium from updating that invalid CNP with Cilium node status. By checking Cilium logs for unable to update CNP, retrying..., it is possible to determine which Cilium Network Policies are considered invalid after updating to Cilium v1.0.0-rc3.

To verify that the CNP resource definition contains the validation schema, run the following command:

kubectl get crd ciliumnetworkpolicies.cilium.io -o json

kubectl get crd ciliumnetworkpolicies.cilium.io -o json | grep -A 12 openAPIV3Schema
    "openAPIV3Schema": {
        "oneOf": [
            {
                "required": [
                    "spec"
                ]
            },
            {
                "required": [
                    "specs"
                ]
            }
        ],

In case the user writes a policy that does not conform to the schema, Kubernetes will return an error, e.g.:

cat <<EOF > ./bad-cnp.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
description: "Policy to test multiple rules in a single file"
metadata:
  name: my-new-cilium-object
spec:
  endpointSelector:
    matchLabels:
      app: details
      track: stable
      version: v1
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: reviews
        track: stable
        version: v1
    toPorts:
    - ports:
      - port: '65536'
        protocol: TCP
      rules:
        http:
        - method: GET
          path: "/health"
EOF

kubectl create -f ./bad-cnp.yaml
...
spec.ingress.toPorts.ports.port in body should match '^(6553[0-5]|655[0-2][0-9]|65[0-4][0-9]{2}|6[0-4][0-9]{3}|[1-5][0-9]{4}|[0-9]{1,4})$'

In this case, the policy has a port out of the 0-65535 range.

Mounting the BPF FS (Optional)

This step is optional but recommended. It allows the cilium-agent to pin BPF resources to a persistent filesystem and make them persistent across restarts of the agent. If the BPF filesystem is not mounted in the host filesystem, Cilium will automatically mount the filesystem in the mount namespace of the container when the agent starts. This will allow operation of Cilium but will result in unmounting of the filesystem when the pod is restarted. This in turn will cause resources such as the connection tracking table of the BPF programs to be released which will cause all connections into local containers to be dropped. Mounting the BPF filesystem in the host mount namespace will ensure that the agent can be restarted without affecting connectivity of any pods.

In order to mount the BPF filesystem, the following command must be run in the host mount namespace. The command must only be run once during the boot process of the machine.

mount bpffs /sys/fs/bpf -t bpf

A portable way to achieve this with persistence is to add the following line to /etc/fstab and then run mount /sys/fs/bpf. This will cause the filesystem to be automatically mounted when the node boots.

bpffs                      /sys/fs/bpf             bpf     defaults 0 0

If you are using systemd to manage the kubelet, another option is to add a mountd systemd service on all hosts:

Due to how systemd mounts filesystems, the mount point path must be reflected in the unit filename.

cat <<EOF | sudo tee /etc/systemd/system/sys-fs-bpf.mount
[Unit]
Description=Cilium BPF mounts
Documentation=http://docs.cilium.io/
DefaultDependencies=no
Before=local-fs.target umount.target
After=swap.target

[Mount]
What=bpffs
Where=/sys/fs/bpf
Type=bpf

[Install]
WantedBy=multi-user.target
EOF

CNI Configuration

CNI - Container Network Interface is the plugin layer used by Kubernetes to delegate networking configuration. You can find additional information on the CNI project website.

Note

Kubernetes `` >= 1.3.5`` requires the loopback CNI plugin to be installed on all worker nodes. The binary is typically provided by most Kubernetes distributions. See section Installing CNI and loopback for instructions on how to install CNI in case the loopback binary is not already installed on your worker nodes.

CNI configuration is automatically being taken care of when deploying Cilium via the provided DaemonSet. The script cni-install.sh is automatically run via the postStart mechanism when the cilium pod is started.

Note

In order for the the cni-install.sh script to work properly, the kubelet task must either be running on the host filesystem of the worker node, or the /etc/cni/net.d and /opt/cni/bin directories must be mounted into the container where kubelet is running. This can be achieved with Volumes mounts.

The CNI auto installation is performed as follows:

1. The /etc/cni/net.d and /opt/cni/bin directories are mounted from the host filesystem into the pod where Cilium is running.
2. The file /etc/cni/net.d/00-cilium.conf is written in case it does not exist yet.
3. The binary cilium-cni is installed to /opt/cni/bin. Any existing binary with the name cilium-cni is overwritten.

Installing CNI and loopback

Since Kubernetes v1.3.5 the loopback CNI plugin must be installed. There are many ways to install CNI, the following is an example:

sudo mkdir -p /opt/cni
wget https://storage.googleapis.com/kubernetes-release/network-plugins/cni-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz
sudo tar -xvf cni-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz -C /opt/cni
rm cni-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz

Adjusting CNI configuration

The CNI installation can be configured with environment variables. These environment variables can be specified in the DaemonSet file like this:

env:
  - name: "CNI_CONF_NAME"
    value: "00-cilium.conf"

The following variables are supported:

Option	Description	Default
HOST_PREFIX	Path prefix of all host mounts	/host
CNI_DIR	Path to mounted CNI directory	${HOST_PREFIX}/opt/cni
CNI_CONF_NAME	Name of configuration file	00-cilium.conf

If you want to further adjust the CNI configuration you may do so by creating the CNI configuration /etc/cni/net.d/00-cilium.conf manually:

sudo mkdir -p /etc/cni/net.d
sudo sh -c 'echo "{
    "name": "cilium",
    "type": "cilium-cni"
}
" > /etc/cni/net.d/00-cilium.conf'

Cilium will use any existing /etc/cni/net.d/00-cilium.conf file if it already exists on a worker node and only creates it if it does not exist yet.

Deploying the DaemonSet

K8s 1.7

K8s 1.8

K8s 1.9

K8s 1.10

K8s 1.11

$ wget https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.7/cilium.yaml

$ wget https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.8/cilium.yaml

$ wget https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.9/cilium.yaml

$ wget https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.10/cilium.yaml

$ wget https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.11/cilium.yaml

Adjusting the ConfigMap

After downloading the cilium.yaml file, open it with your text editor and change the ConfigMap based on the following instructions.

Adjusting etcd-config

First, make sure the etcd-config endpoints have the correct addresses of your etcd nodes.

If you are running more than one node simply specify the complete of endpoints. The list of endpoints can accept both domain names or IP addresses. Make sure you specify the correct port used in your etcd node.

If etcd is running with TLS, there are a couple of changes that you need to do.

1. Make sure you have https in all endpoints;

2. Uncomment the line #ca-file: '/var/lib/etcd-secrets/etcd-ca' so that the certificate authority of the servers are known to Cilium;

3. Create a kubernetes secret with certificate authority file in kubernetes;

   1. Use certificate authority file, with the name ca.crt, used to create in etcd;

   2. Create the secret by executing:

      $ kubectl create secret generic -n kube-system cilium-etcd-secrets \
          --from-file=etcd-ca=ca.crt
      

If etcd is running with client to server authentication, you need make more changes to the ConfigMap:

1. Uncomment both lines #key-file: '/var/lib/etcd-secrets/etcd-client-key' and #cert-file: '/var/lib/etcd-secrets/etcd-client-crt';

2. Create a kubernetes secret with client.key and client.crt files in kubernetes.

   1. Use the file with the name client.key that contains the client key;

   2. Use the file with the name client.crt that contains the client certificate;

   3. Create the secret by executing:

      $ kubectl create secret generic -n kube-system cilium-etcd-secrets \
          --from-file=etcd-ca=ca.crt \
          --from-file=etcd-client-key=client.key \
          --from-file=etcd-client-crt=client.crt
      

Note

If you have set up the secret before you might see the error Error from server (AlreadyExists): secrets "cilium-etcd-secrets" already exists you can simply delete it with kubectl delete secret -n kube-system cilium-etcd-secrets and re-create it again.

Note

When creating the kubernetes secrets just make sure you create it with all necessary files, ca.crt, client.key and client.crt in a single kubectl create.

Regarding the etcd configuration that is all you need to change in the ConfigMap.

Adjusting Cilium Options

In the ConfigMap there are a couple of options that can be changed accordingly with your changes.

• debug - Sets to run Cilium in full debug mode, it can be changed at runtime;
• disable-ipv4 - Disables IPv4 in Cilium and endpoints managed by Cilium;
• clean-cilium-state - Removes any Cilium state, e.g. BPF policy maps, before starting the Cilium agent;
• legacy-host-allows-world - If true, the policy with the entity reserved:host allows traffic from world. If false, the policy needs to explicitly have the entity reserved:world to allow traffic from world. It is recommended to set it to false. This option provides compatibility with Cilium 1.0 which was not able to differentiate between NodePort traffic and traffic from the host.

Any changes that you perform in the Cilium ConfigMap and in cilium-etcd-secrets Secret will require you to restart any existing Cilium pods in order for them to pick the latest configuration.

The following ConfigMap is an example where the etcd cluster is running in 2 nodes, node-1 and node-2 with TLS, and client to server authentication enabled.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
data:
    endpoints:
    - https://node-1:31079
    - https://node-2:31079
    #
    # In case you want to use TLS in etcd, uncomment the 'ca-file' line
    # and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    ca-file: '/var/lib/etcd-secrets/etcd-ca'
    #
    # In case you want client to server authentication, uncomment the following
    # lines and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    key-file: '/var/lib/etcd-secrets/etcd-client-key'
    cert-file: '/var/lib/etcd-secrets/etcd-client-crt'

  # If you want to run cilium in debug mode change this value to true
  debug: "false"
  disable-ipv4: "false"
  # If you want to clean cilium state; change this value to true
  clean-cilium-state: "false"
  legacy-host-allows-world: "false"

After configuring the ConfigMap in cilium.yaml it is time to deploy it using kubectl:

$ kubectl create -f cilium.yaml

Kubernetes will deploy the cilium DaemonSet as a pod in the kube-system namespace on all worker nodes. This operation is performed in the background. Run the following command to check the progress of the deployment:

$ kubectl --namespace kube-system get ds
NAME            DESIRED   CURRENT   READY     NODE-SELECTOR   AGE
cilium          4         4         4         <none>          2m

As the pods are deployed, the number in the ready column will increase and eventually reach the desired count.

$ kubectl --namespace kube-system describe ds cilium
Name:           cilium
Image(s):       cilium/cilium:stable
Selector:       io.cilium.admin.daemon-set=cilium,name=cilium
Node-Selector:  <none>
Labels:         io.cilium.admin.daemon-set=cilium
                name=cilium
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Misscheduled: 0
Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Events:
  FirstSeen     LastSeen        Count   From            SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----            -------------   --------        ------                  -------
  35s           35s             1       {daemon-set }                   Normal          SuccessfulCreate        Created pod: cilium-2xzqm

We can now check the logfile of a particular cilium agent:

$ kubectl --namespace kube-system get pods
NAME           READY     STATUS    RESTARTS   AGE
cilium-2xzqm   1/1       Running   0          41m

$ kubectl --namespace kube-system logs cilium-2xzqm
INFO      _ _ _
INFO  ___|_| |_|_ _ _____
INFO |  _| | | | | |     |
INFO |___|_|_|_|___|_|_|_|
INFO Cilium 0.8.90 f022e2f Thu, 27 Apr 2017 23:17:56 -0700 go version go1.7.5 linux/amd64
INFO clang and kernel versions: OK!
INFO linking environment: OK!
[...]

Deploying to selected nodes

To deploy Cilium only to a selected list of worker nodes, you can add a NodeSelector to the cilium.yaml file like this:

spec:
  template:
    spec:
      nodeSelector:
        with-network-plugin: cilium

And then label each node where Cilium should be deployed:

kubectl label node worker0 with-network-plugin=cilium
kubectl label node worker1 with-network-plugin=cilium
kubectl label node worker2 with-network-plugin=cilium

Networking For Existing Pods

In case pods were already running before the Cilium DaemonSet was deployed, these pods will still be connected using the previous networking plugin according to the CNI configuration. A typical example for this is the kube-dns service which runs in the kube-system namespace by default.

A simple way to change networking for such existing pods is to rely on the fact that Kubernetes automatically restarts pods in a Deployment if they are deleted, so we can simply delete the original kube-dns pod and the replacement pod started immediately after will have networking managed by Cilium. In a production deployment, this step could be performed as a rolling update of kube-dns pods to avoid downtime of the DNS service.

$ kubectl --namespace kube-system delete pods -l k8s-app=kube-dns
pod "kube-dns-268032401-t57r2" deleted

Running kubectl get pods will show you that Kubernetes started a new set of kube-dns pods while at the same time terminating the old pods:

$ kubectl --namespace kube-system get pods
NAME                          READY     STATUS        RESTARTS   AGE
cilium-5074s                  1/1       Running       0          58m
kube-addon-manager-minikube   1/1       Running       0          59m
kube-dns-268032401-j0vml      3/3       Running       0          9s
kube-dns-268032401-t57r2      3/3       Terminating   0          57m

Removing the cilium daemon

All cilium agents are managed as a DaemonSet which means that deleting the DaemonSet will automatically stop and remove all pods which run Cilium on each worker node:

$ kubectl --namespace kube-system delete ds cilium