Installation with external etcd

This guide walks you through the steps required to set up Cilium on Kubernetes using an external etcd. Use of an external etcd provides better performance and is suitable for larger environments. If you are looking for a simple installation method to get started, refer to the section Installation with managed etcd.

Should you encounter any issues during the installation, please refer to the Troubleshooting section and / or seek help on Slack.

When do I need to use a kvstore?

Unlike the section Quick Installation, this guide explains how to configure Cilium to use an external kvstore such as etcd. If you are unsure whether you need to use a kvstore at all, the following is a list of reasons when to use a kvstore:

  • If you want to use the Cluster Mesh functionality.
  • If you are running in an environment with more than 250 nodes, 5k pods, or if you observe a high overhead in state propagation caused by Kubernetes events.
  • If you do not want Cilium to store state in Kubernetes custom resources (CRDs).

Requirements

Make sure your Kubernetes environment is meeting the requirements:

  • Kubernetes >= 1.9
  • Linux kernel >= 4.9
  • Kubernetes in CNI mode
  • Mounted BPF filesystem mounted on all worker nodes
  • Recommended: Enable PodCIDR allocation (--allocate-node-cidrs) in the kube-controller-manager (recommended)

Refer to the section Requirements for detailed instruction on how to prepare your Kubernetes environment.

Configure the External Etcd

When using an external kvstore, the address of the external kvstore needs to be configured in the ConfigMap. Download the base YAML and configure it with Helm:

Download the Cilium release tarball and change to the kubernetes install directory:

curl -LO https://github.com/cilium/cilium/archive/v1.6.tar.gz
tar xzvf v1.6.tar.gz
cd cilium-1.6/install/kubernetes

Install Helm to prepare generating the deployment artifacts based on the Helm templates.

Change the etcd endpoints accordingly:

helm template cilium \
  --namespace kube-system \
  --set global.etcd.enabled=true \
  --set global.etcd.endpoints[0]=http://etcd-endpoint1:2379 \
  --set global.etcd.endpoints[1]=http://etcd-endpoint2:2379 \
  > cilium.yaml

Optional: Configure the SSL certificates

Create a Kubernetes secret with the root certificate authority, and client-side key and certificate of etcd:

kubectl create secret generic -n kube-system cilium-etcd-secrets \
     --from-file=etcd-client-ca.crt=ca.crt \
     --from-file=etcd-client.key=client.key \
     --from-file=etcd-client.crt=client.crt

Adjust the helm template generation to enable SSL for etcd and use https instead of http for the etcd endpoint URLs:

helm template cilium \
  --namespace kube-system \
  --set global.etcd.enabled=true \
  --set global.etcd.ssl=true \
  --set global.etcd.endpoints[0]=https://etcd-endpoint1:2379 \
  --set global.etcd.endpoints[1]=https://etcd-endpoint2:2379 \
  > cilium.yaml

Deploy Cilium

kubectl create -f cilium.yaml

Validate the Installation

Verify that Cilium pods were started on each of your worker nodes

kubectl --namespace kube-system get ds cilium
NAME            DESIRED   CURRENT   READY     NODE-SELECTOR   AGE
cilium          4         4         4         <none>          2m

kubectl -n kube-system get deployments cilium-operator
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
cilium-operator   1/1     1            1           5m25s