Installation with external etcd

This guide walks you through the steps required to set up Cilium on Kubernetes using an external etcd. Use of an external etcd provides better performance and is suitable for larger environments. If you are looking for a simple installation method to get started, refer to the section Installation with managed etcd.

Should you encounter any issues during the installation, please refer to the Troubleshooting section and / or seek help on Slack.

When do I need to use a kvstore?

Unlike the section Quick Installation, this guide explains how to configure Cilium to use an external kvstore such as etcd. If you are unsure whether you need to use a kvstore at all, the following is a list of reasons when to use a kvstore:

  • If you want to use the Cluster Mesh functionality.
  • If you are running in an environment with more than 250 nodes, 5k pods, or if you observe a high overhead in state propagation caused by Kubernetes events.
  • If you do not want Cilium to store state in Kubernetes custom resources (CRDs).

Requirements

Make sure your Kubernetes environment is meeting the requirements:

  • Kubernetes >= 1.9
  • Linux kernel >= 4.9
  • Kubernetes in CNI mode
  • Mounted BPF filesystem mounted on all worker nodes
  • Recommended: Enable PodCIDR allocation (--allocate-node-cidrs) in the kube-controller-manager (recommended)

Refer to the section Requirements for detailed instruction on how to prepare your Kubernetes environment.

You will also need an external etcd version 3.1.0 or higher

Configure Cilium

When using an external kvstore, the address of the external kvstore needs to be configured in the ConfigMap. Download the base YAML and configure it with Helm:

Note

First, make sure you have Helm 3 installed.

If you have (or planning to have) Helm 2 charts (and Tiller) in the same cluster, there should be no issue as both version are mutually compatible in order to support gradual migration. Cilium chart is targeting Helm 3 (v3.0.3 and above).

Setup helm repository:

helm repo add cilium https://helm.cilium.io/

Deploy Cilium release via Helm:

helm install cilium cilium/cilium --version 1.7.6 \
  --namespace kube-system \
  --set global.etcd.enabled=true \
  --set global.etcd.endpoints[0]=http://etcd-endpoint1:2379 \
  --set global.etcd.endpoints[1]=http://etcd-endpoint2:2379

Optional: Configure the SSL certificates

Create a Kubernetes secret with the root certificate authority, and client-side key and certificate of etcd:

kubectl create secret generic -n kube-system cilium-etcd-secrets \
     --from-file=etcd-client-ca.crt=ca.crt \
     --from-file=etcd-client.key=client.key \
     --from-file=etcd-client.crt=client.crt

Adjust the helm template generation to enable SSL for etcd and use https instead of http for the etcd endpoint URLs:

helm install cilium cilium/cilium --version 1.7.6 \
  --namespace kube-system \
  --set global.etcd.enabled=true \
  --set global.etcd.ssl=true \
  --set global.etcd.endpoints[0]=https://etcd-endpoint1:2379 \
  --set global.etcd.endpoints[1]=https://etcd-endpoint2:2379

Validate the Installation

Verify that Cilium pods were started on each of your worker nodes

kubectl --namespace kube-system get ds cilium
NAME            DESIRED   CURRENT   READY     NODE-SELECTOR   AGE
cilium          4         4         4         <none>          2m

kubectl -n kube-system get deployments cilium-operator
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
cilium-operator   1/1     1            1           5m25s