Installation on AWS EKS using etcd operator


This is a beta level feature. Be cautious and verify the installation before you run this in production.

Create EKS Cluster

  1. Create an EKS cluster and add some worker nodes by following steps 1-3 in the official EKS documentation:

    Getting Started with Amazon EKS

    When asked to select an AMI. We recommend to use the EKS-optimized AMI as suggested by the guide itself.

    After following the guide, you should have a cluster up and running:

    kubectl get nodes
    NAME                                            STATUS    ROLES     AGE       VERSION     Ready     <none>    3m        v1.10.3   Ready     <none>    2m        v1.10.3    Ready     <none>    2m        v1.10.3
    kubectl -n kube-system get pods
    NAME                       READY     STATUS    RESTARTS   AGE
    aws-node-4wbp6             1/1       Running   1          2m
    aws-node-d5fb2             1/1       Running   1          2m
    aws-node-mxwfb             1/1       Running   0          2m
    kube-dns-7cc87d595-sjcgw   3/3       Running   0          27m
    kube-proxy-jk4lk           1/1       Running   0          2m
    kube-proxy-phn6c           1/1       Running   0          2m
    kube-proxy-rctvn           1/1       Running   0          2m
  2. Tell the aws-node agent to disable SNAT for all traffic

    kubectl -n kube-system set env ds aws-node AWS_VPC_K8S_CNI_EXTERNALSNAT=true
  3. Restart kube-dns to ensure that it is being managed by Cilium.

    kubectl -n kube-system delete pod -l k8s-app=kube-dns

Prepare etcd operator

  1. Dependencies

    The certificate generation scripts have dependencies on cfssl and cfssljson, which can be downloaded from here . Make sure to copy the binaries in a directory which is in your PATH variable. Alternatively, if you have Go installed, then you can also get the libraries using go get -u and go get -u

  2. Generate and deploy etcd certificates

    This certificate will be used to secure the communication between Cilium agents and the etcd cluster.

    cd examples/kubernetes/addons/etcd-operator
    tls/certs/ cluster.local

    Deploy the etcd certificates:

  3. Deploy the etcd operator

    kubectl apply -f 00-crd-etcd.yaml

Deploy Cilium + etcd

Deploy Cilium including an etcd deployment:

cd examples/kubernetes/addons/etcd-operator
kubectl apply -f .

Give it some time to come up as both the etcd cluster and Cilium are being deployed in parallel. Cilium will provide basic networking to etcd in a heavily restricted policy environment and then automatically connect to etcd as soon as the cluster becomes available.

Verify installation

Verify that everything is up and running:

kubectl -n kube-system get pods
NAME                            READY     STATUS    RESTARTS   AGE
aws-node-9tj2v                  1/1       Running   0          1h
aws-node-gt8gt                  1/1       Running   0          1h
aws-node-xx8sc                  1/1       Running   0          1h
cilium-54gxk                    1/1       Running   0          9m
cilium-etcd-5t2cvng8jw          1/1       Running   0          8m
cilium-etcd-f2rlpccpcq          1/1       Running   0          7m
cilium-etcd-rh66gsbgqb          1/1       Running   0          8m
cilium-qjqv8                    1/1       Running   0          9m
cilium-sfjd2                    1/1       Running   0          9m
etcd-operator-84dd99cfd-69q4b   1/1       Running   0          8m
kube-dns-7cc87d595-sjcgw        3/3       Running   0          1h
kube-proxy-jk4lk                1/1       Running   0          1h
kube-proxy-phn6c                1/1       Running   0          1h
kube-proxy-rctvn                1/1       Running   0          1h
kubectl -n kube-system exec -ti cilium-qjqv8 cilium-health status
Probe time:   2018-08-20T14:37:50Z
Nodes: (localhost):
    Host connectivity to
      ICMP:          OK, RTT=250.203µs
      HTTP via L3:   OK, RTT=427.923µs
    Endpoint connectivity to
      ICMP:   OK, RTT=257.911µs
    Host connectivity to
      ICMP:          OK, RTT=831.244µs
      HTTP via L3:   OK, RTT=1.746408ms
    Endpoint connectivity to
      ICMP:          OK, RTT=860.772µs
      HTTP via L3:   OK, RTT=1.848061ms
    Host connectivity to
      ICMP:          OK, RTT=530.695µs
      HTTP via L3:   OK, RTT=1.234267ms
    Endpoint connectivity to
      ICMP:          OK, RTT=669.397µs
      HTTP via L3:   OK, RTT=1.273788ms