Installation on Google GKE

GKE Requirements

  1. Install the Google Cloud SDK (gcloud) see Installing Google Cloud SDK.
  2. Create a project or use an existing one
export GKE_PROJECT=gke-clusters
gcloud projects create $GKE_PROJECT
gcloud config set project $GKE_PROJECT
  1. Enable the GKE API for the project if not already done
gcloud services enable container.googleapis.com

Create a GKE Cluster

You can apply any method to create a GKE cluster. The example given here is using the Google Cloud SDK.

Note

Either of the cluster zone or region must be specified in gcloud commands below. The full list of locations is available on this page. This guide uses --zone to specify the zone but you may replace this flag with --region instead.

export CLUSTER_NAME=cluster1
export CLUSTER_ZONE=us-west2-a
gcloud container clusters create $CLUSTER_NAME --image-type COS --num-nodes 2 --machine-type n1-standard-4 --zone $CLUSTER_ZONE

Retrieve the credentials to access the cluster:

gcloud container clusters get-credentials $CLUSTER_NAME --zone $CLUSTER_ZONE

When done, you should be able to access your cluster like this:

kubectl get nodes
NAME                                      STATUS   ROLES    AGE   VERSION
gke-cluster1-default-pool-a63a765c-flr2   Ready    <none>   6m    v1.14.10-gke.36
gke-cluster1-default-pool-a63a765c-z73c   Ready    <none>   6m    v1.14.10-gke.36

Deploy Cilium

Extract the Cluster CIDR to enable native-routing:

NATIVE_CIDR="$(gcloud container clusters describe $CLUSTER_NAME --zone $CLUSTER_ZONE --format 'value(clusterIpv4Cidr)')"
echo $NATIVE_CIDR

Note

First, make sure you have Helm 3 installed.

If you have (or planning to have) Helm 2 charts (and Tiller) in the same cluster, there should be no issue as both version are mutually compatible in order to support gradual migration. Cilium chart is targeting Helm 3 (v3.0.3 and above).

Setup Helm repository:

helm repo add cilium https://helm.cilium.io/

Deploy Cilium release via Helm:

If you are ready to restart existing pods when initializing the node, you can also pass the --set nodeinit.restartPods=true flag to the helm command below. This will ensure all pods are managed by Cilium.

kubectl create namespace cilium
helm install cilium cilium/cilium --version 1.8.3 \
  --namespace cilium \
  --set global.nodeinit.enabled=true \
  --set nodeinit.reconfigureKubelet=true \
  --set nodeinit.removeCbrBridge=true \
  --set global.cni.binPath=/home/kubernetes/bin \
  --set global.gke.enabled=true \
  --set config.ipam=kubernetes \
  --set global.nativeRoutingCIDR=$NATIVE_CIDR

The NodeInit DaemonSet is required to prepare the GKE nodes as nodes are added to the cluster. The NodeInit DaemonSet will perform the following actions:

  • Reconfigure kubelet to run in CNI mode
  • Mount the BPF filesystem

Restart unmanaged Pods

If you did not use the nodeinit.restartPods=true in the Helm options when deploying Cilium, then unmanaged pods need to be restarted manually. Restart all already running pods which are not running in host-networking mode to ensure that Cilium starts managing them. This is required to ensure that all pods which have been running before Cilium was deployed have network connectivity provided by Cilium and NetworkPolicy applies to them:

kubectl get pods --all-namespaces -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork --no-headers=true | grep '<none>' | awk '{print "-n "$1" "$2}' | xargs -L 1 -r kubectl delete pod
pod "event-exporter-v0.2.3-f9c896d75-cbvcz" deleted
pod "fluentd-gcp-scaler-69d79984cb-nfwwk" deleted
pod "heapster-v1.6.0-beta.1-56d5d5d87f-qw8pv" deleted
pod "kube-dns-5f8689dbc9-2nzft" deleted
pod "kube-dns-5f8689dbc9-j7x5f" deleted
pod "kube-dns-autoscaler-76fcd5f658-22r72" deleted
pod "kube-state-metrics-7d9774bbd5-n6m5k" deleted
pod "l7-default-backend-6f8697844f-d2rq2" deleted
pod "metrics-server-v0.3.1-54699c9cc8-7l5w2" deleted

Validate the Installation

You can monitor as Cilium and all required components are being installed:

kubectl -n cilium get pods --watch
NAME                               READY   STATUS            RESTARTS   AGE
cilium-bbpwg                       0/1     PodInitializing   0          27s
cilium-node-init-jwtw6             1/1     Running           0          27s
cilium-node-init-t5cm9             1/1     Running           0          27s
cilium-operator-7967c75f94-ckd5g   0/1     Pending           0          27s
cilium-rnrxr                       0/1     Running           0          27s

It may take a couple of minutes for all components to come up:

kubectl -n cilium get pods
NAME                               READY   STATUS    RESTARTS   AGE
cilium-bbpwg                       1/1     Running   0          70s
cilium-node-init-jwtw6             1/1     Running   0          70s
cilium-node-init-t5cm9             1/1     Running   0          70s
cilium-operator-7967c75f94-ckd5g   1/1     Running   0          70s
cilium-rnrxr                       1/1     Running   0          70s

Deploy the connectivity test

You can deploy the “connectivity-check” to test connectivity between pods.

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.8.3/examples/kubernetes/connectivity-check/connectivity-check.yaml

It will deploy a series of deployments which will use various connectivity paths to connect to each other. Connectivity paths include with and without service load-balancing and various network policy combinations. The pod name indicates the connectivity variant and the readiness and liveness gate indicates success or failure of the test:

$ kubectl get pods -n cilium-test
NAME                                                    READY   STATUS    RESTARTS   AGE
echo-a-6788c799fd-42qxx                                 1/1     Running   0          69s
echo-b-59757679d4-pjtdl                                 1/1     Running   0          69s
echo-b-host-f86bd784d-wnh4v                             1/1     Running   0          68s
host-to-b-multi-node-clusterip-585db65b4d-x74nz         1/1     Running   0          68s
host-to-b-multi-node-headless-77c64bc7d8-kgf8p          1/1     Running   0          67s
pod-to-a-allowed-cnp-87b5895c8-bfw4x                    1/1     Running   0          68s
pod-to-a-b76ddb6b4-2v4kb                                1/1     Running   0          68s
pod-to-a-denied-cnp-677d9f567b-kkjp4                    1/1     Running   0          68s
pod-to-b-intra-node-nodeport-8484fb6d89-bwj8q           1/1     Running   0          68s
pod-to-b-multi-node-clusterip-f7655dbc8-h5bwk           1/1     Running   0          68s
pod-to-b-multi-node-headless-5fd98b9648-5bjj8           1/1     Running   0          68s
pod-to-b-multi-node-nodeport-74bd8d7bd5-kmfmm           1/1     Running   0          68s
pod-to-external-1111-7489c7c46d-jhtkr                   1/1     Running   0          68s
pod-to-external-fqdn-allow-google-cnp-b7b6bcdcb-97p75   1/1     Running   0          68s

Note

If you deploy the connectivity check to a single node cluster, pods that check multi-node functionalities will remain in the Pending state. This is expected since these pods need at least 2 nodes to be scheduled successfully.

Specify Environment Variables

Specify the namespace in which Cilium is installed as CILIUM_NAMESPACE environment variable. Subsequent commands reference this environment variable.

export CILIUM_NAMESPACE=cilium

Enable Hubble

Hubble is a fully distributed networking and security observability platform for cloud native workloads. It is built on top of Cilium and eBPF to enable deep visibility into the communication and behavior of services as well as the networking infrastructure in a completely transparent manner.

  • Hubble can be configured to be in local mode or distributed mode (beta).

    In local mode, Hubble listens on a UNIX domain socket. You can connect to a Hubble instance by running hubble command from inside the Cilium pod. This provides networking visibility for traffic observed by the local Cilium agent.

    helm upgrade cilium cilium/cilium --version 1.8.3 \
       --namespace $CILIUM_NAMESPACE \
       --reuse-values \
       --set global.hubble.enabled=true \
       --set global.hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}"
    

    In distributed mode (beta), Hubble listens on a TCP port on the host network. This allows Hubble Relay to communicate with all the Hubble instances in the cluster. Hubble CLI and Hubble UI in turn connect to Hubble Relay to provide cluster-wide networking visibility.

    Warning

    In Distributed mode, Hubble runs a gRPC service over plain-text HTTP on the host network without any authentication/authorization. The main consequence is that anybody who can reach the Hubble gRPC service can obtain all the networking metadata from the host. It is therefore strongly discouraged to enable distributed mode in a production environment.

    helm upgrade cilium cilium/cilium --version 1.8.3 \
       --namespace $CILIUM_NAMESPACE \
       --reuse-values \
       --set global.hubble.enabled=true \
       --set global.hubble.listenAddress=":4244" \
       --set global.hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}" \
       --set global.hubble.relay.enabled=true \
       --set global.hubble.ui.enabled=true
    
  • Restart the Cilium daemonset to allow Cilium agent to pick up the ConfigMap changes:

    kubectl rollout restart -n $CILIUM_NAMESPACE ds/cilium
    
  • To pick one Cilium instance and validate that Hubble is properly configured to listen on a UNIX domain socket:

    kubectl exec -n $CILIUM_NAMESPACE -t ds/cilium -- hubble observe
    
  • (Distributed mode only) To validate that Hubble Relay is running, install the hubble CLI:

    Download the latest hubble release:

    export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
    curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz"
    curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz.sha256sum"
    sha256sum --check hubble-linux-amd64.tar.gz.sha256sum
    tar zxf hubble-linux-amd64.tar.gz
    

    and move the hubble CLI to a directory listed in the $PATH environment variable. For example:

    sudo mv hubble /usr/local/bin
    

    Download the latest hubble release:

    export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
    curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-amd64.tar.gz"
    curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-amd64.tar.gz.sha256sum"
    shasum -a 256 -c hubble-darwin-amd64.tar.gz.sha256sum
    tar zxf hubble-darwin-amd64.tar.gz
    

    and move the hubble CLI to a directory listed in the $PATH environment variable. For example:

    sudo mv hubble /usr/local/bin
    

    Download the latest hubble release:

    curl -LO "https://raw.githubusercontent.com/cilium/hubble/master/stable.txt"
    set /p HUBBLE_VERSION=<stable.txt
    curl -LO "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz"
    curl -LO "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz.sha256sum"
    certutil -hashfile hubble-windows-amd64.tar.gz SHA256
    type hubble-windows-amd64.tar.gz.sha256sum
    :: verify that the checksum from the two commands above match
    tar zxf hubble-windows-amd64.tar.gz
    

    and move the hubble.exe CLI to a directory listed in the %PATH% environment variable after extracting it from the tarball.

    Once the hubble CLI is installed, set up a port forwarding for hubble-relay service and run hubble observe command:

    kubectl port-forward -n $CILIUM_NAMESPACE svc/hubble-relay 4245:80
    hubble observe --server localhost:4245
    

    (For Linux / MacOS) For convenience, you may set and export the HUBBLE_DEFAULT_SOCKET_PATH environment variable:

    $ export HUBBLE_DEFAULT_SOCKET_PATH=localhost:4245
    

    This will allow you to use hubble status and hubble observe commands without having to specify the server address via the --server flag.

  • (Distributed mode only) To validate that Hubble UI is properly configured, set up a port forwarding for hubble-ui service:

    kubectl port-forward -n $CILIUM_NAMESPACE svc/hubble-ui 12000:80
    

    and then open http://localhost:12000/.