Setting up Cilium in AWS ENI mode¶
Note
The AWS ENI integration is still subject to some limitations. See Limitations for details.
Create an AWS cluster¶
Setup a Kubernetes on AWS. You can use any method you prefer, but for the
simplicity of this tutorial, we are going to use eksctl. For more details on how to set up an
EKS cluster using eksctl, see the section Installation on AWS EKS.
eksctl create cluster --name test-cluster --without-nodegroup
Disable VPC CNI (aws-node DaemonSet) (EKS only)¶
If you are running an EKS cluster, you should delete the aws-node DaemonSet.
Cilium will manage ENIs instead of VPC CNI, so the aws-node DaemonSet
has to be deleted to prevent conflict behavior.
Note
Once aws-node DaemonSet is deleted, EKS will not try to restore it.
kubectl -n kube-system delete daemonset aws-node
Deploy Cilium¶
Note
First, make sure you have Helm 3 installed. Helm 2 is no longer supported.
Setup Helm repository:
helm repo add cilium https://helm.cilium.io/
Deploy Cilium release via Helm:
helm install cilium cilium/cilium --version 1.9.5 \ --namespace kube-system \ --set eni=true \ --set ipam.mode=eni \ --set egressMasqueradeInterfaces=eth0 \ --set tunnel=disabled \ --set nodeinit.enabled=true
Note
The above options are assuming that masquerading is desired and that the VM
is connected to the VPC using eth0. It will route all traffic that does
not stay in the VPC via eth0 and masquerade it.
If you want to avoid masquerading, set masquerade=false. You must
ensure that the security groups associated with the ENIs (eth1,
eth2, …) allow for egress traffic to outside of the VPC. By default,
the security groups for pod ENIs are derived from the primary ENI
(eth0).
Create a node group¶
eksctl create nodegroup --cluster test-cluster --nodes 2
Validate the Installation¶
You can monitor as Cilium and all required components are being installed:
kubectl -n kube-system get pods --watch
NAME READY STATUS RESTARTS AGE
cilium-operator-cb4578bc5-q52qk 0/1 Pending 0 8s
cilium-s8w5m 0/1 PodInitializing 0 7s
coredns-86c58d9df4-4g7dd 0/1 ContainerCreating 0 8m57s
coredns-86c58d9df4-4l6b2 0/1 ContainerCreating 0 8m57s
It may take a couple of minutes for all components to come up:
cilium-operator-cb4578bc5-q52qk 1/1 Running 0 4m13s
cilium-s8w5m 1/1 Running 0 4m12s
coredns-86c58d9df4-4g7dd 1/1 Running 0 13m
coredns-86c58d9df4-4l6b2 1/1 Running 0 13m
Deploy the connectivity test¶
You can deploy the “connectivity-check” to test connectivity between pods. It is recommended to create a separate namespace for this.
kubectl create ns cilium-test
Deploy the check with:
kubectl apply -n cilium-test -f https://raw.githubusercontent.com/cilium/cilium/1.9.5/examples/kubernetes/connectivity-check/connectivity-check.yaml
It will deploy a series of deployments which will use various connectivity paths to connect to each other. Connectivity paths include with and without service load-balancing and various network policy combinations. The pod name indicates the connectivity variant and the readiness and liveness gate indicates success or failure of the test:
$ kubectl get pods -n cilium-test
NAME READY STATUS RESTARTS AGE
echo-a-76c5d9bd76-q8d99 1/1 Running 0 66s
echo-b-795c4b4f76-9wrrx 1/1 Running 0 66s
echo-b-host-6b7fc94b7c-xtsff 1/1 Running 0 66s
host-to-b-multi-node-clusterip-85476cd779-bpg4b 1/1 Running 0 66s
host-to-b-multi-node-headless-dc6c44cb5-8jdz8 1/1 Running 0 65s
pod-to-a-79546bc469-rl2qq 1/1 Running 0 66s
pod-to-a-allowed-cnp-58b7f7fb8f-lkq7p 1/1 Running 0 66s
pod-to-a-denied-cnp-6967cb6f7f-7h9fn 1/1 Running 0 66s
pod-to-b-intra-node-nodeport-9b487cf89-6ptrt 1/1 Running 0 65s
pod-to-b-multi-node-clusterip-7db5dfdcf7-jkjpw 1/1 Running 0 66s
pod-to-b-multi-node-headless-7d44b85d69-mtscc 1/1 Running 0 66s
pod-to-b-multi-node-nodeport-7ffc76db7c-rrw82 1/1 Running 0 65s
pod-to-external-1111-d56f47579-d79dz 1/1 Running 0 66s
pod-to-external-fqdn-allow-google-cnp-78986f4bcf-btjn7 1/1 Running 0 66s
Note
If you deploy the connectivity check to a single node cluster, pods that check multi-node
functionalities will remain in the Pending state. This is expected since these pods
need at least 2 nodes to be scheduled successfully.
Specify Environment Variables¶
Specify the namespace in which Cilium is installed as CILIUM_NAMESPACE
environment variable. Subsequent commands reference this environment variable.
export CILIUM_NAMESPACE=kube-system
Enable Hubble for Cluster-Wide Visibility¶
Hubble is the component for observability in Cilium. To obtain cluster-wide visibility into your network traffic, deploy Hubble Relay and the UI as follows on your existing installation:
If you installed Cilium via helm install, you may enable Hubble
Relay and UI with the following command:
helm upgrade cilium cilium/cilium --version 1.9.5 \ --namespace $CILIUM_NAMESPACE \ --reuse-values \ --set hubble.listenAddress=":4244" \ --set hubble.relay.enabled=true \ --set hubble.ui.enabled=true
On Cilium 1.9.1 and older, the Cilium agent pods will be restarted in the process.
If you installed Cilium 1.9.2 or newer via the provided
quick-install.yaml, you may deploy Hubble Relay and UI on top of
your existing installation with the following command:
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.9.5/install/kubernetes/quick-hubble-install.yaml
Installation via quick-hubble-install.yaml only works if the
installed Cilium version is 1.9.2 or newer. Users of Cilium 1.9.0
or 1.9.1 are encouraged to upgrade to a newer version by applying the
most recent Cilium quick-install.yaml first.
Alternatively, it is possible to manually generate a YAML manifest for the Cilium DaemonSet and Hubble Relay/UI as follows. The generated YAML can be applied on top of an existing installation:
# Set this to your installed Cilium version
export CILIUM_VERSION=1.9.1
# Please set any custom Helm values you may need for Cilium,
# such as for example `--set operator.replicas=1` on single-cluster nodes.
helm template cilium cilium/cilium --version $CILIUM_VERSION \\
--namespace $CILIUM_NAMESPACE \\
--set hubble.tls.auto.method="cronJob" \\
--set hubble.listenAddress=":4244" \\
--set hubble.relay.enabled=true \\
--set hubble.ui.enabled=true > cilium-with-hubble.yaml
# This will modify your existing Cilium DaemonSet and ConfigMap
kubectl apply -f cilium-with-hubble.yaml
The Cilium agent pods will be restarted in the process.
Once the Hubble UI pod is started, use port forwarding for the hubble-ui
service. This allows opening the UI locally on a browser:
kubectl port-forward -n $CILIUM_NAMESPACE svc/hubble-ui --address 0.0.0.0 --address :: 12000:80
And then open http://localhost:12000/ to access the UI.
Hubble UI is not the only way to get access to Hubble data. A command line tool, the Hubble CLI, is also available. It can be installed by following the instructions below:
Download the latest hubble release:
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz"
curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz.sha256sum"
sha256sum --check hubble-linux-amd64.tar.gz.sha256sum
tar zxf hubble-linux-amd64.tar.gz
and move the hubble CLI to a directory listed in the $PATH environment variable. For example:
sudo mv hubble /usr/local/bin
Download the latest hubble release:
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-amd64.tar.gz"
curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-amd64.tar.gz.sha256sum"
shasum -a 256 -c hubble-darwin-amd64.tar.gz.sha256sum
tar zxf hubble-darwin-amd64.tar.gz
and move the hubble CLI to a directory listed in the $PATH environment variable. For example:
sudo mv hubble /usr/local/bin
Download the latest hubble release:
curl -LO "https://raw.githubusercontent.com/cilium/hubble/master/stable.txt"
set /p HUBBLE_VERSION=<stable.txt
curl -LO "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz"
curl -LO "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz.sha256sum"
certutil -hashfile hubble-windows-amd64.tar.gz SHA256
type hubble-windows-amd64.tar.gz.sha256sum
:: verify that the checksum from the two commands above match
tar zxf hubble-windows-amd64.tar.gz
and move the hubble.exe CLI to a directory listed in the %PATH% environment variable after
extracting it from the tarball.
Similarly to the UI, use port forwarding for the hubble-relay service to
make it available locally:
kubectl port-forward -n $CILIUM_NAMESPACE svc/hubble-relay --address 0.0.0.0 --address :: 4245:80
In a separate terminal window, run the hubble status command specifying the
Hubble Relay address:
$ hubble --server localhost:4245 status
Healthcheck (via localhost:4245): Ok
Current/Max Flows: 5455/16384 (33.29%)
Flows/s: 11.30
Connected Nodes: 4/4
If Hubble Relay reports that all nodes are connected, as in the example output above, you can now use the CLI to observe flows of the entire cluster:
hubble --server localhost:4245 observe
If you encounter any problem at this point, you may seek help on Slack.
Tip
Hubble CLI configuration can be persisted using a configuration file or
environment variables. This avoids having to specify options specific to a
particular environment every time a command is run. Run hubble help
config for more information.
For more information about Hubble and its components, see the Observability section.
Limitations¶
The AWS ENI integration of Cilium is currently only enabled for IPv4.
When applying L7 policies at egress, the source identity context is lost as it is currently not carried in the packet. This means that traffic will look like it is coming from outside of the cluster to the receiving pod.
HostPort type services additionally require either of the following configurations:
Troubleshooting¶
Make sure to disable DHCP on ENIs¶
Cilium will use both the primary and secondary IP addresses assigned to ENIs.
Use of the primary IP address optimizes the number of IPs available to pods but
can conflict with a DHCP agent running on the node and assigning the primary IP
of the ENI to the interface of the node. A common scenario where this happens
is if NetworkManager is running on the node and automatically performing
DHCP on all network interfaces of the VM. Be sure to disable DHCP on any ENIs
that get attached to the node or disable NetworkManager entirely.