End-To-End Connectivity Testing
Introduction
Cilium uses cilium-cli connectivity tests for implementing and running end-to-end tests which test Cilium all the way from the API level (for example, importing policies, CLI) to the datapath (in order words, whether policy that is imported is enforced accordingly in the datapath).
Running End-To-End Connectivity Tests
The connectivity tests are implemented in such a way that they can be run against any K8s cluster running Cilium. The built-in feature detection allows the testing framework to automatically skip tests when a required test condition cannot be met (for example, skip the Egress Gateway tests if the Egress Gateway feature is disabled).
Running tests locally
Warning
Make sure you install cilium-cli v0.15.0 or later. The rest of instructions do not work with older versions of cilium-cli. To confirm the cilium-cli version that’s installed in your system, run:
cilium version --client
See Cilium CLI upgrade notes for more details.
Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "arm64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
shasum -a 256 -c cilium-darwin-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-darwin-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
See the full page of releases.
Alternatively, Cilium CLI
can be manually built and installed by fetching
https://github.com/cilium/cilium-cli
, and then running make install
.
Next, you need a Kubernetes cluster to run Cilium. The easiest way to create one
is to use kind. Cilium provides
a wrapper script which simplifies creating K8s cluster with kind
. For example,
to create a cluster consisting of 1 control-plane node, 3 worker nodes, without
kube-proxy, and with DualStack
enabled:
$ cd cilium/
$ ./contrib/scripts/kind.sh "" 3 "" "" "none" "dual"
...
Kind is up! Time to install cilium:
make kind-image
make kind-install-cilium
Afterwards, you need to install Cilium. The preferred way is to use
cilium-cli install,
as it is able to automate some steps (e.g., detecting kube-apiserver
endpoint
address which otherwise needs to be specified when running w/o kube-proxy
, or
set an annotation to a K8s worker node to prevent Cilium from being scheduled on it).
Assuming that Cilium was built with:
$ cd cilium/
$ make kind-image
...
^^^ Images pushed, multi-arch manifest should be above. ^^^
You can install Cilium with the following command:
$ cilium install --wait \
--chart-directory=$GOPATH/src/github.com/cilium/cilium/install/kubernetes/cilium \
--helm-set=image.override=localhost:5000/cilium/cilium-dev:local \
--helm-set=image.pullPolicy=Never \
--helm-set=operator.image.override=localhost:5000/cilium/operator-generic:local \
--helm-set=operator.image.pullPolicy=Never \
--helm-set-string=tunnel=vxlan \
--nodes-without-cilium=kind-worker3
...
⌛ Waiting for Cilium to be installed and ready...
✅ Cilium was successfully installed! Run 'cilium status' to view installation health
Finally, to run tests:
$ cilium connectivity test
...
✅ All 32 tests (263 actions) successful, 2 tests skipped, 1 scenarios skipped.
Alternatively, you can select which tests to run:
$ cilium connectivity test --test north-south-loadbalancing
...
[=] Test [north-south-loadbalancing]
Running tests in VM
To run Cilium and the connectivity tests in a virtual machine, one can use little-vm-helper (LVH). The project provides a runner of qemu-based VMs, a builder of VM images, and a registry containing pre-built VM images.
First, install the LVH cli tool:
$ go install github.com/cilium/little-vm-helper/cmd/lvh@latest
$ lvh --help
...
Use "lvh [command] --help" for more information about a command.
Second, fetch a VM image:
$ mkdir images/
$ docker run -v $(pwd)/images:/mnt/images \
quay.io/lvh-images/kind:6.0-main \
cp /data/images/kind_6.0.qcow2.zst /mnt/images
$ cd images/
$ zstd -d kind_6.0.qcow2.zst
Alternatively, you can use the scripts/pull_image.sh
:
$ mkdir images/
$ git clone https://github.com/cilium/little-vm-helper
$ IMAGE_DIR=./images ./little-vm-helper/scripts/pull_image.sh quay.io/lvh-images/kind:6.0-main
See https://quay.io/organization/lvh-images/kind?tab=tags for all available images. To build a new VM image (or to update any existing) please refer to little-vm-helper-images.
Next, start a VM:
$ lvh run --image ./images/kind_6.0.qcow2 --host-mount $GOPATH/src/github.com/cilium/ --daemonize -p 2222:22 --cpu=3 --mem=6G
Finally, you can SSH into the VM to start a K8s cluster, install Cilium, and finally run the connectivity tests:
$ ssh -p 2222 -o "StrictHostKeyChecking=no" root@localhost
# echo "nameserver 1.1.1.1" > /etc/resolv.conf
# cd /host/cilium
# git config --global --add safe.directory /host/cilium
# ./contrib/scripts/kind.sh "" 3 "" "" "none" "dual"
# cd /host/cilium-cli
# ./cilium install --wait \
--chart-directory=../cilium/install/kubernetes/cilium \
--version=v1.13.2 \
--helm-set-string=tunnel=vxlan \
--nodes-without-cilium=kind-worker3
# ./cilium connectivity test
...
✅ All 32 tests (263 actions) successful, 2 tests skipped, 1 scenarios skipped.
To stop the VM, run from the host:
$ pkill qemu-system-x86