Kata Containers with Cilium¶
Kata Containers is an open source project that
provides a secure container runtime with lightweight virtual machines that feel
and perform like containers, but provide stronger workload isolation using
hardware virtualization technology as a second layer of defense. Kata
Containers implements OCI runtime spec, just like
runc that is used by
Docker. Cilium can be used along with Kata Containers, using both enables
higher degree of security. Kata Containers enhances security in the compute
layer, while Cilium provides policy and observability in the networking layer.
This guide shows how to install Cilium along with Kata Containers. It assumes that you have already followed the official Kata Containers installation user guide to get the Kata Containers runtime up and running on your platform of choice but that you haven’t yet setup Kubernetes.
This guide has been validated by following the Kata Containers guide for Google Compute Engine (GCE) and using Ubuntu 18.04 LTS with the packaged version of Kata Containers, CRI-containerd and Kubernetes 1.18.3.
Setup Kubernetes with CRI¶
Kata Containers runtime is an OCI compatible runtime and cannot directly interact with the CRI API level. For this reason, it relies on a CRI implementation to translate CRI into OCI. At the time of writing this guide, there are two supported ways called CRI-O and CRI-containerd. It is up to you to choose the one that you want, but you have to pick one.
Refer to the section Requirements for detailed instruction on how to prepare your Kubernetes environment and make sure to use Kubernetes >= 1.12. Then, follow the official guide to run Kata Containers with Kubernetes.
Minimum version of kubernetes 1.12 is required to use the RuntimeClass Feature for Kata Container runtime described below.
With your Kubernetes cluster ready, you can now proceed to deploy Cilium.
Setup Helm repository:
helm repo add cilium https://helm.cilium.io/
Deploy Cilium release via Helm:
Kata containers do not work with Host-Reachable Services, or with
kube-proxy replacement in strict mode. These
features should be disabled with
--set kubeProxyReplacement=disabled (or
Both features rely on socket-based load-balancing, which is not possible given that Kata containers are virtual machines running with their own kernel. For kube-proxy replacement, this limitation is tracked with GitHub issue 15437.
Validate the Installation¶
You can monitor as Cilium and all required components are being installed:
kubectl -n kube-system get pods --watch NAME READY STATUS RESTARTS AGE cilium-operator-cb4578bc5-q52qk 0/1 Pending 0 8s cilium-s8w5m 0/1 PodInitializing 0 7s coredns-86c58d9df4-4g7dd 0/1 ContainerCreating 0 8m57s coredns-86c58d9df4-4l6b2 0/1 ContainerCreating 0 8m57s
It may take a couple of minutes for all components to come up:
cilium-operator-cb4578bc5-q52qk 1/1 Running 0 4m13s cilium-s8w5m 1/1 Running 0 4m12s coredns-86c58d9df4-4g7dd 1/1 Running 0 13m coredns-86c58d9df4-4l6b2 1/1 Running 0 13m
Deploy the connectivity test¶
You can deploy the “connectivity-check” to test connectivity between pods. It is recommended to create a separate namespace for this.
kubectl create ns cilium-test
Deploy the check with:
kubectl apply -n cilium-test -f https://raw.githubusercontent.com/cilium/cilium/v1.9/examples/kubernetes/connectivity-check/connectivity-check.yaml
It will deploy a series of deployments which will use various connectivity paths to connect to each other. Connectivity paths include with and without service load-balancing and various network policy combinations. The pod name indicates the connectivity variant and the readiness and liveness gate indicates success or failure of the test:
$ kubectl get pods -n cilium-test NAME READY STATUS RESTARTS AGE echo-a-76c5d9bd76-q8d99 1/1 Running 0 66s echo-b-795c4b4f76-9wrrx 1/1 Running 0 66s echo-b-host-6b7fc94b7c-xtsff 1/1 Running 0 66s host-to-b-multi-node-clusterip-85476cd779-bpg4b 1/1 Running 0 66s host-to-b-multi-node-headless-dc6c44cb5-8jdz8 1/1 Running 0 65s pod-to-a-79546bc469-rl2qq 1/1 Running 0 66s pod-to-a-allowed-cnp-58b7f7fb8f-lkq7p 1/1 Running 0 66s pod-to-a-denied-cnp-6967cb6f7f-7h9fn 1/1 Running 0 66s pod-to-b-intra-node-nodeport-9b487cf89-6ptrt 1/1 Running 0 65s pod-to-b-multi-node-clusterip-7db5dfdcf7-jkjpw 1/1 Running 0 66s pod-to-b-multi-node-headless-7d44b85d69-mtscc 1/1 Running 0 66s pod-to-b-multi-node-nodeport-7ffc76db7c-rrw82 1/1 Running 0 65s pod-to-external-1111-d56f47579-d79dz 1/1 Running 0 66s pod-to-external-fqdn-allow-google-cnp-78986f4bcf-btjn7 1/1 Running 0 66s
If you deploy the connectivity check to a single node cluster, pods that check multi-node
functionalities will remain in the
Pending state. This is expected since these pods
need at least 2 nodes to be scheduled successfully.