Setting Up Cilium in AlibabaCloud ENI Mode (beta)

Note

This is a beta feature. Please provide feedback and file a GitHub issue if you experience any problems.

Note

The AlibabaCloud ENI integration is still subject to some limitations. See Limitations for details.

Create a Cluster on AlibabaCloud

Setup a Kubernetes on AlibabaCloud. You can use any method you prefer. The quickest way is to create an ACK (Alibaba Cloud Container Service for Kubernetes) cluster and to replace the CNI plugin with Cilium. For more details on how to set up an ACK cluster please follow the official documentation.

Disable ACK CNI (ACK Only)

If you are running an ACK cluster, you should delete the ACK CNI.

Cilium will manage ENIs instead of the ACK CNI, so any running DaemonSet from the list below has to be deleted to prevent conflicts.

  • kube-flannel-ds
  • terway
  • terway-eni
  • terway-eniip

Note

If you are using ACK with Flannel (DaemonSet kube-flannel-ds), the Cloud Controller Manager (CCM) will create route (Pod CIDR) in VPC. If your cluster is an Managed Kubernetes you cannot disable this behavior. Please consider creating a new cluster.

kubectl -n kube-system delete daemonset <terway>

Create AlibabaCloud Secrets

Before installing Cilium, a new Kubernetes Secret with the AlibabaCloud Tokens needs to be added to your Kubernetes cluster. This Secret will allow Cilium to gather information from the AlibabaCloud API which is needed to implement ToGroups policies.

AlibabaCloud Access Keys

To create a new access token the following guide can be used. These keys need to have certain RAM Permissions:

{
  "Version": "1",
  "Statement": [{
      "Action": [
        "ecs:CreateNetworkInterface",
        "ecs:DescribeNetworkInterfaces",
        "ecs:AttachNetworkInterface",
        "ecs:DetachNetworkInterface",
        "ecs:DeleteNetworkInterface",
        "ecs:DescribeInstanceAttribute",
        "ecs:DescribeInstanceTypes",
        "ecs:AssignPrivateIpAddresses",
        "ecs:UnassignPrivateIpAddresses",
        "ecs:DescribeInstances"
      ],
      "Resource": [
        "*"
      ],
      "Effect": "Allow"
    },
    {
      "Action": [
        "vpc:DescribeVSwitches",
        "vpc:ListTagResources",
      ],
      "Resource": [
        "*"
      ],
      "Effect": "Allow"
    }
  ]
}

As soon as you have the access tokens, the following secret needs to be added, with each empty string replaced by the associated value as a base64-encoded string:

apiVersion: v1
kind: Secret
metadata:
  name: cilium-alibabacloud
  namespace: kube-system
type: Opaque
data:
  ALIBABA_CLOUD_ACCESS_KEY_ID: ""
  ALIBABA_CLOUD_ACCESS_KEY_SECRET: ""

The base64 command line utility can be used to generate each value, for example:

$ echo -n "access_key" | base64
YWNjZXNzX2tleQ==

This secret stores the AlibabaCloud credentials, which will be used to connect to the AlibabaCloud API.

$ kubectl create -f cilium-secret.yaml

Deploy Cilium

Note

Make sure you have Helm 3 installed. Helm 2 is no longer supported.

Setup Helm repository:

helm repo add cilium https://helm.cilium.io/

Deploy Cilium release via Helm:

helm install cilium cilium/cilium --version 1.10.0 \
  --namespace kube-system \
  --set alibabacloud.enabled=true \
  --set ipam.mode=alibabacloud \
  --set enableIPv4Masquerade=false \
  --set tunnel=disabled \
  --set nodeinit.enabled=true

Note

You must ensure that the security groups associated with the ENIs (eth1, eth2, …) allow for egress traffic to go outside of the VPC. By default, the security groups for pod ENIs are derived from the primary ENI (eth0).

Validate the Installation

Install the latest version of the Cilium CLI on your local machine. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable a variety of functionality.

curl -LO https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz
curl -LO https://github.com/cilium/cilium-cli/releases/latest/download/cilium-darwin-amd64.tar.gz
sudo tar xzvfC cilium-darwin-amd64.tar.gz /usr/local/bin
rm cilium-darwin-amd64.tar.gz
See the full page of releases.

To validate that Cilium has been properly installed, you can run

cilium status --wait

The output should be similar to the following one:

   /¯¯\
/¯¯\__/¯¯\    Cilium:         OK
\__/¯¯\__/    Operator:       OK
/¯¯\__/¯¯\    Hubble:         disabled
\__/¯¯\__/    ClusterMesh:    disabled
   \__/

DaemonSet         cilium             Desired: 2, Ready: 2/2, Available: 2/2
Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
Containers:       cilium-operator    Running: 2
                  cilium             Running: 2
Image versions    cilium             quay.io/cilium/cilium:v1.9.5: 2
                  cilium-operator    quay.io/cilium/operator-generic:v1.9.5: 2

Deploy the connectivity test

Run the following command to validate that your cluster has proper network connectivity:

cilium connectivity test

The output should be similar to the following one:

ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [k8s-cluster] Creating namespace for connectivity check...
(...)
---------------------------------------------------------------------------------------------------------------------
📋 Test Report
---------------------------------------------------------------------------------------------------------------------
✅ 69/69 tests successful (0 warnings)

Congratulations! You have a fully functional Kubernetes cluster with Cilium. 🎉

Limitations

  • The Alibaba ENI integration of Cilium is currently only enabled for IPv4.
  • Only work with instance support ENI, refer to Instance families.