Setting Up Cilium in AlibabaCloud ENI Mode (beta)

Note

This is a beta feature. Please provide feedback and file a GitHub issue if you experience any problems.

Note

The AlibabaCloud ENI integration is still subject to some limitations. See Limitations for details.

Create a Cluster on AlibabaCloud

Setup a Kubernetes on AlibabaCloud. You can use any method you prefer. The quickest way is to create an ACK (Alibaba Cloud Container Service for Kubernetes) cluster and to replace the CNI plugin with Cilium. For more details on how to set up an ACK cluster please follow the official documentation.

Disable ACK CNI (ACK Only)

If you are running an ACK cluster, you should delete the ACK CNI.

Cilium will manage ENIs instead of the ACK CNI, so any running DaemonSet from the list below has to be deleted to prevent conflicts.

  • kube-flannel-ds

  • terway

  • terway-eni

  • terway-eniip

Note

If you are using ACK with Flannel (DaemonSet kube-flannel-ds), the Cloud Controller Manager (CCM) will create route (Pod CIDR) in VPC. If your cluster is an Managed Kubernetes you cannot disable this behavior. Please consider creating a new cluster.

kubectl -n kube-system delete daemonset <terway>

The next step is to remove CRD below created by terway* CNI

kubectl delete crd \
    ciliumclusterwidenetworkpolicies.cilium.io \
    ciliumendpoints.cilium.io \
    ciliumidentities.cilium.io \
    ciliumnetworkpolicies.cilium.io \
    ciliumnodes.cilium.io \
    bgpconfigurations.crd.projectcalico.org \
    clusterinformations.crd.projectcalico.org \
    felixconfigurations.crd.projectcalico.org \
    globalnetworkpolicies.crd.projectcalico.org \
    globalnetworksets.crd.projectcalico.org \
    hostendpoints.crd.projectcalico.org \
    ippools.crd.projectcalico.org \
    networkpolicies.crd.projectcalico.org

Create AlibabaCloud Secrets

Before installing Cilium, a new Kubernetes Secret with the AlibabaCloud Tokens needs to be added to your Kubernetes cluster. This Secret will allow Cilium to gather information from the AlibabaCloud API which is needed to implement ToGroups policies.

AlibabaCloud Access Keys

To create a new access token the following guide can be used. These keys need to have certain RAM Permissions:

{
  "Version": "1",
  "Statement": [{
      "Action": [
        "ecs:CreateNetworkInterface",
        "ecs:DescribeNetworkInterfaces",
        "ecs:AttachNetworkInterface",
        "ecs:DetachNetworkInterface",
        "ecs:DeleteNetworkInterface",
        "ecs:DescribeInstanceAttribute",
        "ecs:DescribeInstanceTypes",
        "ecs:AssignPrivateIpAddresses",
        "ecs:UnassignPrivateIpAddresses",
        "ecs:DescribeInstances",
        "ecs:DescribeSecurityGroups"
      ],
      "Resource": [
        "*"
      ],
      "Effect": "Allow"
    },
    {
      "Action": [
        "vpc:DescribeVSwitches",
        "vpc:ListTagResources",
        "vpc:DescribeVpcs"
      ],
      "Resource": [
        "*"
      ],
      "Effect": "Allow"
    }
  ]
}

As soon as you have the access tokens, the following secret needs to be added, with each empty string replaced by the associated value as a base64-encoded string:

apiVersion: v1
kind: Secret
metadata:
  name: cilium-alibabacloud
  namespace: kube-system
type: Opaque
data:
  ALIBABA_CLOUD_ACCESS_KEY_ID: ""
  ALIBABA_CLOUD_ACCESS_KEY_SECRET: ""

The base64 command line utility can be used to generate each value, for example:

$ echo -n "access_key" | base64
YWNjZXNzX2tleQ==

This secret stores the AlibabaCloud credentials, which will be used to connect to the AlibabaCloud API.

$ kubectl create -f cilium-secret.yaml

Deploy Cilium

Note

Make sure you have Helm 3 installed. Helm 2 is no longer supported.

Setup Helm repository:

helm repo add cilium https://helm.cilium.io/

Deploy Cilium release via Helm:

helm install cilium cilium/cilium --version 1.12.2 \
  --namespace kube-system \
  --set alibabacloud.enabled=true \
  --set ipam.mode=alibabacloud \
  --set enableIPv4Masquerade=false \
  --set tunnel=disabled

Note

You must ensure that the security groups associated with the ENIs (eth1, eth2, …) allow for egress traffic to go outside of the VPC. By default, the security groups for pod ENIs are derived from the primary ENI (eth0).

Validate the Installation

Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

To validate that Cilium has been properly installed, you can run

$ cilium status --wait
   /¯¯\
/¯¯\__/¯¯\    Cilium:         OK
\__/¯¯\__/    Operator:       OK
/¯¯\__/¯¯\    Hubble:         disabled
\__/¯¯\__/    ClusterMesh:    disabled
   \__/

DaemonSet         cilium             Desired: 2, Ready: 2/2, Available: 2/2
Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
Containers:       cilium-operator    Running: 2
                  cilium             Running: 2
Image versions    cilium             quay.io/cilium/cilium:v1.9.5: 2
                  cilium-operator    quay.io/cilium/operator-generic:v1.9.5: 2

Run the following command to validate that your cluster has proper network connectivity:

$ cilium connectivity test
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [k8s-cluster] Creating namespace for connectivity check...
(...)
---------------------------------------------------------------------------------------------------------------------
📋 Test Report
---------------------------------------------------------------------------------------------------------------------
✅ 69/69 tests successful (0 warnings)

Congratulations! You have a fully functional Kubernetes cluster with Cilium. 🎉

Next Steps

Limitations

  • The Alibaba ENI integration of Cilium is currently only enabled for IPv4.

  • Only work with instance support ENI, refer to Instance families.