Upgrade Guide

This upgrade guide is intended for Cilium running on Kubernetes. If you have questions, feel free to ping us on Cilium Slack.

Warning

Read the full upgrade guide to understand all the necessary steps before performing them.

Do not upgrade to 1.17 before reading the section 1.17.4 Upgrade Notes and completing the required steps. Skipping this step may lead to an non-functional upgrade.

The only tested rollback and upgrade path is between consecutive minor releases. Always perform rollbacks and upgrades between one minor release at a time. This means that going from (a hypothetical) 1.1 to 1.2 and back is supported while going from 1.1 to 1.3 and back is not.

Always update to the latest patch release of your current version before attempting an upgrade.

Running pre-flight check (Required)

When rolling out an upgrade with Kubernetes, Kubernetes will first terminate the pod followed by pulling the new image version and then finally spin up the new image. In order to reduce the downtime of the agent and to prevent ErrImagePull errors during upgrade, the pre-flight check pre-pulls the new image version. If you are running in Kubernetes Without kube-proxy mode you must also pass on the Kubernetes API Server IP and / or the Kubernetes API Server Port when generating the cilium-preflight.yaml file.

helm template cilium/cilium --version 1.17.5 \
  --namespace=kube-system \
  --set preflight.enabled=true \
  --set agent=false \
  --set operator.enabled=false \
  > cilium-preflight.yaml
kubectl create -f cilium-preflight.yaml

After applying the cilium-preflight.yaml, ensure that the number of READY pods is the same number of Cilium pods running.

$ kubectl get daemonset -n kube-system | sed -n '1p;/cilium/p'
NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
cilium                    2         2         2       2            2           <none>          1h20m
cilium-pre-flight-check   2         2         2       2            2           <none>          7m15s

Once the number of READY pods are equal, make sure the Cilium pre-flight deployment is also marked as READY 1/1. If it shows READY 0/1, consult the CNP Validation section and resolve issues with the deployment before continuing with the upgrade.

$ kubectl get deployment -n kube-system cilium-pre-flight-check -w
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cilium-pre-flight-check   1/1     1            0           12s

Clean up pre-flight check

Once the number of READY for the preflight DaemonSet is the same as the number of cilium pods running and the preflight Deployment is marked as READY 1/1 you can delete the cilium-preflight and proceed with the upgrade.

kubectl delete -f cilium-preflight.yaml

Upgrading Cilium

During normal cluster operations, all Cilium components should run the same version. Upgrading just one of them (e.g., upgrading the agent without upgrading the operator) could result in unexpected cluster behavior. The following steps will describe how to upgrade all of the components from one stable release to a later stable release.

Warning

Read the full upgrade guide to understand all the necessary steps before performing them.

Do not upgrade to 1.17 before reading the section 1.17.4 Upgrade Notes and completing the required steps. Skipping this step may lead to an non-functional upgrade.

The only tested rollback and upgrade path is between consecutive minor releases. Always perform rollbacks and upgrades between one minor release at a time. This means that going from (a hypothetical) 1.1 to 1.2 and back is supported while going from 1.1 to 1.3 and back is not.

Always update to the latest patch release of your current version before attempting an upgrade.

Step 1: Upgrade to latest patch version

When upgrading from one minor release to another minor release, for example 1.x to 1.y, it is recommended to upgrade to the latest patch release for a Cilium release series first. Upgrading to the latest patch release ensures the most seamless experience if a rollback is required following the minor release upgrade. The upgrade guides for previous versions can be found for each minor version at the bottom left corner.

Step 2: Use Helm to Upgrade your Cilium deployment

Helm can be used to either upgrade Cilium directly or to generate a new set of YAML files that can be used to upgrade an existing deployment via kubectl. By default, Helm will generate the new templates using the default values files packaged with each new release. You still need to ensure that you are specifying the equivalent options as used for the initial deployment, either by specifying a them at the command line or by committing the values to a YAML file.

Setup Helm repository:

helm repo add cilium https://helm.cilium.io/

To minimize datapath disruption during the upgrade, the upgradeCompatibility option should be set to the initial Cilium version which was installed in this cluster.

Generate the required YAML file and deploy it:

helm template cilium/cilium --version 1.17.5 \
  --set upgradeCompatibility=1.X \
  --namespace kube-system \
  > cilium.yaml
kubectl apply -f cilium.yaml

Note

Instead of using --set, you can also save the values relative to your deployment in a YAML file and use it to regenerate the YAML for the latest Cilium version. Running any of the previous commands will overwrite the existing cluster’s ConfigMap so it is critical to preserve any existing options, either by setting them at the command line or storing them in a YAML file, similar to:

agent: true
upgradeCompatibility: "1.8"
ipam:
  mode: "kubernetes"
k8sServiceHost: "API_SERVER_IP"
k8sServicePort: "API_SERVER_PORT"
kubeProxyReplacement: "true"

You can then upgrade using this values file by running:

helm upgrade cilium cilium/cilium --version 1.17.5 \
  --namespace=kube-system \
  -f my-values.yaml

When upgrading from one minor release to another minor release using helm upgrade, do not use Helm’s --reuse-values flag. The --reuse-values flag ignores any newly introduced values present in the new release and thus may cause the Helm template to render incorrectly. Instead, if you want to reuse the values from your existing installation, save the old values in a values file, check the file for any renamed or deprecated values, and then pass it to the helm upgrade command as described above. You can retrieve and save the values from an existing installation with the following command:

helm get values cilium --namespace=kube-system -o yaml > old-values.yaml

The --reuse-values flag may only be safely used if the Cilium chart version remains unchanged, for example when helm upgrade is used to apply configuration changes without upgrading Cilium.

Step 3: Rolling Back

Occasionally, it may be necessary to undo the rollout because a step was missed or something went wrong during upgrade. To undo the rollout run:

kubectl rollout undo daemonset/cilium -n kube-system

This will revert the latest changes to the Cilium DaemonSet and return Cilium to the state it was in prior to the upgrade.

Note

When rolling back after new features of the new minor version have already been consumed, consult the Version Specific Notes to check and prepare for incompatible feature use before downgrading/rolling back. This step is only required after new functionality introduced in the new minor version has already been explicitly used by creating new resources or by opting into new features via the ConfigMap.

Version Specific Notes

This section details the upgrade notes specific to 1.17. Read them carefully and take the suggested actions before upgrading Cilium to 1.17. For upgrades to earlier releases, see the upgrade notes to the previous version.

The only tested upgrade and rollback path is between consecutive minor releases. Always perform upgrades and rollbacks between one minor release at a time. Additionally, always update to the latest patch release of your current version before attempting an upgrade.

Tested upgrades are expected to have minimal to no impact on new and existing connections matched by either no Network Policies, or L3/L4 Network Policies only. Any traffic flowing via user space proxies (for example, because an L7 policy is in place, or using Ingress/Gateway API) will be disrupted during upgrade. Endpoints communicating via the proxy must reconnect to re-establish connections.

1.17.4 Upgrade Notes

  • The check for connectivity to the Kubernetes apiserver has been removed from the cilium-agent liveness probe. This can be turned back on by setting the helm option livenessProbe.requireK8sConnectivity to true.

1.17 Upgrade Notes

  • Operating Cilium in --datapath-mode=lb-only for plain Docker mode now requires to add an additional --enable-k8s=false to the command line, otherwise it is assumed that Kubernetes is present.

  • The Kubernetes clients used by Cilium Agent and Cilium Operator now have separately configurable rate limits. The default rate limit for Cilium Operator K8s clients has been increased to 100 QPS/200 Burst. To configure the rate limit for Cilium Operator, use the --operator-k8s-client-qps and --operator-k8s-client-burst flags or the corresponding Helm values.

  • Support for Consul, deprecated since v1.12, has been removed.

  • Cilium now supports services protocol differentiation, which allows the agent to distinguish two services on the same port with different protocols (e.g. TCP and UDP). This feature, enabled by default, can be controlled with the --bpf-lb-proto-diff flag. After the upgrade, existing services without a protocol set will be preserved as such, to avoid any connection disruptions, and will need to be deleted and recreated in order for their protocol to be taken into account by the agent. In case of downgrades to a version that doesn’t support services protocol differentiation, existing services with the protocol set will be deleted and recreated, without the protocol, by the agent, causing connection disruptions for such services.

  • MTU auto-detection is now continuous during agent lifetime, changing device MTU no longer requires restarting the agent to pick up the new MTU.

  • MTU auto-detection will now use the lowest MTU of all external interfaces. Before, only the primary interface was considered. One exception to this is in ENI mode where the secondary interfaces are not considered for MTU auto-detection. MTU can still be configured manually via the MTU helm option, --mtu agent flag or mtu option in CNI configuration.

  • Support for L7 protocol visibility using Pod annotations (policy.cilium.io/proxy-visibility), deprecated since v1.15, has been removed.

  • The Cilium cluster name validation cannot be bypassed anymore, both for the local and remote clusters. The cluster name is strictly enforced to consist of at most 32 lower case alphanumeric characters and ‘-’, start and end with an alphanumeric character.

  • Cilium could previously be run in a configuration where the Etcd instances that distribute Cilium state between nodes would be managed in pod network by Cilium itself. This support, which had been previously deprecated as complicated and error prone, has now been removed. Refer to Installation with external etcd for alternatives for running Cilium with Etcd.

  • For IPsec, support for a single key has been removed. Per-tunnel keys will now be used regardless of the presence of the + sign in the secret.

  • The option to run a synchronous probe using cilium-health status --probe is no longer supported, and is now a hidden option that returns the results of the most recent cached probe. It will be removed in a future release.

  • The Cilium status API now reports the KVStore subsystem with Disabled state when disabled, instead of OK state and Disabled message.

  • Support for metallb-bgp, deprecated since 1.14, has been removed.

  • Layer 7 policy support for Cassandra and Memcached have been deprecated and their getting started guides have been removed.

  • TLS Visibility can now pass secrets to Envoy via SDS instead of inline via NPDS. This comes with some configuration changes:

    • The Helm value tls.secretsBackend, which could be set to local or k8s (which would grant the Agent read access to all Secrets in the cluster so it could send Secrets inline) has been deprecated and replaced with tls.readSecretsOnlyFromSecretsNamespace, which defaults to true (the agent can only read TLS Visibility Secrets from the namespace configured in the new tls.secretsNamespace setting) or false (the agent can read TLS Visibility Secrets from anywhere in the cluster, equivalent to tls.secretsBackend: k8s)

    • A new Helm value tls.secretSync has been added, which controls if TLS Visibility Secrets will be synchronized to the configured secrets namespace (tls.secretsNamespace.name) by the Cilium Operator, and if SDS will be used. SDS usage requires that secret synchronization be enabled.

    • The defaults for new clusters enable SDS via tls.readSecretsOnlyFromSecretsNamespace: true and tls.secretSync.enabled: true

    • The defaults for upgraded clusters (where upgradeCompatibility is v1.16) do not enable SDS. They are: tls.readSecretsOnlyFromSecretsNamespace: true and tls.secretSync.enabled: false

Removed Options

  • The previously deprecated clustermesh-ip-identities-sync-timeout flag has been removed in favor of clustermesh-sync-timeout.

  • The previously deprecated built-in WireGuard userspace-mode fallback (Helm wireguard.userspaceFallback) has been removed. Users of WireGuard transparent encryption are required to use a Linux kernel with WireGuard support.

  • The previously deprecated metallb-bgp flags bgp-config-path, bgp-announce-lb-ip and bgp-announce-pod-cidr have been removed. Users are now required to use Cilium BGP control plane for BGP advertisements.

Deprecated Options

  • The high-scale mode for ipcache has been deprecated and will be removed in v1.18.

  • The hubble-relay flag --dial-timeout has been deprecated (now a no-op) and will be removed in Cilium 1.18.

  • The External Workloads feature has been deprecated and will be removed in v1.18.

  • The cilium-docker-plugin has been deprecated and is planned to be removed in v1.18.

  • The --k8s-watcher-endpoint-selector cilium-agent flag didn’t have any effect since v1.14 and has been deprecated. It will be remove in v1.18.

  • The --enable-k8s-terminating-endpoint cilium-agent flag (enableK8sTerminatingEndpoint in Helm) has been deprecated. The feature will be unconditionally enabled in v1.18.

  • The --datapath-mode=lb-only has been deprecated and will be removed in v1.18.

Helm Options

  • The Helm options hubble.tls.server.cert, hubble.tls.server.key, hubble.relay.tls.client.cert, hubble.relay.tls.client.key, hubble.relay.tls.server.cert, hubble.relay.tls.server.key, hubble.ui.tls.client.cert, and hubble.ui.tls.client.key have been deprecated in favor of the associated existingSecret options and will be removed in a future release.

  • The default value of hubble.tls.auto.certValidityDuration has been lowered from 1095 days to 365 days because recent versions of MacOS will fail to validate certificates with expirations longer than 825 days.

  • The Helm option hubble.relay.dialTimeout has been deprecated (now a no-op) and will be removed in Cilium 1.18.

  • The metallb-bgp integration Helm options bgp.enabled, bgp.announce.podCIDR, and bgp.announce.loadbalancerIP have been removed. Users are now required to use Cilium BGP control plane options available under bgpControlPlane for BGP announcements.

  • The default value of dnsProxy.endpointMaxIpPerHostname and its corresponding agent option has been increased from 50 to 1000 to reflect improved scaling of toFQDNs policies and to better handle domains which return a large number of IPs with short TTLs.

Agent Options

  • The CONNTRACK_LOCAL option has been deprecated and will be removed in a future release.

Bugtool Options

  • The flag k8s-mode (and related flags cilium-agent-container-name, k8s-namespace & k8s-label) have been deprecated and will be removed in a Cilium 1.18. Cilium CLI should be used to gather a sysdump from a K8s cluster.

Added Metrics

  • cilium_node_health_connectivity_status

  • cilium_node_health_connectivity_latency_seconds

  • cilium_operator_unmanaged_pods

  • cilium_policy_selector_match_count_max

  • cilium_identity_cache_timer_duration

  • cilium_identity_cache_timer_trigger_latency

  • cilium_identity_cache_timer_trigger_folds

Removed Metrics

  • cilium_cidrgroup_translation_time_stats_seconds has been removed, as the measured code path no longer exists.

  • cilium_triggers_policy_update_total has been removed.

  • cilium_triggers_policy_update_folds has been removed.

  • cilium_triggers_policy_update_call_duration has been removed.

Changed Metrics

The metrics prefix of all Envoy NPDS (NetworkPolicy discovery service) metrics has been renamed from envoy_cilium_policymap_<node-ip>_<node-id>_ to envoy_cilium_npds_.

  • envoy_cilium_policymap_<node-ip>_<node-id>_control_plane_rate_limit_enforced -> envoy_cilium_npds_control_plane_rate_limit_enforced

  • envoy_cilium_policymap_<node-ip>_<node-id>_control_plane_connected_state -> envoy_cilium_npds_control_plane_connected_state

  • envoy_cilium_policymap_<node-ip>_<node-id>_control_plane_pending_requests -> envoy_cilium_npds_control_plane_pending_requests

  • envoy_cilium_policymap_<node-ip>_<node-id>_init_fetch_timeout -> envoy_cilium_npds_init_fetch_timeout

  • envoy_cilium_policymap_<node-ip>_<node-id>_update_attempt -> envoy_cilium_npds_update_attempt

  • envoy_cilium_policymap_<node-ip>_<node-id>_update_failure -> envoy_cilium_npds_update_failure

  • envoy_cilium_policymap_<node-ip>_<node-id>_update_rejected -> envoy_cilium_npds_update_rejected

  • envoy_cilium_policymap_<node-ip>_<node-id>_update_success -> envoy_cilium_npds_update_success

  • envoy_cilium_policymap_<node-ip>_<node-id>_update_time -> envoy_cilium_npds_update_time

  • envoy_cilium_policymap_<node-ip>_<node-id>_update_duration -> envoy_cilium_npds_update_duration

  • envoy_cilium_policymap_<node-ip>_<node-id>_version -> envoy_cilium_npds_version

  • doublewrite_identity_crd_total_count has been renamed to doublewrite_identity_crd_total

  • doublewrite_identity_kvstore_total_count has been renamed to doublewrite_identity_kvstore_total

  • doublewrite_identity_crd_only_count has been renamed to doublewrite_identity_crd_only_total

  • doublewrite_identity_kvstore_only_count has been renamed to doublewrite_identity_kvstore_only_total

  • lbipam_conflicting_pools_total has been renamed to lbipam_conflicting_pools

  • lbipam_ips_available_total has been renamed to lbipam_ips_available

  • lbipam_ips_used_total has been renamed to lbipam_ips_used

  • lbipam_services_matching_total has been renamed to lbipam_services_matching

  • lbipam_services_unsatisfied_total has been renamed to lbipam_services_unsatisfied

  • doublewrite_identity_crd_total_count has been renamed to doublewrite_crd_identities

  • doublewrite_identity_kvstore_total_count has been renamed to doublewrite_kvstore_identities

  • doublewrite_identity_crd_only_count has been renamed to doublewrite_crd_only_identities

  • doublewrite_identity_kvstore_only_count has been renamed to doublewrite_kvstore_only_identities

Deprecated Metrics

  • cilium_node_connectivity_status is now deprecated. Please use cilium_node_health_connectivity_status instead.

  • cilium_node_connectivity_latency_seconds is now deprecated. Please use cilium_node_health_connectivity_latency_seconds instead.

Hubble CLI

  • the --cluster behavior changed to show flows emitted from nodes outside of the provided cluster name (either coming from or going to the target cluster). This change brings consistency between the --cluster and --namespace flags and removed the incompatibility between the --cluster and --node-name flags. The previous behavior of --cluster foo can be reproduced with --node-name foo/ (shows all flows emitted from a node in cluster foo).

Advanced

Upgrade Impact

Upgrades are designed to have minimal impact on your running deployment. Networking connectivity, policy enforcement and load balancing will remain functional in general. The following is a list of operations that will not be available during the upgrade:

  • API-aware policy rules are enforced in user space proxies and are running as part of the Cilium pod. Upgrading Cilium causes the proxy to restart, which results in a connectivity outage and causes the connection to reset.

  • Existing policy will remain effective but implementation of new policy rules will be postponed to after the upgrade has been completed on a particular node.

  • Monitoring components such as cilium-dbg monitor will experience a brief outage while the Cilium pod is restarting. Events are queued up and read after the upgrade. If the number of events exceeds the event buffer size, events will be lost.

Rebasing a ConfigMap

This section describes the procedure to rebase an existing ConfigMap to the template of another version.

Export the current ConfigMap

$ kubectl get configmap -n kube-system cilium-config -o yaml --export > cilium-cm-old.yaml
$ cat ./cilium-cm-old.yaml
apiVersion: v1
data:
  clean-cilium-state: "false"
  debug: "true"
  disable-ipv4: "false"
  etcd-config: |-
    ---
    endpoints:
    - https://192.168.60.11:2379
    #
    # In case you want to use TLS in etcd, uncomment the 'trusted-ca-file' line
    # and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    trusted-ca-file: '/var/lib/etcd-secrets/etcd-client-ca.crt'
    #
    # In case you want client to server authentication, uncomment the following
    # lines and add the certificate and key in cilium-etcd-secrets below
    key-file: '/var/lib/etcd-secrets/etcd-client.key'
    cert-file: '/var/lib/etcd-secrets/etcd-client.crt'
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: cilium-config
  selfLink: /api/v1/namespaces/kube-system/configmaps/cilium-config

In the ConfigMap above, we can verify that Cilium is using debug with true, it has a etcd endpoint running with TLS, and the etcd is set up to have client to server authentication.

Generate the latest ConfigMap

helm template cilium \
  --namespace=kube-system \
  --set agent=false \
  --set config.enabled=true \
  --set operator.enabled=false \
  > cilium-configmap.yaml

Add new options

Add the new options manually to your old ConfigMap, and make the necessary changes.

In this example, the debug option is meant to be kept with true, the etcd-config is kept unchanged, and monitor-aggregation is a new option, but after reading the Version Specific Notes the value was kept unchanged from the default value.

After making the necessary changes, the old ConfigMap was migrated with the new options while keeping the configuration that we wanted:

$ cat ./cilium-cm-old.yaml
apiVersion: v1
data:
  debug: "true"
  disable-ipv4: "false"
  # If you want to clean cilium state; change this value to true
  clean-cilium-state: "false"
  monitor-aggregation: "medium"
  etcd-config: |-
    ---
    endpoints:
    - https://192.168.60.11:2379
    #
    # In case you want to use TLS in etcd, uncomment the 'trusted-ca-file' line
    # and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    trusted-ca-file: '/var/lib/etcd-secrets/etcd-client-ca.crt'
    #
    # In case you want client to server authentication, uncomment the following
    # lines and add the certificate and key in cilium-etcd-secrets below
    key-file: '/var/lib/etcd-secrets/etcd-client.key'
    cert-file: '/var/lib/etcd-secrets/etcd-client.crt'
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: cilium-config
  selfLink: /api/v1/namespaces/kube-system/configmaps/cilium-config

Apply new ConfigMap

After adding the options, manually save the file with your changes and install the ConfigMap in the kube-system namespace of your cluster.

$ kubectl apply -n kube-system -f ./cilium-cm-old.yaml

As the ConfigMap is successfully upgraded we can start upgrading Cilium DaemonSet and RBAC which will pick up the latest configuration from the ConfigMap.

Migrating from kvstore-backed identities to Kubernetes CRD-backed identities

Beginning with Cilium 1.6, Kubernetes CRD-backed security identities can be used for smaller clusters. Along with other changes in 1.6, this allows kvstore-free operation if desired. It is possible to migrate identities from an existing kvstore deployment to CRD-backed identities. This minimizes disruptions to traffic as the update rolls out through the cluster.

Migration

When identities change, existing connections can be disrupted while Cilium initializes and synchronizes with the shared identity store. The disruption occurs when new numeric identities are used for existing pods on some instances and others are used on others. When converting to CRD-backed identities, it is possible to pre-allocate CRD identities so that the numeric identities match those in the kvstore. This allows new and old Cilium instances in the rollout to agree.

There are two ways to achieve this: you can either run a one-off cilium preflight migrate-identity script which will perform a point-in-time copy of all identities from the kvstore to CRDs (added in Cilium 1.6), or use the “Double Write” identity allocation mode which will have Cilium manage identities in both the kvstore and CRD at the same time for a seamless migration (added in Cilium 1.17).

Migration with the cilium preflight migrate-identity script

The cilium preflight migrate-identity script is a one-off tool that can be used to copy identities from the kvstore into CRDs. It has a couple of limitations:

  • If an identity is created in the kvstore after the one-off migration has been completed, it will not be copied into a CRD. This means that you need to perform the migration on a cluster with no identity churn.

  • There is no easy way to revert back to --identity-allocation-mode=kvstore if something goes wrong after Cilium has been migrated to --identity-allocation-mode=crd

If these limitations are not acceptable, it is recommended to use the “Double Write” identity allocation mode instead.

The following steps show an example of performing the migration using the cilium preflight migrate-identity script. It is safe to re-run the command if desired. It will identify already allocated identities or ones that cannot be migrated. Note that identity 34815 is migrated, 17003 is already migrated, and 11730 has a conflict and a new ID allocated for those labels.

The steps below assume a stable cluster with no new identities created during the rollout. Once Cilium using CRD-backed identities is running, it may begin allocating identities in a way that conflicts with older ones in the kvstore.

The cilium preflight manifest requires etcd support and can be built with:

helm template cilium \
  --namespace=kube-system \
  --set preflight.enabled=true \
  --set agent=false \
  --set config.enabled=false \
  --set operator.enabled=false \
  --set etcd.enabled=true \
  --set etcd.ssl=true \
  > cilium-preflight.yaml
kubectl create -f cilium-preflight.yaml

Example migration

$ kubectl exec -n kube-system cilium-pre-flight-check-1234 -- cilium-dbg preflight migrate-identity
INFO[0000] Setting up kvstore client
INFO[0000] Connecting to etcd server...                  config=/var/lib/cilium/etcd-config.yml endpoints="[https://192.168.60.11:2379]" subsys=kvstore
INFO[0000] Setting up kubernetes client
INFO[0000] Establishing connection to apiserver          host="https://192.168.60.11:6443" subsys=k8s
INFO[0000] Connected to apiserver                        subsys=k8s
INFO[0000] Got lease ID 29c66c67db8870c8                 subsys=kvstore
INFO[0000] Got lock lease ID 29c66c67db8870ca            subsys=kvstore
INFO[0000] Successfully verified version of etcd endpoint  config=/var/lib/cilium/etcd-config.yml endpoints="[https://192.168.60.11:2379]" etcdEndpoint="https://192.168.60.11:2379" subsys=kvstore version=3.3.13
INFO[0000] CRD (CustomResourceDefinition) is installed and up-to-date  name=CiliumNetworkPolicy/v2 subsys=k8s
INFO[0000] Updating CRD (CustomResourceDefinition)...    name=v2.CiliumEndpoint subsys=k8s
INFO[0001] CRD (CustomResourceDefinition) is installed and up-to-date  name=v2.CiliumEndpoint subsys=k8s
INFO[0001] Updating CRD (CustomResourceDefinition)...    name=v2.CiliumNode subsys=k8s
INFO[0002] CRD (CustomResourceDefinition) is installed and up-to-date  name=v2.CiliumNode subsys=k8s
INFO[0002] Updating CRD (CustomResourceDefinition)...    name=v2.CiliumIdentity subsys=k8s
INFO[0003] CRD (CustomResourceDefinition) is installed and up-to-date  name=v2.CiliumIdentity subsys=k8s
INFO[0003] Listing identities in kvstore
INFO[0003] Migrating identities to CRD
INFO[0003] Skipped non-kubernetes labels when labelling ciliumidentity. All labels will still be used in identity determination  labels="map[]" subsys=crd-allocator
INFO[0003] Skipped non-kubernetes labels when labelling ciliumidentity. All labels will still be used in identity determination  labels="map[]" subsys=crd-allocator
INFO[0003] Skipped non-kubernetes labels when labelling ciliumidentity. All labels will still be used in identity determination  labels="map[]" subsys=crd-allocator
INFO[0003] Migrated identity                             identity=34815 identityLabels="k8s:class=tiefighter;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=default;k8s:io.kubernetes.pod.namespace=default;k8s:org=empire;"
WARN[0003] ID is allocated to a different key in CRD. A new ID will be allocated for the this key  identityLabels="k8s:class=deathstar;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=default;k8s:io.kubernetes.pod.namespace=default;k8s:org=empire;" oldIdentity=11730
INFO[0003] Reusing existing global key                   key="k8s:class=deathstar;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=default;k8s:io.kubernetes.pod.namespace=default;k8s:org=empire;" subsys=allocator
INFO[0003] New ID allocated for key in CRD               identity=17281 identityLabels="k8s:class=deathstar;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=default;k8s:io.kubernetes.pod.namespace=default;k8s:org=empire;" oldIdentity=11730
INFO[0003] ID was already allocated to this key. It is already migrated  identity=17003 identityLabels="k8s:class=xwing;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=default;k8s:io.kubernetes.pod.namespace=default;k8s:org=alliance;"

Note

It is also possible to use the --k8s-kubeconfig-path and --kvstore-opt cilium CLI options with the preflight command. The default is to derive the configuration as cilium-agent does.

cilium preflight migrate-identity --k8s-kubeconfig-path /var/lib/cilium/cilium.kubeconfig --kvstore etcd --kvstore-opt etcd.config=/var/lib/cilium/etcd-config.yml

Once the migration is complete, confirm the endpoint identities match by listing the endpoints stored in CRDs and in etcd:

$ kubectl get ciliumendpoints -A # new CRD-backed endpoints
$ kubectl exec -n kube-system cilium-1234 -- cilium-dbg endpoint list # existing etcd-backed endpoints

Clearing CRD identities

If a migration has gone wrong, it possible to start with a clean slate. Ensure that no Cilium instances are running with --identity-allocation-mode=crd and execute:

$ kubectl delete ciliumid --all
Migration with the “Double Write” identity allocation mode

Note

This is a beta feature. Please provide feedback and file a GitHub issue if you experience any problems.

The “Double Write” Identity Allocation Mode allows Cilium to allocate identities as KVStore values and as CRDs at the same time. This mode also has two versions: one where the source of truth comes from the kvstore (--identity-allocation-mode=doublewrite-readkvstore), and one where the source of truth comes from CRDs (--identity-allocation-mode=doublewrite-readcrd).

Note

“Double Write” mode is not compatible with Consul as the KVStore

The high-level migration plan looks as follows:

  1. Starting state: Cilium is running in KVStore mode.

  2. Switch Cilium to “Double Write” mode with all reads happening from the KVStore. This is almost the same as the pure KVStore mode with the only difference being that all identities are duplicated as CRDs but are not used.

  3. Switch Cilium to “Double Write” mode with all reads happening from CRDs. This is equivalent to Cilium running in pure CRD mode but identities will still be updated in the KVStore to allow for the possibility of a fast rollback.

  4. Switch Cilium to CRD mode. The KVStore will no longer be used and will be ready for decommission.

This will allow you to perform a gradual and seamless migration with the possibility of a fast rollback at steps two or three.

Furthermore, when the “Double Write” mode is enabled, the Operator will emit additional metrics to help monitor the migration progress. These metrics can be used for alerting about identity inconsistencies between the KVStore and CRDs.

Note that you can also use this to migrate from CRD to KVStore mode. All operations simply need to be repeated in reverse order.

Rollout Instructions

  1. Re-deploy first the Operator and then the Agents with --identity-allocation-mode=doublewrite-readkvstore.

  2. Monitor the Operator metrics and logs to ensure that all identities have converged between the KVStore and CRDs. The relevant metrics emitted by the Operator are:

    • cilium_operator_identity_crd_total_count and cilium_operator_identity_kvstore_total_count report the total number of identities in CRDs and KVStore respectively.

    • cilium_operator_identity_crd_only_count and cilium_operator_identity_kvstore_only_count report the number of identities that are only in CRDs or only in the KVStore respectively, to help detect inconsistencies.

    In case further investigation is needed, the Operator logs will contain detailed information about the discrepancies between KVStore and CRD identities. Note that Garbage Collection for KVStore identities and CRD identities happens at slightly different times, so it is possible to see discrepancies in the metrics for certain periods of time, depending on --identity-gc-interval and --identity-heartbeat-timeout settings.

  3. Once all identities have converged, re-deploy the Operator and the Agents with --identity-allocation-mode=doublewrite-readcrd. This will cause Cilium to read identities only from CRDs, but continue to write them to the KVStore.

  4. Once you are ready to decommission the KVStore, re-deploy first the Agents and then the Operator with --identity-allocation-mode=crd. This will make Cilium read and write identities only to CRDs.

  5. You can now decommission the KVStore.

CNP Validation

Running the CNP Validator will make sure the policies deployed in the cluster are valid. It is important to run this validation before an upgrade so it will make sure Cilium has a correct behavior after upgrade. Avoiding doing this validation might cause Cilium from updating its NodeStatus in those invalid Network Policies as well as in the worst case scenario it might give a false sense of security to the user if a policy is badly formatted and Cilium is not enforcing that policy due a bad validation schema. This CNP Validator is automatically executed as part of the pre-flight check Running pre-flight check (Required).

Start by deployment the cilium-pre-flight-check and check if the Deployment shows READY 1/1, if it does not check the pod logs.

$ kubectl get deployment -n kube-system cilium-pre-flight-check -w
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cilium-pre-flight-check   0/1     1            0           12s

$ kubectl logs -n kube-system deployment/cilium-pre-flight-check -c cnp-validator --previous
level=info msg="Setting up kubernetes client"
level=info msg="Establishing connection to apiserver" host="https://172.20.0.1:443" subsys=k8s
level=info msg="Connected to apiserver" subsys=k8s
level=info msg="Validating CiliumNetworkPolicy 'default/cidr-rule': OK!
level=error msg="Validating CiliumNetworkPolicy 'default/cnp-update': unexpected validation error: spec.labels: Invalid value: \"string\": spec.labels in body must be of type object: \"string\""
level=error msg="Found invalid CiliumNetworkPolicy"

In this example, we can see the CiliumNetworkPolicy in the default namespace with the name cnp-update is not valid for the Cilium version we are trying to upgrade. In order to fix this policy we need to edit it, we can do this by saving the policy locally and modify it. For this example it seems the .spec.labels has set an array of strings which is not correct as per the official schema.

$ kubectl get cnp -n default cnp-update -o yaml > cnp-bad.yaml
$ cat cnp-bad.yaml
  apiVersion: cilium.io/v2
  kind: CiliumNetworkPolicy
  [...]
  spec:
    endpointSelector:
      matchLabels:
        id: app1
    ingress:
    - fromEndpoints:
      - matchLabels:
          id: app2
      toPorts:
      - ports:
        - port: "80"
          protocol: TCP
    labels:
    - custom=true
  [...]

To fix this policy we need to set the .spec.labels with the right format and commit these changes into Kubernetes.

$ cat cnp-bad.yaml
  apiVersion: cilium.io/v2
  kind: CiliumNetworkPolicy
  [...]
  spec:
    endpointSelector:
      matchLabels:
        id: app1
    ingress:
    - fromEndpoints:
      - matchLabels:
          id: app2
      toPorts:
      - ports:
        - port: "80"
          protocol: TCP
    labels:
    - key: "custom"
      value: "true"
  [...]
$
$ kubectl apply -f ./cnp-bad.yaml

After applying the fixed policy we can delete the pod that was validating the policies so that Kubernetes creates a new pod immediately to verify if the fixed policies are now valid.

$ kubectl delete pod -n kube-system -l k8s-app=cilium-pre-flight-check-deployment
pod "cilium-pre-flight-check-86dfb69668-ngbql" deleted
$ kubectl get deployment -n kube-system cilium-pre-flight-check
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cilium-pre-flight-check   1/1     1            1           55m
$ kubectl logs -n kube-system deployment/cilium-pre-flight-check -c cnp-validator
level=info msg="Setting up kubernetes client"
level=info msg="Establishing connection to apiserver" host="https://172.20.0.1:443" subsys=k8s
level=info msg="Connected to apiserver" subsys=k8s
level=info msg="Validating CiliumNetworkPolicy 'default/cidr-rule': OK!
level=info msg="Validating CiliumNetworkPolicy 'default/cnp-update': OK!
level=info msg="All CCNPs and CNPs valid!"

Once they are valid you can continue with the upgrade process. Clean up pre-flight check