Multi-Cluster Services API (Beta)

Note

This is a beta feature. Please provide feedback and file a GitHub issue if you experience any problems.

This tutorial will guide you to through the support of Multi-Cluster Services API (MCS-API) in Cilium.

Prerequisites

You need to have a functioning Cluster Mesh setup, please follow the Setting up Cluster Mesh guide to set it up.

Make sure you are running CoreDNS 1.12.2 or later (installed by default from Kubernetes 1.35).

To install Cilium with MCS-API support, run:

helm install cilium ./cilium \
   --namespace kube-system \
   --set clustermesh.mcsapi.enabled=true

To enable MCS-API support on an existing Cilium installation, run:

helm upgrade cilium ./cilium \
   --namespace kube-system \
   --reuse-values \
   --set clustermesh.mcsapi.enabled=true

Also checkout the EndpointSlice synchronization feature if you need Headless Services support.

If you set clustermesh.mcsapi.corednsAutoConfigure.enabled to true, Cilium will automatically configure and rollout CoreDNS for MCS-API support. Otherwise to configure CoreDNS manually, you need to execute the following steps:

# (optional) Install MCS-API CRDs if Cilium has not yet started or automatic CRDs installation is disabled
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/vendor/sigs.k8s.io/mcs-api/config/crd/multicluster.x-k8s.io_serviceexports.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/vendor/sigs.k8s.io/mcs-api/config/crd/multicluster.x-k8s.io_serviceimports.yaml

# Adding RBAC to read ServiceImports
kubectl create clusterrole coredns-mcsapi \
   --verb=list,watch --resource=serviceimports.multicluster.x-k8s.io
kubectl create clusterrolebinding coredns-mcsapi \
   --clusterrole=coredns-mcsapi --serviceaccount=kube-system:coredns

# Configure CoreDNS to support MCS-API
kubectl get configmap -n kube-system coredns -o yaml | \
   sed -e 's/cluster.local/cluster.local clusterset.local/g' | \
   sed -E 's/^(.*)kubernetes(.*){/1kubernetes2{n1   multicluster clusterset.local/' | \
   kubectl replace -f-

# Rollout CoreDNS to apply the change
kubectl rollout deployment -n kube-system coredns

Exporting a Service

To export a service you should create a ServiceExport resource. As a result your Service will be exported to all clusters, provided that the Service Namespace is present on those clusters.

apiVersion: multicluster.x-k8s.io/v1beta1
kind: ServiceExport
metadata:
   name: rebel-base

In all the clusters and for each set of exported Services that have the same name and namespace, a ServiceImport resource will be automatically created. All the Endpoints from those exported Services with the same name and namespace will be merged and made globally available.

An exported Service through MCS-API is available by default on the <svc>.<ns>.svc.clusterset.local domain. If you have defined any hostname (via a Statefulset for instance) on your pods each pods would also be available available through the <hostname>.<clustername>.<svc>.<ns>.svc.clusterset.local domain.

Note

The <clustername>.<svc>.<ns>.svc.clusterset.local domain that would allow to get all the endpoints of a Service in a specific cluster is not allowed!

We recommend creating one service per cluster and/or region and exporting it accordingly if you do want to have this kind of behavior, for instance creating and exporting services mysvc-eu and mysvc-us instead of only one service. For more information checkout the dedicated section in the MCS-API KEP explaining this behavior.

The ServiceImport has also a logic to merge different Service properties:

SessionAffinity
Ports (Union of the different ServiceExports)
Type (ClusterSetIP/Headless)
Annotations & Labels (via the ServiceExport exportedLabels and exportedAnnotations fields)

If any conflict arises on any of these properties, the oldest ServiceExport will have precedence to resolve the conflict. This means that you should get a consistent behavior globally for the same set of exported Services that has the same name and namespace. If any conflicts arises, you would be able to see details about it in the ServiceExport status Conditions.

Deploying a Simple Example Service using MCS-API

In cluster 1, deploy:

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/clustermesh/cluster1.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/clustermesh/mcsapi-example.yaml

In cluster 2, deploy:

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/clustermesh/cluster2.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/clustermesh/mcsapi-example.yaml

From either cluster, access the exported service:
```
kubectl exec -ti deployment/x-wing -- curl rebel-base-mcsapi.default.svc.clusterset.local
```
You will see replies from pods in both clusters.

Gateway-API

Gateway-API has optional support for MCS-API via GEP1748 by specifying a ServiceImport backend, for example:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
   name: rebel-base-mcsapi
   namespace: default
spec:
   parentRefs:
   - group: gateway.networking.k8s.io
      kind: Gateway
      name: my-gateway
      namespace: default
   rules:
   - backendRefs:
      - group: multicluster.x-k8s.io
         kind: ServiceImport
         name: rebel-base-mcsapi
         port: 80
      matches:
      - method: GET
         path:
         type: PathPrefix
         value: /

The Gateway API implementation of Cilium fully support its own MCS-API implementation.

If you want to use another Gateway API implementation with the Cilium MCS-API implementation, the Gateway API implementation you are using should officially support MCS-API / GEP1748.

On the other hands, the Cilium Gateway API implementation only supports MCS-API implementations using an underlying Service associated with a ServiceImport, and with the annotation multicluster.kubernetes.io/derived-service on ServiceImport resources.