Service Affinity

This tutorial will guide you to enable service affinity across multiple Kubernetes clusters.

Prerequisites

You need to have a functioning Cluster Mesh with a Global Service, please follow the guide Setting up Cluster Mesh and Load-balancing & Service Discovery to set it up.

Enabling Global Service Affinity

Load-balancing across multiple clusters might not be ideal in some cases. The annotation service.cilium.io/affinity: "local|remote|none" can be used to specify the preferred endpoint destination.

For example, if the value of annotation service.cilium.io/affinity is local, the Global Service will load-balance across healthy local backends, and only user remote endpoints if and only if all of local backends are not available or unhealthy.

apiVersion: v1
kind: Service
metadata:
  name: rebel-base
  annotations:
     service.cilium.io/global: "true"
     # Possible values:
     # - local
     #    preferred endpoints from local cluster if available
     # - remote
     #    preferred endpoints from remote cluster if available
     # none (default)
     #    no preference. Default behavior if this annotation does not exist
     service.cilium.io/affinity: "local"
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    name: rebel-base

In cluster 1, add service.cilium.io/affinity="local" to existing global service

kubectl annotate service rebel-base service.cilium.io/affinity=local --overwrite

From cluster 1, access the global service:
```
kubectl exec -ti deployment/x-wing -- curl rebel-base
```
You will see replies from pods in cluster 1 only.
From cluster 2, access the global service:
```
kubectl exec -ti deployment/x-wing -- curl rebel-base
```
You will see replies from pods in both clusters as usual.

From cluster 1, check the service endpoints, the local endpoints are marked as preferred.

kubectl exec -n kube-system -ti ds/cilium -- cilium-dbg service list --clustermesh-affinity

ID   Frontend            Service Type   Backend
1    10.96.0.1:443       ClusterIP      1 => 172.18.0.3:6443 (active)
2    10.96.0.10:53       ClusterIP      1 => 10.244.1.171:53 (active)
                                        2 => 10.244.2.206:53 (active)
3    10.96.0.10:9153     ClusterIP      1 => 10.244.1.171:9153 (active)
                                        2 => 10.244.2.206:9153 (active)
4    10.96.210.49:2379   ClusterIP      1 => 10.244.2.216:2379 (active)
5    10.96.173.113:80    ClusterIP      1 => 10.244.2.136:80 (active)
                                        2 => 10.244.1.61:80 (active) (preferred)
                                        3 => 10.244.2.31:80 (active) (preferred)
                                        4 => 10.244.2.200:80 (active)

In cluster 1, change service.cilium.io/affinity value to remote for existing global service
```
kubectl annotate service rebel-base service.cilium.io/affinity=remote --overwrite
```
From cluster 1, access the global service:
```
kubectl exec -ti deployment/x-wing -- curl rebel-base
```
This time, the replies are coming from pods in cluster 2 only.

From cluster 1, check the service endpoints, now the remote endpoints are marked as preferred.

kubectl exec -n kube-system -ti ds/cilium -- cilium-dbg service list --clustermesh-affinity

ID   Frontend            Service Type   Backend
1    10.96.0.1:443       ClusterIP      1 => 172.18.0.3:6443 (active)
2    10.96.0.10:53       ClusterIP      1 => 10.244.1.171:53 (active)
                                        2 => 10.244.2.206:53 (active)
3    10.96.0.10:9153     ClusterIP      1 => 10.244.1.171:9153 (active)
                                        2 => 10.244.2.206:9153 (active)
4    10.96.210.49:2379   ClusterIP      1 => 10.244.2.216:2379 (active)
5    10.96.173.113:80    ClusterIP      1 => 10.244.2.136:80 (active) (preferred)
                                        2 => 10.244.1.61:80 (active)
                                        3 => 10.244.2.31:80 (active)
                                        4 => 10.244.2.200:80 (active) (preferred)

From cluster 2, access the global service:
```
kubectl exec -ti deployment/x-wing -- curl rebel-base
```
You will see replies from pods in both clusters as usual.
In cluster 1, remove service.cilium.io/affinity annotation for existing global service
```
kubectl annotate service rebel-base service.cilium.io/affinity- --overwrite
```
From either cluster, access the global service:
```
kubectl exec -ti deployment/x-wing -- curl rebel-base
```
You will see replies from pods in both clusters again.