Mutual Authentication Example

This example shows you how to enforce mutual authentication between two Pods.

Deploy a client (pod-worker) and a server (echo) using the following manifest:

$ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.16.3/examples/kubernetes/servicemesh/mutual-auth-example.yaml
$ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.16.3/examples/kubernetes/servicemesh/cnp-without-mutual-auth.yaml
service/echo created
deployment.apps/echo created
pod/pod-worker created
ciliumnetworkpolicy.cilium.io/no-mutual-auth-echo created

Verify that the Pods have been successfully deployed:

$ kubectl get svc echo
NAME   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
echo   ClusterIP   10.96.16.90   <none>        8080/TCP   42m
$ kubectl get pod pod-worker
NAME         READY   STATUS    RESTARTS   AGE
pod-worker   1/1     Running   0          40m

Verify that the network policy has been deployed successfully and filters the traffic as expected.

Run the following commands:

$ kubectl exec -it pod-worker -- curl -s -o /dev/null -w "%{http_code}" http://echo:8080/headers
200
$ kubectl exec -it pod-worker -- curl http://echo:8080/headers-1
Access denied

The first request should be successful (the pod-worker Pod is able to connect to the echo Service over a specific HTTP path and the HTTP status code is 200). The second one should be denied (the pod-worker Pod is unable to connect to the echo Service over a specific HTTP path other than ‘/headers’).

Before we enable mutual authentication between pod-worker and echo, let’s verify that the SPIRE server is healthy.

Assuming you have followed the installation instructions and have a SPIRE server serving Cilium, adding mutual authentication simply requires adding authentication.mode: "required" in the ingress/egress block in your network policies.

Verify SPIRE Health

Note

This example assumes a default SPIRE installation.

Let’s first verify that the SPIRE server and agents automatically deployed are working as expected.

The SPIRE server is deployed as a StatefulSet and the SPIRE agents are deployed as a DaemonSet (you should therefore see one SPIRE agent per node).

$ kubectl get all -n cilium-spire
NAME                    READY   STATUS    RESTARTS   AGE
pod/spire-agent-27jd7   1/1     Running   0          144m
pod/spire-agent-qkc8l   1/1     Running   0          144m
pod/spire-server-0      2/2     Running   0          144m

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/spire-server   ClusterIP   10.96.124.177   <none>        8081/TCP   144m

NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/spire-agent   2         2         2       2            2           <none>          144m

NAME                            READY   AGE
statefulset.apps/spire-server   1/1     144m

Run a healthcheck on the SPIRE server.

$ kubectl exec -n cilium-spire spire-server-0 -c spire-server -- /opt/spire/bin/spire-server healthcheck
Server is healthy.

Verify the list of attested agents:

$ kubectl exec -n cilium-spire spire-server-0 -c spire-server -- /opt/spire/bin/spire-server agent list
Found 2 attested agents:

SPIFFE ID         : spiffe://spiffe.cilium/spire/agent/k8s_psat/default/64745bf2-bd9d-4e42-bb2b-e095a6b65121
Attestation type  : k8s_psat
Expiration time   : 2023-07-04 18:39:50 +0000 UTC
Serial number     : 110848236251310359782141595494072495768

SPIFFE ID         : spiffe://spiffe.cilium/spire/agent/k8s_psat/default/d4a8a6da-d808-4993-b67a-bed250bbc53e
Attestation type  : k8s_psat
Expiration time   : 2023-07-04 18:39:55 +0000 UTC
Serial number     : 7806033782886940845084156064765627978

Notice that the SPIRE Server uses Kubernetes Projected Service Account Tokens (PSATs) to verify the Identity of a SPIRE Agent running on a Kubernetes Cluster. Projected Service Account Tokens provide additional security guarantees over traditional Kubernetes Service Account Tokens and when supported by a Kubernetes cluster, PSAT is the recommended attestation strategy.

Verify SPIFFE Identities

Now that we know the SPIRE service is healthy, let’s verify that the Cilium and SPIRE integration has been successful:

  • The Cilium agent and operator should have a registered delegate Identity with the SPIRE Server.

  • The Cilium operator should have registered Identities with the SPIRE server on behalf of the workloads (Kubernetes Pods).

Verify that the Cilium agent and operator have Identities on the SPIRE server:

$ kubectl exec -n cilium-spire spire-server-0 -c spire-server -- /opt/spire/bin/spire-server entry show -parentID spiffe://spiffe.cilium/ns/cilium-spire/sa/spire-agent
Found 2 entries
Entry ID         : b6424c87-4323-4d64-98dd-cd5b51a1fcbb
SPIFFE ID        : spiffe://spiffe.cilium/cilium-agent
Parent ID        : spiffe://spiffe.cilium/ns/cilium-spire/sa/spire-agent
Revision         : 0
X509-SVID TTL    : default
JWT-SVID TTL     : default
Selector         : k8s:ns:kube-system
Selector         : k8s:sa:cilium

Entry ID         : 8aa91d65-16c4-48a0-bc1f-c9bf26e6a25f
SPIFFE ID        : spiffe://spiffe.cilium/cilium-operator
Parent ID        : spiffe://spiffe.cilium/ns/cilium-spire/sa/spire-agent
Revision         : 0
X509-SVID TTL    : default
JWT-SVID TTL     : default
Selector         : k8s:ns:kube-system
Selector         : k8s:sa:cilium-operator

Next, verify that the echo Pod has an Identity registered with the SPIRE server.

To do this, you must first construct the Pod’s SPIFFE ID. The SPIFFE ID for a workload is based on the spiffe://spiffe.cilium/identity/$IDENTITY_ID format, where $IDENTITY_ID is a workload’s Cilium Identity.

Grab the Cilium Identity for the echo Pod;

$ IDENTITY_ID=$(kubectl get cep -l app=echo -o=jsonpath='{.items[0].status.identity.id}')
$ echo $IDENTITY_ID
17947

Use the Cilium Identity for the echo pod to construct its SPIFFE ID and check it is registered on the SPIRE server:

$ kubectl exec -n cilium-spire spire-server-0 -c spire-server -- /opt/spire/bin/spire-server entry show -spiffeID spiffe://spiffe.cilium/identity/$IDENTITY_ID
Found 1 entry
Entry ID         : 9fc13971-fb19-4814-b9f0-737b30e336c6
SPIFFE ID        : spiffe://spiffe.cilium/identity/17947
Parent ID        : spiffe://spiffe.cilium/cilium-operator
Revision         : 0
X509-SVID TTL    : default
JWT-SVID TTL     : default
Selector         : cilium:mutual-auth

You can see the that the cilium-operator was listed in the Parent ID. That is because the Cilium operator creates SPIRE entries for Cilium Identities as they are created.

To get all registered entries, execute the following command:

kubectl exec -n cilium-spire spire-server-0 -c spire-server -- /opt/spire/bin/spire-server entry show -selector cilium:mutual-auth

There are as many entries as there are identities. Verify that these match by running the command:

kubectl get ciliumidentities

The identify ID listed under NAME should match with the digits at the end of the SPIFFE ID executed in the previous command.

Enforce Mutual Authentication

Rolling out mutual authentication with Cilium is as simple as adding the following block to an existing or new CiliumNetworkPolicy egress or ingress rules:

authentication:
    mode: "required"

Update the existing rule to only allow ingress access to mutually authenticated workloads to access echo using:

$ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.16.3/examples/kubernetes/servicemesh/cnp-with-mutual-auth.yaml

Verify Mutual Authentication

Re-try your connectivity tests. They should give similar results as before:

$ kubectl exec -it pod-worker -- curl -s -o /dev/null -w "%{http_code}" http://echo:8080/headers
200
$ kubectl exec -it pod-worker -- curl http://echo:8080/headers-1
Access denied

Verify that mutual authentication has happened by accessing the logs on the agent.

Start by enabling debug level:

cilium config set debug true

Examine the logs on the Cilium agent located in the same node as the echo Pod. For brevity, you can search for some specific log messages:

$ kubectl -n kube-system -c cilium-agent logs cilium-9pshw --timestamps=true | grep "Policy is requiring authentication\|Validating Server SNI\|Validated certificate\|Successfully authenticated"
2023-07-04T17:58:28.795760597Z level=debug msg="Policy is requiring authentication" key="localIdentity=17947, remoteIdentity=39239, remoteNodeID=54264, authType=spire" subsys=auth
2023-07-04T17:58:28.800509503Z level=debug msg="Validating Server SNI" SNI ID=39239 subsys=auth
2023-07-04T17:58:28.800525190Z level=debug msg="Validated certificate" subsys=auth uri-san="[spiffe://spiffe.cilium/identity/39239]"
2023-07-04T17:58:28.801441968Z level=debug msg="Successfully authenticated" key="localIdentity=17947, remoteIdentity=39239, remoteNodeID=54264, authType=spire" remote_node_ip=10.0.1.175 subsys=auth

When you apply a mutual authentication policy, the agent retrieves the identity of the source Pod, connects to the node where the destination Pod is running and performs a mutual TLS handshake (with the log above showing one side of the mutual TLS handshake). As the handshake succeeded, the connection was authenticated and the traffic protected by policy could proceed.

Packets between the two Pods can flow until the network policy is removed or the entry expires.