Layer 7 Protocol Visibility
Note
This feature requires enabling L7 Proxy support. Without it, the visibility annotation is ignored.
While Monitoring Datapath State provides introspection into datapath state, by default it will only provide visibility into L3/L4 packet events. If Layer 7 Examples are configured, one can get visibility into L7 protocols, but this requires the full policy for each selected endpoint to be written. To get more visibility into the application without configuring a full policy, Cilium provides a means of prescribing visibility via annotations when running in tandem with Kubernetes.
Visibility information is represented by a comma-separated list of tuples in the annotation:
<{Traffic Direction}/{L4 Port}/{L4 Protocol}/{L7 Protocol}>
For example:
<Egress/53/UDP/DNS>,<Egress/80/TCP/HTTP>
To do this, you can provide the annotation in your Kubernetes YAMLs, or via the command line, e.g.:
kubectl annotate pod foo -n bar policy.cilium.io/proxy-visibility="<Egress/53/UDP/DNS>,<Egress/80/TCP/HTTP>"
Cilium will pick up that pods have received these annotations, and will
transparently redirect traffic to the proxy such that the output of
cilium monitor shows traffic being redirected to the proxy, e.g.:
-> Request http from 1474 ([k8s:id=app2 k8s:io.kubernetes.pod.namespace=default k8s:appSecond=true k8s:io.cilium.k8s.policy.cluster=default k8s:io.cilium.k8s.policy.serviceaccount=app2-account k8s:zgroup=testapp]) to 244 ([k8s:io.cilium.k8s.policy.cluster=default k8s:io.cilium.k8s.policy.serviceaccount=app1-account k8s:io.kubernetes.pod.namespace=default k8s:zgroup=testapp k8s:id=app1]), identity 30162->42462, verdict Forwarded GET http://app1-service/ => 0
-> Response http to 1474 ([k8s:zgroup=testapp k8s:id=app2 k8s:io.kubernetes.pod.namespace=default k8s:appSecond=true k8s:io.cilium.k8s.policy.cluster=default k8s:io.cilium.k8s.policy.serviceaccount=app2-account]) from 244 ([k8s:io.cilium.k8s.policy.serviceaccount=app1-account k8s:io.kubernetes.pod.namespace=default k8s:zgroup=testapp k8s:id=app1 k8s:io.cilium.k8s.policy.cluster=default]), identity 30162->42462, verdict Forwarded GET http://app1-service/ => 200
You can check the status of the visibility policy by checking the Cilium endpoint of that pod, for example:
$ kubectl get cep -n kube-system
NAME ENDPOINT ID IDENTITY ID INGRESS ENFORCEMENT EGRESS ENFORCEMENT VISIBILITY POLICY ENDPOINT STATE IPV4 IPV6
coredns-7d7f5b7685-wvzwb 1959 104 false false ready 10.16.75.193 f00d::a10:0:0:2c77
$
$ kubectl annotate pod -n kube-system coredns-7d7f5b7685-wvzwb policy.cilium.io/proxy-visibility="<Egress/53/UDP/DNS>,<Egress/80/TCP/HTTP>" --overwrite
pod/coredns-7d7f5b7685-wvzwb annotated
$
$ kubectl get cep -n kube-system
NAME ENDPOINT ID IDENTITY ID INGRESS ENFORCEMENT EGRESS ENFORCEMENT VISIBILITY POLICY ENDPOINT STATE IPV4 IPV6
coredns-7d7f5b7685-wvzwb 1959 104 false false OK ready 10.16.75.193 f00d::a10:0:0:2c7
In order for Cilium to populate the INGRESS ENFORCEMENT, EGRESS ENFORCEMENT
and VISIBILITY POLICY fields, it must run with --endpoint-status=policy
to make field values visible.
Security Implications
Monitoring Layer 7 traffic involves security considerations for handling potentially sensitive information, such as usernames, passwords, query parameters, API keys, and others.
Warning
By default, Hubble does not redact potentially sensitive information present in Layer 7 Hubble Flows.
To harden security, Cilium provides the --hubble-redact-enabled option which
enables Hubble to handle sensitive information present in Layer 7 flows.
More specifically, it offers the following features for supported Layer 7 protocols:
For HTTP: redacting URL query (GET) parameters (
--hubble-redact-http-urlquery)For Kafka: redacting API key (
--hubble-redact-kafka-apikey)
For more information on configuring Cilium, see Cilium Configuration.
Troubleshooting
If L7 visibility is not appearing in cilium monitor or Hubble components,
it is worth double-checking that:
No enforcement policy is applied in the direction specified in the annotation
The “Visibility Policy” column in the CiliumEndpoint shows
OK. If it is blank, then no annotation is configured; if it shows an error then there is a problem with the visibility annotation.
The following example deliberately misconfigures the annotation to demonstrate that the CiliumEndpoint for the pod presents an error when the visibility annotation cannot be implemented:
$ kubectl annotate pod -n kube-system coredns-7d7f5b7685-wvzwb policy.cilium.io/proxy-visibility="<Ingress/53/UDP/DNS>,<Egress/80/TCP/HTTP>"
pod/coredns-7d7f5b7685-wvzwb annotated
$
$ kubectl get cep -n kube-system
NAME ENDPOINT ID IDENTITY ID INGRESS ENFORCEMENT EGRESS ENFORCEMENT VISIBILITY POLICY ENDPOINT STATE IPV4 IPV6
coredns-7d7f5b7685-wvzwb 1959 104 false false dns not allowed with direction Ingress ready 10.16.75.193 f00d::a10:0:0:2c77
Limitations
Visibility annotations do not apply if rules are imported which select the pod which is annotated.
DNS visibility is available on egress only.