Layer 7 Protocol Visibility

Note

This feature requires enabling L7 Proxy support.

While Monitoring Datapath State provides introspection into datapath state, by default, it will only provide visibility into L3/L4 packet events. If you want L7 protocol visibility, you can use L7 Cilium Network Policies (see Layer 7 Examples).

Note

Historically, it had been possible to enable L7 visibility using Pod annotations (policy.cilium.io/proxy-visibility). This method is no longer supported and we recommend users to switch to L7 policies instead.

To enable visibility for L7 traffic, create a CiliumNetworkPolicy that specifies L7 rules. Traffic flows matching a L7 rule in a CiliumNetworkPolicy will become visible to Cilium and, thus, can be exposed to the end user. It’s important to remember that L7 network policies not only enables visibility but also restrict what traffic is allowed to flow in and out of a Pod.

The following example enables visibility for DNS (TCP/UDP/53) and HTTP (ports TCP/80 and TCP/8080) traffic within the default namespace by specifying two L7 rules – one for DNS and one for HTTP. It also restricts egress communication and drops anything that is not matched. L7 matching conditions on the rules have been omitted or wildcarded, which will permit all requests that match the L4 section of each rule:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l7-visibility"
spec:
  endpointSelector:
    matchLabels:
      "k8s:io.kubernetes.pod.namespace": default
  egress:
  - toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": default
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      - port: "8080"
        protocol: TCP
      rules:
        http: [{}]

Based on the above policy, Cilium will pick up all TCP/UDP/53, TCP/80 and TCP/8080 egress traffic from Pods in the default namespace and redirect it to the proxy (see Proxy Injection) such that the output of cilium monitor or hubble observe shows the L7 flow details. Below is the example of running hubble observe -f -t l7 -o compact command:

default/testapp-5b9cc645cb-4slbs:45240 (ID:26450) -> kube-system/coredns-787d4945fb-bdmdq:53 (ID:9313) dns-request proxy FORWARDED (DNS Query web.default.svc.cluster.local. A)
default/testapp-5b9cc645cb-4slbs:45240 (ID:26450) <- kube-system/coredns-787d4945fb-bdmdq:53 (ID:9313) dns-response proxy FORWARDED (DNS Answer "10.96.118.37" TTL: 30 (Proxy web.default.svc.cluster.local. A))
default/testapp-5b9cc645cb-4slbs:33044 (ID:26450) -> default/echo-594485b8dc-fp57l:8080 (ID:32531) http-request FORWARDED (HTTP/1.1 GET http://web/)
default/testapp-5b9cc645cb-4slbs:33044 (ID:26450) <- default/echo-594485b8dc-fp57l:8080 (ID:32531) http-response FORWARDED (HTTP/1.1 200 4ms (GET http://web/))

Security Implications

Monitoring Layer 7 traffic involves security considerations for handling potentially sensitive information, such as usernames, passwords, query parameters, API keys, and others.

Warning

By default, Hubble does not redact potentially sensitive information present in Layer 7 Hubble Flows.

To harden security, Cilium provides the --hubble-redact-enabled option which enables Hubble to handle sensitive information present in Layer 7 flows. More specifically, it offers the following features for supported Layer 7 protocols:

  • For HTTP: redacting URL query (GET) parameters (--hubble-redact-http-urlquery)

  • For HTTP: redacting URL user info (for example, password used in basic auth) (--hubble-redact-http-userinfo)

  • For Kafka: redacting API key (--hubble-redact-kafka-apikey)

  • For HTTP headers: redacting all headers except those defined in the --hubble-redact-http-headers-allow list or redacting only the headers defined in the --hubble-redact-http-headers-deny list

For more information on configuring Cilium, see Cilium Configuration.

Limitations

  • DNS visibility is available on egress only.