Gateway API Support

What is Gateway API?

Gateway API is a Kubernetes SIG-Network subproject to design a successor for the Ingress object. It is a set of resources that model service networking in Kubernetes, and is designed to be role-oriented, portable, expressive, and extensible.

See the Gateway API site for more details.

Cilium Gateway API Support

Cilium supports Gateway API v1.0.0 for below resources, all the Core conformance tests are passed.

Prerequisites

Cilium must be configured with kubeProxyReplacement=true. Please refer to kube-proxy replacement for more details.
Cilium must be configured with the L7 proxy enabled using the --enable-l7-proxy flag (enabled by default).

The below CRDs from Gateway API v1.0.0 must be pre-installed. Please refer to this docs for installation steps. Alternatively, the below snippet could be used.

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml

Similar to Ingress, Gateway API controller creates a service of LoadBalancer type, so your environment will need to support this.

Installation

Warning

Make sure you install cilium-cli v0.15.0 or later. The rest of instructions do not work with older versions of cilium-cli. To confirm the cilium-cli version that’s installed in your system, run:

cilium version --client

See Cilium CLI upgrade notes for more details.

Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "arm64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}
shasum -a 256 -c cilium-darwin-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-darwin-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum}

Clone the Cilium GitHub repository so that the Cilium CLI can access the latest unreleased Helm chart from the main branch:

git clone git@github.com:cilium/cilium.git
cd cilium

Cilium Gateway API Controller can be enabled with helm flag gatewayAPI.enabled set as true. Please refer to Installation using Helm for a fresh installation.

$ helm upgrade cilium ./cilium \
    --namespace kube-system \
    --reuse-values \
    --set kubeProxyReplacement=true \
    --set gatewayAPI.enabled=true

$ kubectl -n kube-system rollout restart deployment/cilium-operator
$ kubectl -n kube-system rollout restart ds/cilium

Next you can check the status of the Cilium agent and operator:

$ cilium status

Cilium Gateway API Controller can be enabled with the below command

$ cilium install --chart-directory ./install/kubernetes/cilium \
    --set kubeProxyReplacement=true \
    --set gatewayAPI.enabled=true

Next you can check the status of the Cilium agent and operator:

$ cilium status

Examples

Please refer to one of the below examples on how to use and leverage Cilium’s Gateway API features:

More examples can be found in the upstream repository.

Troubleshooting

This page guides you through the different mechanics of Gateway API and how to troubleshoot them.

Be sure to follow the Generic and Setup Verification steps from the Troubleshooting Ingress & Service Mesh page.

Checking resources

Check the Gateway resource
$ kubectl get gateway -A NAMESPACE NAME CLASS ADDRESS PROGRAMMED AGE website http-gateway cilium 172.21.255.202 True 5h webshop tls-gateway cilium 172.21.255.203 True 5h
The preceding command returns an overview of all the Gateways in the cluster. Check the following:
- Is the Gateway programmed?
  
  A programmed Gateway means that Cilium prepared a configuration for it.
  - If the Programmed true indicator is missing, make sure that Gateway API is enabled in the Cilium configuration.
- Does the gateway have an address?
You can check the service with kubectl get service. If the gateway has an address, it means that a LoadBalancer service is assigned to the gateway. If no IP appears, you might be missing a LoadBalancer implementation.
- Is the class cilium?
Cilium only programs Gateways with the class cilium.
- If the Gateway API resource type (Gateway, HTTPRoute, etc.) is not found, make sure that the Gateway API CRDs are installed.
You can use kubectl describe gateway to investigate issues more thoroughly.
$ kubectl describe gateway <name> Conditions: Message: Gateway successfully scheduled Reason: Accepted Status: True Type: Accepted Message: Gateway successfully reconciled Reason: Programmed Status: True Type: Programmed [...] Listeners: Attached Routes: 2 Conditions: Message: Listener Ready Reason: Programmed Status: True Type: Programmed Message: Listener Accepted Reason: Accepted Status: True [...]
You can see the general status of the gateway as well as the status of the configured listeners.

Listener status displays the number of routes successfully attached to the listener.

You can see status conditions for both gateway and listener:
Accepted: the Gateway configuration was accepted.

Programmed: the Gateway configuration was programmed into Envoy.

ResolvedRefs: all referenced secrets were found and have permission for use.
If any of these conditions are set to false, the Message and Reason fields give more information.
Check the HTTPRoute resource

When the Gateway is functional, you can check the routes to verify if they are configured correctly. The way to check route status is similar to checking the status of a gateway resource.

While these instructions are written for HTTPRoute, they also apply to other route types.
$ kubectl get httproute -A
NAMESPACE                 NAME              HOSTNAMES         AGE
website                   homepage          www.example.org   17m
webshop                   catalog-service                     17m
webshop                   cart-service                        17m
To get more information, enter kubectl describe httproute <name>.
$ kubectl describe httproute <name>
Status:
  Parents:
    Conditions:
      Last Transition Time:  2023-06-05T15:11:53Z
      Message:               Accepted HTTPRoute
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2023-06-05T15:11:53Z
      Message:               Service reference is valid
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Controller Name:         io.cilium/gateway-controller
    Parent Ref:
      Group:  gateway.networking.k8s.io
      Kind:   Gateway
      Name:   same-namespace
Status lists the conditions that are relevant for the specific HTTPRoute. Conditions are listed by parent reference to the gateway. If you linked the route to multiple gateways, multiple entries appear. Conditions include Reason, Type, Status and Message. Type indicates the condition type, and Status indicates with a boolean whether the condition type is met. Optionally, Message gives you more information about the condition.

Notice the following condition types:

Accepted: The HTTPRoute configuration was correct and accepted.

ResolvedRefs: The referenced services were found and are valid references.

If any of these are set to false, you can get more information by looking at the Message and Reason fields.

Common mistakes

Warning

Gateway API is a recent addition to Kubernetes, and the Cilium project has not yet received much user feedback. If you encounter an issue that is not yet listed in this section, consider opening a PR to add your issue to the list.

The backend service does not exist.

To verify whether the backend service was found, run kubectl describe httproute <name> and inspect the conditions field:

Parents:
  Conditions:
    Last Transition Time:  2023-06-06T13:55:10Z
    Message:               Service "backend" not found
    Observed Generation:   1
    Reason:                BackendNotFound
    Status:                False
    Type:                  ResolvedRefs
    Last Transition Time:  2023-06-06T13:55:10Z
    Message:               Accepted HTTPRoute
    Observed Generation:   1
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
  Controller Name:         io.cilium/gateway-controller

The gateway specified under parentRefs does not exist.

To verify whether the gateway was found, run kubectl describe httproute <name> and inspect the conditions field:

Parents:
  Conditions:
    Last Transition Time:  2023-06-06T13:56:40Z
    Message:               Gateway.gateway.networking.k8s.io "my-gatewai" not found
    Observed Generation:   2
      Reason:                InvalidHTTPRoute
      Status:                False
      Type:                  Accepted

Underlying mechanics: a high level overview

A Cilium deployment has two parts that handle Gateway API resources: the Cilium agent and the Cilium operator.

The Cilium operator watches all Gateway API resources and verifies whether the resources are valid. If resources are valid, the operator marks them as accepted. That starts the process of translation into Cilium Envoy Configuration resources.

The Cilium agent then picks up the Cilium Envoy Configuration resources.

The Cilium agent uses the resources to supply the configuration to the built in Envoy or the Envoy DaemonSet. Envoy handles traffic.