Gateway API Support

What is Gateway API?

Gateway API is a Kubernetes SIG-Network subproject to design a successor for the Ingress object. It is a set of resources that model service networking in Kubernetes, and is designed to be role-oriented, portable, expressive, and extensible.

See the Gateway API site for more details.

Cilium Gateway API Support

Cilium supports Gateway API v1.0.0 for below resources, all the Core conformance tests are passed.

Prerequisites

  • Cilium must be configured with NodePort enabled, using nodePort.enabled=true or by enabling the kube-proxy replacement with kubeProxyReplacement=true. For more information, see kube-proxy replacement.

  • Cilium must be configured with the L7 proxy enabled using the --enable-l7-proxy flag (enabled by default).

  • The below CRDs from Gateway API v1.0.0 must be pre-installed. Please refer to this docs for installation steps. Alternatively, the below snippet could be used.

    $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
    $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
    $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
    $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
    $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml
    $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml
    
  • Similar to Ingress, Gateway API controller creates a service of LoadBalancer type, so your environment will need to support this.

Installation

Warning

Make sure you install cilium-cli v0.15.0 or later. The rest of instructions do not work with older versions of cilium-cli. To confirm the cilium-cli version that’s installed in your system, run:

cilium version --client

See Cilium CLI upgrade notes for more details.

Install the latest version of the Cilium CLI. The Cilium CLI can be used to install Cilium, inspect the state of a Cilium installation, and enable/disable various features (e.g. clustermesh, Hubble).

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

Clone the Cilium GitHub repository so that the Cilium CLI can access the latest unreleased Helm chart from the main branch:

git clone git@github.com:cilium/cilium.git
cd cilium

Cilium Gateway API Controller can be enabled with helm flag gatewayAPI.enabled set as true. Please refer to Installation using Helm for a fresh installation.

$ helm upgrade cilium ./cilium \
    --namespace kube-system \
    --reuse-values \
    --set kubeProxyReplacement=true \
    --set gatewayAPI.enabled=true

$ kubectl -n kube-system rollout restart deployment/cilium-operator
$ kubectl -n kube-system rollout restart ds/cilium

Next you can check the status of the Cilium agent and operator:

$ cilium status

Examples

Please refer to one of the below examples on how to use and leverage Cilium’s Gateway API features:

More examples can be found in the upstream repository.

Troubleshooting

This page guides you through the different mechanics of Gateway API and how to troubleshoot them.

Be sure to follow the Generic and Setup Verification steps from the Troubleshooting Ingress & Service Mesh page.

Checking resources

  1. Check the Gateway resource

    $ kubectl get gateway -A
    NAMESPACE                   NAME                 CLASS    ADDRESS          PROGRAMMED   AGE
    website                     http-gateway         cilium   172.21.255.202   True         5h
    webshop                     tls-gateway          cilium   172.21.255.203   True         5h
    

    The preceding command returns an overview of all the Gateways in the cluster. Check the following:

    • Is the Gateway programmed?

      A programmed Gateway means that Cilium prepared a configuration for it.

      • If the Programmed true indicator is missing, make sure that Gateway API is enabled in the Cilium configuration.

    • Does the gateway have an address?

    You can check the service with kubectl get service. If the gateway has an address, it means that a LoadBalancer service is assigned to the gateway. If no IP appears, you might be missing a LoadBalancer implementation.

    • Is the class cilium?

    Cilium only programs Gateways with the class cilium.

    • If the Gateway API resource type (Gateway, HTTPRoute, etc.) is not found, make sure that the Gateway API CRDs are installed.

    You can use kubectl describe gateway to investigate issues more thoroughly.

    $ kubectl describe gateway <name>
    
      Conditions:
        Message:               Gateway successfully scheduled
        Reason:                Accepted
        Status:                True
        Type:                  Accepted
        Message:               Gateway successfully reconciled
        Reason:                Programmed
        Status:                True
        Type:                  Programmed
        [...]
      Listeners:
        Attached Routes:  2
        Conditions:
          Message:               Listener Ready
          Reason:                Programmed
          Status:                True
          Type:                  Programmed
          Message:               Listener Accepted
          Reason:                Accepted
          Status:                True
          [...]
    

    You can see the general status of the gateway as well as the status of the configured listeners.

    Listener status displays the number of routes successfully attached to the listener.

    You can see status conditions for both gateway and listener:

    • Accepted: the Gateway configuration was accepted.

    • Programmed: the Gateway configuration was programmed into Envoy.

    • ResolvedRefs: all referenced secrets were found and have permission for use.

    If any of these conditions are set to false, the Message and Reason fields give more information.

  2. Check the HTTPRoute resource

When the Gateway is functional, you can check the routes to verify if they are configured correctly. The way to check route status is similar to checking the status of a gateway resource.

While these instructions are written for HTTPRoute, they also apply to other route types.

$ kubectl get httproute -A
NAMESPACE                 NAME              HOSTNAMES         AGE
website                   homepage          www.example.org   17m
webshop                   catalog-service                     17m
webshop                   cart-service                        17m

To get more information, enter kubectl describe httproute <name>.

$ kubectl describe httproute <name>
Status:
  Parents:
    Conditions:
      Last Transition Time:  2023-06-05T15:11:53Z
      Message:               Accepted HTTPRoute
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2023-06-05T15:11:53Z
      Message:               Service reference is valid
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Controller Name:         io.cilium/gateway-controller
    Parent Ref:
      Group:  gateway.networking.k8s.io
      Kind:   Gateway
      Name:   same-namespace

Status lists the conditions that are relevant for the specific HTTPRoute. Conditions are listed by parent reference to the gateway. If you linked the route to multiple gateways, multiple entries appear. Conditions include Reason, Type, Status and Message. Type indicates the condition type, and Status indicates with a boolean whether the condition type is met. Optionally, Message gives you more information about the condition.

Notice the following condition types:

  • Accepted: The HTTPRoute configuration was correct and accepted.

  • ResolvedRefs: The referenced services were found and are valid references.

If any of these are set to false, you can get more information by looking at the Message and Reason fields.

  1. Check Cilium Operator logs

The Cilium Operator logs may contain further debugging information. For example, if the required Custom Resource Definitions (CRDs) are not installed, an error will be logged:

$ kubectl logs -n kube-system deployments/cilium-operator | grep gateway
level=error msg="Required GatewayAPI resources are not found, please
refer to docs for installation instructions" error="customresourcedefinitions.apiextensions.k8s.io \"gatewayclasses.gateway.networking.k8s.io\" not found
customresourcedefinitions.apiextensions.k8s.io \"gateways.gateway.networking.k8s.io\" not found
customresourcedefinitions.apiextensions.k8s.io \"httproutes.gateway.networking.k8s.io\" not found
customresourcedefinitions.apiextensions.k8s.io \"referencegrants.gateway.networking.k8s.io\" not found
customresourcedefinitions.apiextensions.k8s.io \"grpcroutes.gateway.networking.k8s.io\" not found
customresourcedefinitions.apiextensions.k8s.io \"tlsroutes.gateway.networking.k8s.io\" not found" subsys=gateway-api

Common mistakes

Warning

Gateway API is a recent addition to Kubernetes, and the Cilium project has not yet received much user feedback. If you encounter an issue that is not yet listed in this section, consider opening a PR to add your issue to the list.

  • The backend service does not exist.

    To verify whether the backend service was found, run kubectl describe httproute <name> and inspect the conditions field:

    Parents:
      Conditions:
        Last Transition Time:  2023-06-06T13:55:10Z
        Message:               Service "backend" not found
        Observed Generation:   1
        Reason:                BackendNotFound
        Status:                False
        Type:                  ResolvedRefs
        Last Transition Time:  2023-06-06T13:55:10Z
        Message:               Accepted HTTPRoute
        Observed Generation:   1
        Reason:                Accepted
        Status:                True
        Type:                  Accepted
      Controller Name:         io.cilium/gateway-controller
    
  • The gateway specified under parentRefs does not exist.

    To verify whether the gateway was found, run kubectl describe httproute <name> and inspect the conditions field:

Parents:
  Conditions:
    Last Transition Time:  2023-06-06T13:56:40Z
    Message:               Gateway.gateway.networking.k8s.io "my-gatewai" not found
    Observed Generation:   2
      Reason:                InvalidHTTPRoute
      Status:                False
      Type:                  Accepted

Underlying mechanics: a high level overview

A Cilium deployment has two parts that handle Gateway API resources: the Cilium agent and the Cilium operator.

The Cilium operator watches all Gateway API resources and verifies whether the resources are valid. If resources are valid, the operator marks them as accepted. That starts the process of translation into Cilium Envoy Configuration resources.

The Cilium agent then picks up the Cilium Envoy Configuration resources.

The Cilium agent uses the resources to supply the configuration to the built in Envoy or the Envoy DaemonSet. Envoy handles traffic.