Installation on Broadcom VMware ESXi / NSX

Cilium can be installed on VMware ESXi with or without NSX by using official image.

Deploying Cilium on Broadcom VMware vSphere ESXi with or without NSX(-T)

Cilium can be deployed on VMware vSphere ESXi, with or without NSX(-T). However, there are known issues when using tunnel mode with VXLAN as the encapsulation.

Install Cilium via helm install with VXLAN Protocol

helm install cilium ./cilium \
   --namespace kube-system \
   --set image.pullPolicy=IfNotPresent \
   --set ipam.mode=kubernetes \
   --set tunnelProtocol=vxlan

Note

With NSX(-T), use a custom port for the tunnelPort flag, for instance --set tunnelPort=8223. GitHub issue 21801 tracks some reports of problems with offloads when using the VXLAN UDP port standard (4789) or draft (8472).

Install Cilium via helm install with Geneve Protocol

helm install cilium ./cilium \
   --namespace kube-system \
   --set image.pullPolicy=IfNotPresent \
   --set ipam.mode=kubernetes \
   --set tunnelProtocol=geneve

Note

NSX(-T) with Network Virtualization (with Edge T0/T1) also uses Geneve Protocol between Transport Nodes (ESXi, Edge). Be aware when troubleshooting that the Geneve traffic you observe on the network may be generated by either NSX(-T) or Cilium.

Troubleshooting

Pod Communication Failure Across Hosts

When deploying Cilium with some old release ESXi (7) or with NSX-T (3.x/4.x), with VXLAN encapsulation, the inter-host pod communication may fail, except for ICMP (ping), which still functions.

In the Cilium-health status you will see:

==== detail from pod cilium-mvrb6 , on node alg-cilium-cp
Probe time:   2025-03-12T16:55:02Z
Nodes:
alg-cilium-cp (localhost):
    Host connectivity to 10.44.144.20:
    ICMP to stack:   OK, RTT=640.959µs
    HTTP to agent:   OK, RTT=148.15µs
    Endpoint connectivity to 10.42.0.38:
    ICMP to stack:   OK, RTT=632.181µs
    HTTP to agent:   OK, RTT=295.409µs
alg-cilium-wk1:
    Host connectivity to 10.44.144.21:
    ICMP to stack:   OK, RTT=764.463µs
    HTTP to agent:   OK, RTT=1.154573ms
    Endpoint connectivity to 10.42.4.211:
    ICMP to stack:   OK, RTT=765.081µs
    HTTP to agent:   Get "http://10.42.4.211:4240/hello": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

The problem originates from a bug in the VMXNET3 driver related to NIC offload support for VXLAN encapsulation. This is due to the use of an outdated standard port (8472) for VXLAN.

In this case you need to change to VXLAN Port --set tunnelPort=8223 or use Geneve tunnel Protocol --set tunnelProtocol=geneve. There is some workaround about Disable NIC Offload but it is not recommended solution.