LoadBalancer IP Address Management (LB IPAM)
LB IPAM is a feature that allows Cilium to assign IP addresses to Services of
type LoadBalancer. This functionality is usually left up to a cloud provider,
however, when deploying in a private cloud environment, these facilities are not
always available.
LB IPAM works in conjunction with features like the Cilium BGP Control Plane. Where LB IPAM is responsible for allocation and assigning of IPs to Service objects and other features are responsible for load balancing and/or advertisement of these IPs.
LB IPAM is always enabled but dormant. The controller is awoken when the first IP Pool is added to the cluster.
Pools
LB IPAM has the notion of IP Pools which the administrator can create to tell Cilium which IP ranges can be used to allocate IPs from.
A basic IP Pools with both an IPv4 and IPv6 range looks like this:
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
name: "blue-pool"
spec:
cidrs:
- cidr: "10.0.10.0/24"
- cidr: "2004::0/64"
After adding the pool to the cluster, it appears like so.
$ kubectl get ippools
NAME DISABLED CONFLICTING IPS AVAILABLE AGE
blue-pool false False 65788 2s
Note
The amount of available IPs in the pool is lower than the actual sum of all usable IPs in the CIDRs because the allocation logic is limited to 65536 IPs per CIDR. CIDRs containing more than 65536 IPs can be broken down into multiple smaller CIDRs to achieve full utilization.
Service Selectors
IP Pools have an optional .spec.serviceSelector field which allows administrators
to limit which services can get IPs from which pools using a label selector.
The pool will allocate to any service if no service selector is specified.
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
name: "blue-pool"
spec:
cidrs:
- cidr: "20.0.10.0/24"
serviceSelector:
matchExpressions:
- {key: color, operator: In, values: [blue, cyan]}
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
name: "red-pool"
spec:
cidrs:
- cidr: "20.0.10.0/24"
serviceSelector:
matchLabels:
color: red
There are a few special purpose selector fields which don’t match on labels but
instead on other metadata like .meta.name or .meta.namespace.
Selector |
Field |
io.kubernetes.service.namespace |
|
io.kubernetes.service.name |
|
For example:
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
name: "blue-pool"
spec:
cidrs:
- cidr: "20.0.10.0/24"
serviceSelector:
matchLabels:
"io.kubernetes.service.namespace": "tenant-a"
Conflicts
IP Pools are not allowed to have overlapping CIDRs. When an administrator does
create pools which overlap, a soft error is caused. The last added pool will be
marked as Conflicting and no further allocation will happen from that pool.
Therefore, administrators should always check the status of all pools after making
modifications.
For example, if we add 2 pools (blue-pool and red-pool) both with the same
CIDR, we will see the following:
$ kubectl get ippools
NAME DISABLED CONFLICTING IPS AVAILABLE AGE
blue-pool false False 254 25m
red-pool false True 254 11s
The reason for the conflict is stated in the status and can be accessed like so
$ kubectl get ippools/red-pool -o jsonpath='{.status.conditions[?(@.type=="io.cilium/conflict")].message}'
Pool conflicts since CIDR '20.0.10.0/24' overlaps CIDR '20.0.10.0/24' from IP Pool 'blue-pool'
or
$ kubectl describe ippools/red-pool
Name: red-pool
#[...]
Status:
Conditions:
#[...]
Last Transition Time: 2022-10-25T14:09:05Z
Message: Pool conflicts since CIDR '20.0.10.0/24' overlaps CIDR '20.0.10.0/24' from IP Pool 'blue-pool'
Observed Generation: 1
Reason: cidr_overlap
Status: True
Type: io.cilium/conflict
#[...]
Disabling a Pool
IP Pools can be disabled. Disabling a pool will stop LB IPAM from allocating new IPs from the pool, but doesn’t remove existing allocations. This allows an administrator to slowly drain pool or reserve a pool for future use.
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
name: "blue-pool"
spec:
cidrs:
- cidr: "20.0.10.0/24"
disabled: true
$ kubectl get ippools
NAME DISABLED CONFLICTING IPS AVAILABLE AGE
blue-pool true False 254 41m
Status
The IP Pool’s status contains additional counts which can be used to monitor the amount of used and available IPs. A machine parsable output can be obtained like so.
$ kubectl get ippools -o jsonpath='{.items[*].status.conditions[?(@.type!="io.cilium/conflict")]}' | jq
{
"lastTransitionTime": "2022-10-25T14:08:55Z",
"message": "254",
"observedGeneration": 1,
"reason": "noreason",
"status": "Unknown",
"type": "io.cilium/ips-total"
}
{
"lastTransitionTime": "2022-10-25T14:08:55Z",
"message": "254",
"observedGeneration": 1,
"reason": "noreason",
"status": "Unknown",
"type": "io.cilium/ips-available"
}
{
"lastTransitionTime": "2022-10-25T14:08:55Z",
"message": "0",
"observedGeneration": 1,
"reason": "noreason",
"status": "Unknown",
"type": "io.cilium/ips-used"
}
Or human readable output like so
$ kubectl describe ippools/blue-pool
Name: blue-pool
Namespace:
Labels: <none>
Annotations: <none>
API Version: cilium.io/v2alpha1
Kind: CiliumLoadBalancerIPPool
#[...]
Status:
Conditions:
#[...]
Last Transition Time: 2022-10-25T14:08:55Z
Message: 254
Observed Generation: 1
Reason: noreason
Status: Unknown
Type: io.cilium/ips-total
Last Transition Time: 2022-10-25T14:08:55Z
Message: 254
Observed Generation: 1
Reason: noreason
Status: Unknown
Type: io.cilium/ips-available
Last Transition Time: 2022-10-25T14:08:55Z
Message: 0
Observed Generation: 1
Reason: noreason
Status: Unknown
Type: io.cilium/ips-used
Services
Any service with .spec.type=LoadBalancer can get IPs from any pool as long
as the IP Pool’s service selector matches the service.
Lets say we add a simple service.
apiVersion: v1
kind: Service
metadata:
name: service-red
namespace: example
labels:
color: red
spec:
type: LoadBalancer
ports:
- port: 1234
This service will appear like so.
$ kubectl -n example get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service-red LoadBalancer 10.96.192.212 <pending> 1234:30628/TCP 24s
The ExternalIP field has a value of <pending> which means no LB IPs have been assigned.
When LB IPAM is unable to allocate or assign IPs for the service, it will update the service
conditions in the status.
The service conditions can be checked like so:
$ kubectl -n example get svc/service-red -o jsonpath='{.status.conditions}' | jq
[
{
"lastTransitionTime": "2022-10-06T13:40:48Z",
"message": "There are no enabled CiliumLoadBalancerIPPools that match this service",
"reason": "no_pool",
"status": "False",
"type": "io.cilium/lb-ipam-request-satisfied"
}
]
After updating the service labels to match our blue-pool from before we see:
$ kubectl -n example get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service-red LoadBalancer 10.96.192.212 20.0.10.163 1234:30628/TCP 12m
$ kubectl -n example get svc/service-red -o jsonpath='{.status.conditions}' | jq
[
{
"lastTransitionTime": "2022-10-06T13:40:48Z",
"message": "There are no enabled CiliumLoadBalancerIPPools that match this service",
"reason": "no_pool",
"status": "False",
"type": "io.cilium/lb-ipam-request-satisfied"
},
{
"lastTransitionTime": "2022-10-06T13:52:55Z",
"message": "",
"reason": "satisfied",
"status": "True",
"type": "io.cilium/lb-ipam-request-satisfied"
}
]
IPv4 / IPv6 families + policy
LB IPAM supports IPv4 and/or IPv6 in SingleStack or DualStack mode.
Services can use the .spec.ipFamilyPolicy and .spec.ipFamilies fields to change
the requested IPs.
If .spec.ipFamilyPolicy isn’t specified, SingleStack mode is assumed.
If both IPv4 and IPv6 are enabled in SingleStack mode, an IPv4 address is allocated.
If .spec.ipFamilyPolicy is set to PreferDualStack, LB IPAM will attempt to allocate
both an IPv4 and IPv6 address if both are enabled on the cluster. If only IPv4 or only IPv6 is
enabled on the cluster, the service is still considered “satisfied”.
If .spec.ipFamilyPolicy is set to RequireDualStack LB IPAM will attempt to allocate
both an IPv4 and IPv6 address. The service is considered “unsatisfied” If IPv4
or IPv6 is disabled on the cluster.
The order of .spec.ipFamilies has no effect on LB IPAM but is significant for cluster IP
allocation which isn’t handled by LB IPAM.
LoadBalancerClass
Kubernetes >= v1.24 supports multiple load balancers
in the same cluster. Picking between load balancers is done with the .spec.loadBalancerClass field.
When LB IPAM is enabled it allocates and assigns IPs for services with
no load balancer class set.
LB IPAM only does IP allocation and doesn’t provide load balancing services by itself. Therefore, users should pick one of the following Cilium load balancer classes, all of which use LB IPAM for allocation (if the feature is enabled):
loadBalancerClass |
Feature |
|
If the .spec.loadBalancerClass is set to a class which isn’t handled by Cilium’s LB IPAM,
then Cilium’s LB IPAM will ignore the service entirely, not even setting a condition in the status.
Requesting IPs
Services can request specific IPs. The legacy way of doing so is via .spec.loadBalancerIP
which takes a single IP address. This method has been deprecated in k8s v1.24 but is supported
until its future removal.
The new way of requesting specific IPs is to use annotations, io.cilium/lb-ipam-ips in the case
of Cilium LB IPAM. This annotation takes a comma-separated list of IP addresses, allowing for
multiple IPs to be requested at once.
The service selector of the IP Pool still applies, requested IPs will not be allocated or assigned if the services don’t match the pool’s selector.
apiVersion: v1
kind: Service
metadata:
name: service-blue
namespace: example
labels:
color: blue
annotations:
"io.cilium/lb-ipam-ips": "20.0.10.100,20.0.10.200"
spec:
type: LoadBalancer
ports:
- port: 1234
$ kubectl -n example get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service-blue LoadBalancer 10.96.26.105 20.0.10.100,20.0.10.200 1234:30363/TCP 43s