Troubleshooting Cilium deployed with Argo CD
There have been reports from users hitting issues with Argo CD. This documentation page outlines some of the known issues and their solutions.
Argo CD deletes CustomResourceDefinitions
When deploying Cilium with Argo CD, some users have reported that Cilium-generated custom resources disappear, causing one or more of the following issues:
ciliumid
not found (GitHub issue 17614)Argo CD Out-of-sync issues for hubble-generate-certs (GitHub issue 14550)
Out-of-sync issues for Cilium using Argo CD (GitHub issue 18298)
Solution
To prevent these issues, declare resource exclusions in the Argo CD ConfigMap
by following these instructions.
Here is an example snippet:
resource.exclusions: |
- apiGroups:
- cilium.io
kinds:
- CiliumIdentity
clusters:
- "*"
Also, it has been reported that the problem may affect all workloads you deploy with Argo CD in a cluster running Cilium, not just Cilium itself. If so, you will need the following exclusions in your Argo CD application definition to avoid getting “out of sync” when Hubble rotates its certificates.
ignoreDifferences:
- group: ""
kind: ConfigMap
name: hubble-ca-cert
jsonPointers:
- /data/ca.crt
- group: ""
kind: Secret
name: hubble-relay-client-certs
jsonPointers:
- /data/ca.crt
- /data/tls.crt
- /data/tls.key
- group: ""
kind: Secret
name: hubble-server-certs
jsonPointers:
- /data/ca.crt
- /data/tls.crt
- /data/tls.key
Note
After applying the above configurations, for the settings to take effect, you will need to restart the Argo CD deployments.
Helm template with serviceMonitor enabled fails
Some users have reported that when they install Cilium using Argo CD and run helm template
with serviceMonitor
enabled, it fails.
It fails because Argo CD CLI doesn’t pass the --api-versions
flag to Helm upon deployment.
Solution
This pull request fixed this issue in Argo CD’s v2.3.0 release.
Upgrade your Argo CD and check if helm template
with serviceMonitor
enabled still fails.
Note
When using helm template
, it is highly recommended you set
--kube-version
and --api-versions
with the values matching your
target Kubernetes cluster. Helm charts such as Cilium’s often conditionally
enable certain Kubernetes features based on their availability (beta vs
stable) on the target cluster.
By specifying --api-versions=monitoring.coreos.com/v1
you should be
able to pass validation with helm template
.
If you have an issue with Argo CD that’s not outlined above, check this list of Argo CD related issues on GitHub. If you can’t find an issue that relates to yours, create one and/or seek help on Cilium Slack.
Application chart for Cilium deployed to Talos Linux fails with: field not declared in schema
When deploying Cilium to Talos Linux with ArgoCD, some users have reported issues due to Talos Security configuration. ArgoCD may fail to deploy the application with the message:
Failed to compare desired state to live state: failed to calculate diff:
error calculating structured merge diff: error building typed value from live
resource: .spec.template.spec.securityContext.appArmorProfile: field not
declared in schema
Solution
Add option ServerSideApply=true
to list syncPolicy.syncOptions
for the Application.
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
syncPolicy:
syncOptions:
- ServerSideApply=true
Visit the ArgoCD documentation for further details.