Developer / Contributor Guide¶

Requirements¶

You need to have the following tools available in order to effectively contribute to Cilium:

To run Cilium locally on VMs, you need:

Finally, in order to build the documentation, you should have Sphinx installed:

$ sudo pip install sphinx

You should start with the Getting Started Guides, which walks you through the set-up, such as installing Vagrant, getting the Cilium sources, and going through some Cilium basics.

Vagrant Setup¶

While the Getting Started Guides uses a Vagrantfile tuned for the basic walk through, the setup for the Vagrantfile in the root of the Cilium tree depends on a number of environment variables and network setup that are managed via contrib/vagrant/start.sh.

$ contrib/vagrant/start.sh

$ RUNTIME=containerd K8S=1 NWORKERS=1 contrib/vagrant/start.sh

$ RUNTIME=containerd K8S=1 NWORKERS=1 vagrant status

$ export KUBECONFIG=$KUBECONFIG:$GOPATH/src/github.com/cilium/cilium/vagrant.kubeconfig

$ vagrant init cilium/ubuntu
$ vagrant up
$ vagrant ssh [...]
$ cd go/src/github.com/cilium/cilium/
$ make
$ sudo make install
$ sudo mkdir -p /etc/sysconfig/
$ sudo cp contrib/systemd/cilium.service /etc/systemd/system/
$ sudo cp contrib/systemd/cilium  /etc/sysconfig/cilium
$ sudo usermod -a -G cilium vagrant
$ sudo systemctl enable cilium
$ sudo systemctl restart cilium

$ vagrant halt

Local Development in Vagrant Box¶

See Setting up the development environment for information on how to setup the development environment.

When the development VM is provisioned, it builds and installs Cilium. After the initial build and install you can do further building and testing incrementally inside the VM. vagrant ssh takes you to the Cilium source tree directory (/home/vagrant/go/src/github.com/cilium/cilium) by default, and the following commands assume that you are working within that directory.

$ make

$ sudo -E make install

$ sudo systemctl restart cilium

$ sudo systemctl status cilium
$ cilium status

Making Changes¶

1. Create a topic branch: git checkout -b myBranch master
2. Make the changes you want
3. Separate the changes into logical commits.
   1. Describe the changes in the commit messages. Focus on answering the question why the change is required and document anything that might be unexpected.
   2. If any description is required to understand your code changes, then those instructions should be code comments instead of statements in the commit description.
4. Make sure your changes meet the following criteria:
   1. New code is covered by Unit Testing.
   2. End to end integration / runtime tests have been extended or added. If not required, mention in the commit message what existing test covers the new code.
   3. Follow-up commits are squashed together nicely. Commits should separate logical chunks of code and not represent a chronological list of changes.
5. Run git diff --check to catch obvious white space violations
6. Run make to build your changes. This will also run go fmt and error out on any golang formatting errors.
7. See Unit Testing on how to run unit tests.
8. See End-To-End Testing Framework for information how to run the end to end integration tests

Unit Testing¶

Cilium uses the standard go test framework in combination with gocheck for richer testing functionality.

$ make start-kvstores

$ make unit-tests

$ cd pkg/kvstore
$ go test

$ cd pkg/kvstore
$ go test -check.v -check.vv

$ make unit-tests TESTPKGS=github.com/cilium/cilium/pkg/kvstore

$ go test -check.f TestParallelAllocation

$ . contrib/shell/test.sh
$ watchtest pkg/policy

Add/update a golang dependency¶

Once you have downloaded dep make sure you have version >= 0.4.1

$ dep version
dep:
 version     : v0.4.1
 build date  : 2018-01-24
 git hash    : 37d9ea0a
 go version  : go1.9.1
 go compiler : gc
 platform    : linux/amd64

After that, you can edit the Gopkg.toml file, add the library that you want to add. Lets assume we want to add github.com/containernetworking/cni version v0.5.2:

[[constraint]]
  name = "github.com/containernetworking/cni"
  revision = "v0.5.2"

Once you add the libraries that you need you can save the file and run

$ dep ensure -v

For a first run, it can take a while as it will download all dependencies to your local cache but the remaining runs will be faster.

Debugging¶

$ cilium endpoint config 3978 debug=true
Endpoint 3978 configuration updated successfully
$ cilium monitor -v --hex
Listening for events on 2 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit
------------------------------------------------------------------------------
CPU 00: MARK 0x1c56d86c FROM 3978 DEBUG: 70 bytes Incoming packet from container ifindex 85
00000000  33 33 00 00 00 02 ae 45  75 73 11 04 86 dd 60 00  |33.....Eus....`.|
00000010  00 00 00 10 3a ff fe 80  00 00 00 00 00 00 ac 45  |....:..........E|
00000020  75 ff fe 73 11 04 ff 02  00 00 00 00 00 00 00 00  |u..s............|
00000030  00 00 00 00 00 02 85 00  15 b4 00 00 00 00 01 01  |................|
00000040  ae 45 75 73 11 04 00 00  00 00 00 00              |.Eus........|
CPU 00: MARK 0x1c56d86c FROM 3978 DEBUG: Handling ICMPv6 type=133
------------------------------------------------------------------------------
CPU 00: MARK 0x1c56d86c FROM 3978 Packet dropped 131 (Invalid destination mac) 70 bytes ifindex=0 284->0
00000000  33 33 00 00 00 02 ae 45  75 73 11 04 86 dd 60 00  |33.....Eus....`.|
00000010  00 00 00 10 3a ff fe 80  00 00 00 00 00 00 ac 45  |....:..........E|
00000020  75 ff fe 73 11 04 ff 02  00 00 00 00 00 00 00 00  |u..s............|
00000030  00 00 00 00 00 02 85 00  15 b4 00 00 00 00 01 01  |................|
00000040  00 00 00 00                                       |....|
------------------------------------------------------------------------------
CPU 00: MARK 0x7dc2b704 FROM 3978 DEBUG: 86 bytes Incoming packet from container ifindex 85
00000000  33 33 ff 00 8a d6 ae 45  75 73 11 04 86 dd 60 00  |33.....Eus....`.|
00000010  00 00 00 20 3a ff fe 80  00 00 00 00 00 00 ac 45  |... :..........E|
00000020  75 ff fe 73 11 04 ff 02  00 00 00 00 00 00 00 00  |u..s............|
00000030  00 01 ff 00 8a d6 87 00  20 40 00 00 00 00 fd 02  |........ @......|
00000040  00 00 00 00 00 00 c0 a8  21 0b 00 00 8a d6 01 01  |........!.......|
00000050  ae 45 75 73 11 04 00 00  00 00 00 00              |.Eus........|
CPU 00: MARK 0x7dc2b704 FROM 3978 DEBUG: Handling ICMPv6 type=135
CPU 00: MARK 0x7dc2b704 FROM 3978 DEBUG: ICMPv6 neighbour soliciation for address b21a8c0:d68a0000

$ cilium endpoint list
ENDPOINT   POLICY        IDENTITY   LABELS (source:key[=value])   IPv6                     IPv4            STATUS
           ENFORCEMENT
48896      Disabled      266        container:id.server           fd02::c0a8:210b:0:bf00   10.11.13.37     not-ready
60670      Disabled      267        container:id.client           fd02::c0a8:210b:0:ecfe   10.11.167.158   not-ready

# for log in /var/run/cilium/state/*.log; do echo "cat $log"; cat $log; done
cat /var/run/cilium/state/bpf_features.log
BPF/probes: CONFIG_CGROUP_BPF=y is not in kernel configuration
BPF/probes: CONFIG_LWTUNNEL_BPF=y is not in kernel configuration
HAVE_LPM_MAP_TYPE: Your kernel doesn't support LPM trie maps for BPF, thus disabling CIDR policies. Recommendation is to run 4.11+ kernels.
HAVE_LRU_MAP_TYPE: Your kernel doesn't support LRU maps for BPF, thus switching back to using hash table for the cilium connection tracker. Recommendation is to run 4.10+ kernels.

# ls /sys/fs/bpf/tc/globals/
cilium_calls_15124  cilium_calls_48896        cilium_ct4_global       cilium_lb4_rr_seq       cilium_lb6_services  cilium_policy_25729  cilium_policy_60670       cilium_proxy6
cilium_calls_25729  cilium_calls_60670        cilium_ct6_global       cilium_lb4_services     cilium_lxc           cilium_policy_3978   cilium_policy_reserved_1  cilium_reserved_policy
cilium_calls_3978   cilium_calls_netdev_ns_1  cilium_events           cilium_lb6_reverse_nat  cilium_policy        cilium_policy_4314   cilium_policy_reserved_2  cilium_tunnel_map
cilium_calls_4314   cilium_calls_overlay_2    cilium_lb4_reverse_nat  cilium_lb6_rr_seq       cilium_policy_15124  cilium_policy_48896  cilium_proxy4
# bpf-map info /sys/fs/bpf/tc/globals/cilium_policy_15124
Type:           Hash
Key size:       8
Value size:     24
Max entries:    1024
Flags:          0x0
# bpf-map dump /sys/fs/bpf/tc/globals/cilium_policy_15124
Key:
00000000  6a 01 00 00 82 23 06 00                           |j....#..|
Value:
00000000  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00                           |........|

Introduction¶

Cilium uses Ginkgo as a testing framework for writing end-to-end tests which test Cilium all the way from the API level (e.g. importing policies, CLI) to the datapath (i.e, whether policy that is imported is enforced accordingly in the datapath). The tests in the test directory are built on top of Ginkgo. Ginkgo provides a rich framework for developing tests alongside the benefits of Golang (compilation-time checks, types, etc.). To get accustomed to the basics of Ginkgo, we recommend reading the Ginkgo Getting-Started Guide , as well as running example tests to get a feel for the Ginkgo workflow.

These test scripts will invoke vagrant to create virtual machine(s) to run the tests. The tests make heavy use of the Ginkgo focus concept to determine which VMs are necessary to run particular tests. All test names must begin with one of the following prefixes:

• Runtime: Test cilium in a runtime environment running on a single node.
• K8s: Create a small multi-node kubernetes environment for testing features beyond a single host, and for testing kubernetes-specific features.
• Nightly: sets up a multinode Kubernetes cluster to run scale, performance, and chaos testing for Cilium.

Running End-To-End Tests¶

$ sudo make -C test/

ginkgo --focus="Runtime*" -noColor

$ ginkgo --focus="Runtime*" -noColor -v -dryRun
Running Suite: runtime
======================
Random Seed: 1516125117
Will run 42 of 164 specs
................
RuntimePolicyEnforcement Policy Enforcement Always
  Always to Never with policy
  /Users/ianvernon/go/src/github.com/cilium/cilium/test/runtime/Policies.go:258
•
------------------------------
RuntimePolicyEnforcement Policy Enforcement Always
  Always to Never without policy
  /Users/ianvernon/go/src/github.com/cilium/cilium/test/runtime/Policies.go:293
•
------------------------------
RuntimePolicyEnforcement Policy Enforcement Never
  Container creation
  /Users/ianvernon/go/src/github.com/cilium/cilium/test/runtime/Policies.go:332
•
------------------------------
RuntimePolicyEnforcement Policy Enforcement Never
  Never to default with policy
  /Users/ianvernon/go/src/github.com/cilium/cilium/test/runtime/Policies.go:349
.................
Ran 42 of 164 Specs in 0.002 seconds
SUCCESS! -- 0 Passed | 0 Failed | 0 Pending | 122 Skipped PASS

Ginkgo ran 1 suite in 1.830262168s
Test Suite Passed

ginkgo --focus="K8s*" -noColor

K8S_VERSION=<version> ginkgo --focus="K8s*" -noColor

Vagrant.configure('2') do |config|
  if Vagrant.has_plugin?("vagrant-vbguest") then
    config.vbguest.auto_update = false
  end

  config.vm.provider :virtualbox do |vbox|
    vbox.check_guest_additions = false
  end
end

ginkgo --focus="Nightly*"  -noColor

$ cd test/
$ ginkgo . -- --help | grep -A 1 cilium
  -cilium.SSHConfig string
        Specify a custom command to fetch SSH configuration (eg: 'vagrant ssh-config')
  -cilium.holdEnvironment
        On failure, hold the environment in its current state
  -cilium.provision
        Provision Vagrant boxes and Cilium before running test (default true)
  -cilium.showCommands
        Output which commands are ran to stdout
  -cilium.testScope string
        Specifies scope of test to be ran (k8s, Nightly, runtime)
$ ginkgo --focus "Policies*" -- --cilium.provision=false

It("Example test", func(){
    Expect(true).Should(BeTrue())
})

FIt("Example focused test", func(){
    Expect(true).Should(BeTrue())
})

ginkgo --focus "Run*" --focus "L7 "

Test Reports¶

The Cilium Ginkgo framework formulates JUnit reports for each test. The following files currently are generated depending upon the test suite that is ran:

• runtime.xml
• K8s.xml

Best Practices for Writing Tests¶

• Provide informative output to console during a test using the By construct. This helps with debugging and gives those who did not write the test a good idea of what is going on. The lower the barrier of entry is for understanding tests, the better our tests will be!
• Leave the testing environment in the same state that it was in when the test started by deleting resources, resetting configuration, etc.
• Gather logs in the case that a test fails. If a test fails while running on Jenkins, a postmortem needs to be done to analyze why. So, dumping logs to a location where Jenkins can pick them up is of the highest imperative. Use the following code in an AfterFailed method:

AfterFailed(func() {
        vm.ReportFailed()
})

Ginkgo Extensions¶

In Cilium, some Ginkgo features are extended to cover some uses cases that are useful for testing Cilium.

Describe
    BeforeAll(A)
    AfterAll(A)
    AfterFailed(A)
    AfterEach(A)
    JustAfterEach(A)
    TESTA1
    TESTA2
    TESTA3
    Context
        BeforeAll(B)
        AfterAll(B)
        AfterFailed(B)
        AfterEach(B)
        JustAfterEach(B)
        TESTB1
        TESTB2
        TESTB3

Describe
    BeforeAll
    TESTA1; JustAfterEach(A), AfterFailed(A), AfterEach(A)
    TESTA2; JustAfterEach(A), AfterFailed(A), AfterEach(A)
    TESTA3; JustAfterEach(A), AfterFailed(A), AfterEach(A)
    Context
        BeforeAll(B)
        TESTB1:
           JustAfterEach(B); JustAfterEach(A)
           AfterFailed(B); AfterFailed(A);
           AfterEach(B) ; AfterEach(A);
        TESTB2:
           JustAfterEach(B); JustAfterEach(A)
           AfterFailed(B); AfterFailed(A);
           AfterEach(B) ; AfterEach(A);
        TESTB3:
           JustAfterEach(B); JustAfterEach(A)
           AfterFailed(B); AfterFailed(A);
           AfterEach(B) ; AfterEach(A);
        AfterAll(B)
    AfterAll(A)

Debugging:¶

Ginkgo provides to us different ways of debugging. In case that you want to see all the logs messages in the console you can run the test in verbose mode using the option -v:

ginkgo --focus "Runtime*" -v

In case that the verbose mode is not enough, you can retrieve all run commands and their output in the report directory (./test/test_results). Each test creates a new folder, which contains a file called log where all information is saved, in case of a failing test an exhaustive data will be added.

$ head test/test_results/RuntimeKafkaKafkaPolicyIngress/logs
level=info msg=Starting testName=RuntimeKafka
level=info msg="Vagrant: running command \"vagrant ssh-config runtime\""
cmd: "sudo cilium status" exitCode: 0
 KVStore:            Ok         Consul: 172.17.0.3:8300
ContainerRuntime:   Ok
Kubernetes:         Disabled
Kubernetes APIs:    [""]
Cilium:             Ok   OK
NodeMonitor:        Disabled
Allocated IPv4 addresses:

dlv test . -- --ginkgo.focus="Runtime" -ginkgo.v=true --cilium.provision=false

Running End-To-End Tests In Other Environments¶

If you want to run tests in a different VM, you can use --cilium.SSHConfig to provide the SSH configuration of the endpoint on which tests will be ran. The tests presume the following on the remote instance:

• Cilium source code is located in the directory /home/vagrant/go/src/github.com/cilium/cilium/.
• Cilium is installed and running.

The ssh connection needs to be defined as a ssh-config file and need to have the following targets:

• runtime: To run runtime tests
• k8s{1..2}-${K8S_VERSION}: to run Kubernetes tests. These instances must have Kubernetes installed and running as a prerequisite for running tests.

An example ssh-config can be the following:

Host runtime
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /home/eloy/.go/src/github.com/cilium/cilium/test/.vagrant/machines/runtime/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

To run this you can use the following command:

ginkgo  -v -- --cilium.provision=false --cilium.SSHConfig="cat ssh-config"

VMs for Testing¶

The VMs used for testing are defined in test/Vagrantfile. There are a variety of configuration options that can be passed as environment variables:

Further Assistance¶

Have a question about how the tests work or want to chat more about improving the testing infrastructure for Cilium? Hop on over to the testing channel on Slack.

Getting Started¶

1. Make sure you have a GitHub account

2. Clone the cilium repository

   go get -d github.com/cilium/cilium
   cd $GOPATH/src/github.com/cilium/cilium
   

3. Set up your Setting up the development environment

4. Check the GitHub issues for good tasks to get started.

Submitting a pull request¶

Contributions must be submitted in the form of pull requests against the github repository at: https://github.com/cilium/cilium

1. Fork the Cilium repository to your own personal GitHub space or request access to a Cilium developer account on Slack
2. Push your changes to the topic branch in your fork of the repository.
3. Submit a pull request on https://github.com/cilium/cilium.

Before hitting the submit button, please make sure that the following requirements have been met:

1. Each commit compiles and is functional on its own to allow for bisecting of commits.

2. All code is covered by unit and/or runtime tests where feasible.

3. All changes have been tested and checked for regressions by running the existing testsuite against your changes. See the End-To-End Testing Framework section for additional details.

4. All commits contain a well written commit description including a title, description and a Fixes: #XXX line if the commit addresses a particular GitHub issue. Note that the GitHub issue will be automatically closed when the commit is merged.

   apipanic: Log stack at debug level
   
   Previously, it was difficult to debug issues when the API panicked
   because only a single line like the following was printed:
   
   level=warning msg="Cilium API handler panicked" client[email protected] method=GET
   panic_message="write unix /var/run/cilium/cilium.sock->@: write: broken
   pipe"
   
   This patch logs the stack at this point at debug level so that it can at
   least be determined in developer environments.
   
   Fixes: #4191
   
   Signed-off-by: Joe Stringer <joe@covalent.io>
   

5. If any of the commits fixes a particular commit already in the tree, that commit is referenced in the commit message of the bugfix. This ensures that whoever performs a backport will pull in all required fixes:

   daemon: use endpoint RLock in HandleEndpoint
   
   Fixes: a804c7c7dd9a ("daemon: wait for endpoint to be in ready state if specified via EndpointChangeRequest")
   
   Signed-off-by: André Martins <andre@cilium.io>
   

6. All commits are signed off. See the section Developer’s Certificate of Origin.

7. Pick the appropriate milestone for which this PR is being targeted to, e.g. 1.1, 1.2. This is in particular important in the time frame between the feature freeze and final release date.

8. Pick the right release-note label

   Labels	When to set
   release-note/bug	This is a non-trivial bugfix
   release-note/major	This is a major feature addition, e.g. Add MongoDB support
   release-note/minor	This is a minor feature addition, e.g. Refactor endpoint package

9. Verify the release note text. If not explicitly changed, the title of the PR will be used for the release notes. If you want to change this, you can add a special section to the description of the PR.

   ```release-note
   This is a release note text
   ```
   

   Note

   If multiple lines are provided, then the first line serves as the high level bullet point item and any additional line will be added as a sub item to the first line.

10. Pick the right labels for your PR:

    Labels	When to set
    kind/bug	This is a bugfix worth mentioning in the release notes
    kind/enhancement	This is an enhancement/feature
    priority/release-blocker	This PR should block the current release
    area/*	Code area this PR covers
    needs-backport/X.Y	PR needs to be backported to these stable releases
    pending-review	PR is immediately ready for review
    wip	PR is still work in progress, signals reviewers to hold.
    backport/X.Y	This is backport PR, may only be set as part of Backporting process
    upgrade-impact	The code changes have a potential upgrade impact

Getting a pull request merged¶

1. As you submit the pull request as described in the section Submitting a pull request. One of the reviewers will start a CI run by replying with a comment test-me-please as described in Triggering Pull-Request Builds With Jenkins. If you are a core team member, you may trigger the CI run yourself.
   1. Hound: basic golang/lint static code analyzer. You need to make the puppy happy.
   2. CI / Jenkins: Will run a series of tests:
      1. Unit tests
      2. Single node runtime tests
      3. Multi node Kubernetes tests
2. As part of the submission, GitHub will have requested a review from the respective code owners according to the CODEOWNERS file in the repository.
   1. Address any feedback received from the reviewers
   2. You can push individual commits to address feedback and then rebase your branch at the end before merging.
3. Owners of the repository will automatically adjust the labels on the pull request to track its state and progress towards merging.
4. Once the PR has been reviewed and the CI tests have passed, the PR will be merged by one of the repository owners. In case this does not happen, ping us on Slack.

Developer images¶

DOCKER_IMAGE_TAG=jane-developer-my-fix make dev-docker-image

You can then push the image tag to the registry for development builds:

docker push cilium/cilium-dev:jane-developer-my-fix

Access to the developer builds registry is restricted but access is granted liberally. Join the #development channel in Slack and ask for permission to push builds.

Official release images¶

Anyone can build official release images using the make target below but pushing to the official registries is restricted to Cilium maintainers. Ask in the #launchpad Slack channels for the exact details.

DOCKER_IMAGE_TAG=v1.4.0 make docker-image

You can then push the image tag to the registry:

docker push cilium/cilium:v1.4.0

Building¶

The documentation has several dependencies which can be installed using pip:

$ pip install -r Documentation/requirements.txt

Whenever making changes to Cilium documentation you should check that you did not introduce any new warnings or errors, and also check that your changes look as you intended. To do this you can build the docs:

$ make -C Documentation html

After this you can browse the updated docs as HTML starting at Documentation\_build\html\index.html.

Alternatively you can use a Docker container to build the pages:

$ make render-docs

This builds the docs in a container and builds and starts a web server with your document changes.

Now the documentation page should be browsable on http://localhost:9080.

Jobs Overview¶

Triggering Pull-Request Builds With Jenkins¶

To ensure that build resources are used judiciously, builds on Jenkins are manually triggered via comments on each pull-request that contain “trigger-phrases”. Only members of the Cilium GitHub organization are allowed to trigger these jobs. Refer to the table below for information regarding which phrase triggers which build, which build is required for a pull-request to be merged, etc. Each linked job contains a description illustrating which subset of tests the job runs.

There are some feature flags based on Pull Requests labels, the list of labels are the following:

• area/containerd: Enable containerd runtime on all Kubernetes test.
• ci/next-next: Run tests on net-next kernel. This causes the test-me-please target to only run on the net-next kernel. It is purely for testing on a different kernel, to merge a PR it must pass the CI without this flag.

Using Jenkins for testing¶

Typically when running Jenkins tests via one of the above trigger phases, it will run all of the tests in that particular category. However, there may be cases where you just want to run a single test quickly on Jenkins and observe the test result. To do so, you need to update the relevant test to have a custom name, and to update the Jenkins file to focus that test. Below is an example patch that shows how this can be achieved.

diff --git a/ginkgo.Jenkinsfile b/ginkgo.Jenkinsfile
index ee17808748a6..637f99269a41 100644
--- a/ginkgo.Jenkinsfile
+++ b/ginkgo.Jenkinsfile
@@ -62,10 +62,10 @@ pipeline {
             steps {
                 parallel(
                     "Runtime":{
-                        sh 'cd ${TESTDIR}; ginkgo --focus="RuntimeValidated*" -v -noColor'
+                        sh 'cd ${TESTDIR}; ginkgo --focus="XFoooo*" -v -noColor'
                     },
                     "K8s-1.9":{
-                        sh 'cd ${TESTDIR}; K8S_VERSION=1.9 ginkgo --focus=" K8sValidated*" -v -noColor ${FAILFAST}'
+                        sh 'cd ${TESTDIR}; K8S_VERSION=1.9 ginkgo --focus=" K8sFooooo*" -v -noColor ${FAILFAST}'
                     },
                     failFast: true
                 )
diff --git a/test/k8sT/Nightly.go b/test/k8sT/Nightly.go
index 62b324619797..3f955c73a818 100644
--- a/test/k8sT/Nightly.go
+++ b/test/k8sT/Nightly.go
@@ -466,7 +466,7 @@ var _ = Describe("NightlyExamples", func() {

                })

-               It("K8sValidated Updating Cilium stable to master", func() {
+               FIt("K8sFooooo K8sValidated Updating Cilium stable to master", func() {
                        podFilter := "k8s:zgroup=testapp"

                        //This test should run in each PR for now.

CI Failure Triage¶

This section describes the process to triage CI failures. We define 3 categories:

Infrastructure details¶

$ vagrant global-status
id       name                          provider   state   directory
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6e68c6c  k8s1-build-PR-1588-6          virtualbox running /root/jenkins/workspace/cilium_cilium_PR-1588-CWL743UTZEF6CPEZCNXQVSZVEW32FR3CMGKGY6667CU7X43AAZ4Q/tests/k8s
ec5962a  cilium-master-build-PR-1588-6 virtualbox running /root/jenkins/workspace/cilium_cilium_PR-1588-CWL743UTZEF6CPEZCNXQVSZVEW32FR3CMGKGY6667CU7X43AAZ4Q
bfaffaa  k8s2-build-PR-1588-6          virtualbox running /root/jenkins/workspace/cilium_cilium_PR-1588-CWL743UTZEF6CPEZCNXQVSZVEW32FR3CMGKGY6667CU7X43AAZ4Q/tests/k8s
3fa346c  k8s1-build-PR-1588-7          virtualbox running /root/jenkins/workspace/cilium_cili[email protected]2/tests/k8s
b7ded3c  cilium-master-build-PR-1588-7 virtualbox running /root/jenkins/workspace/cilium_cili[email protected]2

$ JOB_BASE_NAME=PR-1588 BUILD_NUMBER=6 vagrant ssh 6e68c6c

Release Cadence¶

Cilium schedules a minor release every 6 weeks. Each minor release is performed by incrementing the Y in the version format X.Y.Z. The group of committers can decide to increment X instead to mark major milestones in which case Y is reset to 0.

Stable releases¶

The committers can nominate PRs merged into master as required for backport into the stable release branches. Upon necessity, stable releases are published with the version X.Y.Z+1. Stable releases are regularly released in high frequency or on demand to address major incidents.

In order to guarantee stable production usage while maintaining a high release cadence, the following stable releases will be maintained:

• Stable backports into the last two releases
• LTS release for extended long term backport coverage

LTS¶

The group of committers nominates a release to be a long term stable release. Such releases are guaranteed to receive backports for major and security relevant bugfixes. LTS releases will be declared end of life after 6 months. The group of committers will nominate and start supporting a new LTS release before the current LTS expires. If for some reason, no release can be declared LTS before the current LTS release expires, the current LTS lifetime will be extended.

Given the current 6 weeks release cadence, the development teams will aim at declaring every 3rd release to be an LTS to guarantee enough time overlap between LTS release.

Generic Release Process¶

This process applies to all releases other than minor releases, this includes:

• Stable releases
• Release candidates

If you intent to release a new minor release, see the Minor Release Process section instead.

1. Ensure that the necessary backports have been completed and merged. See Backporting process.

2. Checkout the desired stable branch and pull it:

   git checkout v1.0; git pull
   

3. Create a branch for the release pull request:

   git checkout -b pr/prepare-v1.0.3
   

4. Update the VERSION file to represent X.Y.Z+1

5. If this is the first release after creating a new release branch. Adjust the image pull policy for all .sed files in examples/kubernetes from Always to IfNotPresent.

6. Update the image tag versions in the examples:

   make -C examples/kubernetes clean all
   

7. Update the cilium_version and cilium_tag variables in examples/getting-started/Vagrantfile

8. Update the AUTHORS file

   make update-authors
   

   Note

   Check to see if the AUTHORS file has any formatting errors (for instance, indentation mismatches) as well as duplicate contributor names, and correct them accordingly.

9. Generate the NEWS.rst addition based off of the prior release tag (e.g., if you are generating the NEWS.rst for v1.0.3):

   git shortlog v1.0.2.. > add-to-NEWS.rst
   

10. Add a new section to NEWS.rst:

    v1.0.3
    ======
    
    ::
    
        <<contents of add-to-NEWS.rst>>
        [...]
        <<end of add-to-NEWS.rst>>
    

11. Add all modified files using git add and create a pull request with the title Prepare for release v1.0.3. Add the backport label to the PR which corresponds to the branch for which the release is being performed, e.g. backport/1.0.

    Note

    Make sure to create the PR against the desired stable branch. In this case v1.0

12. Follow standard procedures to get the aforementioned PR merged into the desired stable branch. See Submitting a pull request for more information about this process.

13. Checkout out the stable branch and pull your merged changes:

    git checkout v1.0; git pull
    

14. Create release tags:

    git tag -a v1.0.3 -m 'Release v1.0.3'
    git tag -a 1.0.3 -m 'Release 1.0.3'
    

    Note

    There are two tags that correspond to the same release because GitHub recommends using vx.y.z for release version formatting, and ReadTheDocs, which hosts the Cilium documentation, requires the version to be in format x.y.z For more information about how ReadTheDocs does versioning, you can read their Versions Documentation.

15. Build the binaries and push it to the release bucket:

    DOMAIN=releases.cilium.io ./contrib/release/uploadrev v1.0.3
    

    This step will print a markdown snippet which you will need when crafting the GitHub release so make sure to keep it handy.

16. Build the container images and push them

    DOCKER_IMAGE_TAG=v1.0.3 make docker-image
    docker push cilium/cilium:v1.0.3
    

17. Push the git release tag

    git push --tags
    

18. Create a GitHub release:

    1. Choose the correct target branch, e.g. v1.0

    2. Choose the correct target tag, e.g. v1.0.3

    3. Title: 1.0.3

    4. Check the This is a pre-release box if you are releasing a release candidate.

    5. Fill in the release description:

       Changes
       -------
       
       ```
       << contents of NEWS.rst for this release >>
       ```
       
       Release binaries
       ----------------
       
       << contents of snippet outputed by uploadrev >>
       

    6. Preview the description and then publish the release

19. Announce the release in the #general channel on Slack

20. Bump the version of Cilium used in the Cilium upgrade tests to use the new release

    Please reach out on the #development channel on Slack for assistance with this task.

Minor Release Process¶

Backporting process¶

Cilium PRs that are marked with the label needs-backport/X.Y need to be backported to the stable branch X.Y. The following steps summarize the process for backporting these PRs.

1. Make sure the Github labels are up-to-date, as this process will deal with all commits from PRs that have the needs-backport/X.Y label set (for a stable release version X.Y). If any PRs contain labels such as backport-pending/X.Y, ensure that the backport for that PR have been merged and if so, change the label to backport-done/X.Y.

2. The scripts referred to below need to be run in Linux, they do not work on OSX. You can use the cilium dev VM for this, but you need to configure git to have your name and email address to be used in the commit messages:

   $ git config --global user.name "John Doe"
   $ git config --global user.email [email protected]
   

3. Make sure you have your a GitHub developer access token available. For details, see contrib/backporting/README.md

4. Fetch the repo, e.g., git fetch

5. Check a new branch for your backports based on the stable branch for that version, e.g., git checkout -b pr/v1.0-backport-YY-MM-DD origin/v1.0

6. Run the check-stable script, referring to your Github access token, this will list the commits that need backporting, from the newest to oldest:

   $ GITHUB_TOKEN=xxx contrib/backporting/check-stable 1.0
   

   Note

   contrib/backporting/check-stable accepts a second argument to specify a path to write a nicely-formatted pull request message to. This can be used alongside Github command-line tools to send the pull request from the command line in steps 9-10 via gh pull-request -b vX.Y -l backport/vX.Y -F <path>.

7. Cherry-pick the commits using the master git SHAs listed, starting from the oldest (bottom), working your way up and fixing any merge conflicts as they appear. Note that for PRs that have multiple commits you will want to check that you are cherry-picking oldest commits first. The cherry-pick script accepts multiple arguments, in which case it will attempt to apply each commit in the order specified on the command line until one cherry pick fails or every commit is cherry-picked.

   $ contrib/backporting/cherry-pick <oldest-commit-sha>
   ...
   $ contrib/backporting/cherry-pick <newest-commit-sha>
   

8. Push your backports branch to cilium repo, e.g., git push -u origin pr/v1.0-backports-YY-MM-DD

9. In Github, create a new PR from your branch towards the feature branch you are backporting to. Note that by default Github creates PRs against the master branch, so you will need to change it.

10. Label the new backport PR with the backport label for the stable branch such as backport/X.Y so that it is easy to find backport PRs later.

11. Mark all PRs you backported with the backport pending label backport-pending/X.Y and clear the needs-backport/vX.Y label. This can be via the GitHub interface, or using the backport script contrib/backporting/set-labels.py, e.g.:

    # Set PR 1234's v1.0 backporting labels to pending
    $ contrib/backporting/set-labels.py 1234 pending 1.0
    

    Note

    contrib/backporting/set-labels.py requires Python 3 and PyGithub installed.

12. After the backport PR is merged, mark all backported PRs with backport-done/X.Y label and clear the backport-pending/X.Y label(s).

    # Set PR 1234's v1.0 backporting labels to done
    contrib/backporting/set-labels.py 1234 done 1.0.
    

Update cilium-builder and cilium-runtime images¶

Login to quay.io with your credentials to the repository that you want to update:

cilium-builder - contains Cilium build-time dependencies cilium-runtime - contains Cilium run-time dependencies

0. After login, select the tab “builds” on the left menu.

1. Click on the wheel.
2. Enable the trigger for that build trigger.

3. Confirm that you want to enable the trigger.

4. After enabling the trigger, click again on the wheel.
5. And click on “Run Trigger Now”.

6. A new pop-up will appear and you can select the branch that contains your changes.
7. Select the branch that contains the new changes.

8. After selecting your branch click on “Start Build”.

9. Once the build has started you can disable the Build trigger by clicking on the wheel.
10. And click on “Disable Trigger”.

11. Confirm that you want to disable the build trigger.

12. Once the build is finished click under Tags (on the left menu).
13. Click on the wheel and;
14. Add a new tag to the image that was built.

15. Write the name of the tag that you want to give for the newly built image.
16. Confirm the name is correct and click on “Create Tag”.

17. After the new tag was created you can delete the other tag, which is the name of your branch. Select the tag name.
18. Click in Actions.
19. Click in “Delete Tags”.

20. Confirm that you want to delete tag with your branch name.

You have created a new image build with a new tag. The next steps should be to update the repository root’s Dockerfile so that it points to the new cilium-builder or cilium-runtime image recently created.

21. Update the versions of the images that are pulled into the CI VMs.

• Open a PR against the Packer-CI-Build with an update to said image versions. Once your PR is merged, a new version of the VM will be ready for consumption in the CI.
• Update the SERVER_VERSION field in test/Vagrantfile to contain the new version, which is the build number from the Jenkins Job for the VMs. For example, build 119 from the pipeline would be the value to set for SERVER_VERSION.
• Open a pull request with this version change in the cilium repository.

Nightly Docker image¶

After each successful Nightly build, a cilium/nightly image is pushed to dockerhub.

To use latest nightly build, please use cilium/nightly:latest tag. Nightly images are stored on dockerhub tagged with following format: YYYYMMDD-<job number>. Job number is added to tag for the unlikely event of two consecutive nightly builds being built on the same date.