JJB branch cutting jobs automation

Thanh ha <zxiiro@...>

Hi Everyone,

(Branching off this discussion from the "Aluminium code freeze" thread.)

I had some time this weekend and I figured this is something that was worth spending time on considering it will save us more time in the future as that branch cutting I did last week was a painful process despite the available automation.

So I have been working on a patch [0] and JIRA [1] to rewrite the branch cutting script in releng/builder to be smarter. The current script is an AWK script that parses the YAML files as a plain text document and is not smart enough to parse YAML appropriately. The new script I'm working on is Python based that utilizes a YAML parsing library so it has a much better interface for pattern matching on YAML documents and structure.

With this, we can parse entire sections of YAML and copy configuration sections out to new job configuration and retain the configuration data structure from the section it copied from, unlike the AWK script. I've identified 2 use cases in our JJB configuration structure:

1) project/project.yaml files are where projects store their basic job configuration and are typically in a format where it can have multiple jjb "project" sections in order of latest release to oldest release. So typically what needs to happen in these files is that the top jjb "project" definition is copy and pasted to the top, the new top configuration becomes the next release and the previous top configuration is updated to change "branch: master" to "branch: stable/RELEASE".

There are a few projects that are MRI which we DO NOT want to update however MRI projects typically do not follow the ODL SimRel release code name so what we can do is parse the parameter "stream" and see if it matches the stable branch we are cutting before making modifications to the file.

These are the conditions to modify this type of file.

2) Files where "stream" is a list object, in this case these are typically NOT the project/project.yaml files so those are ignored and the files themselves may have multiple jjb "project" definitions themselves (see integration) but most of the time they are CSIT files that only have a single jjb "project" definition. We want to work whether or not they have multiple project sections defined.

So in this case we want to parse the entire file looking for all configurations that have "stream" definitions that are a list and insert a new list item for the next dev release. Again typically the lists are ordered with the latest release as the top item in the list so with this assumption we can check that the first "stream" is the branch we are cutting and copy that and update it's branch to "stable/RELEASE" while inserting the next release configuration above it.

As far as my investigation went, these are the 2 conditions that we need to update every time we perform branch cutting. I'm not sure if there is more conditions but I've designed the script in such a way that it will be easy for us to add more conditional rules to update the JJB directory if we find out.

One thing the script lacks however is that it is not able to retain the formatting of the YAML files as it seems to strip out whitespace and reformats the files when it dumps the data back into the file. To get around this we can use a library like "prettier" to auto-format our YAML files, I've prepared a patch for that here [2]. So moving forward we will run prettier to automatically handle the YAML formatting so that when the auto-update script runs and messes with the formatting, prettier can run after it to re-adjust the formatting back.

The script in its current form is almost ready and I'll spend some more time over the next week to polish it up and run more tests. I will make sure this script is ready and tested before the Silicon branch cut event and perhaps we can do a test "release" where we run the script to produce a patch that all project PTLs can review to make sure the script updated their project configurations appropriately.


On Wed, Aug 5, 2020 at 3:29 PM Thanh Ha <zxiiro@...> wrote:
So a lot of this is actually automated and we have documentation on the branch cutting process here:

So there's a script that will update the JJB jobs templates to the next release but it is finicky. There's a few things it does wrong though including:

* Setting java-version for all projects but some csit jobs require "jre" instead of java-version.
* Duplicates the gerrit-release-merge jobs into the new branch causing JJB to fail due to duplicate job definition. This is something I can fix by moving release management jobs into their own project config. I'll look into this after branch cutting work is complete.
* Injects "java-version" to all jobs regardless of if they are java jobs at all so that needs to be manually resolved too.
* Does not appropriately update the opflex project (this is a snowflake project)

and a few other small things that need a human to inspect.

As for creating branches in all projects we have an autorelease job that will clone every repo and create stable/aluminium branches https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-branch-cut/ someone just has to manually run it once a release.

Finally the version bump job that we normally use needs to be run to bump the master branch.

So for the most part things are automated but there's a few steps that need a human to 1) decide when to run the job, and 2) inspect job config to fix issues caused by the jjb update script.


On Wed, Aug 5, 2020 at 3:16 PM Luis Gomez <ecelgp@...> wrote:
Not sure if this is tribal knowledge but every release I have to manually patch JJB for some misses:

I have not checked yet the patch for this release.

On Aug 5, 2020, at 11:49 AM, Robert Varga <nite@...> wrote:

On 05/08/2020 19:19, Luis Gomez wrote:
Branching is not a problem, the real work is to update all the jenkins jobs after the branching. Some of them need special adjustments, not easy to automate today.

Well, for most projects I witnessed it's just a copy of the first block
-- do we have those specials documented somewhere, or is it just tribal
knowledge? :)


Join {TSC@lists.opendaylight.org to automatically receive all group messages.