JJB branch cutting jobs automation
|
Thanh ha <zxiiro@...>
Hi Everyone, (Branching off this discussion from the "Aluminium code freeze" thread.) I had some time this weekend and I figured this is something that was worth spending time on considering it will save us more time in the future as that branch cutting I did last week was a painful process despite the available automation. So I have been working on a patch [0] and JIRA [1] to rewrite the branch cutting script in releng/builder to be smarter. The current script is an AWK script that parses the YAML files as a plain text document and is not smart enough to parse YAML appropriately. The new script I'm working on is Python based that utilizes a YAML parsing library so it has a much better interface for pattern matching on YAML documents and structure. With this, we can parse entire sections of YAML and copy configuration sections out to new job configuration and retain the configuration data structure from the section it copied from, unlike the AWK script. I've identified 2 use cases in our JJB configuration structure: 1) project/project.yaml files are where projects store their basic job configuration and are typically in a format where it can have multiple jjb "project" sections in order of latest release to oldest release. So typically what needs to happen in these files is that the top jjb "project" definition is copy and pasted to the top, the new top configuration becomes the next release and the previous top configuration is updated to change "branch: master" to "branch: stable/RELEASE". There are a few projects that are MRI which we DO NOT want to update however MRI projects typically do not follow the ODL SimRel release code name so what we can do is parse the parameter "stream" and see if it matches the stable branch we are cutting before making modifications to the file. These are the conditions to modify this type of file. 2) Files where "stream" is a list object, in this case these are typically NOT the project/project.yaml files so those are ignored and the files themselves may have multiple jjb "project" definitions themselves (see integration) but most of the time they are CSIT files that only have a single jjb "project" definition. We want to work whether or not they have multiple project sections defined. So in this case we want to parse the entire file looking for all configurations that have "stream" definitions that are a list and insert a new list item for the next dev release. Again typically the lists are ordered with the latest release as the top item in the list so with this assumption we can check that the first "stream" is the branch we are cutting and copy that and update it's branch to "stable/RELEASE" while inserting the next release configuration above it. As far as my investigation went, these are the 2 conditions that we need to update every time we perform branch cutting. I'm not sure if there is more conditions but I've designed the script in such a way that it will be easy for us to add more conditional rules to update the JJB directory if we find out. One thing the script lacks however is that it is not able to retain the formatting of the YAML files as it seems to strip out whitespace and reformats the files when it dumps the data back into the file. To get around this we can use a library like "prettier" to auto-format our YAML files, I've prepared a patch for that here [2]. So moving forward we will run prettier to automatically handle the YAML formatting so that when the auto-update script runs and messes with the formatting, prettier can run after it to re-adjust the formatting back. The script in its current form is almost ready and I'll spend some more time over the next week to polish it up and run more tests. I will make sure this script is ready and tested before the Silicon branch cut event and perhaps we can do a test "release" where we run the script to produce a patch that all project PTLs can review to make sure the script updated their project configurations appropriately. Cheers, Thanh On Wed, Aug 5, 2020 at 3:29 PM Thanh Ha <zxiiro@...> wrote:
|
|
|