Date
1 - 9 of 9
Git workflows
Robert Varga
Hello,
I am slowly catching up on things from last month. One item is the subject of Github workflows. There are a number of unresolved issues, some of which may be my ignorance of the outside world (in which case I would *love* to be proven wrong). Here is the list: 1. Support for multiple branches ================================ It is OpenDaylight policy to support up to 3 branches at any given time for any MSI project. For MRI projects, that number gets to 4 for periods last 2-5 months -- as is the case for YANG tools right now, we have: - yangtools-7.0.x for 2022.03 Phosphorus security support - yangtools-8.0.x for 2022.06 Sulfur - yangtools-9.0.x for 2022.09 Chlorine - yangtools-master for 2023.03 Argon As far as I know, Github does not provide the equivalent of Gerrit cherry-picks out of the box. That certainly was the case ~5 years when I investigated this more deeply. The crux of the issue seems to be Change-ID and its tie-in with GH PRs. I was told by Andy Grimberg this is nigh impossible to reconcile. Change-ID is critical for cross-referencing commits, because equivalent patches can look very differently on each supported branch. That having been said, I do believe this is fixable by automation, e.g. having a bot assign Change-IDs for a PR and squashing each PR into a single patch -- which then can be projected to Gerrit, allowing for migration. I am not aware of such a bot existing, so I track this as something would have to be contributed. 2. Permissions ============== Github is a system external to LF. As such, I do not think there is infrastructure present to project each project's INFO.yaml into Github permissions. AFAICT the only existing thing is the 'OpenDaylight project', which is an all-or-nothing thing. That is something LF IT has to tackle before we consider migrating. 3. Verification =============== Our current infrastructure is tied to Jenkins. A switch to GH requires that a PR triggers the appropriate jobs in Jenkins. Unless we are talking a straight-up move to GH Actions, we need point 1. to be solved and drive verification projected from Gerrit back to GH. If GH actions are in the picture, at least maven-verify need to be migrated. Again, this needs a community contribution. Now, I am not a nay-sayer, but we have a sh*tload of things going and migrating it requires some serious man-power and for those uninitiated, I like to quote Thanh: "Getting Eclipse build where it is was a twelve month effort for three dedicated people" (IIRC). We no longer have Thanh and all his dedication and experience. I have used this quote when someone proposed moving to Gradle. Moving to GH workflows is harder. If we are to tackle this, we need to solve above problems in order: 2, 1, 3. I will lend my support to anyone seriously committed to this undertaking. At the end of the day, this is not impossible. The OpenJDK community has executed a full transition from custom Mercurial workflows (webrev et al.) to GitHub PRs -- but that transition includes a metric ton of automation which had to be written from scratch. We as a community are struggling to get to Jenkins Pipelines, which is dead simple in comparison. So, Venkat, as the one proposing this change, are you in a position and willing to drive it to completion? Regards, Robert |
|
Robert Varga
On 09/08/2022 22:58, Robert Varga wrote:
Hello,Sorry to self-reply. [snip] At the end of the day, this is not impossible. The OpenJDK community has executed a full transition from custom Mercurial workflows (webrev et al.) to GitHub PRs -- but that transition includes a metric ton of automation which had to be written from scratch.A typical Java PR looks like this: https://github.com/openjdk/jdk/pull/9812 Note that the actual management is done by "openjdk" bot, including responding to commands by authorized users: https://github.com/openjdk/jdk/pull/9766#issuecomment-1206152322 has the following workflow results: 1. https://github.com/openjdk/jdk/pull/9766#issuecomment-1206153918 2. @openjdk openjdk bot added the integrated label 5 days ago 3. @openjdk openjdk bot closed this 5 days ago 4. @openjdk openjdk bot removed ready rfr labels 5 days ago 5. https://github.com/openjdk/jdk/pull/9766#issuecomment-1206154128 I am not saying we need all of this (although that would be awesome), but we absolutely require 50% of this. Note that inter-release OpenJDK requires explicit backports tracked in JIRA like here: https://bugs.openjdk.org/browse/JDK-8274349. That side-steps a workflow across multiple Git branches. I do not believe our community does not have the resources to support that amount of overhead, so we need something better here (which in turn implies Change-ID). Regards, Robert |
|
Robert Varga
On 09/08/2022 22:58, Robert Varga wrote:
Hello,Sorry, a clarification: Now, I am not a nay-sayer, but we have a sh*tload of things going and migrating it requires some serious man-power and for those uninitiated, I like to quote Thanh: "Getting Eclipse build where it is was a twelve month effort for three dedicated people" (IIRC). We no longer have Thanh and all his dedication and experience.This cost Eclipse 3MY to execute. The lesson Thanh offered here is this (paraphrasing): "The problem is you are not done until all workflows execute perfectly. Making an up-front analysis effectively means a full migration, because you have to see what each and every project does. Projects do weird things." You will not believe that last sentence until you have experienced it. When you do, it will fundamentally change your perspective. Trust me: been there, done that, multiple times. I would never agree before, I cannot over-emphasize now. Based on my recent (past 24 months) experience, the break down goes down like this: 1. migrate first project: 1 month 2. migrate next 4 projects: 3 months 3. migrate next 5 projects: 1 month 4. migrate all projects: 3 months There is a distinct 1/5/10/all separation in ODL projects. The reality is you do not know until you have OpenFlowPlugin integrated, but you can come prepared if you've done bgpcep (and we do because it is MRI). Summed up: the risk is "you do not know what you do not know". The only way to know more is migrate more projects. Bye, Robert P.S.: I will freely admit the most severe single hit this community has taken over all of those years is loosing Thanh. Two reasons: dedication and experience. I hope we will find someone to replace him, the sooner the better. P.P.S.: It took 4 months to develop a major YANG Tools change. It took another 15 man-months to fully integrate that change. Executed as two steps. Both were a major pain. We are past that now. The mop-up is mostly peanuts. |
|
Robert Varga
On 09/08/2022 22:58, Robert Varga wrote:
Hello,[snip] I have used this quote when someone proposed moving to Gradle. Moving to GH workflows is harder.Regarding Gradle: it was proposed first in 2015 (if memory serves). We are now at a place where MD-SAL Binding codegen is completely independent of Maven and the interface (yangtools' codegen-api) gives complete control over file lifecycle to the driver (e.g. build system plugin). Gradle/SBT/whatever plugins are now simple, if you just do (and contribute!) the equivalent of yang-maven-plugin (sans yang-maven-plugin-spi, which is unused and going away). Yeah, there is no abstract support to make that easy, but those bits should be clear when there are 2 plugins. Regards, Robert |
|
Guillaume Lambert
Hi all
De : TSC@... <TSC@...> de la part de Robert Varga <nite@...>
Envoyé : mardi 9 août 2022 22:58 À : tsc@... Objet : [OpenDaylight TSC] Git workflows Hello,
I am slowly catching up on things from last month. One item is the subject of Github workflows. There are a number of unresolved issues, some of which may be my ignorance of the outside world (in which case I would *love* to be proven wrong). Here is the list: 1. Support for multiple branches ================================ It is OpenDaylight policy to support up to 3 branches at any given time for any MSI project. For MRI projects, that number gets to 4 for periods last 2-5 months -- as is the case for YANG tools right now, we have: - yangtools-7.0.x for 2022.03 Phosphorus security support - yangtools-8.0.x for 2022.06 Sulfur - yangtools-9.0.x for 2022.09 Chlorine - yangtools-master for 2023.03 Argon As far as I know, Github does not provide the equivalent of Gerrit cherry-picks out of the box. That certainly was the case ~5 years when I investigated this more deeply. The crux of the issue seems to be Change-ID and its tie-in with GH PRs. I was told by Andy Grimberg this is nigh impossible to reconcile. Change-ID is critical for cross-referencing commits, because equivalent patches can look very differently on each supported branch. That having been said, I do believe this is fixable by automation, e.g. having a bot assign Change-IDs for a PR and squashing each PR into a single patch -- which then can be projected to Gerrit, allowing for migration. I am not aware of such a bot existing, so I track this as something would have to be contributed. 2. Permissions ============== Github is a system external to LF. As such, I do not think there is infrastructure present to project each project's INFO.yaml into Github permissions. AFAICT the only existing thing is the 'OpenDaylight project', which is an all-or-nothing thing. That is something LF IT has to tackle before we consider migrating. 3. Verification =============== Our current infrastructure is tied to Jenkins. A switch to GH requires that a PR triggers the appropriate jobs in Jenkins. Unless we are talking a straight-up move to GH Actions, we need point 1. to be solved and drive verification projected from Gerrit back to GH. If GH actions are in the picture, at least maven-verify need to be migrated. Again, this needs a community contribution. Now, I am not a nay-sayer, but we have a sh*tload of things going and migrating it requires some serious man-power and for those uninitiated, I like to quote Thanh: "Getting Eclipse build where it is was a twelve month effort for three dedicated people" (IIRC). We no longer have Thanh and all his dedication and experience. I have used this quote when someone proposed moving to Gradle. Moving to GH workflows is harder. If we are to tackle this, we need to solve above problems in order: 2, 1, 3. I will lend my support to anyone seriously committed to this undertaking. At the end of the day, this is not impossible. The OpenJDK community has executed a full transition from custom Mercurial workflows (webrev et al.) to GitHub PRs -- but that transition includes a metric ton of automation which had to be written from scratch. We as a community are struggling to get to Jenkins Pipelines, which is dead simple in comparison. So, Venkat, as the one proposing this change, are you in a position and willing to drive it to completion? Regards, Robert _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. |
|
Greetings,
toggle quoted message
Show quoted text
Coming into this a little late. On 8/9/22 13:58, Robert Varga wrote:
--<snip>-- 1. Support for multiple branchesThis is still accurate AFAIK. Change-ID is very much a Gerrit concept which they leverage heavily to understand the status of any given change across the status of all branches. That all being said, there is the ability to pull GitHub PRs into Gerrit. The problem with this being as follows: 1. It's a manual process by way of a plugin in Gerrit that has to be initiated by someone that is expectings a PR that needs to come in. 2. Change-ID isn't something that is enforced by GitHub, it could be semi-enforced by a GitHub Action, but all that would do is mark the PR as not passing, so it's not true enforcement. GitLab, on the other hand, _could_ enforce this because it's possible to setup a regex filter on commit messages that must be passed to even raise an MR, but there's other downsides to that (including LFRE not having any job integration currently with GitLab and Jenkins, though it's possible). Doing the Change-ID work by a GitHub Action or bot may be a solution for reflecting changes back into Gerrit, but you're still going to run into some weird edge cases. 2. PermissionsTechnically, we have something comprable on the GitHub side using INFO.yaml. However, absolutely no project that we support in GitHub has elected to utilize it as it ended up being harder to work with than the INFO files as they currently exist in repo. Mostly because the implementation pulled it out of the repos themselves and stuck it into a side repo inside the Org. The reasons for this are varied but mostly come down to how easy it was to detect changes to the remote INFO files that needed to be then be shadowed into the LF's releng/info-master repository. 3. VerificationLF managed Jenkins already supports GitHub as a source SCM triggering into Jenkins. In point of fact all of the global-jjb core jobs that ODL utilizes have two variants, a Gerrit variant and a GitHub variant. The primary issue being that you can't have both variants active for a given repository at the same time because namespace collisions. There was an idea floated not long ago internally about if it would be possible to sort of go the other way Gerrit -> GitHub with work still primarily happening in Gerrit (changes raised, etc) but cause Gerrit changes to raise PRs into the GitHub mirror to then trigger GitHub Actions that would then somehow have the results shuttled back to the Gerrit Change. I believe this would be doable, but nobody has had the time to sit down and evaluate how to actually make it work. The best scenario would be that changes could be raised on either side (Gerrit or GitHub) and that review itself would just continue to happen in Gerrit along with the final merges. Getting bi-directionality would be a major project though. --<snip>-- -Andy- -- Andrew J Grimberg Manager Release Engineering The Linux Foundation NOTICE: The Linux Foundation supports their employees with flexible work hours. If you recieve mail from me outside of standard business hours please be aware that I do not expect a response until the next standard business day. |
|
Robert Varga
On 12/08/2022 19:30, Andrew Grimberg wrote:
Greetings,Hello, Coming into this a little late.and welcome :) On 8/9/22 13:58, Robert Varga wrote:[snip] I think getting PRs into Gerrit would be a very useful first step. If there is genuine interest, we should see the facility get used for contributions -- and we can hash things out and we use it.That having been said, I do believe this is fixable by automation, e.g. having a bot assign Change-IDs for a PR and squashing each PR into a single patch -- which then can be projected to Gerrit, allowing for migration. I am not aware of such a bot existing, so I track this as something would have to be contributed.This is still accurate AFAIK. Change-ID is very much a Gerrit concept which they leverage heavily to understand the status of any given change across the status of all branches. What is the paperwork we need to do to get this rolling? :) Regards, Robert |
|
On 8/12/22 13:10, Robert Varga wrote:
On 12/08/2022 19:30, Andrew Grimberg wrote::) Thanks!Greetings,Hello,Coming into this a little late.and welcome :) As always, open a support ticket at https://support.linuxfoundation.orgOn 8/9/22 13:58, Robert Varga wrote:[snip] Please be aware that this may take a bit to get working as we haven't set it up ourselves. My experience with it was testing out the workflow on GerritHub [0]. Since the folks that run GerritHub (GerritForge [1]) are the creators and maintainers of the plugin that allows for this workflow to even exist at present. You can test out the workflow with some personal GitHub repos by logging into GerritHub. [0] https://gerrithub.io [1] https://gerritforge.com -- Andrew J Grimberg Manager Release Engineering The Linux Foundation NOTICE: The Linux Foundation supports their employees with flexible work hours. If you recieve mail from me outside of standard business hours please be aware that I do not expect a response until the next standard business day. |
|
Robert Varga
On 15/08/2022 17:48, Andrew Grimberg wrote:
On 8/12/22 13:10, Robert Varga wrote:[snip]On 12/08/2022 19:30, Andrew Grimberg wrote:Greetings,Hello,Coming into this a little late. https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-24404What is the paperwork we need to do to get this rolling? :)As always, open a support ticket at https://support.linuxfoundation.org Please be aware that this may take a bit to get working as we haven't set it up ourselves. My experience with it was testing out the workflow on GerritHub [0]. Since the folks that run GerritHub (GerritForge [1]) are the creators and maintainers of the plugin that allows for this workflow to even exist at present.I'll take a peek later and follow up on the issue if it does not align with what we want to do. Thanks, Robert |
|