Date
1 - 12 of 12
Widening reach for ... code?
|
Robert Varga
Hello,
in today's (lightly advertised) TWS call, Casey raised the question of migrating our infrastructure to GitLab in a bid to attract more contributions. Unfortunately a straight-up migration seems unfeasible to me due to a multitude for things that come into play: 1) Jenkins integration, or are we saying we would also switch CI? If so, how does that work with our JJB infra/CSIT? 2) Patch review process. Let's not kid ourselves, GitLab and GitHub patch review workflow is utterly different from Gerrit. I do not know what the multi-branch support looks like these days, but last time I checked GitHub's workflow was woefully inadequate to support multiple concurrent streams (to test: do you have 'cherry-pick'-like option?) 3) The straight-up choice of GitLab -- I think this is something the community should have a choice in. Then again, we probably have far less technical participation than would be needed to make a well-informed decision. With that out of the way, I would like to make a counter-proposal: How about LF(N) investing some dev/test resources into making it possible for outside contributors to be mirrored back to Gerrit? I mean we already have a read-only mirror on GitHub for each and every ODL mirror, and we should be able to make the same thing work on GitLab (or any other force). We are utterly lacking the ability to review pull requests -- in any way, shape or form! What I mean is that while the repository mirrors are read-only, at least GitHub provides API access, which should allow our infra to monitor those repositories for incoming PRs. When a PR is created, updated, closed, these actions should automagically be turned into equivalent patch (series) being created in Gerrit. I do believe this is feasible, as Gerrit provides a superset of required operations. It would also mean any such PR would be serviced by our CI. This would allow people to contribute, potentially provide Git{Hub,Lab} reviews of PRs, extending the reach, while still not requiring any sort of infrastructure or workflow migration. It also is very much aligned with LF's mission to promote use of free software -- Gerrit is (Apache-licensed) FOSS after all! Hence, Casey, is this something LF has any capability/interest of picking up in a bid to actively make the world a better place? Regards, Robert |
|
|
|
Robert Varga
Sorry, thinko fixes:
On 24/08/2020 22:06, Robert Varga wrote: How about LF(N) investing some dev/test resources into making its/contributors/contributions/ s/ODL mirror/ODL project/ Regards, Robert |
|
|
|
Thanh ha <zxiiro@...>
See inline. (Include's Robert's fixes) On Mon, Aug 24, 2020 at 4:06 PM Robert Varga <nite@...> wrote: Hello, Either way would require modifying JJB to support the new SCM system GitLab and the trigger mechanism would be quite a bit different than what we have with Gerrit (PRs vs Gerrit Trigger) so this wouldn't be a simple change of backends. Concern here for me is who would lead this effort. I think we'd need LF support considering the community does not have the available cycles to do this. 2) Patch review process. Let's not kid ourselves, GitLab and GitHub The cherry-pick option in GitHub (likely GitLab too but I'm not familiar with it) is CLI with a command like `git cherry-pick master` or similar. Personally I like this method but that's because I'm in the weeds of Git often, I can see the value in an easy button for this like Gerrit has. Sadly, the GitHub / GitLab model is via PRs (basically branches) which are not equivalent to patch reviews like Gerrit. So a change of system would require all the committers to unlearn Gerrit. 3) The straight-up choice of GitLab -- I think this is something the I think this would be better received if the LF did the background work to make this work for the community, however I feel LF is no longer actively engaged in LFN project's day to day development cycle. Meaning LF is going to drop GitLab on us and tell us to do the legwork to make all of our projects work with that. This kind of transition would require a significant amount of work for the community to take on to transition the CI bits and adjust our workflows. Considering there's a lack of people to lead this kind of work... I worry that this is an exercise in frustration. With that out of the way, I would like to make a counter-proposal: This is possible as I've used GerritHub (http://gerrithub.io/) before and it worked (although over 8 years ago when I tried it), it was quite clunky but maybe it's better now. I'm not sure if GerritHub is proprietary or if there's available a Gerrit plugin for this feature. Not sure how hard it would be to replicate this feature but at least it seems to be possible. Regards, Thanh I mean we already have a read-only mirror on GitHub for each and every |
|
|
|
I can weigh in a little bit on this. I'm not going to go in depth here,
but I can definitely answer a few of the questions raised. On 2020-08-24 13:06, Robert Varga wrote: Hello,I migration to GitLab would basically mean throwing the Jenkins infra out the door and having to rebuild all jobs as GitLab pipelines. Technically we could make MRs (GitLab's parlance of Merge Request vs GitHub's of Pull Requests / PR) work in Jenkins, but we've not done any work along that in that direction. We do, however, have PRs working with Jenkins (I'll get more into this in a bit) 2) Patch review process. Let's not kid ourselves, GitLab and GitHubTo be brutally honest (and this it truly my personal opinion here) both GitLab and GitHub suck when it comes to code review. The review process that Gerrit pushes creates a much better experience IMO. It is, however, a really huge hurdle for folks and a lot our time with new developers seems to be spent on just getting them familiar with Gerrit Way 3) The straight-up choice of GitLab -- I think this is something theFor various reasons I'm not going to into the whys or wherefores of why GitLab is the current preferred choice of LFIT for SaaS SCM/CI solutions. I'll gladly talk to you off record on this one ;) But to name just a few * We have a way to enforce DCO requirements that stops changes from coming in without them. GitHub is an after the fact CI validator which makes it really hard to actually enforce and very easy to game * GitLab supports repo hierarchies because of how they built their system. GitHub is a flat namespace. We already flatten our repo hierarchies (not that ODL has many, but it does have a few) by translating the '/' to '-'. ONAP and other LFN properties already have issues being able to really understand where a particular repo originally came from in the GitHub mirror because of this flattening. * GitLab CI is more flexible in that we can leverage pipeline templates / libraries. GitHub Actions is still lacking in this though they do have the GitHub Actions marketplace, it makes it harder to have standard job designs. * GitLab CI allows us to do dynamic builders ala Jenkins as long as we're doing it against AWS or GCE. * It's possible with GitHub to do some amount of dynamic builders by leveraging AWS CodeBuild but then you're splitting your GitHub Actions into two logically distinct parts. The GHA workflow itself, and then the CodeBuild "job" portion that runs out in a CodeBuild defined environment. Each CodeBuild project has to be pre-created before it can be used and we would essentially be back to having to come up with a nice way (ala jjb) to manage those CodeBuild projects and their life cycle :-/ (yes they can be managed from the aws cli, but we're still then having to basically write a jjb type replacement that does stuff agaisnt CodeBuild) * One pro for GitHub that is a con for GitLab (not this affect ODL, but it does affect some other LFN properties) EasyCLA support exists for GitHub, it does not currently exist for GitLab. With that out of the way, I would like to make a counter-proposal:Technically we can do this already. There is a GitHub plugin for Gerrit that will allow us to create Gerrit side changes if a PR is opened a GitHub repository. That being said, I've played with this workflow over on GerritHub and it seriously hurts. The workflow operates in the following manner: * PR is created in GitHub * For every commit in the PR a new Gerrit change is created * If changes are requested on the PR then the modified code ends up coming in as a _separate_ Gerrit Change. The requested changes are _not_ communicated back to the open PR, they happen internal to the Gerrit system. If the system is configured to not send mail to unknown entities (as all of ours are) then the person that raised the PR, if they have never logged into the Gerrit system, will _not_ get notified. * Once the change is merged on the Gerrit side, the code gets mirrored back to GitHub but the PR does not get closed. Additionally, and I don't know if this is just a limitation of how GerritHub operates, the PRs don't automatically create Gerrit Changes. You have to explicitly force a PR refresh cycle. Now, some of this is obviously just a failing in how the plugin operates, but the biggest issue here is that the GitHub/GitLab workflow is very much an anti-pattern in Gerrit. Gerrit requires that Change-ID to be able to track different patches across the life of a single change. You don't get that in GitHub/GitLab, you just got a very messy patch series to contend with. I mean we already have a read-only mirror on GitHub for each and every-- Andrew J Grimberg Manager Release Engineering The Linux Foundation |
|
|
|
On 2020-08-24 13:48, Thanh ha wrote:
See inline.We are doing some leg work on this. We've been getting semi-deep in the weeds the following of late: GitHub + Azure Pipelines GitHub + Jenkins (we already support several projects in this config) GitHub + GitHub Actions GitLab + GitLab CI You'll note a distinct lack of GitLab + Jenkins in that list ;) OPNFV had a proposal just last week for an RE managed lift from Gerrit + Jenkins -> GitLab + GitLab CI transition. The reason they were targetted is because they don't really have that many jobs to transition and in all honesty, it wouldn't be much of a change in how they do things now outside of the use of GitLab instead of Gerrit. The reason being that most of the work in OPNFV actually happens in third-party CI systems. With that out of the way, I would like to make a counter-proposal:No, it's still clunky and nasty IMO. That's me having just tried it again about 1.5 - 2 months ago. The plugin that GerritHub uses is available to us, but I've not tried using it directly, just via their system. -Andy- I mean we already have a read-only mirror on GitHub for each and every-- Andrew J Grimberg Manager Release Engineering The Linux Foundation |
|
|
|
Robert Varga
On 24/08/2020 23:07, Andrew Grimberg wrote:
We are doing some leg work on this. We've been getting semi-deep in theRight-o. We are in a completely different position: we have *hundreds* if not *thousands* of jobs to migrate. Granted, majority of those are duds, but my guesstimate is we are looking at ~200 jobs that have to work on day one. That makes the migration Not Feasible at the Time of This Writing (tm), at least in my book. Regards, Robert |
|
|
|
Thanh ha <zxiiro@...>
On Mon, Aug 24, 2020 at 5:07 PM Andrew Grimberg <agrimberg@...> wrote: On 2020-08-24 13:48, Thanh ha wrote: On the ODL side of things, moving the regular build jobs will be the "easy" part as it's migrating some maven shell scripts to the new CI system. CSIT on the other hand would be the difficult part. I think not only to get the shell scripts working on the new platform but also figuring out how to re-route the triggering mechanisms. Like when autorelease passes, build the integration-test project with a specific parameter that is different than when autorelease fails. There's a few specific trigger paths that would require someone to fully understand how all the ODL jobs are mapped to carry out. Also since CSIT also spins up a lab using heat, if we're lucky the heat templates will work out of the box but if not it would take time to figure out the workflow here. Another thing would be there're several ODL jobs that are tied directly to Gerrit comment triggers, this would need to be rewritten for GitLab I would assume. In any case, while I wouldn't be against moving to a new CI system. However with consideration of ODL's current available resources... It would be a significant disruption for the ODL community and would require either LF to lead the work, or someone is able to step up to do that and see it to completion. Thanh > With that out of the way, I would like to make a counter-proposal: |
|
|
|
Robert Varga
On 24/08/2020 22:59, Andrew Grimberg wrote:
So far so good.How about LF(N) investing some dev/test resources into making itTechnically we can do this already. There is a GitHub plugin for Gerrit * If changes are requested on the PR then the modified code ends upOkay, so this is where LF(N) or ODL budgeting gets involved. The plugin needs to be improved to mirror any comments back to the PR. IIRC at least GH has APIs to do this. Same as above, this is just being plain stupid. Except for your final point... Additionally, and I don't know if this is just a limitation of how... right, which is where our pre-commit hooks et al. get into play. A very obvious solution is to: 1) require all PRs to have 'Allow edits by maintainers' enabled, which is the default at GH at least. 2) have a bot that updates all PR patches to include Change-ID (yeah, the bot has to be a maintainer, but LF should have zero problems running at that privilege level) 3) that effectively means we have bridged the PR/patch gap, hence re-pushes end up correlating correctly -- i.e. the bot knows the PR number and patch (seqnr|headline|...) hence can (re-)assign correct change-IDs and/or issue warnings/whatever. Yes, it is not plain GH/GL workflow, but hey, I'll bet anyone a (non-sour!) beer it can be made to work 95% of time. But that circles back to allocating budget (== people) to taking this analysis to a solution. It seems we have a fruitful exploration here and nothing says 'utterly unfeasible' so far -- perhaps we need an RFP or similar? Regards, Robert |
|
|
|
Robert Varga
On 24/08/2020 23:21, Thanh Ha wrote:
Like when autorelease passesAh, and related, how would autorelease/maven-stage jobs' Sigul signing work? Regards, Robert |
|
|
|
Olivier Dugeon
@Cédric, OPNFV is mention in this thread ... Olivier Le 24/08/2020 à 23:07, Andrew Grimberg
a écrit :
On 2020-08-24 13:48, Thanh ha wrote:See inline. (Include's Robert's fixes) On Mon, Aug 24, 2020 at 4:06 PM Robert Varga <nite@... <mailto:nite@...>> wrote: Hello, in today's (lightly advertised) TWS call, Casey raised the question of migrating our infrastructure to GitLab in a bid to attract more contributions. Unfortunately a straight-up migration seems unfeasible to me due to a multitude for things that come into play: 1) Jenkins integration, or are we saying we would also switch CI? If so, how does that work with our JJB infra/CSIT? Either way would require modifying JJB to support the new SCM system GitLab and the trigger mechanism would be quite a bit different than what we have with Gerrit (PRs vs Gerrit Trigger) so this wouldn't be a simple change of backends. Concern here for me is who would lead this effort. I think we'd need LF support considering the community does not have the available cycles to do this. 2) Patch review process. Let's not kid ourselves, GitLab and GitHub patch review workflow is utterly different from Gerrit. I do not know what the multi-branch support looks like these days, but last time I checked GitHub's workflow was woefully inadequate to support multiple concurrent streams (to test: do you have 'cherry-pick'-like option?) The cherry-pick option in GitHub (likely GitLab too but I'm not familiar with it) is CLI with a command like `git cherry-pick master` or similar. Personally I like this method but that's because I'm in the weeds of Git often, I can see the value in an easy button for this like Gerrit has. Sadly, the GitHub / GitLab model is via PRs (basically branches) which are not equivalent to patch reviews like Gerrit. So a change of system would require all the committers to unlearn Gerrit. 3) The straight-up choice of GitLab -- I think this is something the community should have a choice in. Then again, we probably have far less technical participation than would be needed to make a well-informed decision. I think this would be better received if the LF did the background work to make this work for the community, however I feel LF is no longer actively engaged in LFN project's day to day development cycle. Meaning LF is going to drop GitLab on us and tell us to do the legwork to make all of our projects work with that. This kind of transition would require a significant amount of work for the community to take on to transition the CI bits and adjust our workflows. Considering there's a lack of people to lead this kind of work... I worry that this is an exercise in frustration.We are doing some leg work on this. We've been getting semi-deep in the weeds the following of late: GitHub + Azure Pipelines GitHub + Jenkins (we already support several projects in this config) GitHub + GitHub Actions GitLab + GitLab CI You'll note a distinct lack of GitLab + Jenkins in that list ;) OPNFV had a proposal just last week for an RE managed lift from Gerrit + Jenkins -> GitLab + GitLab CI transition. The reason they were targetted is because they don't really have that many jobs to transition and in all honesty, it wouldn't be much of a change in how they do things now outside of the use of GitLab instead of Gerrit. The reason being that most of the work in OPNFV actually happens in third-party CI systems. --
Olivier Dugeon Orange Expert, Future Networks Open Source Referent Orange/IMT/OLN/WTC/IEE/iTeQ fixe : +33 2 96 07 28 80 mobile : +33 6 82 90 37 85 olivier.dugeon@... _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. |
|
|
|
On 2020-08-24 14:35, Robert Varga wrote:
On 24/08/2020 22:59, Andrew Grimberg wrote:There's an additional issue with that. In the Gerrit workflow you amendSo far so good.How about LF(N) investing some dev/test resources into making itTechnically we can do this already. There is a GitHub plugin for Gerrit your change, effectively creating a brand new commit object (different commit sha) which _replaces_ the previous patch in your change series. Doing something with a PR breaks down because by default a PR is a sequence of changes that are completely chronological, so a fix in it means you end up with a new commit object with just that change and it gets added to the end of the PR commit chain. To properly work with Gerrit in the work flow would require folks to force push their fixes back to their repo destroying (or at least orphaning) the old versions of their prior commits in the chain. Now, if we can get the PR -> Gerrit thing working correctly, that may not be too much of an issue as the prior patches would still live in the Gerrit change history but getting people on the GitHub side to do things cleanly is going to be difficult. I suppose Ideally what would happen would be that if a change was requested then the PR would be closed and the contributor would have to open a new PR, but if they don't have the same Change-ID (which we _might_ be able to force into their previous commit message) it's still going to disassociate and become a different Gerrit Change. --* Once the change is merged on the Gerrit side, the code gets mirroredSame as above, this is just being plain stupid. Except for your final Andrew J Grimberg Manager Release Engineering The Linux Foundation |
|
|
|
cedric.ollivier@...
Hi,
It’s known that LFN selected OPNFV as first candidate because it was (falsy?) considered as the simplest LFN project. And it’s fully driven by cost savings. I quickly highlighted lots of technical drawbacks mostly mentioned here as well and lots of incompatibilities with the overall OPNFV planning.
Right now, nothing is approved in OPNFV ! I think everybody agrees that it must be a community decision here, not an LFN-only one. I clearly refused it on behalf of Functest (from far the most active OPNFV project). And if I remember well, our TSC chair suggested another LFN project instead of OPNFV (e.g. ODL).
BYW I think the Functest jjb should be correctly evaluated. Functest offers lots of OpenStack and Kubernetes test cases, in 5 active branches, 3 different architectures. You could easily imagine a huge number of jjbs to guarantee the constant stability (I think thousands was mentioned by LFN during the meeting …)
I would have prefer that LFN helps the community and fix and maintain the Releng projects (see last Dev event). We could have rather implemented the job management per branch model in Releng as proposed by all modern CI tools.
I was very surprised that Functest was used as an example during OPNFV Weekly meeting and I was not aware than LFN was working on porting Functest jobs. For sure, the LFN team has to contribute to Functest first before publishing any CI changes related to Functest (and we all know GitlabCI, right?).
Switching to Gitlab doesn’t solve our current model based on ticket (compatible with community principles? See current action to Arpit by LFN Board). We would face with our current problems via GitlabCI as well or any tool as soon as it’s not maintained enough.
My 2 cents: there should be only one review tool even if the git repositories could be replicated.
Cédric
De : DUGEON Olivier TGI/OLN
@Cédric, OPNFV is mention in this thread ... Olivier Le 24/08/2020 à 23:07, Andrew Grimberg a écrit :
On 2020-08-24 13:48, Thanh ha wrote:See inline.(Include's Robert's fixes)On Mon, Aug 24, 2020 at 4:06 PM Robert Varga <nite@...<mailto:nite@...>> wrote:Hello,in today's (lightly advertised) TWS call, Casey raised the question ofmigrating our infrastructure to GitLab in a bid to attract morecontributions.Unfortunately a straight-up migration seems unfeasible to me due to amultitude for things that come into play:1) Jenkins integration, or are we saying we would also switch CI? If so,how does that work with our JJB infra/CSIT?Either way would require modifying JJB to support the new SCM systemGitLab and the trigger mechanism would be quite a bit different thanwhat we have with Gerrit (PRs vs Gerrit Trigger) so this wouldn't be asimple change of backends. Concern here for me is who would lead thiseffort. I think we'd need LF support considering the community does nothave the available cycles to do this.2) Patch review process. Let's not kid ourselves, GitLab and GitHubpatch review workflow is utterly different from Gerrit. I do not knowwhat the multi-branch support looks like these days, but last time Ichecked GitHub's workflow was woefully inadequate to support multipleconcurrent streams (to test: do you have 'cherry-pick'-like option?)The cherry-pick option in GitHub (likely GitLab too but I'm not familiarwith it) is CLI with a command like `git cherry-pick master` or similar.Personally I like this method but that's because I'm in the weeds of Gitoften, I can see the value in an easy button for this like Gerrit has.Sadly, the GitHub / GitLab model is via PRs (basically branches) whichare not equivalent to patch reviews like Gerrit. So a change of systemwould require all the committers to unlearn Gerrit.3) The straight-up choice of GitLab -- I think this is something thecommunity should have a choice in. Then again, we probably have far lesstechnical participation than would be needed to make a well-informeddecision.I think this would be better received if the LF did the background workto make this work for the community, however I feel LF is no longeractively engaged in LFN project's day to day development cycle. MeaningLF is going to drop GitLab on us and tell us to do the legwork to makeall of our projects work with that. This kind of transition wouldrequire a significant amount of work for the community to take on totransition the CI bits and adjust our workflows. Considering there's alack of people to lead this kind of work... I worry that this is anexercise in frustration.We are doing some leg work on this. We've been getting semi-deep in theweeds the following of late:GitHub + Azure PipelinesGitHub + Jenkins (we already support several projects in this config)GitHub + GitHub ActionsGitLab + GitLab CIYou'll note a distinct lack of GitLab + Jenkins in that list ;)OPNFV had a proposal just last week for an RE managed lift from Gerrit +Jenkins -> GitLab + GitLab CI transition. The reason they were targettedis because they don't really have that many jobs to transition and inall honesty, it wouldn't be much of a change in how they do things nowoutside of the use of GitLab instead of Gerrit. The reason being thatmost of the work in OPNFV actually happens in third-party CI systems.With that out of the way, I would like to make a counter-proposal:How about LF(N) investing some dev/test resources into making itpossible for outside contributions to be mirrored back to Gerrit?This is possible as I've used GerritHub (http://gerrithub.io/) beforeand it worked (although over 8 years ago when I tried it), it was quiteclunky but maybe it's better now. I'm not sure if GerritHub isproprietary or if there's available a Gerrit plugin for this feature.Not sure how hard it would be to replicate this feature but at least itseems to be possible.No, it's still clunky and nasty IMO. That's me having just tried itagain about 1.5 - 2 months ago. The plugin that GerritHub uses isavailable to us, but I've not tried using it directly, just via theirsystem.-Andy-I mean we already have a read-only mirror on GitHub for each and everyODL project, and we should be able to make the same thing work on GitLab(or any other force). We are utterly lacking the ability to review pullrequests -- in any way, shape or form!What I mean is that while the repository mirrors are read-only, at leastGitHub provides API access, which should allow our infra to monitorthose repositories for incoming PRs. When a PR is created, updated,closed, these actions should automagically be turned into equivalentpatch (series) being created in Gerrit. I do believe this is feasible,as Gerrit provides a superset of required operations. It would also meanany such PR would be serviced by our CI. --
Olivier Dugeon
fixe : +33 2 96 07 28 80 _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. |
|
|
