(Long rant warning. Proposal is by no means complete, but attempts to propose general mechanics and highlights challenges which are intertwined.)
Unfortunately, we have a larger inter-project problem in the OpenDaylight ecosystem, where managing the inter-project version skew is actually the major pain point. We essentially are integrated via snapshot revisions, which are a moving target and do not allow downstream projects to control what is they are getting. In order to get to something we can manage manually, the best practice is to maintain a single version per project, which runs contrary to semver and does not really solve anything.
I think the path forward lies in an automated, continuous release process, supported by proper project version exports. This is a combination of something we already have, namely the autobuild (which just needs to be refactored), something that we need to develop (a dashboard for version skew) and something we need to add to our projects (maven dependency poms).
The first problem we are facing is that we are integrating multiple projects across SNAPSHOT revisions. This makes it impossible to gate downstream projects from changes happening upstream, nor 'go back in time'. We clearly need to do inter-project integration on "release" artifacts. Manual (or SCM-based) version bumps are not feasible, simply because of the churn introduced to pom.xmls in the upstream project. We can easily fix this by having each merge build perform the equivalent of what autorelease does for all projects: replace -SNAPSHOT with a time-based snapshot -- 0.6.2-SNAPSHOT becomes 0.6.2-20141007-062302. The format is significant, as it does not contain anything which can be mistaken lexicographically, thus artifacts form a monotonic sequence no matter what (well, unless NTP decides otherwise). In order to mark the revision from which a particular set of artifacts was created we'd use a git tag. The HEAD version (and SCM coordinates) would be carried inside the jar (in META-INF), not in the filename.
The second is the version skew problem. Once we have stable versions, we need a simple way of how downstream projects are updated. As it turns out, this is a *lot* easier from mechanical point (it typically changes just a single line), but explodes in validation complexity, as you need to play out a 'what-if' scenario across all projects and then either propose or directly apply the version bump to affected projects. Obviously we can have a guard in place, which will not perform this action if the downstream (consumer) project's committers have gone back in time to hotfix a regression coming from downstream. These should generally be exceptions and should be tracked automatically in some sort of dashboard, as the presence of version skew is a blocking problem for a simultaneous release.
The third problem is the non-possibility of semantic versioning in current setup. This stems from the need for downstream project to have simple dependency specifications, e.g. we strive to use only a single per-project version, which can be easily bumped. In a project of non-trivial size, where there are multiple API components, this prevents them from being individually-versioned. If my understanding of maven documentation is correct, this can be alleviated by upstream projects publishing an "artifacts pom", which contains only a dependencyManagement section listing all of that projects published artifacts along with their version. A downstream project can use a dependency with scope "import" on that pom, inheriting those dependencyManagement declarations. With this mechanism in place, our current "one version per project" would be mapped onto the version of the artifacts pom (and can thus follow the Helium/Lithium/whatever versioning cycle) and all other artifacts coming from a particular project would be free to have their individual versions.
I think we need to deploy the solutions to 1) and 2) in an orchestrated manner, and they will provide the stepping stone towards 3). Ultimately this will drive the way we are doing releases -- I think it has all the ingredients we need to make "simultaneous release day" be just like any other day, without any drama at all.
As stated previously, this is by no means a complete proposal. Obvious friction surface is in 3) where upstream bumps the version, but downstream is not ready to adopt it -- but that can be handled by best practices, scripting smarts and policy -- for example the automated machinery in 2) will analyze (which is quite easy) the version changes and will not apply the bump, but rather submit it to downstream's gerrit for review.
The bottom line is: before we embark on wild tooling chases and build system rewrites, we have to be in the clear what we envision our workflows to be and what requirements we have. The devil is in the detail, and thus we should have an excellent understanding of any proposal's pros and cons before we embark on the journey, simply because it is a lot of work and has the potential to cause a lot of grief.
Bye,
Robert
toggle quoted message
Show quoted text
On 10/06/2014 02:41 PM, Jamie G. wrote: Semantic versioning is the best way to go. We use it successfully at
Apache Servicemix (and Karaf) to build very large containers . The
trick is to specify version ranges that are reasonable/generous (ex
import version range 2.0 up to but not including 3.0) in your
applications that rely on services from features.
Having a bundle version bump will necessitate a corresponding bump in
your feature version - regardless of the use of semantic versioning on
the bundle.
Is there something we can do to improve the features commands to make
using them easier for ODL's user base?
Cheers,
Jamie
On Mon, Oct 6, 2014 at 1:32 AM, Raghurama Bhat <raghu.odl@...> wrote:
Colin/Devin,
Surekha and I had made a proposal very similar to what you have described
back in March at the Napa ODL HackFest. ODL Root Parent project was created
as a result of that. Slide Deck from that presentation is at:
https://wiki.opendaylight.org/view/File:Versioning-Updated.ppt
How ever, the momentum was lost and not all projects implemented all the
suggested changes. Thank you for picking up the discussion again. To get
this implemented fully, I believe we will need to have an all hands on deck
approach similar to what was done for supporting Karaf Container and also
create automated triggers to identify and highlight any deviations.
While I understand the value of semantic versioning in general, IMHO
applying it at the bundle level when we have hundreds of bundle just makes
it very confusing for the consumers. Just look at the mishmash of versions
displayed when you run feature:list command in Karaf.
Thanks,
—Raghu
From: Colin Dixon <colin@...>
Date: Saturday, October 4, 2014 at 1:25 PM
To: Devin Avery <davery@...>
Cc: Vaishali Mithbaokar <vmithbao@...>,
"discuss@..." <discuss@...>, "Surekha
Bejgam (sbejgam)" <sbejgam@...>, "tsc@..."
<tsc@...>
Subject: Re: [OpenDaylight TSC] [OpenDaylight Discuss] Pom / Version
management & ODL Parent Project - Proposal
I've been advocating that we at least consider this approach for a while
now, but I'm don't have the maven knowledge to understand where it falls on
the spectrum of good idea vs. bad idea. I think the idea of making it work
for one project first and expanding is a really good one.
Some ideas (which I think Devin covers most of) have been aggregated in this
bug:
https://bugs.opendaylight.org/show_bug.cgi?id=1494
There are 3 other related bugs:
https://bugs.opendaylight.org/show_bug.cgi?id=1272 (creating odlparent and
moving stuff there)
https://bugs.opendaylight.org/show_bug.cgi?id=1599 (providing information
about the source code used for weekly builds, we might have solved this with
the taglist, but I'm not sure if that's in the weeklies as well as the RC
auto-release)
https://bugs.opendaylight.org/show_bug.cgi?id=1450 (document how to use the
weekly releases)
I'm also direcly cc'ing the two committers on the odlparent project so that
we can get their input.
--Colin
On Fri, Oct 3, 2014 at 4:00 PM, Devin Avery <davery@...> wrote:
Hello All -
During the “Inter-Project Infrastructure Services” meeting hosted by Colin
Dixon, Robert Varga and Vasu Srinivasan some problems and questions were
raised about the pom structure of our projects and how we might clean them
up and possible consolidate to using ODLparent etc. During that session I
offered up to start on the first part of the problem which was cleaning up
pom structures. During the second day I was very thankful to have 3 folks
from the lispflowmapping team join me and together we walked through their
pom structure clean up unnecessary code etc. You can see some of the
progress on this gerrit: https://git.opendaylight.org/gerrit/#/c/11704.
During our clean up of the poms we discussed how we might leverage
ODLParent and a daily or weekly build system to make it easier for projects
to “roll back” if something is introduced that broke things. So, here is a
rough outline of the proposal of best practices with regards to ODLParent in
hopes that we can ultimately get an easy way to “role back” code:
EVERY bundle that is brought in as a dependency in ANY PROJECT should be
listed in the dependency management version of the ODL Parent project (both
third party and ODL). We should avoid declaring ANY version in projects
dependency management or dependency section in any child project or bundle
unless absolutely necessary.
Advantage 1: We can identify if there are third party version conflicts
Advantage 2: Single point definitions allow for reuse and simplified child
poms.
Variables should be used for all dependency version (especially for ODL
project dependencies) so they can be overridden in child poms if needed (say
to roll back)
We need to make sure that we have a daily stable or weekly stable release
artifact getting created for every project, that have a UNIQUE version.
If a problem is introduced into a logical parent project, then developers
could temporarily override the version in their parent project until the
problem is resolved (hence the previous requirement for unique versions for
a weekly/daily stable build).
If we go this route, with a little work I think we could achieve:
a) greatly simplified pom files for all projects
b) the ability to quickly and easily role back version if there is a
problem
If we want to try and go this route, then one major requirement would be
that we make it much easier to contribute to the odl parent pom then it is
now. I say this because if we are adding every dependencies version
management to the odlparent pom then we can’t rely on just one or two
committers to push those changes. In order for this to really work I believe
we would want to make every committer a committer on the odlparent project.
That way, a committer who is reviewing a change to a child project’s POM
file could go review and push a change on the ODLparent pom file allowing
for a much faster turn around.
So, to summarize, here is the proposed approach to cleaning up poms and
optionally getting ready for the ability to easily roll back:
Encourage cleaner poms / structure by placing default configurations for
plugins into ODLparent for common plugins (Note: all deps and default
bundles would be commented to provide reasoning for their purpose)
Set up a model project (lispflowmapping) and change existing projects as
time goes.
Change the ODL parent project such that any committer on any ODL project
is also a committer on ODL parent
Work towards easy version “role-back” by placing ALL dependencies into
ODLParent with variables, and provide instructions on how developers can
role-back to weekly releases
May need to look at setting up stable release builds…
I would appreciate peoples thoughts and opinions on this proposal.
Ultimately I am going to bring it in front of the TSC to accept the change
of committers on the ODL parent project (and as such would appreciate
ODLParent committers +1 on this approach). I am happy to go into more
details if folks want as well and discussed this further. It is a first
pass.
Thank you!
Devin
_______________________________________________
Discuss mailing list
Discuss@...
https://lists.opendaylight.org/mailman/listinfo/discuss
_______________________________________________ TSC mailing list
TSC@...
https://lists.opendaylight.org/mailman/listinfo/tsc
_______________________________________________
Discuss mailing list
Discuss@...
https://lists.opendaylight.org/mailman/listinfo/discuss
_______________________________________________
TSC mailing list
TSC@...
https://lists.opendaylight.org/mailman/listinfo/tsc
|
