Inline.

On Wed, Oct 15, 2014 at 12:08 PM, Robert Varga <nite@...> wrote:

Hello,

On 10/14/2014 11:54 PM, Colin Dixon wrote:

I agree with most of what's said above, but I also feel like it misses much of the original intent from what Devin said which I think are motherhood and apple pie statements about pom files.

1.) Unless there's a good reason, your parent pom should be the pom.xml in the parent directory. (And your root pom's parent should be odlparent.)

There are plenty of good reasons to not do this. All of them stem from the fact that the root pom is where your build project-wide build starts, whereas a pom's parent specifies the equivalent of an abstract class (or a template) for a particular type of artifact. One area where you will notice the difference is as soon as you start performing project-wide processing, like javadoc:aggregate. You want to perform these only when the build is triggered in the top-level directory, but not in any of the subdirectories. If you make your artifacts children of your root, you will spend battling maven until you realize that you need to restructure (which is major pain).

If we want to make it easy for projects to perform the various tasks, we actually need to start providing pre-cooked parent poms, which acts as templates. This includes things like features-parent (e.g. how do I generate features), it-parent (how do I run integration tests), mdsal-parent (e.g. how do I generate model bindings), etc. These will lower the amount of copy&paste going on, hopefully making these more end-user friendly and actually maintainable.

If it turns out that the "unless there's a good reason" is nearly always triggered because of the need to generate javadoc and things like that, then we just need to provide a good canonical structure for projects to follow and show them how to start with that (for new projects), migrate to it (for current projects), and maintain it.

Does maven offer a way to have something in a pom file that does run there, but is explicitly not run when that pom file is included via a parent relationship? If so, that would solve some of our problem by reducing them number of exceptions to the rule that your directory parent should be your pom parent.

If we could do that, I think we could then make it so that all pom files had their parent as either one of the fixed number of templates or their directory parent. The fixed number of templates could have their parent be the logical and directory-based root pom because they wouldn't have to worry about bulling in any root build behavior.

Again, this is all predicated on functionality that I'm not sure maven has. We could ask for the functionality though.

 

2.) Unless there's a good reason, your artifact ID should be the same as the directory containing the pom file.

Yes.

3.) Unless there's a good reason, all versions should be specified by a combination of the odlparent (for third-party dependencies) and our root pom file (for internal dependencies).

Yes, except it's should not be "root pom", but "a centralized place". That centralized place can be the project-specific parent for now, but as we start shipping the pre-cooked parents, that advice will shift, simply because you will suddenly find yourself needing to import that dependencyManagement section into a pom, which has a pre-cooked parent (since maven, like Java, has single inheritance).

See comments above.
 

These are all orthogonal to allowing version skew. I think, however, these give us some really, really nice things.

1.) They make it slightly harder to do bad things and a lot easier to do the right thing, e.g., it's pretty easy to leave <version>blah</version> out of pom files.

2.) They make it easier to for people to interact with other project's code bases. That is, I can get in, debug a problem and maybe even push a fix more of the time.

3.) They make it easier for release engineers and good Samaritans to catch bugs and problems in pom files.

Not disagreeing on the merits, just want to make sure we do not paint ourselves into a corner.

Thanks,
Robert

--Colin

On Wed, Oct 8, 2014 at 1:58 AM, Robert Varga <nite@...> wrote:

(Long rant warning. Proposal is by no means complete, but attempts to propose general mechanics and highlights challenges which are intertwined.)

Unfortunately, we have a larger inter-project problem in the OpenDaylight ecosystem, where managing the inter-project version skew is actually the major pain point. We essentially are integrated via snapshot revisions, which are a moving target and do not allow downstream projects to control what is they are getting. In order to get to something we can manage manually, the best practice is to maintain a single version per project, which runs contrary to semver and does not really solve anything.

I think the path forward lies in an automated, continuous release process, supported by proper project version exports. This is a combination of something we already have, namely the autobuild (which just needs to be refactored), something that we need to develop (a dashboard for version skew) and something we need to add to our projects (maven dependency poms).

The first problem we are facing is that we are integrating multiple projects across SNAPSHOT revisions. This makes it impossible to gate downstream projects from changes happening upstream, nor 'go back in time'. We clearly need to do inter-project integration on "release" artifacts. Manual (or SCM-based) version bumps are not feasible, simply because of the churn introduced to pom.xmls in the upstream project. We can easily fix this by having each merge build perform the equivalent of what autorelease does for all projects: replace -SNAPSHOT with a time-based snapshot -- 0.6.2-SNAPSHOT becomes 0.6.2-20141007-062302. The format is significant, as it does not contain anything which can be mistaken lexicographically, thus artifacts form a monotonic sequence no matter what (well, unless NTP decides otherwise). In order to mark the revision from which a particular set of artifacts was created we'd use a git tag. The HEAD version (and SCM coordinates) would be carried inside the jar (in META-INF), not in the filename.

The second is the version skew problem. Once we have stable versions, we need a simple way of how downstream projects are updated. As it turns out, this is a *lot* easier from mechanical point (it typically changes just a single line), but explodes in validation complexity, as you need to play out a 'what-if' scenario across all projects and then either propose or directly apply the version bump to affected projects. Obviously we can have a guard in place, which will not perform this action if the downstream (consumer) project's committers have gone back in time to hotfix a regression coming from downstream. These should generally be exceptions and should be tracked automatically in some sort of dashboard, as the presence of version skew is a blocking problem for a simultaneous release.

The third problem is the non-possibility of semantic versioning in current setup. This stems from the need for downstream project to have simple dependency specifications, e.g. we strive to use only a single per-project version, which can be easily bumped. In a project of non-trivial size, where there are multiple API components, this prevents them from being individually-versioned. If my understanding of maven documentation is correct, this can be alleviated by upstream projects publishing an "artifacts pom", which contains only a dependencyManagement section listing all of that projects published artifacts along with their version. A downstream project can use a dependency with scope "import" on that pom, inheriting those dependencyManagement declarations. With this mechanism in place, our current "one version per project" would be mapped onto the version of the artifacts pom (and can thus follow the Helium/Lithium/whatever versioning cycle) and all other artifacts coming from a particular project would be free to have their individual versions.

I think we need to deploy the solutions to 1) and 2) in an orchestrated manner, and they will provide the stepping stone towards 3). Ultimately this will drive the way we are doing releases -- I think it has all the ingredients we need to make "simultaneous release day" be just like any other day, without any drama at all.

As stated previously, this is by no means a complete proposal. Obvious friction surface is in 3) where upstream bumps the version, but downstream is not ready to adopt it -- but that can be handled by best practices, scripting smarts and policy -- for example the automated machinery in 2) will analyze (which is quite easy) the version changes and will not apply the bump, but rather submit it to downstream's gerrit for review.

The bottom line is: before we embark on wild tooling chases and build system rewrites, we have to be in the clear what we envision our workflows to be and what requirements we have. The devil is in the detail, and thus we should have an excellent understanding of any proposal's pros and cons before we embark on the journey, simply because it is a lot of work and has the potential to cause a lot of grief.

Bye,
Robert