So, as a nascent project, I think there is a balance point in there between eliminating the technical debt, and maintaining forward momentum for the project and providing a place for folks to come and do cool new things.

The compromise was to use the Stable Release to clean up Hydrogen, then use Helium as the vehicle for new projects. Granted, the process has not been ideal, but I would chalk that up to project growing pains. The dialog on these lists would seem to indicate we have a better handle on how to structure things moving forward.

O

Sent from my iPad

On May 6, 2014, at 1:30 PM, "Moiz Raja (moraja)" <moraja@...> wrote:

I agree that we should not incur further technical debt. I don't want a single new feature to be added to Helium. We should fix what is broken and make Helium for the lack of a better word a "stable" release. All I am questioning is the need for producing a Hydrogen stable release.

Next time around let's NOT release till we are stable. That is the best way to avoid the problems we are in.

-Moiz
________________________________________
From: Omar Sultan (osultan)
Sent: Tuesday, May 06, 2014 12:41 PM
To: Moiz Raja (moraja)
Cc: Ed Warnicke (eaw); Chris Wright; <discuss@...>; tsc@...
Subject: Re: [OpenDaylight Discuss] [OpenDaylight TSC] Hydrogen Stable Release Proposal

So, I think the road to software hell is paved with such intentions, that we will fix things in the next release. My experience is the shiny new stuff gets more attention then fixing and documenting the old stuff.

If we have accumulated some technical debt, it would prefer we clear it and have something clean and stable that folks can use for a basis for Helium. Otherwise, it seems to me like we are starting work on the second story of house when the walls are still wobbly on the first floor.

O

Sent from my iPad

On May 6, 2014, at 12:21 PM, "Moiz Raja (moraja)" <moraja@...> wrote:

An honest question. Why do we need to do a stable release for Hydrogen? Wouldn't it be just easier to get to Helium and thereafter make sure that we start doing stable releases. It seems like we're using more energy in creating the stable release than what it's worth. AFAIK Hydrogen itself was a developer-only release (or atleast that's what it turned out to be) so we do not have any customers waiting for our fixes.

my 20 cents,
-Moiz


________________________________________
From: discuss-bounces@... [discuss-bounces@...] on behalf of Ed Warnicke (eaw)
Sent: Tuesday, May 06, 2014 10:22 AM
To: Chris Wright
Cc: <discuss@...>; tsc@...
Subject: Re: [OpenDaylight Discuss] [OpenDaylight TSC] Hydrogen Stable Release Proposal

On May 5, 2014, at 10:55 PM, Chris Wright <chrisw@...> wrote:

* Ed Warnicke (eaw) (eaw@...) wrote:

Trying to replay the sequences of patches, across all of our (12 I believe) projects with various
interdependencies between them has proven to be a herculean task, particularly in light of global
timezones and Jenkins verifies/merges. Lots of projects are getting wedged on other projects
as they attempt to cherry pick down historical patches to a relatively new stable branch.

Please be more specific. What issues, which projects, where are the
bugs...details…

Sure… controller has been blocking almost daily on some issues or other related to
yangtools (either it played forward to fast, or it needs to catch up).

I’ve talked to the openflowplugin guys who are having blocking issues with synchronization
with controller.

Add timezone delays and Jenkins verify and merge delays… and everything slows to a crawl.

Mostly folks have been busy trying to solve these issues as they arise… making progress not reporting it...

It appears to me that the root issue is attempting to replay the past.
To resolve this I would suggest that we adopt the following tact:

1) Projects (if they so desire) pull their stable branches from a current point in time (today,or
if it takes us a while to discuss this a little later this week).
2) Projects update their versions to append the -1-SNAPSHOT (we’ve done this already in our first attempt
at this).
3) Projects revert the patches inappropriate for a stable branch (I think most projects have done the triage on this,
so its just a bit of doing, and since most projects have put almost all of their effort on master into bug fixing, is a much
smaller number of things).
4) Going forward we follow Chris’s recommendation to commit *new* bug fixes first to master and then (if appropriate)
to stable (such double committing should be done contemporaneously so we don’t fall back into needing to replay the past).

This should allow us to get unwedged on the ongoing ‘replay the past’ problem and get down to testing and hardening
the stable release.

Thoughts?

I think you risk trading one problem for another and in a few weeks back
to where we are now.

I don’t think so… the fundamental problem is replaying the past, and the fact that we are
playing back a large number of bug fixes from the past to avoid a small number of non-bug fixes. And that introduces (unsurprisingly
when you think about it) a large number of ordering issues. At current rate of process, I suspect we would take
months of stop start stop start stop start trying to replay the past… with this approach I suspect we could
pull today, and revert the few patches that need reverting over the course of a week or so and get onto focusing
on hardening the release.

Think of it this way… we have a time ordered stream of patches, some of which depend on each other.
Those time ordered streams get distributed across our 12 projects in Hydrogen. This goes on for hundreds of patches,
all but tens of them bug fixes appropriate for stable.

Then we pull the stable branch and try to put humpty dumpty back together again. We still have hundreds of patches
to replay, in the correct dependency order, across 12 projects… and tens to exclude. Some projects get ahead, and thus
other projects can’t verify. Some projects are behind, and thus other projects must then coordinate and block waiting for them
to catch up.

When you look at what is actually implied by the current approach (the full problems of which were not immediately obvious to me until we
started hitting them)… I’m actually amazed we aren’t more blocked than we are.

The proposal is quite simple… come to a state we know is currently working (now), and revert the tens of patches that are not appropriate
to stable and proceed with the double commit strategy for things that need to go into stable going forward. Much smaller surface area to create difficulty.
Much fewer issues. From down here in the trenches its a much smaller chance of hitting these kinds of issues again.

Its basically putting us on the footing we would have been if we adopted the original plan *at the time of Hydrogen Release* instead of months later
when we had hundreds of patches worth of past to figure out how to replay in the right order.

To date there are exactly two stated issues:

1) bgpcep relying on config-manager change in controller as captured in Bug 903.
2) ovsdb relying on some unspecified change in controller

There have been way more than this… but as mentioned above, folks have been focused
on solving those problems rather than reporting them. Rather than filing bugs and reporting status
folks have been working on solving the problems and getting unblocked.

This is not "lots" it's only two. So either there are "lots" in which
case we need real details (and we have a very fundamental issue that
projects are not speaking up), or it's only two in which case we may
be able to work through those two issues.

Again, when you are closer to the code, and have your hands deeper in getting folks unblocked… you see
*lots*.

Ed


_______________________________________________
Discuss mailing list
Discuss@...
https://lists.opendaylight.org/mailman/listinfo/discuss
_______________________________________________
Discuss mailing list
Discuss@...
https://lists.opendaylight.org/mailman/listinfo/discuss

_______________________________________________
Discuss mailing list
Discuss@...
https://lists.opendaylight.org/mailman/listinfo/discuss