1. Topics
  2. [controller-dev] [yangtools-dev] [releng] Distribution creation job

[controller-dev] [yangtools-dev] [releng] Distribution creation job

Vratko Polak -X (vrpolak - Pantheon Technologies SRO@Cisco) <vrpolak@...>
 

Hi Luis.

 

As your TWS talk is nearing, I would like to present my views.

 

I think ${project}-integration job does basically the same thing

as ${project}-merge, only the triggers (and notifications) are different.
Build from source in ${project} git, run unit tests, upload artifacts.

*-merge is triggered by gerrit; *-integration should be triggered

by ${dependency}-merge and ${dependency}-integration jobs.

Maybe we should merge them to a single ${project}-publish job if possible.

 

> Is there going to be a centralized 'this is what integration looks like' job?

 

Yes, in integration project, but it would generally be too slow

and prone to failures.

That is why there will be ${project}-distribution jobs,

which pack only subtree of features relevant to that project.

They will be triggered by ${project}-publish,

and they will be triggering ${project}-scit jobs with end-to-end tests.

The same tests will be run also on the full integration build,

but once again that job would be slower.

 

> So I merge stuff to yangtools

 

From point of view of BGPCEP project, the following would happen.

1. yangtools-merge triggers controller-integration

(and nothing else yet if Jenkins is good at dependency handling).

2. controller-integration triggers tcpmd5-integration (and unrelated jobs)

3. tcpmd5-integration finally triggers bgpcep-integration

4. bgpcep-integration triggers bgpcep-distribution

5. bgpcep-distribution triggers bgpcep-csit jobs (several configurations).

6. bgpcep-csit-3nodes fails miserably, sends e-mails

to every upstream project. Probably including odlparent,

as Jenkins is not that good at back-tracing triggers.

…

96. Ultimately, integration-integration is triggered.
97. integration-integration triggers integration-distribution.

98. integration-distribution triggers integration-csit jobs.

99. After few hours all suites are finished (no idea who will be notified).

 

Question to Luis:
Is Jenkins that good at dependency handling,

or would bgpcep-integration be triggered three times

(yangtools-merge, controller-integration and tcpmd5-integration)?
If no, that motivates the strategy of triggering from *-merge jobs only,

and that would create the artifact race issue Robert was talking about.

But we will have race issues anyway, as yangtools-merge

will not wait patiently for integration-distribution to finish

before merging the next yangtools change.


Vratko.

 

From: controller-dev-bounces@... [mailto:controller-dev-bounces@...] On Behalf Of Robert Varga
Sent: Wednesday, February 25, 2015 10:00 AM
To: Luis Gomez; yangtools-dev; opendaylight; openflowplugin-dev; openflowjava-dev; aaa-dev@...; vtn-dev@...; ovsdb-dev; l2switch-dev
Cc: dev (dev@...); 'integration-dev@...' (integration-dev@...) (integration-dev@...) (integration-dev@...)
Subject: Re: [controller-dev] [yangtools-dev] [releng] Distribution creation job

 

Hello,

I am not sure I follow what exactly is the plan here. Is there going to be a centralized 'this is what integration looks like' job? Furthermore how does this interact with the ${project}-integration jobs, which also publish artifacts, but are not configured to trigger according to the dependency chain?

So I merge stuff to yangtools, which triggers a merge job, which will complete. Now two jobs will trigger: the controller integration job and this distribution job ... controller integration job publishes artifacts, so ... which artifacts is the distribution job going to pick up?

Bye,
Robert

On 02/25/2015 03:20 AM, Luis Gomez wrote:

Hi all,

 

I am pushing following patches to releng repo:

 

 

This patch will create a job that downloads and builds the integration repository whenever there is a merge in the above projects. You can check the job template here:

 

 

You will need this job for 3 reasons:

 

1) To verify your changes are compatible with the integration code

2) To make sure your changes are immediately included in the integration distribution

3) To run some system test after the distribution is built

 

Please let me know if you have questions or concerns about merging this.

 

PS - these jobs will replace the old integration-polling jobs + integration-centralized-integration job

 

BR/Luis




_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev

 
Luis Gomez
 

Hi Vrakto,

Thanks for your mail, see my answers in-line.


On Feb 26, 2015, at 9:46 AM, Vratko Polak -X (vrpolak - Pantheon Technologies SRO at Cisco) <vrpolak@...> wrote:

Hi Luis.
 
As your TWS talk is nearing, I would like to present my views.
 
I think ${project}-integration job does basically the same thing
as ${project}-merge, only the triggers (and notifications) are different.
Build from source in ${project} git, run unit tests, upload artifacts.
*-merge is triggered by gerrit; *-integration should be triggered
by ${dependency}-merge and ${dependency}-integration jobs.
Maybe we should merge them to a single ${project}-publish job if possible.

Hi Vrakto, these jobs are releng responsibility (not integration), so the question is for Andy and Thahn :)

 
> Is there going to be a centralized 'this is what integration looks like' job?
 
Yes, in integration project, but it would generally be too slow
and prone to failures.
That is why there will be ${project}-distribution jobs,
which pack only subtree of features relevant to that project.
They will be triggered by ${project}-publish,
and they will be triggering ${project}-scit jobs with end-to-end tests.
The same tests will be run also on the full integration build,
but once again that job would be slower.
 
> So I merge stuff to yangtools
 
From point of view of BGPCEP project, the following would happen.
1. yangtools-merge triggers controller-integration

and ideally a yangtools-distribution job which builds integration karaf distribution

(and nothing else yet if Jenkins is good at dependency handling).
2. controller-integration triggers tcpmd5-integration (and unrelated jobs)

I do not think integration job calls another integration job, only a merge calls an integration

3. tcpmd5-integration finally triggers bgpcep-integration

bgpcep-integration should trigger directly from yangtools merge if you have declared the dependency

4. bgpcep-integration triggers bgpcep-distribution

This is not correct, bgpcep-distribution is only triggered on own merges.

5. bgpcep-distribution triggers bgpcep-csit jobs (several configurations).

bgpcep-csit jobs will be triggered after bgpcep-distribution yes but also after any <project>-distribution in the upstream like yangtools-distribution, this is why it is very important for projects in upstream to define their distribution job

6. bgpcep-csit-3nodes fails miserably, sends e-mails
to every upstream project. Probably including odlparent,
as Jenkins is not that good at back-tracing triggers.

So far I think it is only the bgpcep project that gets notified and the reason is I think it is better the project checks first why the system test is failing before it reports the issue to the upstream project, but we can change this if people think different.

…
96. Ultimately, integration-integration is triggered.
97. integration-integration triggers integration-distribution.
98. integration-distribution triggers integration-csit jobs.
99. After few hours all suites are finished (no idea who will be notified).

The projects will be the owners of the ODL system test so no plan to trigger anything in integration this release

 
Question to Luis:
Is Jenkins that good at dependency handling,
or would bgpcep-integration be triggered three times
(yangtools-merge, controller-integration and tcpmd5-integration)?

Should be only after merge

If no, that motivates the strategy of triggering from *-merge jobs only,
and that would create the artifact race issue Robert was talking about.
But we will have race issues anyway, as yangtools-merge
will not wait patiently for integration-distribution to finish
before merging the next yangtools change.

Although it is not the case here, according to Andy there is no issues in having 2 jobs publishing at the same time in Nexus as this last uses timestamps at millisecond precision


Vratko.
 
From: controller-dev-bounces@... [mailto:controller-dev-bounces@...]On Behalf Of Robert Varga
Sent: Wednesday, February 25, 2015 10:00 AM
To: Luis Gomez; yangtools-dev; opendaylight; openflowplugin-dev; openflowjava-dev; aaa-dev@...; vtn-dev@...; ovsdb-dev; l2switch-dev
Cc: dev (dev@...); 'integration-dev@...' (integration-dev@...) (integration-dev@...) (integration-dev@...)
Subject: Re: [controller-dev] [yangtools-dev] [releng] Distribution creation job
 
Hello,

I am not sure I follow what exactly is the plan here. Is there going to be a centralized 'this is what integration looks like' job? Furthermore how does this interact with the ${project}-integration jobs, which also publish artifacts, but are not configured to trigger according to the dependency chain?

So I merge stuff to yangtools, which triggers a merge job, which will complete. Now two jobs will trigger: the controller integration job and this distribution job ... controller integration job publishes artifacts, so ... which artifacts is the distribution job going to pick up?

Bye,
Robert

On 02/25/2015 03:20 AM, Luis Gomez wrote:
Hi all,
 
I am pushing following patches to releng repo:
 
 
This patch will create a job that downloads and builds the integration repository whenever there is a merge in the above projects. You can check the job template here:
 
 
You will need this job for 3 reasons:
 
1) To verify your changes are compatible with the integration code
2) To make sure your changes are immediately included in the integration distribution
3) To run some system test after the distribution is built
 
Please let me know if you have questions or concerns about merging this.
 
PS - these jobs will replace the old integration-polling jobs + integration-centralized-integration job
 
BR/Luis



_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev
 
Robert Varga
 

Hello Luis,

On 02/26/2015 07:28 PM, Luis Gomez wrote:

(and nothing else yet if Jenkins is good at dependency handling).
2. controller-integration triggers tcpmd5-integration (and unrelated jobs)

I do not think integration job calls another integration job, only a merge calls an integration

... and this is wrong, as shown by the way the jobs behaved when we changed guava version. I ended up re-merging the last odlparent change about 4 times just to trigger integration jobs, which needed to pick up things published by other integration jobs running in parallel.

Since integration jobs publish artifacts, they introduce changes, which absolutely be acted upon by downstream projects. Even when the source code has not changed, the resulting binary artifacts have, which specify things like method signatures, imports, etc. etc. all of which are a cause for downstream rebuilds.

Bye,
Robert
1 - 3 of 3