[yangtools-dev] [releng] Distribution creation job


Robert Varga <nite@...>
 

Hello,

I am not sure I follow what exactly is the plan here. Is there going to be a centralized 'this is what integration looks like' job? Furthermore how does this interact with the ${project}-integration jobs, which also publish artifacts, but are not configured to trigger according to the dependency chain?

So I merge stuff to yangtools, which triggers a merge job, which will complete. Now two jobs will trigger: the controller integration job and this distribution job ... controller integration job publishes artifacts, so ... which artifacts is the distribution job going to pick up?

Bye,
Robert

On 02/25/2015 03:20 AM, Luis Gomez wrote:

Hi all,

I am pushing following patches to releng repo:


This patch will create a job that downloads and builds the integration repository whenever there is a merge in the above projects. You can check the job template here:


You will need this job for 3 reasons:

1) To verify your changes are compatible with the integration code
2) To make sure your changes are immediately included in the integration distribution
3) To run some system test after the distribution is built

Please let me know if you have questions or concerns about merging this.

PS - these jobs will replace the old integration-polling jobs + integration-centralized-integration job

BR/Luis


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev


Luis Gomez <ecelgp@...>
 

For some reason I neglected this mail, see my answers in-line:


On Feb 25, 2015, at 1:00 AM, Robert Varga <nite@...> wrote:

Hello,

I am not sure I follow what exactly is the plan here. Is there going to be a centralized 'this is what integration looks like' job? Furthermore how does this interact with the ${project}-integration jobs, which also publish artifacts, but are not configured to trigger according to the dependency chain?

${project}-integration builds the project code on upstream merges, ${project}-distribution builds the integration project on own project merges so they are very different

So I merge stuff to yangtools, which triggers a merge job, which will complete. Now two jobs will trigger: the controller integration job and this distribution job ... controller integration job publishes artifacts, so ... which artifacts is the distribution job going to pick up?

integration artifacts = karaf distribution


Bye,
Robert

On 02/25/2015 03:20 AM, Luis Gomez wrote:
Hi all,

I am pushing following patches to releng repo:


This patch will create a job that downloads and builds the integration repository whenever there is a merge in the above projects. You can check the job template here:


You will need this job for 3 reasons:

1) To verify your changes are compatible with the integration code
2) To make sure your changes are immediately included in the integration distribution
3) To run some system test after the distribution is built

Please let me know if you have questions or concerns about merging this.

PS - these jobs will replace the old integration-polling jobs + integration-centralized-integration job

BR/Luis


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev



Robert Varga <nite@...>
 

So the problem I see is the following:

1) a controller committer merges a patch
2) controller-merge-master publishes artifacts
3a) controller-distribution triggers
3b) openflowjava-integration-master triggers

Now we have two jobs running (3a, 3b) and based on the layout of job queue, slave spinup, etc. etc., three things can happen:
- 3a may pick up artifacts from 3b, or
- 3a picks up artifacts before 3b, or
- 3a picks up some artifacts before 3b and some after that

The same thing is going to happen if I quickly merge a bunch of patches...

Bye,
Robert

On 02/26/2015 06:56 PM, Luis Gomez wrote:

For some reason I neglected this mail, see my answers in-line:


On Feb 25, 2015, at 1:00 AM, Robert Varga <nite@...> wrote:

Hello,

I am not sure I follow what exactly is the plan here. Is there going to be a centralized 'this is what integration looks like' job? Furthermore how does this interact with the ${project}-integration jobs, which also publish artifacts, but are not configured to trigger according to the dependency chain?

${project}-integration builds the project code on upstream merges, ${project}-distribution builds the integration project on own project merges so they are very different

So I merge stuff to yangtools, which triggers a merge job, which will complete. Now two jobs will trigger: the controller integration job and this distribution job ... controller integration job publishes artifacts, so ... which artifacts is the distribution job going to pick up?

integration artifacts = karaf distribution


Bye,
Robert

On 02/25/2015 03:20 AM, Luis Gomez wrote:
Hi all,

I am pushing following patches to releng repo:


This patch will create a job that downloads and builds the integration repository whenever there is a merge in the above projects. You can check the job template here:


You will need this job for 3 reasons:

1) To verify your changes are compatible with the integration code
2) To make sure your changes are immediately included in the integration distribution
3) To run some system test after the distribution is built

Please let me know if you have questions or concerns about merging this.

PS - these jobs will replace the old integration-polling jobs + integration-centralized-integration job

BR/Luis


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev




Luis Gomez <ecelgp@...>
 

Hi Robert, see my comments in-line:

On Feb 26, 2015, at 10:33 AM, Robert Varga <nite@...> wrote:

So the problem I see is the following:

1) a controller committer merges a patch
2) controller-merge-master publishes artifacts
3a) controller-distribution triggers
3b) openflowjava-integration-master triggers

Now we have two jobs running (3a, 3b) and based on the layout of job queue, slave spinup, etc. etc., three things can happen:
- 3a may pick up artifacts from 3b, or
- 3a picks up artifacts before 3b, or
- 3a picks up some artifacts before 3b and some after that

The only requirement for 3a) is to pick new controller artifacts, whether or not we use old or new artifacts from openflowjava is not that important IMO if we consider the integration job will not modify much the existing project artifacts (i.e. no project code change) unless there is a change in a 3rd party version in odlparent, I think this last was the reason we added the publish step in the integration job.


The same thing is going to happen if I quickly merge a bunch of patches...

Bye,
Robert

On 02/26/2015 06:56 PM, Luis Gomez wrote:
For some reason I neglected this mail, see my answers in-line:


On Feb 25, 2015, at 1:00 AM, Robert Varga <nite@...> wrote:

Hello,

I am not sure I follow what exactly is the plan here. Is there going to be a centralized 'this is what integration looks like' job? Furthermore how does this interact with the ${project}-integration jobs, which also publish artifacts, but are not configured to trigger according to the dependency chain?

${project}-integration builds the project code on upstream merges, ${project}-distribution builds the integration project on own project merges so they are very different

So I merge stuff to yangtools, which triggers a merge job, which will complete. Now two jobs will trigger: the controller integration job and this distribution job ... controller integration job publishes artifacts, so ... which artifacts is the distribution job going to pick up?

integration artifacts = karaf distribution


Bye,
Robert

On 02/25/2015 03:20 AM, Luis Gomez wrote:
Hi all,

I am pushing following patches to releng repo:


This patch will create a job that downloads and builds the integration repository whenever there is a merge in the above projects. You can check the job template here:


You will need this job for 3 reasons:

1) To verify your changes are compatible with the integration code
2) To make sure your changes are immediately included in the integration distribution
3) To run some system test after the distribution is built

Please let me know if you have questions or concerns about merging this.

PS - these jobs will replace the old integration-polling jobs + integration-centralized-integration job

BR/Luis


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev





Robert Varga <nite@...>
 

On 02/26/2015 07:53 PM, Luis Gomez wrote:
Hi Robert, see my comments in-line:

On Feb 26, 2015, at 10:33 AM, Robert Varga <nite@...> wrote:

So the problem I see is the following:

1) a controller committer merges a patch
2) controller-merge-master publishes artifacts
3a) controller-distribution triggers
3b) openflowjava-integration-master triggers

Now we have two jobs running (3a, 3b) and based on the layout of job queue, slave spinup, etc. etc., three things can happen:
- 3a may pick up artifacts from 3b, or
- 3a picks up artifacts before 3b, or
- 3a picks up some artifacts before 3b and some after that

The only requirement for 3a) is to pick new controller artifacts, whether or not we use old or new artifacts from openflowjava is not that important IMO if we consider the integration job will not modify much the existing project artifacts (i.e. no project code change) unless there is a change in a 3rd party version in odlparent, I think this last was the reason we added the publish step in the integration job.


Yeah, but the point is that anything that uses that distribution may end up with an inconsistent image, so you may end up with various false positives. And those will be transient and nigh-impossible to track down.

Bye,
Robert


The same thing is going to happen if I quickly merge a bunch of patches...

Bye,
Robert

On 02/26/2015 06:56 PM, Luis Gomez wrote:
For some reason I neglected this mail, see my answers in-line:


On Feb 25, 2015, at 1:00 AM, Robert Varga <nite@...> wrote:

Hello,

I am not sure I follow what exactly is the plan here. Is there going to be a centralized 'this is what integration looks like' job? Furthermore how does this interact with the ${project}-integration jobs, which also publish artifacts, but are not configured to trigger according to the dependency chain?

${project}-integration builds the project code on upstream merges, ${project}-distribution builds the integration project on own project merges so they are very different

So I merge stuff to yangtools, which triggers a merge job, which will complete. Now two jobs will trigger: the controller integration job and this distribution job ... controller integration job publishes artifacts, so ... which artifacts is the distribution job going to pick up?

integration artifacts = karaf distribution


Bye,
Robert

On 02/25/2015 03:20 AM, Luis Gomez wrote:
Hi all,

I am pushing following patches to releng repo:


This patch will create a job that downloads and builds the integration repository whenever there is a merge in the above projects. You can check the job template here:


You will need this job for 3 reasons:

1) To verify your changes are compatible with the integration code
2) To make sure your changes are immediately included in the integration distribution
3) To run some system test after the distribution is built

Please let me know if you have questions or concerns about merging this.

PS - these jobs will replace the old integration-polling jobs + integration-centralized-integration job

BR/Luis


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev






Luis Gomez <ecelgp@...>
 

I am not sure I follow here, can you explain more:

1) false positives, from where?

2) high-impossible to track down, the only artifact that really changes is the controller artifact, is that so difficult to figure out?

BR/Luis


On Feb 26, 2015, at 11:03 AM, Robert Varga <nite@...> wrote:

On 02/26/2015 07:53 PM, Luis Gomez wrote:
Hi Robert, see my comments in-line:

On Feb 26, 2015, at 10:33 AM, Robert Varga <nite@...> wrote:

So the problem I see is the following:

1) a controller committer merges a patch
2) controller-merge-master publishes artifacts
3a) controller-distribution triggers
3b) openflowjava-integration-master triggers

Now we have two jobs running (3a, 3b) and based on the layout of job queue, slave spinup, etc. etc., three things can happen:
- 3a may pick up artifacts from 3b, or
- 3a picks up artifacts before 3b, or
- 3a picks up some artifacts before 3b and some after that

The only requirement for 3a) is to pick new controller artifacts, whether or not we use old or new artifacts from openflowjava is not that important IMO if we consider the integration job will not modify much the existing project artifacts (i.e. no project code change) unless there is a change in a 3rd party version in odlparent, I think this last was the reason we added the publish step in the integration job.


Yeah, but the point is that anything that uses that distribution may end up with an inconsistent image, so you may end up with various false positives. And those will be transient and nigh-impossible to track down.

Bye,
Robert


The same thing is going to happen if I quickly merge a bunch of patches...

Bye,
Robert

On 02/26/2015 06:56 PM, Luis Gomez wrote:
For some reason I neglected this mail, see my answers in-line:


On Feb 25, 2015, at 1:00 AM, Robert Varga <nite@...> wrote:

Hello,

I am not sure I follow what exactly is the plan here. Is there going to be a centralized 'this is what integration looks like' job? Furthermore how does this interact with the ${project}-integration jobs, which also publish artifacts, but are not configured to trigger according to the dependency chain?

${project}-integration builds the project code on upstream merges, ${project}-distribution builds the integration project on own project merges so they are very different

So I merge stuff to yangtools, which triggers a merge job, which will complete. Now two jobs will trigger: the controller integration job and this distribution job ... controller integration job publishes artifacts, so ... which artifacts is the distribution job going to pick up?

integration artifacts = karaf distribution


Bye,
Robert

On 02/25/2015 03:20 AM, Luis Gomez wrote:
Hi all,

I am pushing following patches to releng repo:


This patch will create a job that downloads and builds the integration repository whenever there is a merge in the above projects. You can check the job template here:


You will need this job for 3 reasons:

1) To verify your changes are compatible with the integration code
2) To make sure your changes are immediately included in the integration distribution
3) To run some system test after the distribution is built

Please let me know if you have questions or concerns about merging this.

PS - these jobs will replace the old integration-polling jobs + integration-centralized-integration job

BR/Luis


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev







Robert Varga <nite@...>
 

Note that I am not questioning the intent, but am worried about the results as-currently-implemented. It is very important to understand what artifacts will get into that distribution, since if that image will undergo CSIT to see whether a controller change did not break OF, that OF piece needs to be self-consistent.

Jenkins are just like threads, with Nexus being memory. What I am asking is: what are the locks the reader (controller-distribution job) holds to prevent writers (all the merge and integration jobs) from modifying state (artifacts in Nexus) as it is being read to assemble the distribution?

If we have no such assurances, what assurance do we have in the reliability and repeatability departments? When things break, how do we track which source code corresponds to that particular controller-build?

Thanks,
Robert

On 02/26/2015 08:09 PM, Luis Gomez wrote:

I am not sure I follow here, can you explain more:

1) false positives, from where?

2) high-impossible to track down, the only artifact that really changes is the controller artifact, is that so difficult to figure out?

BR/Luis


On Feb 26, 2015, at 11:03 AM, Robert Varga <nite@...> wrote:

On 02/26/2015 07:53 PM, Luis Gomez wrote:
Hi Robert, see my comments in-line:

On Feb 26, 2015, at 10:33 AM, Robert Varga <nite@...> wrote:

So the problem I see is the following:

1) a controller committer merges a patch
2) controller-merge-master publishes artifacts
3a) controller-distribution triggers
3b) openflowjava-integration-master triggers

Now we have two jobs running (3a, 3b) and based on the layout of job queue, slave spinup, etc. etc., three things can happen:
- 3a may pick up artifacts from 3b, or
- 3a picks up artifacts before 3b, or
- 3a picks up some artifacts before 3b and some after that

The only requirement for 3a) is to pick new controller artifacts, whether or not we use old or new artifacts from openflowjava is not that important IMO if we consider the integration job will not modify much the existing project artifacts (i.e. no project code change) unless there is a change in a 3rd party version in odlparent, I think this last was the reason we added the publish step in the integration job.


Yeah, but the point is that anything that uses that distribution may end up with an inconsistent image, so you may end up with various false positives. And those will be transient and nigh-impossible to track down.

Bye,
Robert


The same thing is going to happen if I quickly merge a bunch of patches...

Bye,
Robert

On 02/26/2015 06:56 PM, Luis Gomez wrote:
For some reason I neglected this mail, see my answers in-line:


On Feb 25, 2015, at 1:00 AM, Robert Varga <nite@...> wrote:

Hello,

I am not sure I follow what exactly is the plan here. Is there going to be a centralized 'this is what integration looks like' job? Furthermore how does this interact with the ${project}-integration jobs, which also publish artifacts, but are not configured to trigger according to the dependency chain?

${project}-integration builds the project code on upstream merges, ${project}-distribution builds the integration project on own project merges so they are very different

So I merge stuff to yangtools, which triggers a merge job, which will complete. Now two jobs will trigger: the controller integration job and this distribution job ... controller integration job publishes artifacts, so ... which artifacts is the distribution job going to pick up?

integration artifacts = karaf distribution


Bye,
Robert

On 02/25/2015 03:20 AM, Luis Gomez wrote:
Hi all,

I am pushing following patches to releng repo:


This patch will create a job that downloads and builds the integration repository whenever there is a merge in the above projects. You can check the job template here:


You will need this job for 3 reasons:

1) To verify your changes are compatible with the integration code
2) To make sure your changes are immediately included in the integration distribution
3) To run some system test after the distribution is built

Please let me know if you have questions or concerns about merging this.

PS - these jobs will replace the old integration-polling jobs + integration-centralized-integration job

BR/Luis


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev