Silicon General Release CSIT check


Daniel de la Rosa
 


Robert Varga
 

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>
It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.
Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert


Luis Gomez
 

I think you patch [1] fixed the issue:


We can maybe pick next AR build to see all netconf tests passing.

Also we need another patch [2] to move aaa and netconf CSIT to MRI once this issue [3] gets addressed in global-jib.

BR/Luis


On Mar 23, 2021, at 12:03 AM, Robert Varga <nite@...> wrote:

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>

It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.

Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert







Daniel de la Rosa
 

Hello all.. Sounds like we need to wait for @Anil Belur  to work on [2] and [3] to pick the next Silicon AR?  or we can move forward with just the NETCONF fix

Thanks

On Tue, Mar 23, 2021 at 1:08 PM Luis Gomez <ecelgp@...> wrote:
I think you patch [1] fixed the issue:


We can maybe pick next AR build to see all netconf tests passing.

Also we need another patch [2] to move aaa and netconf CSIT to MRI once this issue [3] gets addressed in global-jib.

BR/Luis


On Mar 23, 2021, at 12:03 AM, Robert Varga <nite@...> wrote:

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>

It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.

Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert







Luis Gomez
 

Anil, please let us know when global-jib patch is merged as this is blocking CSIT jobs update.

BR/Luis

On Mar 24, 2021, at 8:59 PM, Daniel de la Rosa <ddelarosa0707@...> wrote:

Hello all.. Sounds like we need to wait for @Anil Belur  to work on [2] and [3] to pick the next Silicon AR?  or we can move forward with just the NETCONF fix

Thanks

On Tue, Mar 23, 2021 at 1:08 PM Luis Gomez <ecelgp@...> wrote:
I think you patch [1] fixed the issue:


We can maybe pick next AR build to see all netconf tests passing.

Also we need another patch [2] to move aaa and netconf CSIT to MRI once this issue [3] gets addressed in global-jib.

BR/Luis


On Mar 23, 2021, at 12:03 AM, Robert Varga <nite@...> wrote:

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>

It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.

Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert








Anil Belur
 


On Wed, Mar 24, 2021 at 6:08 AM Luis Gomez <ecelgp@...> wrote:
I think you patch [1] fixed the issue:


We can maybe pick next AR build to see all netconf tests passing.

Also we need another patch [2] to move aaa and netconf CSIT to MRI once this issue [3] gets addressed in global-jib.

BR/Luis





Hello team:

The change in [3.] is no longer required as it's already addressed in releng/global-jjb CR (https://gerrit.linuxfoundation.org/infra/c/releng/global-jjb/+/67014).
The CR was waiting on a review/merge on the CR 67014 before proceeding with the releng/global-jjb release. 

Now CR 67014 is merged and releng/global-jjb v0.62.0 is released. Please merge releng/builder CR for the global-jjb 

Cheers,
Anil

 
On Mar 23, 2021, at 12:03 AM, Robert Varga <nite@...> wrote:

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>

It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.

Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert










Daniel de la Rosa
 

Great, thanks for the update @Anil Belur  I have picked Silicon AR#236 and here is the update CSIT spreadsheet

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing

It looks like we only have one issue with BGPCEP and JSONRPC. The rest (GENIUS, LISP) can be ignored... Richard, are you still supporting JSONRPC? Rangan, can you check the BGP issue.

Thanks




On Thu, Mar 25, 2021 at 6:01 PM Anil Belur <abelur@...> wrote:

On Wed, Mar 24, 2021 at 6:08 AM Luis Gomez <ecelgp@...> wrote:
I think you patch [1] fixed the issue:


We can maybe pick next AR build to see all netconf tests passing.

Also we need another patch [2] to move aaa and netconf CSIT to MRI once this issue [3] gets addressed in global-jib.

BR/Luis





Hello team:

The change in [3.] is no longer required as it's already addressed in releng/global-jjb CR (https://gerrit.linuxfoundation.org/infra/c/releng/global-jjb/+/67014).
The CR was waiting on a review/merge on the CR 67014 before proceeding with the releng/global-jjb release. 

Now CR 67014 is merged and releng/global-jjb v0.62.0 is released. Please merge releng/builder CR for the global-jjb 

Cheers,
Anil

 
On Mar 23, 2021, at 12:03 AM, Robert Varga <nite@...> wrote:

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>

It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.

Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert










Luis Gomez
 

I already added some comments to both failing CSIT, nothing looks critical or blocking the release.

Now, since we already have a good amount of MRI CSIT, going forward I would recommend to manual trigger this test for our release candidates:


Looking at past runs of MRI CSIT, only the cluster netty replicate test fails consistently:


16:41:52 2021-03-20T23:39:15,618 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSink             | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSink(45)] : Constructor with 0 arguments not found. Component will fail.
16:41:52 2021-03-20T23:39:15,619 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSink             | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSink(45)] : Error during instantiation of the implementation object: Constructor not found.
16:41:52 2021-03-20T23:39:15,621 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSource           | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSource(46)] : Constructor with 0 arguments not found. Component will fail.
16:41:52 2021-03-20T23:39:15,621 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSource           | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSource(46)] : Error during instantiation of the implementation object: Constructor not found.

Not sure if this is something we have to take a look.

BR/Luis



On Mar 26, 2021, at 9:29 PM, Daniel de la Rosa <ddelarosa0707@...> wrote:

Great, thanks for the update @Anil Belur  I have picked Silicon AR#236 and here is the update CSIT spreadsheet

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing

It looks like we only have one issue with BGPCEP and JSONRPC. The rest (GENIUS, LISP) can be ignored... Richard, are you still supporting JSONRPC? Rangan, can you check the BGP issue.

Thanks




On Thu, Mar 25, 2021 at 6:01 PM Anil Belur <abelur@...> wrote:

On Wed, Mar 24, 2021 at 6:08 AM Luis Gomez <ecelgp@...> wrote:
I think you patch [1] fixed the issue:


We can maybe pick next AR build to see all netconf tests passing.

Also we need another patch [2] to move aaa and netconf CSIT to MRI once this issue [3] gets addressed in global-jib.

BR/Luis





Hello team:

The change in [3.] is no longer required as it's already addressed in releng/global-jjb CR (https://gerrit.linuxfoundation.org/infra/c/releng/global-jjb/+/67014).
The CR was waiting on a review/merge on the CR 67014 before proceeding with the releng/global-jjb release. 

Now CR 67014 is merged and releng/global-jjb v0.62.0 is released. Please merge releng/builder CR for the global-jjb 

Cheers,
Anil

 
On Mar 23, 2021, at 12:03 AM, Robert Varga <nite@...> wrote:

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>

It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.

Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert











Robert Varga
 

On 28/03/2021 21:47, Luis Gomez wrote:
Looking at past runs of MRI CSIT, only the cluster netty replicate test
fails consistently:

https://jenkins.opendaylight.org/releng/job/mdsal-csit-3node-netty-replicate-only-silicon/10/
<https://jenkins.opendaylight.org/releng/job/mdsal-csit-3node-netty-replicate-only-silicon/10/>

*16:41:52* 2021-03-20T23:39:15,618 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSink | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSink(45)] : Constructor with 0 arguments not found. Component will fail.
*16:41:52* 2021-03-20T23:39:15,619 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSink | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSink(45)] : Error during instantiation of the implementation object: Constructor not found.
*16:41:52* 2021-03-20T23:39:15,621 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSource | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSource(46)] : Constructor with 0 arguments not found. Component will fail.
*16:41:52* 2021-03-20T23:39:15,621 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSource | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSource(46)] : Error during instantiation of the implementation object: Constructor not found.


Not sure if this is something we have to take a look.
I think we can safely punt this to SR1.

Regards,
Robert