[release] [OpenDaylight TSC] Neon MRI CSITs


Vishal Thapar <vthapar@...>
 

https://git.opendaylight.org/gerrit/75824 had broken Genius' mri patch. Stephen pushed a fix https://git.opendaylight.org/gerrit/77245 which is merged and MRI patch rebased. I've kicked off another multipatch build.


On Tue, Oct 23, 2018 at 11:58 PM Sam Hague <shague@...> wrote:


On Tue, Oct 23, 2018 at 2:13 PM Luis Gomez <ecelgp@...> wrote:
Neon MRI is failing on genius,
Vishal has rebased the genius patch below and restarted the multipatch. Let's see how that goes. Currently on netconf and hasn't hit genius yet.
 
I cherry picked the change to stable/fluorine that is also failing now:


BR/Luis

On Oct 22, 2018, at 12:42 PM, Luis Gomez <ecelgp@...> wrote:

Thanks Robert, I triggered a new multi-patch build to verify:


BR/Luis


On Oct 22, 2018, at 3:26 AM, Robert Varga <nite@...> wrote:

On 22/10/2018 12:13, Robert Varga wrote:
On 20/10/2018 00:11, Luis Gomez wrote:
For netconf, I see 2 problems:

1) Issue installing the call home bundle:

2018-10-19T11:30:44,285 | ERROR | Blueprint Extender: 2 | NetconfCallHomeServer            | 279 - org.opendaylight.netconf.callhome-protocol - 1.6.0.SNAPSHOT | Unable to start NETCONF CallHome Service
java.io.IOException: java.lang.IllegalStateException: incompatible event loop type: io.netty.channel.DefaultEventLoop

See full exception in:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-callhome-only-neon/73/odl_1/odl1_karaf.log.gz
This looks like a mismatch after upgrade, I'll check it out later today.

https://git.opendaylight.org/gerrit/#/c/74966/51..52 should address
this, I think.

Regards,
Robert


_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release


Sam Hague <shague@...>
 



On Tue, Oct 23, 2018 at 2:13 PM Luis Gomez <ecelgp@...> wrote:
Neon MRI is failing on genius,
Vishal has rebased the genius patch below and restarted the multipatch. Let's see how that goes. Currently on netconf and hasn't hit genius yet.
 
I cherry picked the change to stable/fluorine that is also failing now:


BR/Luis

On Oct 22, 2018, at 12:42 PM, Luis Gomez <ecelgp@...> wrote:

Thanks Robert, I triggered a new multi-patch build to verify:


BR/Luis


On Oct 22, 2018, at 3:26 AM, Robert Varga <nite@...> wrote:

On 22/10/2018 12:13, Robert Varga wrote:
On 20/10/2018 00:11, Luis Gomez wrote:
For netconf, I see 2 problems:

1) Issue installing the call home bundle:

2018-10-19T11:30:44,285 | ERROR | Blueprint Extender: 2 | NetconfCallHomeServer            | 279 - org.opendaylight.netconf.callhome-protocol - 1.6.0.SNAPSHOT | Unable to start NETCONF CallHome Service
java.io.IOException: java.lang.IllegalStateException: incompatible event loop type: io.netty.channel.DefaultEventLoop

See full exception in:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-callhome-only-neon/73/odl_1/odl1_karaf.log.gz
This looks like a mismatch after upgrade, I'll check it out later today.

https://git.opendaylight.org/gerrit/#/c/74966/51..52 should address
this, I think.

Regards,
Robert



Luis Gomez
 

Neon MRI is failing on genius, I cherry picked the change to stable/fluorine that is also failing now:


BR/Luis

On Oct 22, 2018, at 12:42 PM, Luis Gomez <ecelgp@...> wrote:

Thanks Robert, I triggered a new multi-patch build to verify:


BR/Luis


On Oct 22, 2018, at 3:26 AM, Robert Varga <nite@...> wrote:

On 22/10/2018 12:13, Robert Varga wrote:
On 20/10/2018 00:11, Luis Gomez wrote:
For netconf, I see 2 problems:

1) Issue installing the call home bundle:

2018-10-19T11:30:44,285 | ERROR | Blueprint Extender: 2 | NetconfCallHomeServer            | 279 - org.opendaylight.netconf.callhome-protocol - 1.6.0.SNAPSHOT | Unable to start NETCONF CallHome Service
java.io.IOException: java.lang.IllegalStateException: incompatible event loop type: io.netty.channel.DefaultEventLoop

See full exception in:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-callhome-only-neon/73/odl_1/odl1_karaf.log.gz
This looks like a mismatch after upgrade, I'll check it out later today.

https://git.opendaylight.org/gerrit/#/c/74966/51..52 should address
this, I think.

Regards,
Robert



Luis Gomez
 

Thanks Robert, I triggered a new multi-patch build to verify:


BR/Luis


On Oct 22, 2018, at 3:26 AM, Robert Varga <nite@...> wrote:

On 22/10/2018 12:13, Robert Varga wrote:
On 20/10/2018 00:11, Luis Gomez wrote:
For netconf, I see 2 problems:

1) Issue installing the call home bundle:

2018-10-19T11:30:44,285 | ERROR | Blueprint Extender: 2 | NetconfCallHomeServer            | 279 - org.opendaylight.netconf.callhome-protocol - 1.6.0.SNAPSHOT | Unable to start NETCONF CallHome Service
java.io.IOException: java.lang.IllegalStateException: incompatible event loop type: io.netty.channel.DefaultEventLoop

See full exception in:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-callhome-only-neon/73/odl_1/odl1_karaf.log.gz
This looks like a mismatch after upgrade, I'll check it out later today.

https://git.opendaylight.org/gerrit/#/c/74966/51..52 should address
this, I think.

Regards,
Robert


Robert Varga
 

On 22/10/2018 12:13, Robert Varga wrote:
On 20/10/2018 00:11, Luis Gomez wrote:
For netconf, I see 2 problems:

1) Issue installing the call home bundle:

2018-10-19T11:30:44,285 | ERROR | Blueprint Extender: 2 | NetconfCallHomeServer | 279 - org.opendaylight.netconf.callhome-protocol - 1.6.0.SNAPSHOT | Unable to start NETCONF CallHome Service
java.io.IOException: java.lang.IllegalStateException: incompatible event loop type: io.netty.channel.DefaultEventLoop

See full exception in:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-callhome-only-neon/73/odl_1/odl1_karaf.log.gz
This looks like a mismatch after upgrade, I'll check it out later today.
https://git.opendaylight.org/gerrit/#/c/74966/51..52 should address
this, I think.

Regards,
Robert


Robert Varga
 

On 20/10/2018 00:11, Luis Gomez wrote:
For netconf, I see 2 problems:

1) Issue installing the call home bundle:

2018-10-19T11:30:44,285 | ERROR | Blueprint Extender: 2 | NetconfCallHomeServer | 279 - org.opendaylight.netconf.callhome-protocol - 1.6.0.SNAPSHOT | Unable to start NETCONF CallHome Service
java.io.IOException: java.lang.IllegalStateException: incompatible event loop type: io.netty.channel.DefaultEventLoop

See full exception in:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-callhome-only-neon/73/odl_1/odl1_karaf.log.gz
This looks like a mismatch after upgrade, I'll check it out later today.

Regards,
Robert


Robert Varga
 

On 19/10/2018 21:02, Luis Gomez wrote:
I quickly checked OFP and the issue is that before the patch the operational inventory used to return {"nodes":{}} when no OF nodes are connect and after the patch it returns nothing:

{"errors":{"error":[{"error-type":"application","error-tag":"data-missing","error-message":"Request could not be completed because the relevant data model content does not exist"}]}}

I can quickly change the test if this is intended new behavior.
Since the list is empty, it disappears, I think a 404 is is appropriate
here.

Regards,
Robert


Luis Gomez
 

For netconf, I see 2 problems:

1) Issue installing the call home bundle:

2018-10-19T11:30:44,285 | ERROR | Blueprint Extender: 2 | NetconfCallHomeServer | 279 - org.opendaylight.netconf.callhome-protocol - 1.6.0.SNAPSHOT | Unable to start NETCONF CallHome Service
java.io.IOException: java.lang.IllegalStateException: incompatible event loop type: io.netty.channel.DefaultEventLoop

See full exception in:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-callhome-only-neon/73/odl_1/odl1_karaf.log.gz

2) Impact on the RESTCONF to NETCONF performance test:

REST client tool gets timeout reading device config:

12:14:18.927 [pool-4-thread-1] WARN o.o.n.t.t.c.h.p.SyncExecutionStrategy - Failed to execute request
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Read timeout to /10.30.170.124:8181 of 60000 ms
at com.ning.http.client.providers.netty.future.NettyResponseFuture.abort(NettyResponseFuture.java:231)
at com.ning.http.client.providers.netty.request.NettyRequestSender.abort(NettyRequestSender.java:422)
at com.ning.http.client.providers.netty.request.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:47)
at com.ning.http.client.providers.netty.request.timeout.ReadTimeoutTimerTask.run(ReadTimeoutTimerTask.java:57)
at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:556)
at org.jboss.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:632)
at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:369)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Read timeout to /10.30.170.124:8181 of 60000 ms

See full test in:

https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-1node-scale-only-neon

BR/Luis

On Oct 19, 2018, at 12:02 PM, Luis Gomez <ecelgp@...> wrote:

I quickly checked OFP and the issue is that before the patch the operational inventory used to return {"nodes":{}} when no OF nodes are connect and after the patch it returns nothing:

{"errors":{"error":[{"error-type":"application","error-tag":"data-missing","error-message":"Request could not be completed because the relevant data model content does not exist"}]}}

I can quickly change the test if this is intended new behavior.

BR/Luis


On Oct 19, 2018, at 11:53 AM, Sam Hague <shague@...> wrote:



On Fri, Oct 19, 2018 at 10:49 AM Stephen Kitt <skitt@...> wrote:
Hi,

There is currently a series of CSITs running with the Neon MRI on
https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-neon/92

So far there are lots of yellows, some blues, no reds. I’d appreciate
it if people who know the various CSITs could take a look at the
results to see how close we are to being able to merge the MRI patches.
Looked through a few. Created two groups below: 1. good to merge, 2. not good to merge. Looks about 50/50 where things stand.

1. good:
aaa: same failing suite, so ignore
coe: same failing suite, so ignore
controller: same failing suite, so ignore. tell job had a couple extra test failures but it might be OK since the previous results bounced by a couple also
ovsdb: passing 100%

2. not good
genius: failures and all in jobs started with mri patches.
netconf: more failures in the mri jobs
netvirt: failures and all in jobs started with mri patches. mainly exceptions from genius, though so maybe fixing genius will get netvirt much closer
openflowplugin: 2 more failures in the mri jobs
sfc: failures and all in jobs started with mri patches.


Regards,

Stephen
_______________________________________________
TSC mailing list
TSC@...
https://lists.opendaylight.org/mailman/listinfo/tsc
_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release


Luis Gomez
 

I quickly checked OFP and the issue is that before the patch the operational inventory used to return {"nodes":{}} when no OF nodes are connect and after the patch it returns nothing:

{"errors":{"error":[{"error-type":"application","error-tag":"data-missing","error-message":"Request could not be completed because the relevant data model content does not exist"}]}}

I can quickly change the test if this is intended new behavior.

BR/Luis

On Oct 19, 2018, at 11:53 AM, Sam Hague <shague@...> wrote:



On Fri, Oct 19, 2018 at 10:49 AM Stephen Kitt <skitt@...> wrote:
Hi,

There is currently a series of CSITs running with the Neon MRI on
https://jenkins.opendaylight.org/releng/view/integration/job/integration-multipatch-test-neon/92

So far there are lots of yellows, some blues, no reds. I’d appreciate
it if people who know the various CSITs could take a look at the
results to see how close we are to being able to merge the MRI patches.
Looked through a few. Created two groups below: 1. good to merge, 2. not good to merge. Looks about 50/50 where things stand.

1. good:
aaa: same failing suite, so ignore
coe: same failing suite, so ignore
controller: same failing suite, so ignore. tell job had a couple extra test failures but it might be OK since the previous results bounced by a couple also
ovsdb: passing 100%

2. not good
genius: failures and all in jobs started with mri patches.
netconf: more failures in the mri jobs
netvirt: failures and all in jobs started with mri patches. mainly exceptions from genius, though so maybe fixing genius will get netvirt much closer
openflowplugin: 2 more failures in the mri jobs
sfc: failures and all in jobs started with mri patches.


Regards,

Stephen
_______________________________________________
TSC mailing list
TSC@...
https://lists.opendaylight.org/mailman/listinfo/tsc
_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release