Sodium SR1 release candidate


JamO Luhrsen
 

In our weekly integration call we decided to use the most recent successful
Sodium autorelease build as our release candidate for SR1 which is due next
week on October 24th.

Here is the build:
https://nexus.opendaylight.org/content/repositories//autorelease-3438/org/opendaylight/integration/karaf/0.11.1/karaf-0.11.1.zip

Here is the integration-test job using that build:
https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-test-sodium/366/

Here is the tracking sheet we need to vet:
https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

Projects that have pending failures:
bgpcep, controller, lispflowmapping, netconf, netvirt, openflowplugin, ovsdb, yangtools

Thanks,
JamO


Daniel de la Rosa
 

Team, 

Friendly reminder to review Sodium SR1 tracking sheet so we can release it on time.


On Fri, Oct 18, 2019 at 11:28 AM JamO Luhrsen <jluhrsen@...> wrote:
In our weekly integration call we decided to use the most recent successful
Sodium autorelease build as our release candidate for SR1 which is due next
week on October 24th.

Here is the build:
https://nexus.opendaylight.org/content/repositories//autorelease-3438/org/opendaylight/integration/karaf/0.11.1/karaf-0.11.1.zip

Here is the integration-test job using that build:
https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-test-sodium/366/

Here is the tracking sheet we need to vet:
https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

Projects that have pending failures:
bgpcep, controller, lispflowmapping, netconf, netvirt, openflowplugin, ovsdb, yangtools

Thanks,
JamO
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#13788): https://lists.opendaylight.org/g/integration-dev/message/13788
Mute This Topic: https://lists.opendaylight.org/mt/35075895/1964961
Group Owner: integration-dev+owner@...
Unsubscribe: https://lists.opendaylight.org/g/integration-dev/unsub  [ddelarosa@...]
-=-=-=-=-=-=-=-=-=-=-=-


--
Daniel de la Rosa
Customer Support Manager ( ODL release manager )
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120


Abhinav Gupta <abhinav.gupta@...>
 

Srinivas,

Please look into the CSIT jobs for failures. Please take help from Karthika if required. I’m in parallel onboarding myself with the overall Netvirt CSIT. Let’s discuss and get this moving.
I see 20 CSIT jobs for NetVirt. We may want to eliminate the unnecessary jobs as well.

 

Regards,
Abhinav

 

From: Daniel De La Rosa <ddelarosa@...>
Sent: Tuesday, October 22, 2019 1:49 AM
To: JamO Luhrsen <jluhrsen@...>; Abhinav Gupta <abhinav.gupta@...>
Cc: Release <release@...>; tsc@...; integration-dev@...; netvirt-dev@...; Bgpcep-Dev <bgpcep-dev@...>; controller-dev@...; lispflowmapping-dev@...; netconf-dev@...; openflowplugin-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

Team, 

 

Friendly reminder to review Sodium SR1 tracking sheet so we can release it on time.

 

 

On Fri, Oct 18, 2019 at 11:28 AM JamO Luhrsen <jluhrsen@...> wrote:

In our weekly integration call we decided to use the most recent successful
Sodium autorelease build as our release candidate for SR1 which is due next
week on October 24th.

Here is the build:
https://nexus.opendaylight.org/content/repositories//autorelease-3438/org/opendaylight/integration/karaf/0.11.1/karaf-0.11.1.zip

Here is the integration-test job using that build:
https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-test-sodium/366/

Here is the tracking sheet we need to vet:
https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

Projects that have pending failures:
bgpcep, controller, lispflowmapping, netconf, netvirt, openflowplugin, ovsdb, yangtools

Thanks,
JamO

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#13788): https://lists.opendaylight.org/g/integration-dev/message/13788
Mute This Topic: https://lists.opendaylight.org/mt/35075895/1964961
Group Owner: integration-dev+owner@...
Unsubscribe: https://lists.opendaylight.org/g/integration-dev/unsub  [ddelarosa@...]
-=-=-=-=-=-=-=-=-=-=-=-


 

--

Daniel de la Rosa
Customer Support Manager ( ODL release manager )
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120


JamO Luhrsen
 

We need to get netvirt under control at this point. They have a lot of jobs,
and were never fully healthy but without maintenance they are going to create
a lot of work every release cycle.

I spent some time today going through our sodium sr1 failures and made progress
on the sheet:

https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

there are three jobs with a basic netvirt failure that was ignored in
the original sodium release (openstack instance not getting an ip address)
which seems like a fundamental failure. The original sodium release let
it through under the assumption that it was a test code problem. I don't
think it is. I marked those as BLOCKER.

Should we mark the DCGW jobs as ignore? I don't have time to debug them
and they are saw-tooth failures and inconsistent. No idea if they are
regressions or not.

beyond netvirt, I'm following up on one openflowplugin job and once those
are understood we will only be left with netvirt to be done.

Thanks,
JamO


On 10/21/19 11:41 PM, Abhinav Gupta wrote:

Srinivas,

Please look into the CSIT jobs for failures. Please take help from Karthika if required. I’m in parallel onboarding myself with the overall Netvirt CSIT. Let’s discuss and get this moving.
I see 20 CSIT jobs for NetVirt. We may want to eliminate the unnecessary jobs as well.

 

Regards,
Abhinav

 

From: Daniel De La Rosa <ddelarosa@...>
Sent: Tuesday, October 22, 2019 1:49 AM
To: JamO Luhrsen <jluhrsen@...>; Abhinav Gupta <abhinav.gupta@...>
Cc: Release <release@...>; tsc@...; integration-dev@...; netvirt-dev@...; Bgpcep-Dev <bgpcep-dev@...>; controller-dev@...; lispflowmapping-dev@...; netconf-dev@...; openflowplugin-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

Team, 

 

Friendly reminder to review Sodium SR1 tracking sheet so we can release it on time.

 

 

On Fri, Oct 18, 2019 at 11:28 AM JamO Luhrsen <jluhrsen@...> wrote:

In our weekly integration call we decided to use the most recent successful
Sodium autorelease build as our release candidate for SR1 which is due next
week on October 24th.

Here is the build:
https://nexus.opendaylight.org/content/repositories//autorelease-3438/org/opendaylight/integration/karaf/0.11.1/karaf-0.11.1.zip

Here is the integration-test job using that build:
https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-test-sodium/366/

Here is the tracking sheet we need to vet:
https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

Projects that have pending failures:
bgpcep, controller, lispflowmapping, netconf, netvirt, openflowplugin, ovsdb, yangtools

Thanks,
JamO

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#13788): https://lists.opendaylight.org/g/integration-dev/message/13788
Mute This Topic: https://lists.opendaylight.org/mt/35075895/1964961
Group Owner: integration-dev+owner@...
Unsubscribe: https://lists.opendaylight.org/g/integration-dev/unsub  [ddelarosa@...]
-=-=-=-=-=-=-=-=-=-=-=-


 

--

Daniel de la Rosa
Customer Support Manager ( ODL release manager )
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120



JayaPr
 

Hi Abhinav/Srinivas,

 

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-itm-direct-tunnels-sodium/93/

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-snat-conntrack-sodium/93/

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-sodium/93/

 

The above 3 jobs when run witj 0cmb-1ctl-2cmp cases have 100% pass result and have 2 failures with 1cmb-0ctl-0cmb cases.

Not test code issue but odl code issue.

Can anyone have a look?

 

Regards

Jaya

From: integration-dev@... <integration-dev@...> On Behalf Of JamO Luhrsen via Lists.Opendaylight.Org
Sent: Thursday, October 24, 2019 2:24 AM
To: Abhinav Gupta <abhinav.gupta@...>; Daniel De La Rosa <ddelarosa@...>; Srinivas Rachakonda <srinivas.rachakonda@...>
Cc: integration-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

We need to get netvirt under control at this point. They have a lot of jobs,
and were never fully healthy but without maintenance they are going to create
a lot of work every release cycle.

I spent some time today going through our sodium sr1 failures and made progress
on the sheet:

https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

there are three jobs with a basic netvirt failure that was ignored in
the original sodium release (openstack instance not getting an ip address)
which seems like a fundamental failure. The original sodium release let
it through under the assumption that it was a test code problem. I don't
think it is. I marked those as BLOCKER.

Should we mark the DCGW jobs as ignore? I don't have time to debug them
and they are saw-tooth failures and inconsistent. No idea if they are
regressions or not.

beyond netvirt, I'm following up on one openflowplugin job and once those
are understood we will only be left with netvirt to be done.

Thanks,
JamO

On 10/21/19 11:41 PM, Abhinav Gupta wrote:

Srinivas,

Please look into the CSIT jobs for failures. Please take help from Karthika if required. I’m in parallel onboarding myself with the overall Netvirt CSIT. Let’s discuss and get this moving.
I see 20 CSIT jobs for NetVirt. We may want to eliminate the unnecessary jobs as well.

 

Regards,
Abhinav

 

From: Daniel De La Rosa <ddelarosa@...>
Sent: Tuesday, October 22, 2019 1:49 AM
To: JamO Luhrsen <jluhrsen@...>; Abhinav Gupta <abhinav.gupta@...>
Cc: Release <release@...>; tsc@...; integration-dev@...; netvirt-dev@...; Bgpcep-Dev <bgpcep-dev@...>; controller-dev@...; lispflowmapping-dev@...; netconf-dev@...; openflowplugin-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

Team, 

 

Friendly reminder to review Sodium SR1 tracking sheet so we can release it on time.

 

 

On Fri, Oct 18, 2019 at 11:28 AM JamO Luhrsen <jluhrsen@...> wrote:

In our weekly integration call we decided to use the most recent successful
Sodium autorelease build as our release candidate for SR1 which is due next
week on October 24th.

Here is the build:
https://nexus.opendaylight.org/content/repositories//autorelease-3438/org/opendaylight/integration/karaf/0.11.1/karaf-0.11.1.zip

Here is the integration-test job using that build:
https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-test-sodium/366/

Here is the tracking sheet we need to vet:
https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

Projects that have pending failures:
bgpcep, controller, lispflowmapping, netconf, netvirt, openflowplugin, ovsdb, yangtools

Thanks,
JamO

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#13788): https://lists.opendaylight.org/g/integration-dev/message/13788
Mute This Topic: https://lists.opendaylight.org/mt/35075895/1964961
Group Owner: integration-dev+owner@...
Unsubscribe: https://lists.opendaylight.org/g/integration-dev/unsub  [ddelarosa@...]
-=-=-=-=-=-=-=-=-=-=-=-


 

--

Daniel de la Rosa
Customer Support Manager ( ODL release manager )
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120

 


JamO Luhrsen
 

Hey guys,

at this point, blocker bugs are "blocking" an already late release.
We need to get them resolved asap.

1) are they true regressions and issues that did not exist with the
   original Sodium release?

2) ignoring 1) above, are the bugs truly "blocker" in nature in that
   the netvirt functionality is broken and cannot be used?

Thanks,
JamO

On 10/29/19 12:02 AM, Jaya Priyadarshini wrote:

Hi Abhinav/Srinivas,

 

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-itm-direct-tunnels-sodium/93/

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-snat-conntrack-sodium/93/

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-sodium/93/

 

The above 3 jobs when run witj 0cmb-1ctl-2cmp cases have 100% pass result and have 2 failures with 1cmb-0ctl-0cmb cases.

Not test code issue but odl code issue.

Can anyone have a look?

 

Regards

Jaya

From: integration-dev@... <integration-dev@...> On Behalf Of JamO Luhrsen via Lists.Opendaylight.Org
Sent: Thursday, October 24, 2019 2:24 AM
To: Abhinav Gupta <abhinav.gupta@...>; Daniel De La Rosa <ddelarosa@...>; Srinivas Rachakonda <srinivas.rachakonda@...>
Cc: integration-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

We need to get netvirt under control at this point. They have a lot of jobs,
and were never fully healthy but without maintenance they are going to create
a lot of work every release cycle.

I spent some time today going through our sodium sr1 failures and made progress
on the sheet:

https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

there are three jobs with a basic netvirt failure that was ignored in
the original sodium release (openstack instance not getting an ip address)
which seems like a fundamental failure. The original sodium release let
it through under the assumption that it was a test code problem. I don't
think it is. I marked those as BLOCKER.

Should we mark the DCGW jobs as ignore? I don't have time to debug them
and they are saw-tooth failures and inconsistent. No idea if they are
regressions or not.

beyond netvirt, I'm following up on one openflowplugin job and once those
are understood we will only be left with netvirt to be done.

Thanks,
JamO

On 10/21/19 11:41 PM, Abhinav Gupta wrote:

Srinivas,

Please look into the CSIT jobs for failures. Please take help from Karthika if required. I’m in parallel onboarding myself with the overall Netvirt CSIT. Let’s discuss and get this moving.
I see 20 CSIT jobs for NetVirt. We may want to eliminate the unnecessary jobs as well.

 

Regards,
Abhinav

 

From: Daniel De La Rosa <ddelarosa@...>
Sent: Tuesday, October 22, 2019 1:49 AM
To: JamO Luhrsen <jluhrsen@...>; Abhinav Gupta <abhinav.gupta@...>
Cc: Release <release@...>; tsc@...; integration-dev@...; netvirt-dev@...; Bgpcep-Dev <bgpcep-dev@...>; controller-dev@...; lispflowmapping-dev@...; netconf-dev@...; openflowplugin-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

Team, 

 

Friendly reminder to review Sodium SR1 tracking sheet so we can release it on time.

 

 

On Fri, Oct 18, 2019 at 11:28 AM JamO Luhrsen <jluhrsen@...> wrote:

In our weekly integration call we decided to use the most recent successful
Sodium autorelease build as our release candidate for SR1 which is due next
week on October 24th.

Here is the build:
https://nexus.opendaylight.org/content/repositories//autorelease-3438/org/opendaylight/integration/karaf/0.11.1/karaf-0.11.1.zip

Here is the integration-test job using that build:
https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-test-sodium/366/

Here is the tracking sheet we need to vet:
https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

Projects that have pending failures:
bgpcep, controller, lispflowmapping, netconf, netvirt, openflowplugin, ovsdb, yangtools

Thanks,
JamO

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#13788): https://lists.opendaylight.org/g/integration-dev/message/13788
Mute This Topic: https://lists.opendaylight.org/mt/35075895/1964961
Group Owner: integration-dev+owner@...
Unsubscribe: https://lists.opendaylight.org/g/integration-dev/unsub  [ddelarosa@...]
-=-=-=-=-=-=-=-=-=-=-=-


 

--

Daniel de la Rosa
Customer Support Manager ( ODL release manager )
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120

 



Daniel de la Rosa
 

Let me add +<tsc@...> since we need to make a decision on whether to keep holding Sodium SR1 or to move forward with all these issues documented in the release notes


On Tue, Oct 29, 2019 at 10:05 AM Jamo Luhrsen <jluhrsen@...> wrote:
Hey guys,

at this point, blocker bugs are "blocking" an already late release.
We need to get them resolved asap.

1) are they true regressions and issues that did not exist with the
   original Sodium release?

2) ignoring 1) above, are the bugs truly "blocker" in nature in that
   the netvirt functionality is broken and cannot be used?

Thanks,
JamO

On 10/29/19 12:02 AM, Jaya Priyadarshini wrote:

Hi Abhinav/Srinivas,

 

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-itm-direct-tunnels-sodium/93/

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-snat-conntrack-sodium/93/

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-sodium/93/

 

The above 3 jobs when run witj 0cmb-1ctl-2cmp cases have 100% pass result and have 2 failures with 1cmb-0ctl-0cmb cases.

Not test code issue but odl code issue.

Can anyone have a look?

 

Regards

Jaya

From: integration-dev@... <integration-dev@...> On Behalf Of JamO Luhrsen via Lists.Opendaylight.Org
Sent: Thursday, October 24, 2019 2:24 AM
To: Abhinav Gupta <abhinav.gupta@...>; Daniel De La Rosa <ddelarosa@...>; Srinivas Rachakonda <srinivas.rachakonda@...>
Cc: integration-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

We need to get netvirt under control at this point. They have a lot of jobs,
and were never fully healthy but without maintenance they are going to create
a lot of work every release cycle.

I spent some time today going through our sodium sr1 failures and made progress
on the sheet:

https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

there are three jobs with a basic netvirt failure that was ignored in
the original sodium release (openstack instance not getting an ip address)
which seems like a fundamental failure. The original sodium release let
it through under the assumption that it was a test code problem. I don't
think it is. I marked those as BLOCKER.

Should we mark the DCGW jobs as ignore? I don't have time to debug them
and they are saw-tooth failures and inconsistent. No idea if they are
regressions or not.

beyond netvirt, I'm following up on one openflowplugin job and once those
are understood we will only be left with netvirt to be done.

Thanks,
JamO

On 10/21/19 11:41 PM, Abhinav Gupta wrote:

Srinivas,

Please look into the CSIT jobs for failures. Please take help from Karthika if required. I’m in parallel onboarding myself with the overall Netvirt CSIT. Let’s discuss and get this moving.
I see 20 CSIT jobs for NetVirt. We may want to eliminate the unnecessary jobs as well.

 

Regards,
Abhinav

 

From: Daniel De La Rosa <ddelarosa@...>
Sent: Tuesday, October 22, 2019 1:49 AM
To: JamO Luhrsen <jluhrsen@...>; Abhinav Gupta <abhinav.gupta@...>
Cc: Release <release@...>; tsc@...; integration-dev@...; netvirt-dev@...; Bgpcep-Dev <bgpcep-dev@...>; controller-dev@...; lispflowmapping-dev@...; netconf-dev@...; openflowplugin-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

Team, 

 

Friendly reminder to review Sodium SR1 tracking sheet so we can release it on time.

 

 

On Fri, Oct 18, 2019 at 11:28 AM JamO Luhrsen <jluhrsen@...> wrote:

In our weekly integration call we decided to use the most recent successful
Sodium autorelease build as our release candidate for SR1 which is due next
week on October 24th.

Here is the build:
https://nexus.opendaylight.org/content/repositories//autorelease-3438/org/opendaylight/integration/karaf/0.11.1/karaf-0.11.1.zip

Here is the integration-test job using that build:
https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-test-sodium/366/

Here is the tracking sheet we need to vet:
https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

Projects that have pending failures:
bgpcep, controller, lispflowmapping, netconf, netvirt, openflowplugin, ovsdb, yangtools

Thanks,
JamO

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#13788): https://lists.opendaylight.org/g/integration-dev/message/13788
Mute This Topic: https://lists.opendaylight.org/mt/35075895/1964961
Group Owner: integration-dev+owner@...
Unsubscribe: https://lists.opendaylight.org/g/integration-dev/unsub  [ddelarosa@...]
-=-=-=-=-=-=-=-=-=-=-=-


 

--

Daniel de la Rosa
Customer Support Manager ( ODL release manager )
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120

 




--
Daniel de la Rosa
Customer Support Manager
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120


JayaPr
 

Hi Jamo,

 

There are 5 blockers for netvirt in Sodium-SR1 which are of 2 category:

  1. 2 jobs of dcgw –Both the jobs has 3 failures

1 fails for exceptions caught in teardown : broken recently

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/685/JP-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-dcgw-sodium/4/robot-plugin/log_full.html.gz

no exception in sept12 result.

 

2 fails saying association of vpn fails  but surprisingly when I ran the same suite in sandbox it passes totally.

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/JP-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-dcgw-sodium/5/robot-plugin/log_full.html.gz

              Hence something is being left over from previous suite or something like that, needs fix.

              But not a blocker clearly.

 

  1. 3 jobs of rocky with openstack issue : was existing since beginning.

Well the same testcase passes for 0cmb cases, ie, compute and controller separate so clearly not a netvirt functionality issue or major openstack issue.

 

Regards

Jaya

From: Jamo Luhrsen <jluhrsen@...>
Sent: Tuesday, October 29, 2019 10:35 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>; Abhinav Gupta <abhinav.gupta@...>; Daniel De La Rosa <ddelarosa@...>; Srinivas Rachakonda <srinivas.rachakonda@...>
Cc: integration-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

Hey guys,

at this point, blocker bugs are "blocking" an already late release.
We need to get them resolved asap.

1) are they true regressions and issues that did not exist with the
   original Sodium release?

2) ignoring 1) above, are the bugs truly "blocker" in nature in that
   the netvirt functionality is broken and cannot be used?

Thanks,
JamO

On 10/29/19 12:02 AM, Jaya Priyadarshini wrote:

Hi Abhinav/Srinivas,

 

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-itm-direct-tunnels-sodium/93/

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-snat-conntrack-sodium/93/

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-sodium/93/

 

The above 3 jobs when run witj 0cmb-1ctl-2cmp cases have 100% pass result and have 2 failures with 1cmb-0ctl-0cmb cases.

Not test code issue but odl code issue.

Can anyone have a look?

 

Regards

Jaya

From: integration-dev@... <integration-dev@...> On Behalf Of JamO Luhrsen via Lists.Opendaylight.Org
Sent: Thursday, October 24, 2019 2:24 AM
To: Abhinav Gupta <abhinav.gupta@...>; Daniel De La Rosa <ddelarosa@...>; Srinivas Rachakonda <srinivas.rachakonda@...>
Cc: integration-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

We need to get netvirt under control at this point. They have a lot of jobs,
and were never fully healthy but without maintenance they are going to create
a lot of work every release cycle.

I spent some time today going through our sodium sr1 failures and made progress
on the sheet:

https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

there are three jobs with a basic netvirt failure that was ignored in
the original sodium release (openstack instance not getting an ip address)
which seems like a fundamental failure. The original sodium release let
it through under the assumption that it was a test code problem. I don't
think it is. I marked those as BLOCKER.

Should we mark the DCGW jobs as ignore? I don't have time to debug them
and they are saw-tooth failures and inconsistent. No idea if they are
regressions or not.

beyond netvirt, I'm following up on one openflowplugin job and once those
are understood we will only be left with netvirt to be done.

Thanks,
JamO

On 10/21/19 11:41 PM, Abhinav Gupta wrote:

Srinivas,

Please look into the CSIT jobs for failures. Please take help from Karthika if required. I’m in parallel onboarding myself with the overall Netvirt CSIT. Let’s discuss and get this moving.
I see 20 CSIT jobs for NetVirt. We may want to eliminate the unnecessary jobs as well.

 

Regards,
Abhinav

 

From: Daniel De La Rosa <ddelarosa@...>
Sent: Tuesday, October 22, 2019 1:49 AM
To: JamO Luhrsen <jluhrsen@...>; Abhinav Gupta <abhinav.gupta@...>
Cc: Release <release@...>; tsc@...; integration-dev@...; netvirt-dev@...; Bgpcep-Dev <bgpcep-dev@...>; controller-dev@...; lispflowmapping-dev@...; netconf-dev@...; openflowplugin-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

Team, 

 

Friendly reminder to review Sodium SR1 tracking sheet so we can release it on time.

 

 

On Fri, Oct 18, 2019 at 11:28 AM JamO Luhrsen <jluhrsen@...> wrote:

In our weekly integration call we decided to use the most recent successful
Sodium autorelease build as our release candidate for SR1 which is due next
week on October 24th.

Here is the build:
https://nexus.opendaylight.org/content/repositories//autorelease-3438/org/opendaylight/integration/karaf/0.11.1/karaf-0.11.1.zip

Here is the integration-test job using that build:
https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-test-sodium/366/

Here is the tracking sheet we need to vet:
https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

Projects that have pending failures:
bgpcep, controller, lispflowmapping, netconf, netvirt, openflowplugin, ovsdb, yangtools

Thanks,
JamO

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#13788): https://lists.opendaylight.org/g/integration-dev/message/13788
Mute This Topic: https://lists.opendaylight.org/mt/35075895/1964961
Group Owner: integration-dev+owner@...
Unsubscribe: https://lists.opendaylight.org/g/integration-dev/unsub  [ddelarosa@...]
-=-=-=-=-=-=-=-=-=-=-=-


 

--

Daniel de la Rosa
Customer Support Manager ( ODL release manager )
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120

 

 


JamO Luhrsen
 

Thanks Jaya, see inline...

On 10/30/19 4:05 AM, Jaya Priyadarshini wrote:

Hi Jamo,

 

There are 5 blockers for netvirt in Sodium-SR1 which are of 2 category:

  1. 2 jobs of dcgw –Both the jobs has 3 failures

1 fails for exceptions caught in teardown : broken recently

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/685/JP-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-dcgw-sodium/4/robot-plugin/log_full.html.gz

no exception in sept12 result.


Can someone figure out the severity of this newly introduced exception? does it affect
end-user functionality?

 

2 fails saying association of vpn fails  but surprisingly when I ran the same suite in sandbox it passes totally.

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/JP-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-dcgw-sodium/5/robot-plugin/log_full.html.gz

              Hence something is being left over from previous suite or something like that, needs fix.

              But not a blocker clearly.


ok, can you mark it's JIRA as non-blocking? I would suggest someone spend some
time running the previous suites + failing suite in the sandbox to try to narrow down
the root cause. Also, need to make this jira non-blocker status.

 

  1. 3 jobs of rocky with openstack issue : was existing since beginning.

Well the same testcase passes for 0cmb cases, ie, compute and controller separate so clearly not a netvirt functionality issue or major openstack issue.


So this is not a regression since it was failing the same way with original Sodium release right?

I remember when I was working on netvirt jobs that we had some bugs that only showed
up when compute nodes were or were not combined with the control node. My guess is
that it's a legit netvirt or openstack bug.

Thanks,
JamO

 

Regards

Jaya

From: Jamo Luhrsen <jluhrsen@...>
Sent: Tuesday, October 29, 2019 10:35 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>; Abhinav Gupta <abhinav.gupta@...>; Daniel De La Rosa <ddelarosa@...>; Srinivas Rachakonda <srinivas.rachakonda@...>
Cc: integration-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

Hey guys,

at this point, blocker bugs are "blocking" an already late release.
We need to get them resolved asap.

1) are they true regressions and issues that did not exist with the
   original Sodium release?

2) ignoring 1) above, are the bugs truly "blocker" in nature in that
   the netvirt functionality is broken and cannot be used?

Thanks,
JamO

On 10/29/19 12:02 AM, Jaya Priyadarshini wrote:

Hi Abhinav/Srinivas,

 

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-itm-direct-tunnels-sodium/93/

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-snat-conntrack-sodium/93/

BLOCKER

jamo

netvirt

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-sodium/93/

 

The above 3 jobs when run witj 0cmb-1ctl-2cmp cases have 100% pass result and have 2 failures with 1cmb-0ctl-0cmb cases.

Not test code issue but odl code issue.

Can anyone have a look?

 

Regards

Jaya

From: integration-dev@... <integration-dev@...> On Behalf Of JamO Luhrsen via Lists.Opendaylight.Org
Sent: Thursday, October 24, 2019 2:24 AM
To: Abhinav Gupta <abhinav.gupta@...>; Daniel De La Rosa <ddelarosa@...>; Srinivas Rachakonda <srinivas.rachakonda@...>
Cc: integration-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

We need to get netvirt under control at this point. They have a lot of jobs,
and were never fully healthy but without maintenance they are going to create
a lot of work every release cycle.

I spent some time today going through our sodium sr1 failures and made progress
on the sheet:

https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

there are three jobs with a basic netvirt failure that was ignored in
the original sodium release (openstack instance not getting an ip address)
which seems like a fundamental failure. The original sodium release let
it through under the assumption that it was a test code problem. I don't
think it is. I marked those as BLOCKER.

Should we mark the DCGW jobs as ignore? I don't have time to debug them
and they are saw-tooth failures and inconsistent. No idea if they are
regressions or not.

beyond netvirt, I'm following up on one openflowplugin job and once those
are understood we will only be left with netvirt to be done.

Thanks,
JamO

On 10/21/19 11:41 PM, Abhinav Gupta wrote:

Srinivas,

Please look into the CSIT jobs for failures. Please take help from Karthika if required. I’m in parallel onboarding myself with the overall Netvirt CSIT. Let’s discuss and get this moving.
I see 20 CSIT jobs for NetVirt. We may want to eliminate the unnecessary jobs as well.

 

Regards,
Abhinav

 

From: Daniel De La Rosa <ddelarosa@...>
Sent: Tuesday, October 22, 2019 1:49 AM
To: JamO Luhrsen <jluhrsen@...>; Abhinav Gupta <abhinav.gupta@...>
Cc: Release <release@...>; tsc@...; integration-dev@...; netvirt-dev@...; Bgpcep-Dev <bgpcep-dev@...>; controller-dev@...; lispflowmapping-dev@...; netconf-dev@...; openflowplugin-dev@...
Subject: Re: [integration-dev] Sodium SR1 release candidate

 

Team, 

 

Friendly reminder to review Sodium SR1 tracking sheet so we can release it on time.

 

 

On Fri, Oct 18, 2019 at 11:28 AM JamO Luhrsen <jluhrsen@...> wrote:

In our weekly integration call we decided to use the most recent successful
Sodium autorelease build as our release candidate for SR1 which is due next
week on October 24th.

Here is the build:
https://nexus.opendaylight.org/content/repositories//autorelease-3438/org/opendaylight/integration/karaf/0.11.1/karaf-0.11.1.zip

Here is the integration-test job using that build:
https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-test-sodium/366/

Here is the tracking sheet we need to vet:
https://docs.google.com/spreadsheets/d/1CN-ACdIekRKMYRIuREIRALl6S8az7quAcQlRb3KeMJ8/edit?pli=1#gid=1864238939

Projects that have pending failures:
bgpcep, controller, lispflowmapping, netconf, netvirt, openflowplugin, ovsdb, yangtools

Thanks,
JamO

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#13788): https://lists.opendaylight.org/g/integration-dev/message/13788
Mute This Topic: https://lists.opendaylight.org/mt/35075895/1964961
Group Owner: integration-dev+owner@...
Unsubscribe: https://lists.opendaylight.org/g/integration-dev/unsub  [ddelarosa@...]
-=-=-=-=-=-=-=-=-=-=-=-


 

--

Daniel de la Rosa
Customer Support Manager ( ODL release manager )
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120

 

 



Robert Varga
 

On 30/10/2019 12:05, JayaPr via Lists.Opendaylight.Org wrote:
Hi Jamo,

 

There are 5 blockers for netvirt in Sodium-SR1 which are of 2 category:

1. 2 jobs of dcgw –Both the jobs has 3 failures

1 fails for exceptions caught in teardown : broken recently

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/685/JP-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-dcgw-sodium/4/robot-plugin/log_full.html.gz

no exception in sept12 result.
https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-itm-direct-tunnels-sodium/83/
is the first one to show uptick in failures.

2019-10-06T22:11:07,771 | ERROR | org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.access.control.list.rev160218.access.lists.Acl_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | AsyncDataTreeChangeListenerBase | 274 - org.opendaylight.genius.mdsalutil-api - 0.7.1 | Thread terminated due to uncaught exception: org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.access.control.list.rev160218.access.lists.Acl_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0
java.lang.NullPointerException: null
at org.opendaylight.netvirt.aclservice.listeners.AclEventListener.remove(AclEventListener.java:110) ~[?:?]
at org.opendaylight.netvirt.aclservice.listeners.AclEventListener.remove(AclEventListener.java:50) ~[?:?]
at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:158) ~[274:org.opendaylight.genius.mdsalutil-api:0.7.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]
NullPointerExceptions Are Always Bugs (TM)

In this particular case I think what we are looking at is List<Ace>
becoming empty through a transaction, prompting disappearance of
AccessListEntries and thus:

List<Ace> aceList = acl.getAccessListEntries().getAce();

results in NPE. The code needs to take the path equivalent to aceList
being empty and this is nothing new:
https://wiki.opendaylight.org/view/Neon_platform_upgrade#DataTree_removes_empty_lists.2C_choices_and_augmentations

Jamo: please file an issue with Netvirt, the fix needs to go to all
maintained branches.

Looking at
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-itm-direct-tunnels-sodium/93/odl_1/odl1_karaf.log.gz,
there's at least one more:

2019-10-16T16:44:44,433 | ERROR | jobcoordinator-main-task-6 | VpnSubnetRouteHandler | 385 - org.opendaylight.netvirt.vpnmanager-impl - 0.9.1 | SUBNETROUTE: onInterfaceUp: Unable to handle interface up event for port a8a7cd0e-337a-4540-8749-1372ce63f7b2 in subnet 65fe623a-5c49-45ce-a82a-673b7ff99199
java.lang.NullPointerException: null
at org.opendaylight.netvirt.vpnmanager.VpnSubnetRouteHandler.onInterfaceUp(VpnSubnetRouteHandler.java:623) ~[?:?]
at org.opendaylight.netvirt.vpnmanager.SubnetRouteInterfaceStateChangeListener.lambda$add$0(SubnetRouteInterfaceStateChangeListener.java:115) ~[?:?]
at org.opendaylight.infrautils.utils.ClassLoaders.lambda$call$2(ClassLoaders.java:39) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.call(ClassLoaders.java:47) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.call(ClassLoaders.java:39) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:398) ~[?:?]
at org.opendaylight.infrautils.utils.concurrent.LoggingUncaughtThreadDeathContextRunnable.run(LoggingUncaughtThreadDeathContextRunnable.java:60) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.lambda$run$0(ClassLoaders.java:26) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.call(ClassLoaders.java:47) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.run(ClassLoaders.java:25) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.lambda$wrap$3(ClassLoaders.java:54) ~[293:org.opendaylight.infrautils.util:1.6.1]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) [?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) [?:?]
Regards,
Robert


Robert Varga
 

On 30/10/2019 17:48, Robert Varga wrote:
2019-10-16T16:44:44,433 | ERROR | jobcoordinator-main-task-6 | VpnSubnetRouteHandler | 385 - org.opendaylight.netvirt.vpnmanager-impl - 0.9.1 | SUBNETROUTE: onInterfaceUp: Unable to handle interface up event for port a8a7cd0e-337a-4540-8749-1372ce63f7b2 in subnet 65fe623a-5c49-45ce-a82a-673b7ff99199
java.lang.NullPointerException: null
at org.opendaylight.netvirt.vpnmanager.VpnSubnetRouteHandler.onInterfaceUp(VpnSubnetRouteHandler.java:623) ~[?:?]
at org.opendaylight.netvirt.vpnmanager.SubnetRouteInterfaceStateChangeListener.lambda$add$0(SubnetRouteInterfaceStateChangeListener.java:115) ~[?:?]
at org.opendaylight.infrautils.utils.ClassLoaders.lambda$call$2(ClassLoaders.java:39) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.call(ClassLoaders.java:47) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.call(ClassLoaders.java:39) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:398) ~[?:?]
at org.opendaylight.infrautils.utils.concurrent.LoggingUncaughtThreadDeathContextRunnable.run(LoggingUncaughtThreadDeathContextRunnable.java:60) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.lambda$run$0(ClassLoaders.java:26) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.call(ClassLoaders.java:47) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.run(ClassLoaders.java:25) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.lambda$wrap$3(ClassLoaders.java:54) ~[293:org.opendaylight.infrautils.util:1.6.1]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) [?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) [?:?]
This one is:

SubnetOpDataEntryBuilder subOpBuilder = new
SubnetOpDataEntryBuilder(subnetOpDataEntry);

where subnetOpDataEntry is read from the datastore and then:

List<SubnetToDpn> subDpnList = subOpBuilder.getSubnetToDpn();
subDpnList.add(subDpn);

i.e. subDpnList is null, as it has become empty. There should be
nonnullSubnetToDpn() which will return non-null, but will cause .add()
to fail.

I *bet* there is a Magnesium patch which changes this code and that
patch needs to be backported, as it should deal with both nullability
and immutability.

This is pure technical debt which is being uncovered (for whatever
reason, it does not matter).

Regards,
Robert