1. Topics
  2. Karaf stop command not working in Sodium.

Karaf stop command not working in Sodium.

Karthika Panneer
 

Hi,

From today it is observed that in Genius CSIT karaf is not getting killed.

Command executed to stop controller : /tmp/karaf-0.11.0/bin/stop

 

Execution fails at a point where the check is done to ensure karaf is killed.


Executing command 'ps axf | grep org.apache.karaf | grep -v grep | wc -l'.

 

10:07:48.888

INFO

Command exited with return code 0.

 

10:07:48.888

INFO

${stdout} = 1

 

Output expected to be 0.

 

Logs: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/273/robot-plugin/log.html.gz#s1-s6-k1-k2-k7-k1-k1-k1

 

Any pointers regarding the same?

 

P.S : Issue observed only in sodium.

 

Regards,

R P Karthika
Faseela K
 

+odlparent

 

From: integration-dev <integration-dev-bounces@...> On Behalf Of R P Karthika .
Sent: Monday, August 5, 2019 3:38 PM
To: 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: [integration-dev] Karaf stop command not working in Sodium.

 

Hi,

From today it is observed that in Genius CSIT karaf is not getting killed.

Command executed to stop controller : /tmp/karaf-0.11.0/bin/stop

 

Execution fails at a point where the check is done to ensure karaf is killed.


Executing command 'ps axf | grep org.apache.karaf | grep -v grep | wc -l'.

 

10:07:48.888

INFO

Command exited with return code 0.

 

10:07:48.888

INFO

${stdout} = 1

 

Output expected to be 0.

 

Logs: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/273/robot-plugin/log.html.gz#s1-s6-k1-k2-k7-k1-k1-k1

 

Any pointers regarding the same?

 

P.S : Issue observed only in sodium.

 

Regards,

R P Karthika
Robert Varga
 

On 05/08/2019 12:23, Faseela K wrote:
+odlparent

 

*From:* integration-dev <integration-dev-bounces@...>
*On Behalf Of *R P Karthika .
*Sent:* Monday, August 5, 2019 3:38 PM
*To:* 'integration-dev@...'
(integration-dev@...)
(integration-dev@...)
<integration-dev@...>
*Subject:* [integration-dev] Karaf stop command not working in Sodium.

 

Hi,

From today it is observed that in Genius CSIT karaf is not getting killed.

Command executed to stop controller : /tmp/karaf-0.11.0/bin/stop
Not an odlparent problem:

2019-08-05T04:33:48,484 | INFO | Karaf Shutdown Socket Thread | ShutdownSocketThread | - - | Karaf shutdown socket: received shutdown command. Stopping framework...
[...]

2019-08-05T04:33:52,404 | INFO | Framework stop | BlueprintBundleTracker | 210 - org.opendaylight.controller.blueprint - 0.11.0 | Shutdown of blueprint containers complete
The only explanation is that there is a non-daemon thread holding down
the JVM. Please do dump the threads to make sense of this.

Regards,
Robert

?
signature.asc

JamO Luhrsen
 



On 8/5/19 9:12 AM, Robert Varga wrote:

On 05/08/2019 12:23, Faseela K wrote:

+odlparent

 

*From:* integration-dev <integration-dev-bounces@...>
*On Behalf Of *R P Karthika .
*Sent:* Monday, August 5, 2019 3:38 PM
*To:* 'integration-dev@...'
(integration-dev@...)
(integration-dev@...)
<integration-dev@...>
*Subject:* [integration-dev] Karaf stop command not working in Sodium.

 

Hi,

From today it is observed that in Genius CSIT karaf is not getting killed.

Command executed to stop controller : /tmp/karaf-0.11.0/bin/stop

Not an odlparent problem:


2019-08-05T04:33:48,484 | INFO  | Karaf Shutdown Socket Thread | ShutdownSocketThread             |  -  -  | Karaf shutdown socket: received shutdown command. Stopping framework...

[...]


2019-08-05T04:33:52,404 | INFO  | Framework stop   | BlueprintBundleTracker           | 210 - org.opendaylight.controller.blueprint - 0.11.0 | Shutdown of blueprint containers complete

The only explanation is that there is a non-daemon thread holding down
the JVM. Please do dump the threads to make sense of this.

is this enough info?

thread dump before running robot:
  https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_before.log.gz

thread dump after running robot:
  https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.

Thanks,
JamO



Regards,
Robert



_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev

JayaPr
 

Observing the same in netvirt-csit.

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-rocky-upstream-stateful-sodium/56/console.log.gz

 

Take Down ODL1 :: Stop the karaf in First Controller                  | FAIL |

Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 4 minutes. The last error was: Found running Karaf count: 1: 0 != 1

------------------------------------------------------------------------------

Create Bridge Manually and Verify After Fail :: Create bridge with... | FAIL |

 

Regards

Jaya

 

From: integration-dev <integration-dev-bounces@...> On Behalf Of Jamo Luhrsen
Sent: Tuesday, August 6, 2019 1:12 AM
To: Robert Varga <nite@...>; Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

 

 

On 8/5/19 9:12 AM, Robert Varga wrote:

 
 
On 05/08/2019 12:23, Faseela K wrote:
+odlparent
 
 
 
*From:* integration-dev <integration-dev-bounces@...>
*On Behalf Of *R P Karthika .
*Sent:* Monday, August 5, 2019 3:38 PM
*To:* 'integration-dev@...'
(integration-dev@...)
(integration-dev@...)
<integration-dev@...>
*Subject:* [integration-dev] Karaf stop command not working in Sodium.
 
 
 
Hi,
 
From today it is observed that in Genius CSIT karaf is not getting killed.
 
Command executed to stop controller : /tmp/karaf-0.11.0/bin/stop
 
Not an odlparent problem:
 
2019-08-05T04:33:48,484 | INFO  | Karaf Shutdown Socket Thread | ShutdownSocketThread             |  -  -  | Karaf shutdown socket: received shutdown command. Stopping framework...
 
[...]
 
2019-08-05T04:33:52,404 | INFO  | Framework stop   | BlueprintBundleTracker           | 210 - org.opendaylight.controller.blueprint - 0.11.0 | Shutdown of blueprint containers complete
 
The only explanation is that there is a non-daemon thread holding down
the JVM. Please do dump the threads to make sense of this.


is this enough info?

thread dump before running robot:
  https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_before.log.gz

thread dump after running robot:
  https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.

Thanks,
JamO



 
 
Regards,
Robert
 



_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev

 
JamO Luhrsen
 

Jaya,

you can download the same distro locally and start it and try to stop
in the same manner to see what you can learn.

JamO

On 8/5/19 10:09 PM, Jaya Priyadarshini wrote:

Observing the same in netvirt-csit.

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-rocky-upstream-stateful-sodium/56/console.log.gz

 

Take Down ODL1 :: Stop the karaf in First Controller                  | FAIL |

Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 4 minutes. The last error was: Found running Karaf count: 1: 0 != 1

------------------------------------------------------------------------------

Create Bridge Manually and Verify After Fail :: Create bridge with... | FAIL |

 

Regards

Jaya

 

From: integration-dev <integration-dev-bounces@...> On Behalf Of Jamo Luhrsen
Sent: Tuesday, August 6, 2019 1:12 AM
To: Robert Varga <nite@...>; Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

 

 

On 8/5/19 9:12 AM, Robert Varga wrote:

 
 
On 05/08/2019 12:23, Faseela K wrote:
+odlparent
 
 
 
*From:* integration-dev <integration-dev-bounces@...>
*On Behalf Of *R P Karthika .
*Sent:* Monday, August 5, 2019 3:38 PM
*To:* 'integration-dev@...'
(integration-dev@...)
(integration-dev@...)
<integration-dev@...>
*Subject:* [integration-dev] Karaf stop command not working in Sodium.
 
 
 
Hi,
 
From today it is observed that in Genius CSIT karaf is not getting killed.
 
Command executed to stop controller : /tmp/karaf-0.11.0/bin/stop
 
Not an odlparent problem:
 
2019-08-05T04:33:48,484 | INFO  | Karaf Shutdown Socket Thread | ShutdownSocketThread             |  -  -  | Karaf shutdown socket: received shutdown command. Stopping framework...
 
[...]
 
2019-08-05T04:33:52,404 | INFO  | Framework stop   | BlueprintBundleTracker           | 210 - org.opendaylight.controller.blueprint - 0.11.0 | Shutdown of blueprint containers complete
 
The only explanation is that there is a non-daemon thread holding down
the JVM. Please do dump the threads to make sense of this.


is this enough info?

thread dump before running robot:
  https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_before.log.gz

thread dump after running robot:
  https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.

Thanks,
JamO



 
 
Regards,
Robert
 



_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev

 
Robert Varga
 

On 05/08/2019 21:42, Jamo Luhrsen wrote:

is this enough info?

thread dump before running robot:
 
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_before.log.gz
<https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)


thread dump after running robot:
 
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is
creating SchemaContexts. It really should not, as we should be speedily
stopping.

This is probably a side-effect of
https://jira.opendaylight.org/browse/MDSAL-461 fix, we have a slight
optimization already staged for mdsal-4.0.4, but this needs proper
looking after.

Can you file an issue with mdsal?

Also, can you try with https://git.opendaylight.org/gerrit/c/mdsal/+/83493 ?

I'll take a look next week, as I am going offline.

Regards,
Robert

?
signature.asc

Faseela K
 

Robert,
We tried downloading the same distro and tested things manually, but we are not able to reproduce the issue.
But this is failing most of the times in Jenkins.
And can you tell us how to test this with a WIP patch in mdsal? [0]
How do we test this without having version bump patches in all projects?
Also, why did this break all of a sudden?
Thanks,
Faseela

[0] https://protect2.fireeye.com/url?k=c9282245-95a2f684-c92862de-863d9bcb726f-82981d78e3f8a352&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493

-----Original Message-----
From: Robert Varga <nite@...>
Sent: Tuesday, August 6, 2019 11:20 PM
To: Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

On 05/08/2019 21:42, Jamo Luhrsen wrote:

is this enough info?

thread dump before running robot:
 
https://protect2.fireeye.com/url?k=df2b6581-83a1b140-df2b251a-863d9bcb
726f-2daec84e911dd818&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frele
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2F
274%2Fodl_1%2Fkaraf_1_10983_threads_before.log.gz
<https://protect2.fireeye.com/url?k=d87dd00d-84f704cc-d87d9096-863d9bc
b726f-9bd20260a15f63a1&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frel
eng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2
F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)


thread dump after running robot:
 
https://protect2.fireeye.com/url?k=ab2c8fe8-f7a65b29-ab2ccf73-863d9bcb
726f-611852abf170af25&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frele
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2F
274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is creating SchemaContexts. It really should not, as we should be speedily stopping.

This is probably a side-effect of
https://protect2.fireeye.com/url?k=b64f59e5-eac58d24-b64f197e-863d9bcb726f-ae28518222c47c71&q=1&u=https%3A%2F%2Fjira.opendaylight.org%2Fbrowse%2FMDSAL-461 fix, we have a slight optimization already staged for mdsal-4.0.4, but this needs proper looking after.

Can you file an issue with mdsal?

Also, can you try with https://protect2.fireeye.com/url?k=c9282245-95a2f684-c92862de-863d9bcb726f-82981d78e3f8a352&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493 ?

I'll take a look next week, as I am going offline.

Regards,
Robert
Robert Varga
 

Checkout v4.0.3 mdsal tag, cherry-pick the patch, build affected component, replace the jar in the distro.

I am sure there is also an automated way, but Luis knows the details.

And it's not that the instance won't come down, it just takes more time than the tests are waiting for and it obviously depends on the number of features installed.

Bye,
Robert (offline for real)



Sent from my BlackBerry - the most secure mobile device - via the Orange Network


  Original Message  



From: faseela.k@...
Sent: August 7, 2019 8:33 AM
To: nite@...; jluhrsen@...; r.p.karthika@...; integration-dev@...
Cc: odlparent-dev@...; mdsal-dev@...
Subject: RE: [integration-dev] Karaf stop command not working in Sodium.


Robert,
   We tried downloading the same distro and tested things manually, but we are not able to reproduce the issue.
   But this is failing most of the times in Jenkins.
   And can you tell us how to test this with a WIP patch in mdsal? [0]
   How do we test this without having version bump patches in all projects?
   Also, why did this break all of a sudden?
Thanks,
Faseela

[0] https://protect2.fireeye.com/url?k=c9282245-95a2f684-c92862de-863d9bcb726f-82981d78e3f8a352&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493


-----Original Message-----
From: Robert Varga <nite@...>
Sent: Tuesday, August 6, 2019 11:20 PM
To: Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

On 05/08/2019 21:42, Jamo Luhrsen wrote:

is this enough info?

thread dump before running robot:
 
https://protect2.fireeye.com/url?k=df2b6581-83a1b140-df2b251a-863d9bcb
726f-2daec84e911dd818&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frele
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2F
274%2Fodl_1%2Fkaraf_1_10983_threads_before.log.gz
<https://protect2.fireeye.com/url?k=d87dd00d-84f704cc-d87d9096-863d9bc
b726f-9bd20260a15f63a1&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frel
eng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2
F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)


thread dump after running robot:
 
https://protect2.fireeye.com/url?k=ab2c8fe8-f7a65b29-ab2ccf73-863d9bcb
726f-611852abf170af25&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frele
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2F
274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is creating SchemaContexts. It really should not, as we should be speedily stopping.

This is probably a side-effect of
https://protect2.fireeye.com/url?k=b64f59e5-eac58d24-b64f197e-863d9bcb726f-ae28518222c47c71&q=1&u=https%3A%2F%2Fjira.opendaylight.org%2Fbrowse%2FMDSAL-461 fix, we have a slight optimization already staged for mdsal-4.0.4, but this needs proper looking after.

Can you file an issue with mdsal?

Also, can you try with https://protect2.fireeye.com/url?k=c9282245-95a2f684-c92862de-863d9bcb726f-82981d78e3f8a352&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493 ?

I'll take a look next week, as I am going offline.

Regards,
Robert
Karthika Panneer
 

Hi,
When I tried with the same distro locally I could see that it takes 10 minutes for the process to get killed after stopping the controller.
To confirm the same I triggered a job by increasing the timeout to 10 mins and was able to see that the job was successful.

Success logs : https://jenkins.opendaylight.org/sandbox/job/Karthikagenius-csit-1node-gate-only-sodium/6/

Regards,
R P Karthika

-----Original Message-----
From: Robert Varga <nite@...>
Sent: Tuesday, August 6, 2019 11:20 PM
To: Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

On 05/08/2019 21:42, Jamo Luhrsen wrote:

is this enough info?

thread dump before running robot:
 
https://protect2.fireeye.com/url?k=df2b6581-83a1b140-df2b251a-863d9bcb
726f-2daec84e911dd818&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frele
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2F
274%2Fodl_1%2Fkaraf_1_10983_threads_before.log.gz
<https://protect2.fireeye.com/url?k=d87dd00d-84f704cc-d87d9096-863d9bc
b726f-9bd20260a15f63a1&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frel
eng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2
F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)


thread dump after running robot:
 
https://protect2.fireeye.com/url?k=ab2c8fe8-f7a65b29-ab2ccf73-863d9bcb
726f-611852abf170af25&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frele
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2F
274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is creating SchemaContexts. It really should not, as we should be speedily stopping.

This is probably a side-effect of
https://protect2.fireeye.com/url?k=b64f59e5-eac58d24-b64f197e-863d9bcb726f-ae28518222c47c71&q=1&u=https%3A%2F%2Fjira.opendaylight.org%2Fbrowse%2FMDSAL-461 fix, we have a slight optimization already staged for mdsal-4.0.4, but this needs proper looking after.

Can you file an issue with mdsal?

Also, can you try with https://protect2.fireeye.com/url?k=c9282245-95a2f684-c92862de-863d9bcb726f-82981d78e3f8a352&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493 ?

I'll take a look next week, as I am going offline.

Regards,
Robert
Faseela K
 

Thanks for the confirmation Karthika, CSIT overall only takes 20 mins, we cannot put 10 mins for each controller restart though!
But good that you have clarified the same with the job.

@Robert : The patch you have on mdsal is on verify -1. Can we get it verified, will see if we can spend some cycles to test the same.

Thanks
Faseela

-----Original Message-----
From: R P Karthika . <r.p.karthika@...>
Sent: Wednesday, August 7, 2019 4:50 PM
To: Robert Varga <nite@...>; Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: RE: [integration-dev] Karaf stop command not working in Sodium.

Hi,
When I tried with the same distro locally I could see that it takes 10 minutes for the process to get killed after stopping the controller.
To confirm the same I triggered a job by increasing the timeout to 10 mins and was able to see that the job was successful.

Success logs : https://jenkins.opendaylight.org/sandbox/job/Karthikagenius-csit-1node-gate-only-sodium/6/

Regards,
R P Karthika

-----Original Message-----
From: Robert Varga <nite@...>
Sent: Tuesday, August 6, 2019 11:20 PM
To: Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

On 05/08/2019 21:42, Jamo Luhrsen wrote:

is this enough info?

thread dump before running robot:
 
https://protect2.fireeye.com/url?k=df2b6581-83a1b140-df2b251a-863d9bcb
726f-2daec84e911dd818&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frele
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2F
274%2Fodl_1%2Fkaraf_1_10983_threads_before.log.gz
<https://protect2.fireeye.com/url?k=d87dd00d-84f704cc-d87d9096-863d9bc
b726f-9bd20260a15f63a1&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frel
eng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2
F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)


thread dump after running robot:
 
https://protect2.fireeye.com/url?k=ab2c8fe8-f7a65b29-ab2ccf73-863d9bcb
726f-611852abf170af25&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frele
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2F
274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is creating SchemaContexts. It really should not, as we should be speedily stopping.

This is probably a side-effect of
https://protect2.fireeye.com/url?k=b64f59e5-eac58d24-b64f197e-863d9bcb726f-ae28518222c47c71&q=1&u=https%3A%2F%2Fjira.opendaylight.org%2Fbrowse%2FMDSAL-461 fix, we have a slight optimization already staged for mdsal-4.0.4, but this needs proper looking after.

Can you file an issue with mdsal?

Also, can you try with https://protect2.fireeye.com/url?k=c9282245-95a2f684-c92862de-863d9bcb726f-82981d78e3f8a352&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493 ?

I'll take a look next week, as I am going offline.

Regards,
Robert
JamO Luhrsen
 

If you have some time, you can go try to find which actual snapshot distro
started having this 10m problem. That will help isolate a small window when
this started happening. knowing that will help find the culprit.

good debugging so far :)

JamO

On 8/7/19 4:26 AM, Faseela K wrote:
Thanks for the confirmation Karthika, CSIT overall only takes 20 mins, we cannot put 10 mins for each controller restart though!
But good that you have clarified the same with the job.

@Robert : The patch you have on mdsal is on verify -1. Can we get it verified, will see if we can spend some cycles to test the same.

Thanks
Faseela

-----Original Message-----
From: R P Karthika . <r.p.karthika@...>
Sent: Wednesday, August 7, 2019 4:50 PM
To: Robert Varga <nite@...>; Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: RE: [integration-dev] Karaf stop command not working in Sodium.

Hi,
When I tried with the same distro locally I could see that it takes 10 minutes for the process to get killed after stopping the controller.
To confirm the same I triggered a job by increasing the timeout to 10 mins and was able to see that the job was successful.

Success logs : https://jenkins.opendaylight.org/sandbox/job/Karthikagenius-csit-1node-gate-only-sodium/6/

Regards,
R P Karthika

-----Original Message-----
From: Robert Varga <nite@...>
Sent: Tuesday, August 6, 2019 11:20 PM
To: Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

On 05/08/2019 21:42, Jamo Luhrsen wrote:
is this enough info?

thread dump before running robot:
https://protect2.fireeye.com/url?k=df2b6581-83a1b140-df2b251a-863d9bcb
726f-2daec84e911dd818&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frele
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2F
274%2Fodl_1%2Fkaraf_1_10983_threads_before.log.gz
<https://protect2.fireeye.com/url?k=d87dd00d-84f704cc-d87d9096-863d9bc
b726f-9bd20260a15f63a1&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frel
eng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2
F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)

thread dump after running robot:
https://protect2.fireeye.com/url?k=ab2c8fe8-f7a65b29-ab2ccf73-863d9bcb
726f-611852abf170af25&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frele
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2F
274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is creating SchemaContexts. It really should not, as we should be speedily stopping.

This is probably a side-effect of
https://protect2.fireeye.com/url?k=b64f59e5-eac58d24-b64f197e-863d9bcb726f-ae28518222c47c71&q=1&u=https%3A%2F%2Fjira.opendaylight.org%2Fbrowse%2FMDSAL-461 fix, we have a slight optimization already staged for mdsal-4.0.4, but this needs proper looking after.

Can you file an issue with mdsal?

Also, can you try with https://protect2.fireeye.com/url?k=c9282245-95a2f684-c92862de-863d9bcb726f-82981d78e3f8a352&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493 ?

I'll take a look next week, as I am going offline.

Regards,
Robert
Karthika Panneer
 

Hi Jamo,
From console log of the job, I found the distro URL which pointed out to,
https://nexus.opendaylight.org/content/repositories/autorelease-3154/org/opendaylight/integration/karaf/0.11.0/
When I tested with this distro I found that 10 min issue.
This is last updated on Aug 3 and from that is when CSIT is broken.
I am not able to find the distro before Aug3 in the above mentioned URL.

But in the URL you had pointed out,
https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.11.0-SNAPSHOT/
on Monday(when CSIT started failing), the distro when tested locally seems to be fine.


Regards,
R P Karthika

-----Original Message-----
From: Jamo Luhrsen <jluhrsen@...>
Sent: Wednesday, August 7, 2019 9:23 PM
To: Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; Robert Varga <nite@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

If you have some time, you can go try to find which actual snapshot distro started having this 10m problem. That will help isolate a small window when this started happening. knowing that will help find the culprit.

good debugging so far :)

JamO

On 8/7/19 4:26 AM, Faseela K wrote:
Thanks for the confirmation Karthika, CSIT overall only takes 20 mins, we cannot put 10 mins for each controller restart though!
But good that you have clarified the same with the job.

@Robert : The patch you have on mdsal is on verify -1. Can we get it verified, will see if we can spend some cycles to test the same.

Thanks
Faseela

-----Original Message-----
From: R P Karthika . <r.p.karthika@...>
Sent: Wednesday, August 7, 2019 4:50 PM
To: Robert Varga <nite@...>; Jamo Luhrsen <jluhrsen@...>;
Faseela K <faseela.k@...>;
'integration-dev@...'
(integration-dev@...)
(integration-dev@...)
<integration-dev@...>
Cc: odlparent-dev@...;
mdsal-dev@...
Subject: RE: [integration-dev] Karaf stop command not working in Sodium.

Hi,
When I tried with the same distro locally I could see that it takes 10 minutes for the process to get killed after stopping the controller.
To confirm the same I triggered a job by increasing the timeout to 10 mins and was able to see that the job was successful.

Success logs :
https://protect2.fireeye.com/url?k=553e533b-09b499f8-553e13a0-86e1ed40
02b1-ee38de85a0ba1099&q=1&u=https%3A%2F%2Fjenkins.opendaylight.org%2Fs
andbox%2Fjob%2FKarthikagenius-csit-1node-gate-only-sodium%2F6%2F

Regards,
R P Karthika

-----Original Message-----
From: Robert Varga <nite@...>
Sent: Tuesday, August 6, 2019 11:20 PM
To: Jamo Luhrsen <jluhrsen@...>; Faseela K
<faseela.k@...>; R P Karthika . <r.p.karthika@...>;
'integration-dev@...'
(integration-dev@...)
(integration-dev@...)
<integration-dev@...>
Cc: odlparent-dev@...;
mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

On 05/08/2019 21:42, Jamo Luhrsen wrote:
is this enough info?

thread dump before running robot:

https://protect2.fireeye.com/url?k=df2b6581-83a1b140-df2b251a-863d9bc
b
726f-2daec84e911dd818&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frel
e
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2
F 274%2Fodl_1%2Fkaraf_1_10983_threads_before.log.gz
<https://protect2.fireeye.com/url?k=d87dd00d-84f704cc-d87d9096-863d9b
c
b726f-9bd20260a15f63a1&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Fre
l
eng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%
2 F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)

thread dump after running robot:

https://protect2.fireeye.com/url?k=ab2c8fe8-f7a65b29-ab2ccf73-863d9bc
b
726f-611852abf170af25&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frel
e
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2
F 274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is creating SchemaContexts. It really should not, as we should be speedily stopping.

This is probably a side-effect of
https://protect2.fireeye.com/url?k=b64f59e5-eac58d24-b64f197e-863d9bcb726f-ae28518222c47c71&q=1&u=https%3A%2F%2Fjira.opendaylight.org%2Fbrowse%2FMDSAL-461 fix, we have a slight optimization already staged for mdsal-4.0.4, but this needs proper looking after.

Can you file an issue with mdsal?

Also, can you try with https://protect2.fireeye.com/url?k=c9282245-95a2f684-c92862de-863d9bcb726f-82981d78e3f8a352&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493 ?

I'll take a look next week, as I am going offline.

Regards,
Robert
JamO Luhrsen
 

Karthika,

some of our jobs run with autorelease distros (your first link) and
some with snapshot distros (your second link). You need to change
the autorelease URL to some number smaller than 3154 (see the
autorelease-3154 in the middle?) to get to older distros.

I'm not sure I understand exactly what we are seeing now. Does the
problem happen on both distros (autorelease and snapshot) if it's
more recent than 8/3?

If so, just use the snapshot distros and go find the two distros
that are next to each other chronologically where the older one
passes and the newer one fails. That will give us the breakage
window of a few hours.

Thanks,
JamO

On 8/8/19 1:31 AM, R P Karthika . wrote:
Hi Jamo,
From console log of the job, I found the distro URL which pointed out to,
https://nexus.opendaylight.org/content/repositories/autorelease-3154/org/opendaylight/integration/karaf/0.11.0/
When I tested with this distro I found that 10 min issue.
This is last updated on Aug 3 and from that is when CSIT is broken.
I am not able to find the distro before Aug3 in the above mentioned URL.

But in the URL you had pointed out,
https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.11.0-SNAPSHOT/
on Monday(when CSIT started failing), the distro when tested locally seems to be fine.


Regards,
R P Karthika

-----Original Message-----
From: Jamo Luhrsen <jluhrsen@...>
Sent: Wednesday, August 7, 2019 9:23 PM
To: Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; Robert Varga <nite@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

If you have some time, you can go try to find which actual snapshot distro started having this 10m problem. That will help isolate a small window when this started happening. knowing that will help find the culprit.

good debugging so far :)

JamO

On 8/7/19 4:26 AM, Faseela K wrote:
Thanks for the confirmation Karthika, CSIT overall only takes 20 mins, we cannot put 10 mins for each controller restart though!
But good that you have clarified the same with the job.

@Robert : The patch you have on mdsal is on verify -1. Can we get it verified, will see if we can spend some cycles to test the same.

Thanks
Faseela

-----Original Message-----
From: R P Karthika . <r.p.karthika@...>
Sent: Wednesday, August 7, 2019 4:50 PM
To: Robert Varga <nite@...>; Jamo Luhrsen <jluhrsen@...>;
Faseela K <faseela.k@...>;
'integration-dev@...'
(integration-dev@...)
(integration-dev@...)
<integration-dev@...>
Cc: odlparent-dev@...;
mdsal-dev@...
Subject: RE: [integration-dev] Karaf stop command not working in Sodium.

Hi,
When I tried with the same distro locally I could see that it takes 10 minutes for the process to get killed after stopping the controller.
To confirm the same I triggered a job by increasing the timeout to 10 mins and was able to see that the job was successful.

Success logs :
https://protect2.fireeye.com/url?k=553e533b-09b499f8-553e13a0-86e1ed40
02b1-ee38de85a0ba1099&q=1&u=https%3A%2F%2Fjenkins.opendaylight.org%2Fs
andbox%2Fjob%2FKarthikagenius-csit-1node-gate-only-sodium%2F6%2F

Regards,
R P Karthika

-----Original Message-----
From: Robert Varga <nite@...>
Sent: Tuesday, August 6, 2019 11:20 PM
To: Jamo Luhrsen <jluhrsen@...>; Faseela K
<faseela.k@...>; R P Karthika . <r.p.karthika@...>;
'integration-dev@...'
(integration-dev@...)
(integration-dev@...)
<integration-dev@...>
Cc: odlparent-dev@...;
mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

On 05/08/2019 21:42, Jamo Luhrsen wrote:
is this enough info?

thread dump before running robot:
https://protect2.fireeye.com/url?k=df2b6581-83a1b140-df2b251a-863d9bc
b
726f-2daec84e911dd818&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frel
e
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2
F 274%2Fodl_1%2Fkaraf_1_10983_threads_before.log.gz
<https://protect2.fireeye.com/url?k=d87dd00d-84f704cc-d87d9096-863d9b
c
b726f-9bd20260a15f63a1&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Fre
l
eng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%
2 F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)

thread dump after running robot:
https://protect2.fireeye.com/url?k=ab2c8fe8-f7a65b29-ab2ccf73-863d9bc
b
726f-611852abf170af25&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Frel
e
ng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium%2
F 274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is creating SchemaContexts. It really should not, as we should be speedily stopping.

This is probably a side-effect of
https://protect2.fireeye.com/url?k=b64f59e5-eac58d24-b64f197e-863d9bcb726f-ae28518222c47c71&q=1&u=https%3A%2F%2Fjira.opendaylight.org%2Fbrowse%2FMDSAL-461 fix, we have a slight optimization already staged for mdsal-4.0.4, but this needs proper looking after.

Can you file an issue with mdsal?

Also, can you try with https://protect2.fireeye.com/url?k=c9282245-95a2f684-c92862de-863d9bcb726f-82981d78e3f8a352&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493 ?

I'll take a look next week, as I am going offline.

Regards,
Robert
Robert Varga
 

On 06/08/2019 19:49, Robert Varga wrote:
On 05/08/2019 21:42, Jamo Luhrsen wrote:

is this enough info?

thread dump before running robot:
 
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_before.log.gz
<https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)


thread dump after running robot:
 
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/genius-csit-1node-upstream-only-sodium/274/odl_1/karaf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is
creating SchemaContexts. It really should not, as we should be speedily
stopping.

This is probably a side-effect of
https://jira.opendaylight.org/browse/MDSAL-461 fix, we have a slight
optimization already staged for mdsal-4.0.4, but this needs proper
looking after.

Can you file an issue with mdsal?
MDSAL-466


Also, can you try with https://git.opendaylight.org/gerrit/c/mdsal/+/83493 ?
Still waiting for feedback...

Regards,
Robert

?
signature.asc

Karthika Panneer
 

Hi Robert,
The latest distro generated in nexus.opendaylight.org does not have your changes.
Will update once I test with your changes.

Regards,
R P Karthika

-----Original Message-----
From: Robert Varga <nite@...>
Sent: Tuesday, August 13, 2019 2:13 AM
To: Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.



On 06/08/2019 19:49, Robert Varga wrote:
On 05/08/2019 21:42, Jamo Luhrsen wrote:

is this enough info?

thread dump before running robot:
 
https://protect2.fireeye.com/url?k=fac52b13-a64cf12e-fac56b88-0cc47ad
93c0c-0eb34c0aa6438595&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Fre
leng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium
%2F274%2Fodl_1%2Fkaraf_1_10983_threads_before.log.gz
<https://protect2.fireeye.com/url?k=aba30614-f72adc29-aba3468f-0cc47a
d93c0c-bed3b20ec1fb3a45&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Fr
eleng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodiu
m%2F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)


thread dump after running robot:
 
https://protect2.fireeye.com/url?k=ec2571ca-b0acabf7-ec253151-0cc47ad
93c0c-6bb8bb5f9d39e525&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Fre
leng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium
%2F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is
creating SchemaContexts. It really should not, as we should be
speedily stopping.

This is probably a side-effect of
https://protect2.fireeye.com/url?k=fe272630-a2aefc0d-fe2766ab-0cc47ad9
3c0c-267f8c88dc3ee479&q=1&u=https%3A%2F%2Fjira.opendaylight.org%2Fbrow
se%2FMDSAL-461 fix, we have a slight optimization already staged for mdsal-4.0.4, but this needs proper looking after.

Can you file an issue with mdsal?
MDSAL-466


Also, can you try with https://protect2.fireeye.com/url?k=42be386c-1e37e251-42be78f7-0cc47ad93c0c-e032d24cd23288cf&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493 ?
Still waiting for feedback...

Regards,
Robert
Robert Varga
 

Baseline distro will not have the change until mdsal-4.0.4 is released and integrated (just as any MRI project), hence I would suggest manual testing, as noted in a separate email in this thread beforehand.

Alternatively, we can try validating this in the integration phase, but I will not have time to spin those patches before next week.



Sent from my BlackBerry - the most secure mobile device - via the Orange Network


  Original Message  



From: r.p.karthika@...
Sent: August 13, 2019 12:33
To: nite@...; jluhrsen@...; faseela.k@...; integration-dev@...
Cc: odlparent-dev@...; mdsal-dev@...
Subject: RE: [integration-dev] Karaf stop command not working in Sodium.


Hi Robert,
The latest distro generated in nexus.opendaylight.org does not have your changes.
Will update once I test with your changes.

Regards,
R P Karthika

-----Original Message-----
From: Robert Varga <nite@...>
Sent: Tuesday, August 13, 2019 2:13 AM
To: Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.



On 06/08/2019 19:49, Robert Varga wrote:
On 05/08/2019 21:42, Jamo Luhrsen wrote:

is this enough info?

thread dump before running robot:
 
https://protect2.fireeye.com/url?k=fac52b13-a64cf12e-fac56b88-0cc47ad
93c0c-0eb34c0aa6438595&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Fre
leng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium
%2F274%2Fodl_1%2Fkaraf_1_10983_threads_before.log.gz
<https://protect2.fireeye.com/url?k=aba30614-f72adc29-aba3468f-0cc47a
d93c0c-bed3b20ec1fb3a45&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Fr
eleng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodiu
m%2F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)


thread dump after running robot:
 
https://protect2.fireeye.com/url?k=ec2571ca-b0acabf7-ec253151-0cc47ad
93c0c-6bb8bb5f9d39e525&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Fre
leng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium
%2F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is
creating SchemaContexts. It really should not, as we should be
speedily stopping.

This is probably a side-effect of
https://protect2.fireeye.com/url?k=fe272630-a2aefc0d-fe2766ab-0cc47ad9
3c0c-267f8c88dc3ee479&q=1&u=https%3A%2F%2Fjira.opendaylight.org%2Fbrow
se%2FMDSAL-461 fix, we have a slight optimization already staged for mdsal-4.0.4, but this needs proper looking after.

Can you file an issue with mdsal?
MDSAL-466


Also, can you try with https://protect2.fireeye.com/url?k=42be386c-1e37e251-42be78f7-0cc47ad93c0c-e032d24cd23288cf&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493 ?
Still waiting for feedback...

Regards,
Robert
Karthika Panneer
 

Hi Robert,
I tried testing it locally with your patch.
But still the karaf process takes time to get killed.
Now it takes approximately 5-8 mins for karaf to stop.
Attaching the karaf logs.

Regards,
R P Karthika

-----Original Message-----
From: Robert Varga <nite@...>
Sent: Tuesday, August 13, 2019 11:01 PM
To: R P Karthika . <r.p.karthika@...>; Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; 'integration-dev_lists.opendaylight.org' (integration-dev_lists.opendaylight.org) (integration-dev_lists.opendaylight.org) <integration-dev@...>
Cc: odlparent-dev_lists.opendaylight.org <odlparent-dev@...>; mdsal-dev_lists.opendaylight.org <mdsal-dev@...>
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

Baseline distro will not have the change until mdsal-4.0.4 is released and integrated (just as any MRI project), hence I would suggest manual testing, as noted in a separate email in this thread beforehand.

Alternatively, we can try validating this in the integration phase, but I will not have time to spin those patches before next week.



Sent from my BlackBerry - the most secure mobile device - via the Orange Network


  Original Message  



From: r.p.karthika@...
Sent: August 13, 2019 12:33
To: nite@...; jluhrsen@...; faseela.k@...; integration-dev@...
Cc: odlparent-dev@...; mdsal-dev@...
Subject: RE: [integration-dev] Karaf stop command not working in Sodium.


Hi Robert,
The latest distro generated in nexus.opendaylight.org does not have your changes.
Will update once I test with your changes.

Regards,
R P Karthika

-----Original Message-----
From: Robert Varga <nite@...>
Sent: Tuesday, August 13, 2019 2:13 AM
To: Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; R P Karthika . <r.p.karthika@...>; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Cc: odlparent-dev@...; mdsal-dev@...
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.



On 06/08/2019 19:49, Robert Varga wrote:
On 05/08/2019 21:42, Jamo Luhrsen wrote:

is this enough info?

thread dump before running robot:
 
https://protect2.fireeye.com/url?k=fac52b13-a64cf12e-fac56b88-0cc47ad
93c0c-0eb34c0aa6438595&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Fre
leng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium
%2F274%2Fodl_1%2Fkaraf_1_10983_threads_before.log.gz
<https://protect2.fireeye.com/url?k=aba30614-f72adc29-aba3468f-0cc47a
d93c0c-bed3b20ec1fb3a45&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Fr
eleng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodiu
m%2F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz>
^^^ your email client has betrayed you :)


thread dump after running robot:
 
https://protect2.fireeye.com/url?k=ec2571ca-b0acabf7-ec253151-0cc47ad
93c0c-6bb8bb5f9d39e525&q=1&u=https%3A%2F%2Flogs.opendaylight.org%2Fre
leng%2Fvex-yul-odl-jenkins-1%2Fgenius-csit-1node-upstream-only-sodium
%2F274%2Fodl_1%2Fkaraf_1_10983_threads_after.log.gz

I'm not really sure what to look for, but really curious.
Yeah, kinda ... since build #273 it looks like "Framework stop" is
creating SchemaContexts. It really should not, as we should be
speedily stopping.

This is probably a side-effect of
https://protect2.fireeye.com/url?k=fe272630-a2aefc0d-fe2766ab-0cc47ad9
3c0c-267f8c88dc3ee479&q=1&u=https%3A%2F%2Fjira.opendaylight.org%2Fbrow
se%2FMDSAL-461 fix, we have a slight optimization already staged for mdsal-4.0.4, but this needs proper looking after.

Can you file an issue with mdsal?
MDSAL-466


Also, can you try with https://protect2.fireeye.com/url?k=42be386c-1e37e251-42be78f7-0cc47ad93c0c-e032d24cd23288cf&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2Fc%2Fmdsal%2F%2B%2F83493 ?
Still waiting for feedback...

Regards,
Robert

?
karaf.log

Robert Varga
 

On 14/08/2019 11:01, R P Karthika . wrote:
Hi Robert,
I tried testing it locally with your patch.
But still the karaf process takes time to get killed.
Now it takes approximately 5-8 mins for karaf to stop.
Attaching the karaf logs.
Thanks for testing,
https://git.opendaylight.org/gerrit/#/c/mdsal/+/83625 on top of the
previous patch should do the trick.

Bye,
Robert

?
signature.asc

Karthika Panneer
 

Hi Robert,
This fix seems to be working fine.
Karaf is getting killed without taking much time.
Thanks!!

Regards,
R P Karthika

-----Original Message-----
From: Robert Varga <nite@...>
Sent: Wednesday, August 14, 2019 9:04 PM
To: R P Karthika . <r.p.karthika@...>; Jamo Luhrsen <jluhrsen@...>; Faseela K <faseela.k@...>; 'integration-dev_lists.opendaylight.org' (integration-dev_lists.opendaylight.org) (integration-dev_lists.opendaylight.org) <integration-dev@...>
Cc: odlparent-dev_lists.opendaylight.org <odlparent-dev@...>; mdsal-dev_lists.opendaylight.org <mdsal-dev@...>
Subject: Re: [integration-dev] Karaf stop command not working in Sodium.

On 14/08/2019 11:01, R P Karthika . wrote:
Hi Robert,
I tried testing it locally with your patch.
But still the karaf process takes time to get killed.
Now it takes approximately 5-8 mins for karaf to stop.
Attaching the karaf logs.
Thanks for testing,
https://protect2.fireeye.com/url?k=8d725d29-d1f87fe6-8d721db2-0cc47ad93ea4-e5a4975ad41ac4d7&q=1&u=https%3A%2F%2Fgit.opendaylight.org%2Fgerrit%2F%23%2Fc%2Fmdsal%2F%2B%2F83625 on top of the previous patch should do the trick.

Bye,
Robert
1 - 20 of 25