[release] [genius-dev] Genius CSIT failing on master


Luis Gomez
 

+1, this will determine whether the issue is related with the MDSAL recent change.

On Apr 20, 2018, at 8:55 AM, Faseela K <faseela.k@...> wrote:

Vishal,
   I think it will be better to trigger a sandbox job for ovsdb CSIT with and without your patch, and see how it goes.
Thanks,
Faseela
 
From: Vishal Thapar [mailto:vthapar@...] 
Sent: Friday, April 20, 2018 7:56 PM
To: Faseela K <faseela.k@...>
Cc: Sam Hague <shague@...>; Swati Udhavrao Niture <swati.udhavrao.niture@...>; K.V Suneelu Verma <k.v.suneelu.verma@...>; integration-dev@...; genius-dev@...; ovsdb-dev@...; release@...; netvirt-dev@...; tsc@...
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Hi Faseela,
 
This sounds more and more related to the IpAddress issue. Can you try the patch I have on OVSDB and see if it works?
 
Regards,
Vishal.
 
On Fri, Apr 20, 2018 at 7:18 PM, Faseela K <faseela.k@...> wrote:
Vishal/Sam,
   The current issue we are seeing is not due to bridge connect/disconnect.
   In a fresh clean setup also, tunnels are not getting configured with the DEBUG logs below.
   Do you think this can be due to the mdsal change, or something else? We don’t hit this issue in stable/oxygen.
   It is coming even in a local setup, with basic configuration.
Thanks,
Faseela
 
From: Vishal Thapar [mailto:vthapar@...] 
Sent: Friday, April 20, 2018 6:56 PM
To: Sam Hague <shague@...>
Cc: Swati Udhavrao Niture <swati.udhavrao.niture@...>; Faseela K <faseela.k@...>; K.V Suneelu Verma <k.v.suneelu.verma@...>; integration-dev@...; genius-dev@...; ovsdb-dev@...; release@...; netvirt-dev@...

Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Hi Sam,
 
Yes, we need a similar patch in OVSDB and there was a discussion on this between Suneelu and Anil. The fix in HWVTEP doesn't apply as-is to OVSDB and we don't have a perfect solution to problem. The issue is seen when switch connects to only one node in cluster. Genius CSIT hardocded to ODL1 IP and I think HWVTEP issue was with HAPRoxy between TOR and ODL. Original mail thread is at [1].
 
Regards,
Vishal.
 
 
On Fri, Apr 20, 2018 at 5:30 PM, Sam Hague <shague@...> wrote:
 
 
On Fri, Apr 20, 2018 at 3:23 AM, Vishal Thapar <vthapar@...> wrote:
I see log full of entries like this:
2018-04-20T14:33:20,490 | DEBUG | opendaylight-cluster-data-notification-dispatcher-56 | OvsdbDataTreeChangeListener      | 418 - org.opendaylight.ovsdb.southbound-impl - 1.7.0.SNAPSHOT | Did not find client for Node{getNodeId=Uri{_value=ovsdb://uuid/9fbcf6cc-e6e3-463b-8993-2a382963fb2b/bridge/br-int}, getTerminationPoint=[TerminationPoint{getTpId=Uri{_value=tunb1071e60d72}, augmentations={interface 
 
The upstream OVSBD CSIT doesn't have any exceptions, and the logging isn't at debug level so we won't see the above logs. I am pushing a patch so we can enable debugs there and have it also for troubleshooting.
This may or may not be related to IpAddress issue in yangtools as connectionInfo uses Ipaddress.equals(). This could also be the old issue where stale connection hangs around. To rule the latter out, can we run only the ITM suite once? Or try one test case manually? Stale connection issue was caused by quick delete/add/delete of OVS connection. If we run the test case that doesn't do delete/add, which is current;y done at end of every suite, we'll come to know if it is yangtools/mdsal API issue or not.
 
If it is stale conneciton issue, share the logs with Suneelu, who can confirm if his proposed fix for OVSDB can address this, or just get him to share his WiP patch.
Vishal, as an aside, the connection code is different for ovsdb and hwvtep, so we want to use the improvements from hwvtep also in ovsdb?
 
Regards,
Vishal.
 
On Fri, Apr 20, 2018 at 2:46 PM, Swati Udhavrao Niture <swati.udhavrao.niture@...> wrote:
Hi Faseela,
 
PFA logs.
 
Thanks,
Swati.
 
From: Faseela K 
Sent: Friday, April 20, 2018 12:05 PM
To: Swati Udhavrao Niture; Sam Hague; Vishal Thapar; K.V Suneelu Verma

Cc: integration-dev@...; genius-dev@...; ovsdb-dev@...; release@...; netvirt-dev@...
Subject: RE: [genius-dev] [release] Genius CSIT failing on master
 
Swati,
   Please pull the latest stable/oxygen, tunnel down issue is already resolved.
   Master has some issues with ovsdb plugin, which we are still debugging, and no handle yet.
   If you are working on latest master, could you please enable TRACE logs for org.opendaylight.ovsdb.southbound and push tunnel configuration and supply the logs?
Thanks,
Faseela
 
From: Swati Udhavrao Niture 
Sent: Friday, April 20, 2018 12:03 PM
To: Faseela K <faseela.k@...>; Sam Hague <shague@...>; Vishal Thapar <vthapar@...>; K.V Suneelu Verma <k.v.suneelu.verma@...>
Cc: integration-dev@...; genius-dev@...; ovsdb-dev@...; release@...; netvirt-dev@...
Subject: RE: [genius-dev] [release] Genius CSIT failing on master
 
Hi all,
 
Tunnels are coming up on the switch now, but I have to restart my OVS every-time after I send a request to bring up tunnels.
Has anybody else faced the same issue?
 
Adding to this, in stable-oxygen, tunnels are coming up on the switch, but I see oper_status as down.
Any pointers on this?
 
Thanks,
Swati.
 
From: genius-dev-bounces@... [mailto:genius-dev-bounces@...] On Behalf Of Faseela K
Sent: Thursday, April 19, 2018 1:35 PM
To: Sam Hague; Vishal Thapar; K.V Suneelu Verma
Cc: integration-dev@...; genius-dev@...; ovsdb-dev@...; release@...
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Yes, it is master.
I hope the maven issues are all resolved now, but we are still hitting the same error.
(karaf SSH issues are gone now, the ovsdb plugin issue still exists)
 
 
Thanks,
Faseela
 
From: Sam Hague [mailto:shague@...] 
Sent: Wednesday, April 18, 2018 6:47 PM
To: Vishal Thapar <vthapar@...>
Cc: Faseela K <faseela.k@...>; genius-dev@...; integration-dev@...; ovsdb-dev@...;release@...
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
If this is master, then likely this is related to the maven issues which is creating bad artifacts so odl is in a bad state. That is being worked now.
 
On Wed, Apr 18, 2018, 8:51 AM Vishal Thapar <vthapar@...> wrote:
No, there hasn't been a functional change in OVSDB plugin for more than 2 releases now. There is a bug with stale connection cleanup on OVS restart, could be related to that. Suneelu had a similar issue and fix in HWVTEP. Can check with him if this is same issue. Josh is also running into some variation of same issue and I think Sathwick was at some point trying to capture some logs for Suneelu too.
 
Regards,
Vishal.
 
On Wed, Apr 18, 2018 at 4:24 PM, Faseela K <faseela.k@...> wrote:
+ovsdb-dev
 
Vishal,
  Since distribution-checks are randomly failing, don’t know whether the tests are running properly or not.
  But we are hitting an issue where tunnels are not coming up on the switch, even though the tunnel information is present in config/network-topology. Is there any recent change in ovsdb plugin?
Thanks,
Faseela
 
From: genius-dev-bounces@... [mailto:genius-dev-bounces@...] On Behalf Of B Sathwik
Sent: Wednesday, April 18, 2018 3:48 PM
To: release@...; integration-dev@...
Cc: genius-dev@...
Subject: [genius-dev] Genius CSIT failing on master
 
Hi
 
Genius CSIT is Failing in master branch with some basic issues, Sometimes not able to login into karaf and sometime tunnels are not shown up on bridge side.
 
Since there are several distribution-check failures also seen intermittently on patches, wanted to know whether we have a stable master distribution, or
We should wait to run CSIT later on.
 
Here are the logs of Genius-CSIT:
 

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/sathwikgenius-csit-1node-gate-all-fluorine/4/robot-plugin/log.html.gz

 

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/sathwikgenius-csit-3node-gate-all-fluorine/3/robot-plugin/log.html.gz

 
 
 
Best Regards
Sathwik
 


_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release

 
_______________________________________________
genius-dev mailing list
genius-dev@...
https://lists.opendaylight.org/mailman/listinfo/genius-dev
 
 
 
 
_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release


Vishal Thapar <vthapar@...>
 

Without my patch compilaiton is failing [whole reason for the patch], so CSIT won't run. Ihad run the CSIT earlier, but ran into infra issues with space. Started it again.

Regards,
Vishal.

On Fri, Apr 20, 2018 at 9:31 PM, Luis Gomez <ecelgp@...> wrote:
+1, this will determine whether the issue is related with the MDSAL recent change.

On Apr 20, 2018, at 8:55 AM, Faseela K <faseela.k@...> wrote:

Vishal,
   I think it will be better to trigger a sandbox job for ovsdb CSIT with and without your patch, and see how it goes.
Thanks,
Faseela
 
From: Vishal Thapar [mailto:vthapar@...] 
Sent: Friday, April 20, 2018 7:56 PM
To: Faseela K <faseela.k@...>
Cc: Sam Hague <shague@...>; Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com>; K.V Suneelu Verma <k.v.suneelu.verma@ericsson.com>; integration-dev@lists.opendaylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@...; release@...; netvirt-dev@lists.opendaylight.org; tsc@lists.opendaylight.org
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Hi Faseela,
 
This sounds more and more related to the IpAddress issue. Can you try the patch I have on OVSDB and see if it works?
 
Regards,
Vishal.
 
On Fri, Apr 20, 2018 at 7:18 PM, Faseela K <faseela.k@...> wrote:
Vishal/Sam,
   The current issue we are seeing is not due to bridge connect/disconnect.
   In a fresh clean setup also, tunnels are not getting configured with the DEBUG logs below.
   Do you think this can be due to the mdsal change, or something else? We don’t hit this issue in stable/oxygen.
   It is coming even in a local setup, with basic configuration.
Thanks,
Faseela
 
From: Vishal Thapar [mailto:vthapar@...] 
Sent: Friday, April 20, 2018 6:56 PM
To: Sam Hague <shague@...>
Cc: Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com>; Faseela K <faseela.k@...>; K.V Suneelu Verma <k.v.suneelu.verma@ericsson.com>; integration-dev@lists.opendaylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@...; release@...; netvirt-dev@lists.opendaylight.org

Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Hi Sam,
 
Yes, we need a similar patch in OVSDB and there was a discussion on this between Suneelu and Anil. The fix in HWVTEP doesn't apply as-is to OVSDB and we don't have a perfect solution to problem. The issue is seen when switch connects to only one node in cluster. Genius CSIT hardocded to ODL1 IP and I think HWVTEP issue was with HAPRoxy between TOR and ODL. Original mail thread is at [1].
 
Regards,
Vishal.
 
 
On Fri, Apr 20, 2018 at 5:30 PM, Sam Hague <shague@...> wrote:
 
 
On Fri, Apr 20, 2018 at 3:23 AM, Vishal Thapar <vthapar@...> wrote:
I see log full of entries like this:
2018-04-20T14:33:20,490 | DEBUG | opendaylight-cluster-data-notification-dispatcher-56 | OvsdbDataTreeChangeListener      | 418 - org.opendaylight.ovsdb.southbound-impl - 1.7.0.SNAPSHOT | Did not find client for Node{getNodeId=Uri{_value=ovsdb://uuid/9fbcf6cc-e6e3-463b-8993-2a382963fb2b/bridge/br-int}, getTerminationPoint=[TerminationPoint{getTpId=Uri{_value=tunb1071e60d72}, augmentations={interface 
 
The upstream OVSBD CSIT doesn't have any exceptions, and the logging isn't at debug level so we won't see the above logs. I am pushing a patch so we can enable debugs there and have it also for troubleshooting.
This may or may not be related to IpAddress issue in yangtools as connectionInfo uses Ipaddress.equals(). This could also be the old issue where stale connection hangs around. To rule the latter out, can we run only the ITM suite once? Or try one test case manually? Stale connection issue was caused by quick delete/add/delete of OVS connection. If we run the test case that doesn't do delete/add, which is current;y done at end of every suite, we'll come to know if it is yangtools/mdsal API issue or not.
 
If it is stale conneciton issue, share the logs with Suneelu, who can confirm if his proposed fix for OVSDB can address this, or just get him to share his WiP patch.
Vishal, as an aside, the connection code is different for ovsdb and hwvtep, so we want to use the improvements from hwvtep also in ovsdb?
 
Regards,
Vishal.
 
On Fri, Apr 20, 2018 at 2:46 PM, Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com> wrote:
Hi Faseela,
 
PFA logs.
 
Thanks,
Swati.
 
From: Faseela K 
Sent: Friday, April 20, 2018 12:05 PM
To: Swati Udhavrao Niture; Sam Hague; Vishal Thapar; K.V Suneelu Verma
 
Swati,
   Please pull the latest stable/oxygen, tunnel down issue is already resolved.
   Master has some issues with ovsdb plugin, which we are still debugging, and no handle yet.
   If you are working on latest master, could you please enable TRACE logs for org.opendaylight.ovsdb.southbound and push tunnel configuration and supply the logs?
Thanks,
Faseela
 
From: Swati Udhavrao Niture 
Sent: Friday, April 20, 2018 12:03 PM
To: Faseela K <faseela.k@...>; Sam Hague <shague@...>; Vishal Thapar <vthapar@...>; K.V Suneelu Verma <k.v.suneelu.verma@ericsson.com>
Cc: integration-dev@lists.opendaylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@...; release@...; netvirt-dev@lists.opendaylight.org
Subject: RE: [genius-dev] [release] Genius CSIT failing on master
 
Hi all,
 
Tunnels are coming up on the switch now, but I have to restart my OVS every-time after I send a request to bring up tunnels.
Has anybody else faced the same issue?
 
Adding to this, in stable-oxygen, tunnels are coming up on the switch, but I see oper_status as down.
Any pointers on this?
 
Thanks,
Swati.
 
From: genius-dev-bounces@lists.opendaylight.org [mailto:genius-dev-bounces@lists.opendaylight.org] On Behalf Of Faseela K
Sent: Thursday, April 19, 2018 1:35 PM
To: Sam Hague; Vishal Thapar; K.V Suneelu Verma
Cc: integration-dev@lists.opendaylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@...; release@...
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Yes, it is master.
I hope the maven issues are all resolved now, but we are still hitting the same error.
(karaf SSH issues are gone now, the ovsdb plugin issue still exists)
 
 
Thanks,
Faseela
 
From: Sam Hague [mailto:shague@...] 
Sent: Wednesday, April 18, 2018 6:47 PM
To: Vishal Thapar <vthapar@...>
Cc: Faseela K <faseela.k@...>; genius-dev@...; integration-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org;release@lists.opendaylight.org
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
If this is master, then likely this is related to the maven issues which is creating bad artifacts so odl is in a bad state. That is being worked now.
 
On Wed, Apr 18, 2018, 8:51 AM Vishal Thapar <vthapar@...> wrote:
No, there hasn't been a functional change in OVSDB plugin for more than 2 releases now. There is a bug with stale connection cleanup on OVS restart, could be related to that. Suneelu had a similar issue and fix in HWVTEP. Can check with him if this is same issue. Josh is also running into some variation of same issue and I think Sathwick was at some point trying to capture some logs for Suneelu too.
 
Regards,
Vishal.
 
On Wed, Apr 18, 2018 at 4:24 PM, Faseela K <faseela.k@...> wrote:
+ovsdb-dev
 
Vishal,
  Since distribution-checks are randomly failing, don’t know whether the tests are running properly or not.
  But we are hitting an issue where tunnels are not coming up on the switch, even though the tunnel information is present in config/network-topology. Is there any recent change in ovsdb plugin?
Thanks,
Faseela
 
From: genius-dev-bounces@lists.opendaylight.org [mailto:genius-dev-bounces@lists.opendaylight.org] On Behalf Of B Sathwik
Sent: Wednesday, April 18, 2018 3:48 PM
To: release@lists.opendaylight.org; integration-dev@...
Cc: genius-dev@lists.opendaylight.org
Subject: [genius-dev] Genius CSIT failing on master
 
Hi
 
Genius CSIT is Failing in master branch with some basic issues, Sometimes not able to login into karaf and sometime tunnels are not shown up on bridge side.
 
Since there are several distribution-check failures also seen intermittently on patches, wanted to know whether we have a stable master distribution, or
We should wait to run CSIT later on.
 
Here are the logs of Genius-CSIT:
 

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/sathwikgenius-csit-1node-gate-all-fluorine/4/robot-plugin/log.html.gz

 

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/sathwikgenius-csit-3node-gate-all-fluorine/3/robot-plugin/log.html.gz

 
 
 
Best Regards
Sathwik
 


_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release

 
_______________________________________________
genius-dev mailing list
genius-dev@....org
https://lists.opendaylight.org/mailman/listinfo/genius-dev
 
 
 
 
_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release



Sam Hague
 



On Fri, Apr 20, 2018 at 9:46 AM, Vishal Thapar <vthapar@...> wrote:
Without my patch compilaiton is failing [whole reason for the patch], so CSIT won't run. Ihad run the CSIT earlier, but ran into infra issues with space. Started it again.
Vishal, how are you running csit? in your patch the csit failed the first time because the bundle_url was never set, or really was set to "to", so nothing is uploaded to nexus. Later when the csit runs, it doesn't have a distro so it has nothing to use. Maybe that is what you thought was a space issue. Luis is taking a look at the bundle_url problem as he thinks it has to do with the maven change.

Regards,
Vishal.

On Fri, Apr 20, 2018 at 9:31 PM, Luis Gomez <ecelgp@...> wrote:
+1, this will determine whether the issue is related with the MDSAL recent change.

On Apr 20, 2018, at 8:55 AM, Faseela K <faseela.k@...> wrote:

Vishal,
   I think it will be better to trigger a sandbox job for ovsdb CSIT with and without your patch, and see how it goes.
Thanks,
Faseela
 
From: Vishal Thapar [mailto:vthapar@...] 
Sent: Friday, April 20, 2018 7:56 PM
To: Faseela K <faseela.k@...>
Cc: Sam Hague <shague@...>; Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com>; K.V Suneelu Verma <k.v.suneelu.verma@...m>; integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org; netvirt-dev@...; tsc@...
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Hi Faseela,
 
This sounds more and more related to the IpAddress issue. Can you try the patch I have on OVSDB and see if it works?
 
Regards,
Vishal.
 
On Fri, Apr 20, 2018 at 7:18 PM, Faseela K <faseela.k@...> wrote:
Vishal/Sam,
   The current issue we are seeing is not due to bridge connect/disconnect.
   In a fresh clean setup also, tunnels are not getting configured with the DEBUG logs below.
   Do you think this can be due to the mdsal change, or something else? We don’t hit this issue in stable/oxygen.
   It is coming even in a local setup, with basic configuration.
Thanks,
Faseela
 
From: Vishal Thapar [mailto:vthapar@...] 
Sent: Friday, April 20, 2018 6:56 PM
To: Sam Hague <shague@...>
Cc: Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com>; Faseela K <faseela.k@...>; K.V Suneelu Verma <k.v.suneelu.verma@...m>; integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org; netvirt-dev@...

Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Hi Sam,
 
Yes, we need a similar patch in OVSDB and there was a discussion on this between Suneelu and Anil. The fix in HWVTEP doesn't apply as-is to OVSDB and we don't have a perfect solution to problem. The issue is seen when switch connects to only one node in cluster. Genius CSIT hardocded to ODL1 IP and I think HWVTEP issue was with HAPRoxy between TOR and ODL. Original mail thread is at [1].
 
Regards,
Vishal.
 
 
On Fri, Apr 20, 2018 at 5:30 PM, Sam Hague <shague@...> wrote:
 
 
On Fri, Apr 20, 2018 at 3:23 AM, Vishal Thapar <vthapar@...> wrote:
I see log full of entries like this:
2018-04-20T14:33:20,490 | DEBUG | opendaylight-cluster-data-notification-dispatcher-56 | OvsdbDataTreeChangeListener      | 418 - org.opendaylight.ovsdb.southbound-impl - 1.7.0.SNAPSHOT | Did not find client for Node{getNodeId=Uri{_value=ovsdb://uuid/9fbcf6cc-e6e3-463b-8993-2a382963fb2b/bridge/br-int}, getTerminationPoint=[TerminationPoint{getTpId=Uri{_value=tunb1071e60d72}, augmentations={interface 
 
The upstream OVSBD CSIT doesn't have any exceptions, and the logging isn't at debug level so we won't see the above logs. I am pushing a patch so we can enable debugs there and have it also for troubleshooting.
This may or may not be related to IpAddress issue in yangtools as connectionInfo uses Ipaddress.equals(). This could also be the old issue where stale connection hangs around. To rule the latter out, can we run only the ITM suite once? Or try one test case manually? Stale connection issue was caused by quick delete/add/delete of OVS connection. If we run the test case that doesn't do delete/add, which is current;y done at end of every suite, we'll come to know if it is yangtools/mdsal API issue or not.
 
If it is stale conneciton issue, share the logs with Suneelu, who can confirm if his proposed fix for OVSDB can address this, or just get him to share his WiP patch.
Vishal, as an aside, the connection code is different for ovsdb and hwvtep, so we want to use the improvements from hwvtep also in ovsdb?
 
Regards,
Vishal.
 
On Fri, Apr 20, 2018 at 2:46 PM, Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com> wrote:
Hi Faseela,
 
PFA logs.
 
Thanks,
Swati.
 
From: Faseela K 
Sent: Friday, April 20, 2018 12:05 PM
To: Swati Udhavrao Niture; Sam Hague; Vishal Thapar; K.V Suneelu Verma
 
Swati,
   Please pull the latest stable/oxygen, tunnel down issue is already resolved.
   Master has some issues with ovsdb plugin, which we are still debugging, and no handle yet.
   If you are working on latest master, could you please enable TRACE logs for org.opendaylight.ovsdb.southbound and push tunnel configuration and supply the logs?
Thanks,
Faseela
 
From: Swati Udhavrao Niture 
Sent: Friday, April 20, 2018 12:03 PM
To: Faseela K <faseela.k@...>; Sam Hague <shague@...>; Vishal Thapar <vthapar@...>; K.V Suneelu Verma <k.v.suneelu.verma@...m>
Cc: integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org; netvirt-dev@...
Subject: RE: [genius-dev] [release] Genius CSIT failing on master
 
Hi all,
 
Tunnels are coming up on the switch now, but I have to restart my OVS every-time after I send a request to bring up tunnels.
Has anybody else faced the same issue?
 
Adding to this, in stable-oxygen, tunnels are coming up on the switch, but I see oper_status as down.
Any pointers on this?
 
Thanks,
Swati.
 
From: genius-dev-bounces@lists.opendaylight.org [mailto:genius-dev-bounces@...ght.org] On Behalf Of Faseela K
Sent: Thursday, April 19, 2018 1:35 PM
To: Sam Hague; Vishal Thapar; K.V Suneelu Verma
Cc: integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Yes, it is master.
I hope the maven issues are all resolved now, but we are still hitting the same error.
(karaf SSH issues are gone now, the ovsdb plugin issue still exists)
 
 
Thanks,
Faseela
 
From: Sam Hague [mailto:shague@...] 
Sent: Wednesday, April 18, 2018 6:47 PM
To: Vishal Thapar <vthapar@...>
Cc: Faseela K <faseela.k@...>; genius-dev@...; integration-dev@...light.org; ovsdb-dev@...ndaylight.org;release@lists.opendaylight.org
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
If this is master, then likely this is related to the maven issues which is creating bad artifacts so odl is in a bad state. That is being worked now.
 
On Wed, Apr 18, 2018, 8:51 AM Vishal Thapar <vthapar@...> wrote:
No, there hasn't been a functional change in OVSDB plugin for more than 2 releases now. There is a bug with stale connection cleanup on OVS restart, could be related to that. Suneelu had a similar issue and fix in HWVTEP. Can check with him if this is same issue. Josh is also running into some variation of same issue and I think Sathwick was at some point trying to capture some logs for Suneelu too.
 
Regards,
Vishal.
 
On Wed, Apr 18, 2018 at 4:24 PM, Faseela K <faseela.k@...> wrote:
+ovsdb-dev
 
Vishal,
  Since distribution-checks are randomly failing, don’t know whether the tests are running properly or not.
  But we are hitting an issue where tunnels are not coming up on the switch, even though the tunnel information is present in config/network-topology. Is there any recent change in ovsdb plugin?
Thanks,
Faseela
 
From: genius-dev-bounces@lists.opendaylight.org [mailto:genius-dev-bounces@...ght.org] On Behalf Of B Sathwik
Sent: Wednesday, April 18, 2018 3:48 PM
To: release@....org; integration-dev@...endaylight.org
Cc: genius-dev@...ght.org
Subject: [genius-dev] Genius CSIT failing on master
 
Hi
 
Genius CSIT is Failing in master branch with some basic issues, Sometimes not able to login into karaf and sometime tunnels are not shown up on bridge side.
 
Since there are several distribution-check failures also seen intermittently on patches, wanted to know whether we have a stable master distribution, or
We should wait to run CSIT later on.
 
Here are the logs of Genius-CSIT:
 

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/sathwikgenius-csit-1node-gate-all-fluorine/4/robot-plugin/log.html.gz

 

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/sathwikgenius-csit-3node-gate-all-fluorine/3/robot-plugin/log.html.gz

 
 
 
Best Regards
Sathwik
 


_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release

 
_______________________________________________
genius-dev mailing list
genius-dev@....org
https://lists.opendaylight.org/mailman/listinfo/genius-dev
 
 
 
 
_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release



_______________________________________________
ovsdb-dev mailing list
ovsdb-dev@...rg
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev



Luis Gomez
 

OK, I am looking at this now, will update shortly.


On Apr 20, 2018, at 10:20 AM, Sam Hague <shague@...> wrote:



On Fri, Apr 20, 2018 at 9:46 AM, Vishal Thapar <vthapar@...> wrote:
Without my patch compilaiton is failing [whole reason for the patch], so CSIT won't run. Ihad run the CSIT earlier, but ran into infra issues with space. Started it again.
Vishal, how are you running csit? in your patch the csit failed the first time because the bundle_url was never set, or really was set to "to", so nothing is uploaded to nexus. Later when the csit runs, it doesn't have a distro so it has nothing to use. Maybe that is what you thought was a space issue. Luis is taking a look at the bundle_url problem as he thinks it has to do with the maven change.

Regards,
Vishal.

On Fri, Apr 20, 2018 at 9:31 PM, Luis Gomez <ecelgp@...> wrote:
+1, this will determine whether the issue is related with the MDSAL recent change.

On Apr 20, 2018, at 8:55 AM, Faseela K <faseela.k@...> wrote:

Vishal,
   I think it will be better to trigger a sandbox job for ovsdb CSIT with and without your patch, and see how it goes.
Thanks,
Faseela
 
From: Vishal Thapar [mailto:vthapar@...] 
Sent: Friday, April 20, 2018 7:56 PM
To: Faseela K <faseela.k@...>
Cc: Sam Hague <shague@...>; Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com>; K.V Suneelu Verma <k.v.suneelu.verma@...m>; integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org; netvirt-dev@...; tsc@...
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Hi Faseela,
 
This sounds more and more related to the IpAddress issue. Can you try the patch I have on OVSDB and see if it works?
 
Regards,
Vishal.
 
On Fri, Apr 20, 2018 at 7:18 PM, Faseela K <faseela.k@...> wrote:
Vishal/Sam,
   The current issue we are seeing is not due to bridge connect/disconnect.
   In a fresh clean setup also, tunnels are not getting configured with the DEBUG logs below.
   Do you think this can be due to the mdsal change, or something else? We don’t hit this issue in stable/oxygen.
   It is coming even in a local setup, with basic configuration.
Thanks,
Faseela
 
From: Vishal Thapar [mailto:vthapar@...] 
Sent: Friday, April 20, 2018 6:56 PM
To: Sam Hague <shague@...>
Cc: Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com>; Faseela K <faseela.k@...>; K.V Suneelu Verma <k.v.suneelu.verma@...m>; integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org; netvirt-dev@...

Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Hi Sam,
 
Yes, we need a similar patch in OVSDB and there was a discussion on this between Suneelu and Anil. The fix in HWVTEP doesn't apply as-is to OVSDB and we don't have a perfect solution to problem. The issue is seen when switch connects to only one node in cluster. Genius CSIT hardocded to ODL1 IP and I think HWVTEP issue was with HAPRoxy between TOR and ODL. Original mail thread is at [1].
 
Regards,
Vishal.
 
 
On Fri, Apr 20, 2018 at 5:30 PM, Sam Hague <shague@...> wrote:
 
 
On Fri, Apr 20, 2018 at 3:23 AM, Vishal Thapar <vthapar@...> wrote:
I see log full of entries like this:
2018-04-20T14:33:20,490 | DEBUG | opendaylight-cluster-data-notification-dispatcher-56 | OvsdbDataTreeChangeListener      | 418 - org.opendaylight.ovsdb.southbound-impl - 1.7.0.SNAPSHOT | Did not find client for Node{getNodeId=Uri{_value=ovsdb://uuid/9fbcf6cc-e6e3-463b-8993-2a382963fb2b/bridge/br-int}, getTerminationPoint=[TerminationPoint{getTpId=Uri{_value=tunb1071e60d72}, augmentations={interface 
 
The upstream OVSBD CSIT doesn't have any exceptions, and the logging isn't at debug level so we won't see the above logs. I am pushing a patch so we can enable debugs there and have it also for troubleshooting.
This may or may not be related to IpAddress issue in yangtools as connectionInfo uses Ipaddress.equals(). This could also be the old issue where stale connection hangs around. To rule the latter out, can we run only the ITM suite once? Or try one test case manually? Stale connection issue was caused by quick delete/add/delete of OVS connection. If we run the test case that doesn't do delete/add, which is current;y done at end of every suite, we'll come to know if it is yangtools/mdsal API issue or not.
 
If it is stale conneciton issue, share the logs with Suneelu, who can confirm if his proposed fix for OVSDB can address this, or just get him to share his WiP patch.
Vishal, as an aside, the connection code is different for ovsdb and hwvtep, so we want to use the improvements from hwvtep also in ovsdb?
 
Regards,
Vishal.
 
On Fri, Apr 20, 2018 at 2:46 PM, Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com> wrote:
Hi Faseela,
 
PFA logs.
 
Thanks,
Swati.
 
From: Faseela K 
Sent: Friday, April 20, 2018 12:05 PM
To: Swati Udhavrao Niture; Sam Hague; Vishal Thapar; K.V Suneelu Verma
 
Swati,
   Please pull the latest stable/oxygen, tunnel down issue is already resolved.
   Master has some issues with ovsdb plugin, which we are still debugging, and no handle yet.
   If you are working on latest master, could you please enable TRACE logs for org.opendaylight.ovsdb.southbound and push tunnel configuration and supply the logs?
Thanks,
Faseela
 
From: Swati Udhavrao Niture 
Sent: Friday, April 20, 2018 12:03 PM
To: Faseela K <faseela.k@...>; Sam Hague <shague@...>; Vishal Thapar <vthapar@...>; K.V Suneelu Verma <k.v.suneelu.verma@...m>
Cc: integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org; netvirt-dev@...
Subject: RE: [genius-dev] [release] Genius CSIT failing on master
 
Hi all,
 
Tunnels are coming up on the switch now, but I have to restart my OVS every-time after I send a request to bring up tunnels.
Has anybody else faced the same issue?
 
Adding to this, in stable-oxygen, tunnels are coming up on the switch, but I see oper_status as down.
Any pointers on this?
 
Thanks,
Swati.
 
From: genius-dev-bounces@lists.opendaylight.org [mailto:genius-dev-bounces@...ght.org] On Behalf Of Faseela K
Sent: Thursday, April 19, 2018 1:35 PM
To: Sam Hague; Vishal Thapar; K.V Suneelu Verma
Cc: integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Yes, it is master.
I hope the maven issues are all resolved now, but we are still hitting the same error.
(karaf SSH issues are gone now, the ovsdb plugin issue still exists)
 
 
Thanks,
Faseela
 
From: Sam Hague [mailto:shague@...] 
Sent: Wednesday, April 18, 2018 6:47 PM
To: Vishal Thapar <vthapar@...>
Cc: Faseela K <faseela.k@...>; genius-dev@...; integration-dev@...light.org; ovsdb-dev@...ndaylight.org;release@lists.opendaylight.org
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
If this is master, then likely this is related to the maven issues which is creating bad artifacts so odl is in a bad state. That is being worked now.
 
On Wed, Apr 18, 2018, 8:51 AM Vishal Thapar <vthapar@...> wrote:
No, there hasn't been a functional change in OVSDB plugin for more than 2 releases now. There is a bug with stale connection cleanup on OVS restart, could be related to that. Suneelu had a similar issue and fix in HWVTEP. Can check with him if this is same issue. Josh is also running into some variation of same issue and I think Sathwick was at some point trying to capture some logs for Suneelu too.
 
Regards,
Vishal.
 
On Wed, Apr 18, 2018 at 4:24 PM, Faseela K <faseela.k@...> wrote:
+ovsdb-dev
 
Vishal,
  Since distribution-checks are randomly failing, don’t know whether the tests are running properly or not.
  But we are hitting an issue where tunnels are not coming up on the switch, even though the tunnel information is present in config/network-topology. Is there any recent change in ovsdb plugin?
Thanks,
Faseela
 
From: genius-dev-bounces@lists.opendaylight.org [mailto:genius-dev-bounces@...ght.org] On Behalf Of B Sathwik
Sent: Wednesday, April 18, 2018 3:48 PM
To: release@....org; integration-dev@...endaylight.org
Cc: genius-dev@...ght.org
Subject: [genius-dev] Genius CSIT failing on master
 
Hi
 
Genius CSIT is Failing in master branch with some basic issues, Sometimes not able to login into karaf and sometime tunnels are not shown up on bridge side.
 
Since there are several distribution-check failures also seen intermittently on patches, wanted to know whether we have a stable master distribution, or
We should wait to run CSIT later on.
 
Here are the logs of Genius-CSIT:
 

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/sathwikgenius-csit-1node-gate-all-fluorine/4/robot-plugin/log.html.gz

 

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/sathwikgenius-csit-3node-gate-all-fluorine/3/robot-plugin/log.html.gz

 
 
 
Best Regards
Sathwik
 


_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release

 
_______________________________________________
genius-dev mailing list
genius-dev@....org
https://lists.opendaylight.org/mailman/listinfo/genius-dev
 
 
 
 
_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release



_______________________________________________
ovsdb-dev mailing list
ovsdb-dev@...rg
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev


Luis Gomez
 

No way to verify in sandbox but this should fix: https://git.opendaylight.org/gerrit/#/c/71180

On Apr 20, 2018, at 11:17 AM, Luis Gomez <ecelgp@...> wrote:

OK, I am looking at this now, will update shortly.


On Apr 20, 2018, at 10:20 AM, Sam Hague <shague@...> wrote:



On Fri, Apr 20, 2018 at 9:46 AM, Vishal Thapar <vthapar@...> wrote:
Without my patch compilaiton is failing [whole reason for the patch], so CSIT won't run. Ihad run the CSIT earlier, but ran into infra issues with space. Started it again.
Vishal, how are you running csit? in your patch the csit failed the first time because the bundle_url was never set, or really was set to "to", so nothing is uploaded to nexus. Later when the csit runs, it doesn't have a distro so it has nothing to use. Maybe that is what you thought was a space issue. Luis is taking a look at the bundle_url problem as he thinks it has to do with the maven change.

Regards,
Vishal.

On Fri, Apr 20, 2018 at 9:31 PM, Luis Gomez <ecelgp@...> wrote:
+1, this will determine whether the issue is related with the MDSAL recent change.

On Apr 20, 2018, at 8:55 AM, Faseela K <faseela.k@...> wrote:

Vishal,
   I think it will be better to trigger a sandbox job for ovsdb CSIT with and without your patch, and see how it goes.
Thanks,
Faseela
 
From: Vishal Thapar [mailto:vthapar@...] 
Sent: Friday, April 20, 2018 7:56 PM
To: Faseela K <faseela.k@...>
Cc: Sam Hague <shague@...>; Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com>; K.V Suneelu Verma <k.v.suneelu.verma@...m>; integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org; netvirt-dev@...; tsc@...
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Hi Faseela,
 
This sounds more and more related to the IpAddress issue. Can you try the patch I have on OVSDB and see if it works?
 
Regards,
Vishal.
 
On Fri, Apr 20, 2018 at 7:18 PM, Faseela K <faseela.k@...> wrote:
Vishal/Sam,
   The current issue we are seeing is not due to bridge connect/disconnect.
   In a fresh clean setup also, tunnels are not getting configured with the DEBUG logs below.
   Do you think this can be due to the mdsal change, or something else? We don’t hit this issue in stable/oxygen.
   It is coming even in a local setup, with basic configuration.
Thanks,
Faseela
 
From: Vishal Thapar [mailto:vthapar@...] 
Sent: Friday, April 20, 2018 6:56 PM
To: Sam Hague <shague@...>
Cc: Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com>; Faseela K <faseela.k@...>; K.V Suneelu Verma <k.v.suneelu.verma@...m>; integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org; netvirt-dev@...

Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Hi Sam,
 
Yes, we need a similar patch in OVSDB and there was a discussion on this between Suneelu and Anil. The fix in HWVTEP doesn't apply as-is to OVSDB and we don't have a perfect solution to problem. The issue is seen when switch connects to only one node in cluster. Genius CSIT hardocded to ODL1 IP and I think HWVTEP issue was with HAPRoxy between TOR and ODL. Original mail thread is at [1].
 
Regards,
Vishal.
 
 
On Fri, Apr 20, 2018 at 5:30 PM, Sam Hague <shague@...> wrote:
 
 
On Fri, Apr 20, 2018 at 3:23 AM, Vishal Thapar <vthapar@...> wrote:
I see log full of entries like this:
2018-04-20T14:33:20,490 | DEBUG | opendaylight-cluster-data-notification-dispatcher-56 | OvsdbDataTreeChangeListener      | 418 - org.opendaylight.ovsdb.southbound-impl - 1.7.0.SNAPSHOT | Did not find client for Node{getNodeId=Uri{_value=ovsdb://uuid/9fbcf6cc-e6e3-463b-8993-2a382963fb2b/bridge/br-int}, getTerminationPoint=[TerminationPoint{getTpId=Uri{_value=tunb1071e60d72}, augmentations={interface 
 
The upstream OVSBD CSIT doesn't have any exceptions, and the logging isn't at debug level so we won't see the above logs. I am pushing a patch so we can enable debugs there and have it also for troubleshooting.
This may or may not be related to IpAddress issue in yangtools as connectionInfo uses Ipaddress.equals(). This could also be the old issue where stale connection hangs around. To rule the latter out, can we run only the ITM suite once? Or try one test case manually? Stale connection issue was caused by quick delete/add/delete of OVS connection. If we run the test case that doesn't do delete/add, which is current;y done at end of every suite, we'll come to know if it is yangtools/mdsal API issue or not.
 
If it is stale conneciton issue, share the logs with Suneelu, who can confirm if his proposed fix for OVSDB can address this, or just get him to share his WiP patch.
Vishal, as an aside, the connection code is different for ovsdb and hwvtep, so we want to use the improvements from hwvtep also in ovsdb?
 
Regards,
Vishal.
 
On Fri, Apr 20, 2018 at 2:46 PM, Swati Udhavrao Niture <swati.udhavrao.niture@ericsson.com> wrote:
Hi Faseela,
 
PFA logs.
 
Thanks,
Swati.
 
From: Faseela K 
Sent: Friday, April 20, 2018 12:05 PM
To: Swati Udhavrao Niture; Sam Hague; Vishal Thapar; K.V Suneelu Verma
 
Swati,
   Please pull the latest stable/oxygen, tunnel down issue is already resolved.
   Master has some issues with ovsdb plugin, which we are still debugging, and no handle yet.
   If you are working on latest master, could you please enable TRACE logs for org.opendaylight.ovsdb.southbound and push tunnel configuration and supply the logs?
Thanks,
Faseela
 
From: Swati Udhavrao Niture 
Sent: Friday, April 20, 2018 12:03 PM
To: Faseela K <faseela.k@...>; Sam Hague <shague@...>; Vishal Thapar <vthapar@...>; K.V Suneelu Verma <k.v.suneelu.verma@...m>
Cc: integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org; netvirt-dev@...
Subject: RE: [genius-dev] [release] Genius CSIT failing on master
 
Hi all,
 
Tunnels are coming up on the switch now, but I have to restart my OVS every-time after I send a request to bring up tunnels.
Has anybody else faced the same issue?
 
Adding to this, in stable-oxygen, tunnels are coming up on the switch, but I see oper_status as down.
Any pointers on this?
 
Thanks,
Swati.
 
From: genius-dev-bounces@lists.opendaylight.org [mailto:genius-dev-bounces@...ght.org] On Behalf Of Faseela K
Sent: Thursday, April 19, 2018 1:35 PM
To: Sam Hague; Vishal Thapar; K.V Suneelu Verma
Cc: integration-dev@...daylight.org; genius-dev@lists.opendaylight.org; ovsdb-dev@lists.opendaylight.org; release@lists.opendaylight.org
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
Yes, it is master.
I hope the maven issues are all resolved now, but we are still hitting the same error.
(karaf SSH issues are gone now, the ovsdb plugin issue still exists)
 
 
Thanks,
Faseela
 
From: Sam Hague [mailto:shague@...] 
Sent: Wednesday, April 18, 2018 6:47 PM
To: Vishal Thapar <vthapar@...>
Cc: Faseela K <faseela.k@...>; genius-dev@...; integration-dev@...light.org; ovsdb-dev@...ndaylight.org;release@lists.opendaylight.org
Subject: Re: [genius-dev] [release] Genius CSIT failing on master
 
If this is master, then likely this is related to the maven issues which is creating bad artifacts so odl is in a bad state. That is being worked now.
 
On Wed, Apr 18, 2018, 8:51 AM Vishal Thapar <vthapar@...> wrote:
No, there hasn't been a functional change in OVSDB plugin for more than 2 releases now. There is a bug with stale connection cleanup on OVS restart, could be related to that. Suneelu had a similar issue and fix in HWVTEP. Can check with him if this is same issue. Josh is also running into some variation of same issue and I think Sathwick was at some point trying to capture some logs for Suneelu too.
 
Regards,
Vishal.
 
On Wed, Apr 18, 2018 at 4:24 PM, Faseela K <faseela.k@...> wrote:
+ovsdb-dev
 
Vishal,
  Since distribution-checks are randomly failing, don’t know whether the tests are running properly or not.
  But we are hitting an issue where tunnels are not coming up on the switch, even though the tunnel information is present in config/network-topology. Is there any recent change in ovsdb plugin?
Thanks,
Faseela
 
From: genius-dev-bounces@lists.opendaylight.org [mailto:genius-dev-bounces@...ght.org] On Behalf Of B Sathwik
Sent: Wednesday, April 18, 2018 3:48 PM
To: release@....org; integration-dev@...endaylight.org
Cc: genius-dev@...ght.org
Subject: [genius-dev] Genius CSIT failing on master
 
Hi
 
Genius CSIT is Failing in master branch with some basic issues, Sometimes not able to login into karaf and sometime tunnels are not shown up on bridge side.
 
Since there are several distribution-check failures also seen intermittently on patches, wanted to know whether we have a stable master distribution, or
We should wait to run CSIT later on.
 
Here are the logs of Genius-CSIT:
 

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/sathwikgenius-csit-1node-gate-all-fluorine/4/robot-plugin/log.html.gz

 

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/sathwikgenius-csit-3node-gate-all-fluorine/3/robot-plugin/log.html.gz

 
 
 
Best Regards
Sathwik
 


_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release

 
_______________________________________________
genius-dev mailing list
genius-dev@....org
https://lists.opendaylight.org/mailman/listinfo/genius-dev
 
 
 
 
_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release



_______________________________________________
ovsdb-dev mailing list
ovsdb-dev@...rg
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev