Re: Upgrade script - CSIT


Dheenadayalan B <dheenadayalan.b@...>
 

Hi Srini/Karthik/Devteam,

                                             Unlike in magnesium branch,  northbound interface addition leads to time delay of more than one minute, leads to upgrade csit failure,

Openflowplugin module was listening to port 6653/6633 , however connection establishment between ovs and plugin made with a delay of one minute through northbound.

 

logs as below:

Openflowplugin startup completed as below:

 

Karaf.log:

020-06-04T02:28:15,339 | INFO  | SystemReadyService-0 | OpenFlowPluginProviderImpl       | 382 - org.opendaylight.openflowplugin.impl - 0.11.0.SNAPSHOT | onSystemBootReady() received, starting the switch connections

2020-06-04T02:28:15,547 | INFO  | OFP-SwitchConnectionProvider-Udp/TcpHandler_6653-0 | TcpHandler                       | 392 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.11.0.SNAPSHOT | Switch listener started and ready to accept incoming tcp/tls connections on port: 6653

2020-06-04T02:28:15,550 | INFO  | OFP-SwitchConnectionProvider-Udp/TcpHandler_6653-0 | OpenflowDiagStatusProviderImpl   | 392 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.11.0.SNAPSHOT | The report status is OPERATIONAL for OPENFLOW_SERVER_6653

2020-06-04T02:28:15,559 | INFO  | SystemReadyService-0 | SouthboundProvider               | 401 - org.opendaylight.ovsdb.southbound-impl - 1.11.0.SNAPSHOT | Started OVSDB Manager (in system ready listener)

2020-06-04T02:28:15,561 | INFO  | OFP-SwitchConnectionProvider-Udp/TcpHandler_6633-0 | OpenFlowPluginProviderImpl       | 382 - org.opendaylight.openflowplugin.impl - 0.11.0.SNAPSHOT | All switchConnectionProviders are up and running (2).

2020-06-04T02:28:15,561 | INFO  | OFP-SwitchConnectionProvider-Udp/TcpHandler_6633-0 | OpenflowDiagStatusProviderImpl   | 392 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.11.0.SNAPSHOT | The report status is OPERATIONAL for OPENFLOW

 

Script for northbound setup  executed @20200604 07:58:43.060 (UTC+05:30) (2020-06-04T02:28:43)

 

sudo ovs-vsctl set-manager tcp:10.30.170.113:6640 ptcp:6641:127.0.0.1

 

karaf.log:

 

2020-06-04T02:28:43,942 | DEBUG | jobcoordinator-main-task-6 | InterfaceTopologyStateListener   | 248 - org.opendaylight.genius.interfacemanager-impl - 0.9.0.SNAPSHOT | adding bridge references for bridge: OvsdbBridgeAugmentation{getBridgeExternalIds={BridgeExternalIdsKey{_bridgeExternalIdKey=created_by}=BridgeExternalIds{getBridgeExternalIdKey=created_by, getBridgeExternalIdValue=odl, augmentation=[]}, BridgeExternalIdsKey{_bridgeExternalIdKey=opendaylight-iid}=BridgeExternalIds{getBridgeExternalIdKey=opendaylight-iid, getBridgeExternalIdValue=/network-topology:network-topology/network-topology:topology[network-topology:topology-id='ovsdb:1']/network-topology:node[network-topology:node-id='ovsdb://uuid/758da02d-7c35-43b6-99aa-c9c57108210f/bridge/br-int'], augmentation=[]}}, getBridgeName=OvsdbBridgeName{_value=br-int}, getBridgeOtherConfigs={BridgeOtherConfigsKey{_bridgeOtherConfigKey=hwaddr}=BridgeOtherConfigs{getBridgeOtherConfigKey=hwaddr, getBridgeOtherConfigValue=ec:47:56:1c:b1:9b, augmentation=[]}, BridgeOtherConfigsKey{_bridgeOtherConfigKey=disable-in-band}=BridgeOtherConfigs{getBridgeOtherConfigKey=disable-in-band, getBridgeOtherConfigValue=true, augmentation=[]}}, getBridgeUuid=Uuid{_value=935221ad-40ad-4aaa-939c-a8654e883bd8}, getDatapathId=DatapathId{_value=00:00:ec:47:56:1c:b1:9b}, getDatapathType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.DatapathTypeSystem, getFailMode=interface org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.OvsdbFailModeSecure, getManagedBy=OvsdbNodeRef{_value=KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey{_topologyId=Uri{_value=ovsdb:1}}], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey{_nodeId=Uri{_value=ovsdb://uuid/758da02d-7c35-43b6-99aa-c9c57108210f}}]]}}, getProtocolEntry={ProtocolEntryKey{_protocol=interface org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.OvsdbBridgeProtocolOpenflow13}=ProtocolEntry{getProtocol=interface org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.OvsdbBridgeProtocolOpenflow13, augmentation=[]}}, isStpEnable=false}, dpn: 259791131554203

 

Openflowplugin and ovs communication time:

Karaf.log:

 

2020-06-04T02:29:56,797 | INFO  | epollEventLoopGroup-7-3 | ContextChainHolderImpl           | 382 - org.opendaylight.openflowplugin.impl - 0.11.0.SNAPSHOT | Device openflow:259791131554203 connected.

 

Ovs-vswitchd.log:

2020-06-04T02:29:56.768Z|01001|rconn|INFO|br-int<->tcp:10.30.170.113:6653: connected

 

Thanks,

Dheena

 

From: srinivas.rachakonda@... <srinivas.rachakonda@...>
Sent: 05 June 2020 11:31
To: 'Dheenadayalan B' <dheenadayalan.b@...>; 'odl netvirt dev' <netvirt-dev@...>
Cc: 'P S Ravikanth' <ravikanth.ps@...>; karthikeyan.k@...; 'Chetan Arakere Gowdru' <chetan.arakere@...>; integration-dev@...
Subject: RE: Upgrade script - CSIT

 

++ Netvirt-Dev

 

Thanks,

Srinivas

+91-9243478719

 

From: srinivas.rachakonda@... <srinivas.rachakonda@...>
Sent: 05 June 2020 11:13
To: 'Dheenadayalan B' <dheenadayalan.b@...>
Cc: 'P S Ravikanth' <ravikanth.ps@...>; 'karthikeyan.k@...' <karthikeyan.k@...>; 'Chetan Arakere Gowdru' <chetan.arakere@...>
Subject: RE: Upgrade script - CSIT

 

Hi Dheena,

 

Thanks for the debug session we had today.

 

The issue here is the bundle came up around 1min 58sec, while my script looks for 1min.

 

Now the question still remains if we need to wait more time for bundle to come up in Aluminium, while for Magnesium it came up within 1 min.

 

So today’s action items are:

  1. Srinivas - will increase the timeout and test it.
  2. Dheena – Will check the difference between Magnesium and Aluminium to find the timing issue.

 

Below is bundle URL.

Magnesium:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upgrade-snat-conntrack-magnesium/263

 

Magnesium ODL:

https://nexus.opendaylight.org/content/repositories//autorelease-4112/org/opendaylight/integration/karaf/0.12.2/karaf-0.12.2.zip

 

 

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Dheenadayalan B <dheenadayalan.b@...>
Sent: 04 June 2020 20:58
To: srinivas.rachakonda@...
Cc: 'P S Ravikanth' <ravikanth.ps@...>; karthikeyan.k@...; 'Chetan Arakere Gowdru' <chetan.arakere@...>
Subject: RE: Upgrade script - CSIT

 

Hi Srini,

      RPC will be registered on openflowdevice(DPN) connected with the controller. From the logs provided observerved

rpc call executed even before dpn  259791131554203 connection established, hence the error.

 

Rpc call execution time:

   20200604 07:58:56.147 / 20200604 07:58:56.160 / 00:00:00.013 (2020-06-04T02:28:56) (UTC+05:30hrs)

 

body={

   "input": {

      "node": "/opendaylight-inventory:nodes/opendaylight-inventory:node[opendaylight-inventory:id='openflow:259791131554203']",

      "node-id": 259791131554203

   }

 

CSIT logs: (UTC time zone)

 

2020-06-04T02:29:56,797 | INFO  | epollEventLoopGroup-7-3 | ContextChainHolderImpl           | 382 - org.opendaylight.openflowplugin.impl - 0.11.0.SNAPSHOT | Device openflow:259791131554203 connected.

 

Could you please delay script execution by one minute and check if the same issue occurring again?

 

Thanks,

Dheena

 

From: srinivas.rachakonda@... <srinivas.rachakonda@...>
Sent: 04 June 2020 09:29
To: 'Dheenadayalan B' <dheenadayalan.b@...>
Cc: 'P S Ravikanth' <ravikanth.ps@...>; karthikeyan.k@...; 'Chetan Arakere Gowdru' <chetan.arakere@...>
Subject: RE: Upgrade script - CSIT

 

https://jira.opendaylight.org/browse/NETVIRT-1672

 

Thanks,

Srinivas

+91-9243478719

 

From: srinivas.rachakonda@... <srinivas.rachakonda@...>
Sent: 04 June 2020 09:19
To: 'Dheenadayalan B' <dheenadayalan.b@...>
Cc: 'P S Ravikanth' <ravikanth.ps@...>; 'karthikeyan.k@...' <karthikeyan.k@...>; 'Chetan Arakere Gowdru' <chetan.arakere@...>
Subject: RE: Upgrade script - CSIT

 

Please see the below error:

 

POST Response : url=http://10.30.170.113:8181/restconf/operations/arbitrator-reconcile:get-active-bundle status=501, reason=Not Implemented

 

Can this be added to Aluminium.

We have aluminium release in next 15 days and this fix is needed.

 

Thanks,

Srinivas

+91-9243478719

 

From: srinivas.rachakonda@... <srinivas.rachakonda@...>
Sent: 04 June 2020 09:16
To: 'Dheenadayalan B' <dheenadayalan.b@...>
Cc: 'P S Ravikanth' <ravikanth.ps@...>; 'karthikeyan.k@...' <karthikeyan.k@...>; 'Chetan Arakere Gowdru' <chetan.arakere@...>
Subject: RE: Upgrade script - CSIT

 

Hi Dheena,

 

The fixes passed in Magnesium but failed in Aluminium.

 

Please kindly look into this on priority and let me know if the below json needs to be changed for Aluminium.

 

 

https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upgrade-snat-conntrack-aluminium/88/

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upgrade-snat-conntrack-aluminium/88/robot-plugin/log_full.html.gz

 

 

 

 

 

Thanks,

Srinivas

+91-9243478719

 

From: srinivas.rachakonda@... <srinivas.rachakonda@...>
Sent: 01 June 2020 14:01
To: 'Dheenadayalan B' <dheenadayalan.b@...>
Cc: 'P S Ravikanth' <ravikanth.ps@...>; 'karthikeyan.k@...' <karthikeyan.k@...>; 'Chetan Arakere Gowdru' <chetan.arakere@...>
Subject: RE: Upgrade script - CSIT

 

Hi Dheena,

 

Thanks a lot for the help.

With your suggestions I was able to run the scripts and it went fine.

 

Can you please mail me in details the reasons for the changes so that I can ask the community for acceptance.

 

Thanks,

Srinivas

+91-9243478719

 

From: srinivas.rachakonda@... <srinivas.rachakonda@...>
Sent: 01 June 2020 10:49
To: 'Dheenadayalan B' <dheenadayalan.b@...>
Cc: 'P S Ravikanth' <ravikanth.ps@...>; 'karthikeyan.k@...' <karthikeyan.k@...>; 'Chetan Arakere Gowdru' <chetan.arakere@...>
Subject: Upgrade script - CSIT

 

Hi Dheena,

 

I am working on reproducing the upgrade issue in CSIT today.

I will ping once I am at failed state.

 

We will debug it together on my local setup.

 

Thanks,

Srinivas

+91-9243478719

 

Join integration-dev@lists.opendaylight.org to automatically receive all group messages.