On Mon, Feb 18, 2019 at 8:17 PM Yi Yang (杨燚)-云服务集团 <yangyi01@...> wrote:

Thanks Sam, I have been able to run csit/suites/openstack/connectivity/l2.robot in my local environment. https://git.opendaylight.org/gerrit/#/c/67681/ assumed ODL and openstack nodes have been ready for test, my concern is how we can automatically deploy and install these ODL and openstack nodes.

Yes, this is always the problem. We have another way to deploy using Apex. With that the openstack vms come with everything installed and in containers - so you just start services and it doesn't use devstack to start them. There is still some scripting around the deployment to work around the infra limitations, like the single interface, but it is much less that what we have with devstack. 

 

Now my local environment has 3 ODL nodes,  they just run in non-openstack node, I used HAproxy, csit/suites/openstack/connectivity/l2.robot  can work well, it is still stable after several days, but java has 100% CPU utilization. I checked 1node-topo and 3node-topo (I know 1 or 3 is number of ODL node/openstack controller node), ODL is installed with Openstack controller together in the same node.

Ok, this is good. L2 usually works well since there are no ha tests. ha_l2 and ha_ l3 are the tests with ha and where you might see issues.

 

 

发件人: Sam Hague [mailto:shague@...]
发送时间: 2019年2月18日 23:23
收件人: Yi Yang (杨燚)-云服务集团 <yangyi01@...>
抄送: vgovindarajan@...; gvrangan@...; jluhrsen@...; integration-dev@...; dev@...
主题: Re: which script/tool is used to create 1node and 3node topologies for ODL csit test?

 

 

 

On Mon, Feb 18, 2019 at 10:05 AM Yi Yang (杨燚)-云服务集团 <yangyi01@...> wrote:

Hi, Venkat

 

Sam mentioned you added https://github.com/opendaylight/integration-test/tree/master/tools/deployment/openstack_ha/deploy#3node-topology, can you tell me which repo or script I can use to launch 3 or 5 VMs in 1node-topology and 3node-topology? my local setup look like https://github.com/yyang13/openstack-netvirt, but many test cases in csit can’t work, so I want to figure out why. My goal is to run all the csit test suites in my local environment which have physical servers and physical external network available. It seems all the csit suites are only for opendaylight.org.

 

I tried to run csit/suites/openstack/connectivity/external_network.robot with the below settings but it failed for floating IP ping, csit/suites/openstack/connectivity/l2.robot is ok with tiny change.

 

EXTERNAL_GATEWAY=192.168.100.1

EXTERNAL_PNF=192.168.100.9

EXTERNAL_SUBNET=192.168.100.0/24

EXTERNAL_SUBNET_ALLOCATION_POOL=start=192.168.100.200,end=192.168.100.250

EXTERNAL_INTERNET_ADDR=192.168.199.199

 

Are there any simple steps to duplicate csit tests in local machine? I mean the same VMs as every csit test suite used, But per Sam’s commnets, it seems every VM which is running openstack and opendaylight only has one network interface, so I’m not sure where 1node-topology and 3node-topology are used.

Janki started a doc for running CSIt locally: https://git.opendaylight.org/gerrit/#/c/67681/. Maybe there are some clues in there.

 

Remember 1node and 3node refers to ODL clustering - either 1 ODL or 3 ODL's so it has nothing to do with interfaces. There is always only one openstack controller, since the {1,3}node refers only to ODL's. On the northbound we use HAProxy from neutron odl_networking ml2. The openstack control node uses only a single VIP and the HAProxy distributes the rest calls to the ODL's. On the ovs side we reuse the managers and controllers, so each ovs node is configured to point to all three ODL's.

 

Recall also that 3node (using 3 ODL's) has issues. openflowplugin gets into a bad state sometimes and so does netvirt. Tests are not too bad if not taking the ODL nodes up and down and work for the most part.

 

发件人: Sam Hague [mailto:shague@...]
发送时间: 2019年2月15日 23:38
收件人: Yi Yang (杨燚)-云服务集团 <yangyi01@...>; gvrangan@...
抄送: jluhrsen@...; integration-dev@...; dev@...
主题: Re: which script/tool is used to create 1node and 3node topologies for ODL csit test?

 

Adding Venkat to thread for openstack_ha stuff.

 

On Fri, Feb 15, 2019 at 8:04 AM Yi Yang (杨燚)-云服务集团 <yangyi01@...> wrote:

Hi, Jamo and Sam

 

I have always floating IP issue in my setup [1], you can see my topo details in [1], so I want to duplicate ODL csit test topo to find the difference, I find 1node topo in [2] and 3node topo in [3], I guess ODL csit test used these two topos for 1node and 3node test suite, respectively, can you tell me we have any script to create these two topos? Or are there Vagrantfiles or VM images available for them?

[16] is the line defining the openstack image name. That vm is built by a different job using packer. It takes a centos base, updates packages adds openstack repos. Then a vm pops out and named as [16]. That is fed into the templates I talk about later. You can download that image. Just look in the job console and see the wget. Recognize it doesn't do the openstack config yet though. Also mentioned later is how that vm is then devstack-ified.

 

[16] https://github.com/opendaylight/releng-builder/blob/master/jjb/defaults.yaml#L86 

 

[2] and [3] are the openstack_ha deployment path - which does not go with [4]. openstack_ha was added by Venkat to do a different deployment scheme that used robot itself to deploy everything. The job in [4] and most of the other jobs that netvirt uses rely on the deployment used in the releng/builder project. The openstack_ha stuff ends up the same, but me and Jamo don't watch that so I don't know it's status.

 

For [4] you need to look at releng/builder sh script [7] for how it deploys everything and sets up devstack. But you asked about 1node and 3node - that corresponds to ODL nodes and not openstack nodes. In that case you want [8] which is the shell script for configuring and starting ODL. All of this under [9] which is the root yaml job config. Notice in [9] there is a 1node and 3node under topology - so 1 or 3 ODL's topology under the odl_system_count value of 1 or 3. That yaml brings in the templates in [10] which adds more configuration using that topology value.

 

So eventually [8] is called to bring up the ODLs. Notice near the end of that script there is an exit before running pybot. From there [7] fires up and does the devstack configuration. Also based on the yaml [9] are the os-ctl-cnt: 1 and os-cmp-cnt: 2. That means one openstack controller and two compute nodes. Those get used in [7] to bring up the openstack side. When using 3node ODL, there is config also in [7] line [11] that configures haproxy to load balance across a single ODL or 3 ODL's. Follow that function in the script and you will see ODL_SYSTEM_{1,2,3}_IP that correspond to the different ODL's.

 

[7] https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-openstack-run-test.sh

[8] https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-controller-run-test.sh

[9] https://github.com/opendaylight/releng-builder/blob/master/jjb/netvirt/netvirt-csit-multi-openstack.yaml

[10] https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-templates.yaml#L1679

[11] https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-openstack-run-test.sh#L466

 

I checked this log [4] and found [5], is physnet1 in [5] “External Network1” or “External Network2”in [2] and [3]? Why does it have two network type vlan and flat?

[12] is where we configure the external networks for use in the integration/test suites. In deployment side we have to fake out the external network since the infrastructure only gives us a single eth interface on the vm - and we can't use that like in a typical openstack setup where you have br-ex and a second interface connected to it going externally. So, what we do is we create a br-physnet1 bridge, then patch it to br-int. Then we add tuntap ports to the br-physnet and a namespace to simulate a gateway and external ips. We also add the br-physnet1 bridge to the compute nodes and then add a vxlan tunnel back to the controller br-physnet so those computes nodes can reach the mock pnf and gw addresses.

 

[13] is in the l3 tests. That is where we call the openstack cli to create the vlan and vxlan networks for the tests. external_network.robot is similar and we create a flat external network there.

[12] https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-openstack-run-test.sh#L1055

[13] https://github.com/opendaylight/integration-test/blob/master/csit/suites/openstack/connectivity/l3.robot#L173

Can you help show subnet IPs for Data and External Network1 and External Networks in 1node and 3node topos?

I would not use the 3node jobs and instead would use the 1node jobs. 3node clustering has issues so they don't fully pass. ofp has issues and netvirt itself has issues and they have not be fully debugged or fixed. [14] is a link to the ovs side debugging showing all the ovs-vsctl/ovs-ofctl commands for the control node. [15] is the link in the mdsal model dumps to the neutron stuff.

 

[14]  https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-fluorine/191/robot-plugin/log_03_external_network.html.gz#s1-t8-k3-k2-k1-k1

[15] https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-fluorine/191/robot-plugin/log_03_external_network.html.gz#s1-t8-k3-k3-k1-k2-k16-k3

Which script/tool used tools/deployment/openstack_ha/deploy/deploy_odl_with_netvirt.robot? but keyword Install ODL From RPM In All ODL Nodes and Install ODL From ZIP In All ODL Nodes aren’t defined in any other robot, please refer to [6].

 

[1] https://github.com/yyang13/openstack-netvirt

[2] https://github.com/opendaylight/integration-test/tree/master/tools/deployment/openstack_ha/deploy#1node-topology

[3] https://github.com/opendaylight/integration-test/tree/master/tools/deployment/openstack_ha/deploy#3node-topology

[4]https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-fluorine/193/robot-plugin/log_full.html.gz

[5]

openstack network create l2_net_1 --provider-network-type vlan --provider-physical-network physnet1 --provider-segment 1121

openstack network create l3_net_1 --provider-network-type vlan --provider-physical-network physnet1 --provider-segment 1131

openstack network create l3_net_4 --provider-network-type vlan --provider-physical-network physnet1 --provider-segment 1132

openstack network create l3_net_5 --provider-network-type vlan --provider-physical-network physnet1 --provider-segment 1133

openstack network create external-net --provider-network-type flat --provider-physical-network physnet1

openstack server add floating ip l3_ext_net_1_fip_vm_1 10.10.10.20

openstack server add floating ip l3_ext_net_1_fip_vm_2 10.10.10.7

openstack server add floating ip l3_ext_net_1_fip_vm_3 10.10.10.5

openstack subnet create --network external-net --subnet-range 10.10.10.0/24 external-subnet --gateway 10.10.10.250 --allocation-pool start=10.10.10.2,end=10.10.10.249

 

[6]

[yangyi@localhost test]$ grep -r "Install_ODL_From" *

[yangyi@localhost test]$ grep -r "Install ODL From" *

tools/deployment/openstack_ha/deploy/00_initial_setup.robot:Install ODL From CBS or Nexus

tools/deployment/openstack_ha/deploy/00_initial_setup.robot:Install ODL From Local Rpm

tools/deployment/openstack_ha/deploy/deploy_odl_with_netvirt.robot:    Run Keyword If    '${ODL_INSTALL_MODE}' == 'RPM'    Install ODL From RPM In All ODL Nodes    ${ODL_RPM}

tools/deployment/openstack_ha/deploy/deploy_odl_with_netvirt.robot:    Run Keyword If    '${ODL_INSTALL_MODE}' == 'ZIP'    Install ODL From ZIP In All ODL Nodes    ${ACTUAL_BUNDLE_URL}    ${BUNDLEFOLDER}

[yangyi@localhost test]$