VM cannot access external network in ODL Carbon L3


Xueyifei (Yifei Xue) <xueyifei@...>
 

Hi all

 

I am integrating ODL carbon with openstack ocata. I have encountered an issue in L3 scenario that the instance cannot access the external network.

 

I am using odl driver v2.

 

VM fixed ip: 10.10.10.12; floating ip: 192.16.1.108; floating ip mac address: fa:16:3e:6e:8d:63

 

First I checked the flows, found that the pipline my packages went in seemed different as descript in netvirt tutorial. My flow tables as below:

 

cookie=0x8000000, duration=3129.203s, table=0, n_packets=617, n_bytes=61534, priority=4,in_port=6,vlan_tci=0x0000/0x1fff actions=write_metadata:0x20000000000/0xffffff0000000001,goto_table:17

 

cookie=0x8000001, duration=3269.305s, table=17, n_packets=762, n_bytes=75520, priority=10,metadata=0x20000000000/0xffffff0000000000 actions=load:0x186a4->NXM_NX_REG3[0..24],write_metadata:0x9000020000030d48/0xfffffffffffffffe,goto_table:19

 

cookie=0x8000009, duration=3296.145s, table=19, n_packets=594, n_bytes=58212, priority=20,metadata=0x30d48/0xfffffe,dl_dst=fa:16:3e:31:18:5d actions=goto_table:21

 

cookie=0x8000004, duration=3345.739s, table=21, n_packets=643, n_bytes=63014, priority=10,ip,metadata=0x30d48/0xfffffe actions=goto_table:26

 

cookie=0x8000004, duration=3354.955s, table=26, n_packets=672, n_bytes=65856, priority=10,ip,metadata=0x30d48/0xfffffe,nw_src=10.10.10.12 actions=set_field:192.16.1.108->ip_src,write_metadata:0x30d40/0xfffffe,goto_table:28

 

cookie=0x8000004, duration=3381.948s, table=28, n_packets=699, n_bytes=68502, priority=10,ip,metadata=0x30d40/0xfffffe,nw_src=192.16.1.108 actions=resubmit(,21) ############# Is this correct?

 

For full table21:

 

cookie=0x8000003, duration=18080.196s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x30d42/0xfffffe,nw_dst=192.16.1.111 actions=write_metadata:0x30d42/0xfffffe,goto_table:44

cookie=0x8000003, duration=18080.196s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x30d42/0xfffffe,nw_dst=192.16.1.101 actions=set_field:fa:16:3e:45:2e:f7->eth_dst,load:0x800->NXM_NX_REG6[],resubmit(,220)

cookie=0x8000003, duration=18080.196s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x30d42/0xfffffe,nw_dst=192.16.1.1 actions=set_field:52:54:00:54:4a:9d->eth_dst,load:0x800->NXM_NX_REG6[],resubmit(,220)

cookie=0x8000003, duration=8168.102s, table=21, n_packets=6, n_bytes=588, priority=42,ip,metadata=0x30d40/0xfffffe,nw_dst=192.16.1.111 actions=load:0x800->NXM_NX_REG6[],resubmit(,220)

cookie=0x8000003, duration=3936.076s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x30d48/0xfffffe,nw_dst=10.10.10.2 actions=set_field:0x60->tun_id,set_field:fa:16:3e:49:c1:9e->eth_dst,load:0x300->NXM_NX_REG6[],resubmit(,220)

cookie=0x8000003, duration=3935.647s, table=21, n_packets=1, n_bytes=98, priority=42,ip,metadata=0x30d48/0xfffffe,nw_dst=10.10.10.12 actions=group:150002

cookie=0x8000003, duration=3936.076s, table=21, n_packets=1, n_bytes=98, priority=42,icmp,metadata=0x30d48/0xfffffe,nw_dst=10.10.10.1,icmp_type=8,icmp_code=0 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:31:18:5d->eth_src,move:NXM_OF_IP_SRC[]->NXM_OF_IP_DST[],set_field:10.10.10.1->ip_src,set_field:0->icmp_type,load:0->NXM_OF_IN_PORT[],resubmit(,21)

cookie=0x8000003, duration=18080.196s, table=21, n_packets=0, n_bytes=0, priority=34,ip,metadata=0x30d42/0xfffffe,nw_dst=192.16.1.0/24 actions=write_metadata:0x138b030d42/0xfffffffffe,goto_table:22

cookie=0x8000004, duration=3936.076s, table=21, n_packets=0, n_bytes=0, priority=10,ip,metadata=0x30d42/0xfffffe actions=group:225000

cookie=0x8000004, duration=3936.076s, table=21, n_packets=1040, n_bytes=101920, priority=10,ip,metadata=0x30d48/0xfffffe actions=goto_table:26

cookie=0x6900000, duration=18080.198s, table=211, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17)

cookie=0x6900000, duration=3936.074s, table=212, n_packets=0, n_bytes=0, priority=50,ip,metadata=0x1389000000/0xffff000000,nw_dst=10.10.10.12 actions=write_metadata:0x2/0xfffffe,goto_table:213

cookie=0x6900000, duration=3936.058s, table=212, n_packets=0, n_bytes=0, priority=50,ipv6,metadata=0x1389000000/0xffff000000,ipv6_dst=fe80::f816:3eff:fe80:b41e actions=write_metadata:0x2/0xfffffe,goto_table:213

 

I checked karaf log and found that: ERRORs occurred while proceeding with the flow table25-27:

 

2017-06-28 01:01:17,660 | ERROR | eChangeHandler-0 | InterfacemgrProvider             | 358 - org.opendaylight.genius.interfacemanager-impl - 0.2.0.Carbon | Interface 814cfd00-3a8a-4ced-a8df-0d10e7dd7561 is not present

2017-06-28 01:01:17,660 | WARN  | eChangeHandler-0 | ElanInterfaceManager             | 374 - org.opendaylight.netvirt.elanmanager-impl - 0.4.0.Carbon | Interface 814cfd00-3a8a-4ced-a8df-0d10e7dd7561 is removed from Interface Oper DS due to port down 

2017-06-28 01:01:17,660 | INFO  | eChangeHandler-0 | VpnInterfaceManager              | 366 - org.opendaylight.netvirt.vpnmanager-impl - 0.4.0.Carbon | VPN Interface add event - intfName 814cfd00-3a8a-4ced-a8df-0d10e7dd7561

2017-06-28 01:01:17,660 | ERROR | eChangeHandler-0 | VpnInterfaceManager              | 366 - org.opendaylight.netvirt.vpnmanager-impl - 0.4.0.Carbon | Handling addition of VPN interface 814cfd00-3a8a-4ced-a8df-0d10e7dd7561 skipped as interfaceState is not available

2017-06-28 01:01:17,661 | WARN  | nPool-1-worker-3 | InterfaceConfigListener          | 358 - org.opendaylight.genius.interfacemanager-impl - 0.2.0.Carbon | parent refs not specified for 814cfd00-3a8a-4ced-a8df-0d10e7dd7561

2017-06-28 01:01:18,067 | WARN  | eChangeHandler-0 | IfMgr                            | 379 - org.opendaylight.netvirt.ipv6service-impl - 0.4.0.Carbon | Update Host interface failed. Could not get Host interface details Uuid [_value=814cfd00-3a8a-4ced-a8df-0d10e7dd7561]

2017-06-28 01:01:31,571 | ERROR | eChangeHandler-3 | FloatingIPListener               | 375 - org.opendaylight.netvirt.natservice-impl - 0.4.0.Carbon | NAT Service : Unable to retrieve floatingIpPortMacAddress from floating IP UUID Uuid [_value=e9ab5471-6cd1-40a9-a932-09809837faf9] for floating IP 192.16.1.108

2017-06-28 01:01:31,571 | ERROR | eChangeHandler-3 | FloatingIPListener               | 375 - org.opendaylight.netvirt.natservice-impl - 0.4.0.Carbon | NAT Service : Flow entity received as NULL. Cannot proceed with installation of Pre-DNAT flow table 25 --> table 27 on DpnId 92588886143807

2017-06-28 01:01:31,571 | INFO  | eChangeHandler-3 | FloatingIPListener               | 375 - org.opendaylight.netvirt.natservice-impl - 0.4.0.Carbon | NAT Service : Building PSNAT Flow entity for ip 10.10.10.12 

2017-06-28 01:01:31,572 | INFO  | eChangeHandler-3 | FloatingIPListener               | 375 - org.opendaylight.netvirt.natservice-impl - 0.4.0.Carbon | Building SNAT Flow entity for ip 10.10.10.12 

2017-06-28 01:01:31,572 | WARN  | eChangeHandler-3 | FloatingIPListener               | 375 - org.opendaylight.netvirt.natservice-impl - 0.4.0.Carbon | No MAC address found for floating IP 192.16.1.108

2017-06-28 01:01:31,572 | WARN  | eChangeHandler-3 | NatUtil                          | 375 - org.opendaylight.netvirt.natservice-impl - 0.4.0.Carbon | getOptionalExternalSubnets - null subnetId

2017-06-28 01:01:31,573 | ERROR | eChangeHandler-3 | AsyncDataTreeChangeListenerBase  | 325 - org.opendaylight.genius.mdsalutil-api - 0.2.0.Carbon | Thread terminated due to uncaught exception: AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-3

java.lang.NullPointerException

     at org.opendaylight.netvirt.natservice.internal.VpnFloatingIpHandler.onAddFloatingIp(VpnFloatingIpHandler.java:134)[375:org.opendaylight.netvirt.natservice-impl:0.4.0.Carbon]

     at org.opendaylight.netvirt.natservice.internal.FloatingIPListener.createNATFlowEntries(FloatingIPListener.java:441)[375:org.opendaylight.netvirt.natservice-impl:0.4.0.Carbon]

     at org.opendaylight.netvirt.natservice.internal.FloatingIPListener.processFloatingIPAdd(FloatingIPListener.java:358)[375:org.opendaylight.netvirt.natservice-impl:0.4.0.Carbon]

     at org.opendaylight.netvirt.natservice.internal.FloatingIPListener.add(FloatingIPListener.java:103)[375:org.opendaylight.netvirt.natservice-impl:0.4.0.Carbon]

     at org.opendaylight.netvirt.natservice.internal.FloatingIPListener.add(FloatingIPListener.java:61)[375:org.opendaylight.netvirt.natservice-impl:0.4.0.Carbon]

     at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:156)[325:org.opendaylight.genius.mdsalutil-api:0.2.0.Carbon]

     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_51]

     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_51]

     at java.lang.Thread.run(Thread.java:745)[:1.8.0_51]

 

Then I searched in neutron-server.log and found that the floating ip of the instance had been posted into odl:

 

2017-06-28 01:01:17.640 9018 DEBUG networking_odl.common.client [-] Sending METHOD (post) URL (http://10.1.0.22:8080/controller/nb/v2/neutron/ports) JSON ({

  "port": {

    "port_security_enabled": false,

    "binding:host_id": "",

    "name": "",

    "allowed_address_pairs": [],

    "admin_state_up": true,

    "network_id": "7af41fd9-30f8-4fc4-bc34-1519be2d133b",

    "tenant_id": "be3cd4a52aac41b8946eac859f1778b5",

    "binding:vif_details": {},

    "binding:vnic_type": "normal",

    "binding:vif_type": "unbound",

    "device_owner": "network:floatingip",

    "mac_address": "fa:16:3e:6e:8d:63",

    "binding:profile": {},

    "project_id": "",

    "fixed_ips": [

      {

        "subnet_id": "f97557c3-d6eb-476c-9da3-ea0a909ba3c3",

        "ip_address": "192.16.1.108"

      }

    ],

    "id": "814cfd00-3a8a-4ced-a8df-0d10e7dd7561",

    "security_groups": [],

    "device_id": "PENDING"

  }

}) request /openstack/venvs/neutron-15.1.4/lib/python2.7/site-packages/networking_odl/common/client.py:86

2017-06-28 01:01:17.657 9018 DEBUG networking_odl.common.client [-] Got response:

({

   "port" : {

      "id" : "814cfd00-3a8a-4ced-a8df-0d10e7dd7561",

      "tenant_id" : "be3cd4a52aac41b8946eac859f1778b5",

      "project_id" : "",

      "name" : "",

      "admin_state_up" : true,

      "status" : "ACTIVE",

      "network_id" : "7af41fd9-30f8-4fc4-bc34-1519be2d133b",

      "mac_address" : "fa:16:3e:6e:8d:63",

      "fixed_ips" : [ {

         "ip_address" : "192.16.1.108",

         "subnet_id" : "f97557c3-d6eb-476c-9da3-ea0a909ba3c3"

      } ],

      "device_id" : "PENDING",

      "device_owner" : "network:floatingip",

      "security_groups" : [ ],

      "allowed_address_pairs" : [ ],

      "binding:host_id" : "",

      "binding:vnic_type" : "normal",

      "binding:vif_type" : "unbound",

      "Map" : {

      },

      "port_security_enabled" : false

   }

 

Then I run curl to get the port information in ODL database and found that the floating ip and its mac address were there:

 

root@host3-utility-container-9b7f2482:~# curl -u admin:admin http://10.1.0.22:8080/controller/nb/v2/neutron/ports | grep 192.16.1.108 -C 15

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100  5408    0  5408    0     0   445k      0 --:--:-- --:--:-- --:--:--  480k

      "binding:host_id" : "",

      "binding:vnic_type" : "normal",

      "binding:vif_type" : "unbound",

      "Map" : {

      },

      "port_security_enabled" : false

   }, {

      "id" : "814cfd00-3a8a-4ced-a8df-0d10e7dd7561",

      "tenant_id" : "be3cd4a52aac41b8946eac859f1778b5",

      "revision_number" : 5,

      "name" : "",

      "admin_state_up" : true,

      "network_id" : "7af41fd9-30f8-4fc4-bc34-1519be2d133b",

      "mac_address" : "fa:16:3e:6e:8d:63",

      "fixed_ips" : [ {

         "ip_address" : "192.16.1.108",

         "subnet_id" : "f97557c3-d6eb-476c-9da3-ea0a909ba3c3"

      } ],

      "device_id" : "e9ab5471-6cd1-40a9-a932-09809837faf9",

      "device_owner" : "network:floatingip",

      "security_groups" : [ ],

      "allowed_address_pairs" : [ ],

      "binding:host_id" : "",

      "binding:vnic_type" : "normal",

      "binding:vif_type" : "unbound",

      "Map" : {

      },

      "extra_dhcp_opts" : [ ],

      "port_security_enabled" : false

   } ]

}

 

I’ve totally followed the guide of netvirt with openstack and networking-odl installation guide. I don’t know what’s wrong with my deployment. Is there anyone can help me with this? Thanks

 

 

BR,

Yifei Xue