ODL disconnects from switch during performance tests


Thomas Ferrandiz
 

Hello all,

 

I am currently developing a stateful firewall application for SDN using ODL as the controller.

The application uses the TCP flags OpenFlow extension to match the successive states when establishing a connection.

This means that a large amount of packets–in is sent to the controller and that in turn the controller adds and deletes many flows from the switches.

I use the MD-SAL API to add and remove flows from the switches without going through the datastore.

The version of ODL used is Boron-SR3.

 

At the moment, I am writing a POC firewall with only one switch and I run into the following problem.

I run a stress test that involves creating TCP connection from a client machine to a server machine (running Apache).

The 2 machines are connected by an OVS switch connected to ODL.

 

When the stress test is run with a low number of connections/second (~50) everything works fine.

When using ~200 TCP connections/second, the firewall seems to work fine for a few seconds.

Then the duration of the TCP connections start to increase and the controller ends up disconnection from the OVS.

 

The following errors appear in the log:

 

2017-04-07 09:42:34,419 | ERROR | pool-15-thread-1 | OutboundQueueProviderImpl        | 192 - org.opendaylight.openflowplugin.impl - 0.3.2.Boron-SR2 | No queue present, failing request

 

And the future returned by salFlowService.addFlow show the following error message:

 

Error: [operation-failed]: Device disconnected

 

The following exception also occurs:

org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: No implementation of RPC AbsoluteSchemaPath{path=[(urn:opendaylight:flow:service?revision=2013-08-19)add-flow]} available

                at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:173)[154:org.opendaylight.controller.sal-broker-impl:1.4.3.Boron-SR3]

                at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102)[154:org.opendaylight.controller.sal-broker-impl:1.4.3.Boron-SR3]

                at Proxy1d02b267_bf62_4c92_a739_263c328d7038.invokeRpc(Unknown Source)[:]

                at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:65)[156:org.opendaylight.controller.sal-binding-broker-impl:1.4.3.Boron-SR3]

                at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:43)[156:org.opendaylight.controller.sal-binding-broker-impl:1.4.3.Boron-SR3]

                at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:159)[156:org.opendaylight.controller.sal-binding-broker-impl:1.4.3.Boron-SR3]

                at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:96)[156:org.opendaylight.controller.sal-binding-broker-impl:1.4.3.Boron-SR3]

                at com.sun.proxy.$Proxy54.addFlow(Unknown Source)[120:org.opendaylight.openflowplugin.model.flow-service:0.3.3.Boron-SR3]

 

The OVS logs also shows that it’s the controller that breaks the connection.

 

The JVM has 8GB of memory allocated and uses at most 3 so it’s not a memory issue.

The machine running ODL has 8 logical cores, of which only one is used.

CPU and memory usage is also pretty low on the machine running OVS.

 

So my question would be whether you have an idea of what I could do to solve this issue or a direction in which to investigate?

 

Thanks in advance,

 

Thomas Ferrandiz

{P}

R&D Engineer

Network Architecture & Security

 

 

1219 AVENUE CHAMPS BLANCS
35510 CESSON-SÉVIGNÉ (FR)

 

 

 

 

 

 


Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES@Cisco) <mirehak@...>
 

Added ofplugin-dev.




From: openflowjava-dev-bounces@... <openflowjava-dev-bounces@...> on behalf of Thomas FERRANDIZ <Thomas.FERRANDIZ@...>
Sent: Wednesday, May 10, 2017 17:36
To: openflowjava-dev@...
Subject: [openflowjava-dev] ODL disconnects from switch during performance tests
 

Hello all,

 

I am currently developing a stateful firewall application for SDN using ODL as the controller.

The application uses the TCP flags OpenFlow extension to match the successive states when establishing a connection.

This means that a large amount of packets–in is sent to the controller and that in turn the controller adds and deletes many flows from the switches.

I use the MD-SAL API to add and remove flows from the switches without going through the datastore.

The version of ODL used is Boron-SR3.

 

At the moment, I am writing a POC firewall with only one switch and I run into the following problem.

I run a stress test that involves creating TCP connection from a client machine to a server machine (running Apache).

The 2 machines are connected by an OVS switch connected to ODL.

 

When the stress test is run with a low number of connections/second (~50) everything works fine.

When using ~200 TCP connections/second, the firewall seems to work fine for a few seconds.

Then the duration of the TCP connections start to increase and the controller ends up disconnection from the OVS.

 

The following errors appear in the log:

 

2017-04-07 09:42:34,419 | ERROR | pool-15-thread-1 | OutboundQueueProviderImpl        | 192 - org.opendaylight.openflowplugin.impl - 0.3.2.Boron-SR2 | No queue present, failing request

 

And the future returned by salFlowService.addFlow show the following error message:

 

Error: [operation-failed]: Device disconnected

 

The following exception also occurs:

org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: No implementation of RPC AbsoluteSchemaPath{path=[(urn:opendaylight:flow:service?revision=2013-08-19)add-flow]} available

                at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRoutingTable.invokeRpc(DOMRpcRoutingTable.java:173)[154:org.opendaylight.controller.sal-broker-impl:1.4.3.Boron-SR3]

                at org.opendaylight.controller.md.sal.dom.broker.impl.DOMRpcRouter.invokeRpc(DOMRpcRouter.java:102)[154:org.opendaylight.controller.sal-broker-impl:1.4.3.Boron-SR3]

                at Proxy1d02b267_bf62_4c92_a739_263c328d7038.invokeRpc(Unknown Source)[:]

                at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke0(RpcServiceAdapter.java:65)[156:org.opendaylight.controller.sal-binding-broker-impl:1.4.3.Boron-SR3]

                at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.access$000(RpcServiceAdapter.java:43)[156:org.opendaylight.controller.sal-binding-broker-impl:1.4.3.Boron-SR3]

                at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter$RpcInvocationStrategy.invoke(RpcServiceAdapter.java:159)[156:org.opendaylight.controller.sal-binding-broker-impl:1.4.3.Boron-SR3]

                at org.opendaylight.controller.md.sal.binding.impl.RpcServiceAdapter.invoke(RpcServiceAdapter.java:96)[156:org.opendaylight.controller.sal-binding-broker-impl:1.4.3.Boron-SR3]

                at com.sun.proxy.$Proxy54.addFlow(Unknown Source)[120:org.opendaylight.openflowplugin.model.flow-service:0.3.3.Boron-SR3]

 

The OVS logs also shows that it’s the controller that breaks the connection.

 

The JVM has 8GB of memory allocated and uses at most 3 so it’s not a memory issue.

The machine running ODL has 8 logical cores, of which only one is used.

CPU and memory usage is also pretty low on the machine running OVS.

 

So my question would be whether you have an idea of what I could do to solve this issue or a direction in which to investigate?

 

Thanks in advance,

 

Thomas Ferrandiz

{P}

R&D Engineer

Network Architecture & Security

 

 

1219 AVENUE CHAMPS BLANCS
35510 CESSON-SÉVIGNÉ (FR)