Date   

Re: [openflowplugin-dev] StackedSegment SEGMENT_SIZE = 4096

Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES@Cisco) <mirehak@...>
 

Hi Mageshwaran,

SEGMENT_SIZE must be power of 2 in order to fit into max long (in case that message xid reaches 2^64). But that is just an implementation detail.

Queue segments are arrays and amount of used instances is driven by demand from consumer (e.g.: ofplugin outbound mechanism). By low traffic there might be only 2 segment instances used. As soon as the first one is full then the overlaying outbound queue takes the second one. And as soon as all messages in the first segment are done (confirmed, failed or covered by subsequent barrier)  then the first segment gets recycled. The original message reference is erased upon being sent and corresponding callback reference is erased upon completion/failure fired. This makes segment entries (OutboundQueueEntry) friendly to garbage collector.


In nutshell :

 Segments are cheap.

 And as soon as a OF message is subject to rpc (or some asynchronous mechanism) then it is essential to detect when device finished processing it.

 In my opinion - until there is no profiling data pointing at memory leak around outbound queue and segments I would postpone any optimization. Especially when based on setups with low utilization.


Regarding the RpcResponseKey - this is the old way of getting device reply within a hardcoded interval. It is still used before the outbound queue mechanism is plugged in (between connection established and protocol handshake finish).



Regards,

Michal




From: openflowjava-dev-bounces@... <openflowjava-dev-bounces@...> on behalf of Anil Vishnoi <vishnoianil@...>
Sent: Monday, February 20, 2017 22:35
To: Mageshwaran Vaithianathan
Cc: openflowjava-dev@...; openflowplugin-dev@...
Subject: Re: [openflowjava-dev] [openflowplugin-dev] StackedSegment SEGMENT_SIZE = 4096
 
+openflowjava

On Wed, Feb 15, 2017 at 2:20 AM, Mageshwaran Vaithianathan <mageshwaran.v@...> wrote:

Hello Team,


    Can you please let me know the reason for setting the SEGMENT_SIZE to 4096 in StackedSegment.java? . What will be the appropriate use-case for the same.


 "/openflowjava/openflow-protocol-impl/src/main/java/org/opendaylight/openflowjava/protocol/impl/core/connection/StackedSegment.java"


I am observing that the queue is not getting completely utilised always. My understanding is only those mesages which expects a Reply from the switch will be using this segment for the RpcResponseKey. Is this interpretation correct? 



Best Regards

Mageshwaran Vaithianathan


_______________________________________________
openflowplugin-dev mailing list
openflowplugin-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev




--
Thanks
Anil


Re: [openflowplugin-dev] StackedSegment SEGMENT_SIZE = 4096

Anil Vishnoi
 

+openflowjava

On Wed, Feb 15, 2017 at 2:20 AM, Mageshwaran Vaithianathan <mageshwaran.v@...> wrote:

Hello Team,


    Can you please let me know the reason for setting the SEGMENT_SIZE to 4096 in StackedSegment.java? . What will be the appropriate use-case for the same.


 "/openflowjava/openflow-protocol-impl/src/main/java/org/opendaylight/openflowjava/protocol/impl/core/connection/StackedSegment.java"


I am observing that the queue is not getting completely utilised always. My understanding is only those mesages which expects a Reply from the switch will be using this segment for the RpcResponseKey. Is this interpretation correct? 



Best Regards

Mageshwaran Vaithianathan


_______________________________________________
openflowplugin-dev mailing list
openflowplugin-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev




--
Thanks
Anil


Carbon M4 offset 1 status - openflowjava

Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES@Cisco) <mirehak@...>
 

Hello,


please see Openflow Protocol Library M4 status below.

 
Openflowjava:

 

1. Please provide updates on any previously-incomplete items from prior milestone readouts.
    ​- None previously incomplete items

2. Has your project achieved API freeze such that all externally accessible Stable or Provisional APIs will not be modified after now?
    - Yes, https://git.opendaylight.org/gerrit/#/q/file:%255Eopenflow-protocol-api/.%252B+status:merged+project:openflowjava​

3. Do you have content in your project documentation?
    - Yes, developer guide - Approximately 4100 words,
      https://git.opendaylight.org/gerrit/gitweb?p=docs.git;a=blob;f=docs/developer-guide/openflow-protocol-library-developer-guide.rst;h=37a8fcd85f0a2111ae59a7e2e74d60cd3ce5e110;hb=HEAD
      http://docs.opendaylight.org/en/latest/developer-guide/openflow-protocol-library-developer-guide.html

4. Has your project met the requirements to be included in Maven Central [2]?
    - Yes, except GPG signing.


5. Were project-specific deliverables planned for this milestone delivered successfully?
    - No Deliverables

6. Have you started automated system testing for your top-level features.
    - No, system test waiver granted.

7. Does your project use any ports, including for testing?
    - Yes: 6653, 6633
    - If yes, have you updated the wiki [3] with all ports used?
        - Yes


8. Does your project build successful in Autorelease?
    - Yes, https://jenkins.opendaylight.org/releng/view/openflowjava/job/openflowjava-validate-autorelease-carbon/24/



Regards,

Michal


Re: Barrier Message Timeout - 500ms - Rational behind the value

Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES@Cisco) <mirehak@...>
 

Hi Muthukrishnan,

there is no golden standard behind this value. Setting it to 100ms or even lower value will result into low latency system but this also decreases throughput of system.


If you have few switches and little no big requirement on flows per second throughput then 100ms might suit better to your setup. On the other hand if there are hundreds of devices and throughput of thousands of flows per second required then 500ms or even higher will improve overall performance.


Also there is a complementary parameter - amount of messages in outbound queue without barrier inside. The more messages kept in queue the higher throughput you shall get. But there is memory cost and also by extending size of those queues (per device) there comes overload risk of md-sal notification queue.


True - this value might be configurable. But that would solve the situation only for static requirements.



Regards,

Michal




From: openflowjava-dev-bounces@... <openflowjava-dev-bounces@...> on behalf of Muthukrishnan Thangasamy <muthukrishnan.t@...>
Sent: Wednesday, February 15, 2017 08:13
To: openflowplugin-dev@...
Cc: openflowjava-dev@...
Subject: [openflowjava-dev] Barrier Message Timeout - 500ms - Rational behind the value
 


Dear Team ,


I have been working in Lithium and Boron Stable Version .


Lets take Boron , In openflowplugin.cfg default timer for barrier message is 500ms is there any

rational behind the same ? why not 100ms ?


If I push via SAL-FLOW RPC , total time taken for flow push = Time taken for flow push + barrier delay  = ~ 30 ms + 520 ms (barrier ).


If I change the value from 500ms to 100 ms I am getting drastic performance improvement in flow push processing . Can anyone tell why we have gone for 500ms as default timer value ?

How we have arrived this golden value of 500ms ?



Reference.


File : openflowplugin.cfg

"#Timeout interval in milliseconds between each barrier message.

#Default value is set to 500 milliseconds
barrier-interval-timeout-limit=500"

Regards
Muthukrishnan
9952012433


Barrier Message Timeout - 500ms - Rational behind the value

Muthukrishnan T
 


Dear Team ,


I have been working in Lithium and Boron Stable Version .


Lets take Boron , In openflowplugin.cfg default timer for barrier message is 500ms is there any

rational behind the same ? why not 100ms ?


If I push via SAL-FLOW RPC , total time taken for flow push = Time taken for flow push + barrier delay  = ~ 30 ms + 520 ms (barrier ).


If I change the value from 500ms to 100 ms I am getting drastic performance improvement in flow push processing . Can anyone tell why we have gone for 500ms as default timer value ?

How we have arrived this golden value of 500ms ?



Reference.


File : openflowplugin.cfg

"#Timeout interval in milliseconds between each barrier message.

#Default value is set to 500 milliseconds
barrier-interval-timeout-limit=500"

Regards
Muthukrishnan
9952012433


Re: [openflowplugin-dev] OF TLS support in Boron

Luis Gomez <ecelgp@...>
 

Thanks Vishal,

It works with the new config files after changing TCP->TLS and removing the last line: <cipher-suites></cipher-suites>

I just updated TLS instruction:


BR/Luis



On Feb 13, 2017, at 7:59 PM, Vishal Thapar <vishal.thapar@...> wrote:

Hi Luis,
 
Based off the patch, the file where you configure TLS is now different. The file location will be:
 
<ODLINSTALL>/etc/opendaylight/datastore/initial/config/default-openflow-connection-config.xml
<ODLINSTALL>/etc/opendaylight/datastore/initial/config/legacy-openflow-connection-config.xml
 
These files will be created on startup, though you can manually create the folder and files before starting ODL too. The parameters are practically same. Hope this helps.
 
Regards,
Vishal.
 
From: openflowplugin-dev-bounces@... [mailto:openflowplugin-dev-bounces@...] On Behalf Of Luis Gomez
Sent: 14 February 2017 09:22
To: Abhijit Kumbhare <abhijitkoss@...>; Mohamed ElSerngawy <melserngawy@...>; Jozef Bacigál <jozef.bacigal@...>; Michal Polkoráb <michal.polkorab@...>; Shuva Jyoti Kar <shuva.jyoti.kar@...>
Cc: openflowjava-dev@...; Openflowplugin-dev <openflowplugin-dev@...>
Subject: Re: [openflowplugin-dev] OF TLS support in Boron
 
Actually, I think these are the patches that changed the TLS configuration:
 
 
Can anyone please explain how does it work now?
 
BR/Lus
 
 
On Feb 13, 2017, at 7:29 PM, Luis Gomez <ecelgp@...> wrote:
 
I see something changed here:
 
 
Can anyone explain how to configure TLS after these changes?
 
BR/Luis
 
 
 
On Feb 13, 2017, at 4:56 PM, Abhijit Kumbhare <abhijitkoss@...> wrote:
 
Yes - please open a bug. We can always increase or  lower the priority later based on more investigations.
 
On Mon, Feb 13, 2017 at 3:14 PM Luis Gomez <ecelgp@...> wrote:
FYI, I have just tried this instruction below on Boron and it does not work (it works in Beryllium). Does anybody know if something changed Beryllium -> Boron with regards to TLS support or should I open a blocker bug?
 


Re: [openflowplugin-dev] OF TLS support in Boron

Vishal Thapar <vishal.thapar@...>
 

Hi Luis,

 

Based off the patch, the file where you configure TLS is now different. The file location will be:

 

<ODLINSTALL>/etc/opendaylight/datastore/initial/config/default-openflow-connection-config.xml

<ODLINSTALL>/etc/opendaylight/datastore/initial/config/legacy-openflow-connection-config.xml

 

These files will be created on startup, though you can manually create the folder and files before starting ODL too. The parameters are practically same. Hope this helps.

 

Regards,

Vishal.

 

From: openflowplugin-dev-bounces@... [mailto:openflowplugin-dev-bounces@...] On Behalf Of Luis Gomez
Sent: 14 February 2017 09:22
To: Abhijit Kumbhare <abhijitkoss@...>; Mohamed ElSerngawy <melserngawy@...>; Jozef Bacigál <jozef.bacigal@...>; Michal Polkoráb <michal.polkorab@...>; Shuva Jyoti Kar <shuva.jyoti.kar@...>
Cc: openflowjava-dev@...; Openflowplugin-dev <openflowplugin-dev@...>
Subject: Re: [openflowplugin-dev] OF TLS support in Boron

 

Actually, I think these are the patches that changed the TLS configuration:

 

 

Can anyone please explain how does it work now?

 

BR/Lus

 

 

On Feb 13, 2017, at 7:29 PM, Luis Gomez <ecelgp@...> wrote:

 

I see something changed here:

 

 

Can anyone explain how to configure TLS after these changes?

 

BR/Luis

 

 

 

On Feb 13, 2017, at 4:56 PM, Abhijit Kumbhare <abhijitkoss@...> wrote:

 

Yes - please open a bug. We can always increase or  lower the priority later based on more investigations.

 

On Mon, Feb 13, 2017 at 3:14 PM Luis Gomez <ecelgp@...> wrote:

FYI, I have just tried this instruction below on Boron and it does not work (it works in Beryllium). Does anybody know if something changed Beryllium -> Boron with regards to TLS support or should I open a blocker bug?

 

 

 


Re: OF TLS support in Boron

Luis Gomez <ecelgp@...>
 

Actually, I think these are the patches that changed the TLS configuration:


Can anyone please explain how does it work now?

BR/Lus


On Feb 13, 2017, at 7:29 PM, Luis Gomez <ecelgp@...> wrote:

I see something changed here:


Can anyone explain how to configure TLS after these changes?

BR/Luis



On Feb 13, 2017, at 4:56 PM, Abhijit Kumbhare <abhijitkoss@...> wrote:

Yes - please open a bug. We can always increase or  lower the priority later based on more investigations.

On Mon, Feb 13, 2017 at 3:14 PM Luis Gomez <ecelgp@...> wrote:
FYI, I have just tried this instruction below on Boron and it does not work (it works in Beryllium). Does anybody know if something changed Beryllium -> Boron with regards to TLS support or should I open a blocker bug?




Re: OF TLS support in Boron

Luis Gomez <ecelgp@...>
 

I see something changed here:


Can anyone explain how to configure TLS after these changes?

BR/Luis



On Feb 13, 2017, at 4:56 PM, Abhijit Kumbhare <abhijitkoss@...> wrote:

Yes - please open a bug. We can always increase or  lower the priority later based on more investigations.

On Mon, Feb 13, 2017 at 3:14 PM Luis Gomez <ecelgp@...> wrote:
FYI, I have just tried this instruction below on Boron and it does not work (it works in Beryllium). Does anybody know if something changed Beryllium -> Boron with regards to TLS support or should I open a blocker bug?



Re: OF TLS support in Boron

Abhijit Kumbhare
 

Yes - please open a bug. We can always increase or  lower the priority later based on more investigations.

On Mon, Feb 13, 2017 at 3:14 PM Luis Gomez <ecelgp@...> wrote:
FYI, I have just tried this instruction below on Boron and it does not work (it works in Beryllium). Does anybody know if something changed Beryllium -> Boron with regards to TLS support or should I open a blocker bug?


OF TLS support in Boron

Luis Gomez <ecelgp@...>
 

FYI, I have just tried this instruction below on Boron and it does not work (it works in Beryllium). Does anybody know if something changed Beryllium -> Boron with regards to TLS support or should I open a blocker bug?


OpenDaylight packet distribution algorithm

Mohamad Darianian <mohamad.drnn@...>
 

​​Hi Everyone,

I was googling for a while but was not successful to find a solid answer for my question, and I would greatly appreciate your comments. 
I was wondering to know what 'switch partitioning' algorithm or techniques ODL employs for distribution and allocation of connected OpenFlow switches to worker threads running in a single instance of ODL? Say we have a bunch of switches that are sending packet_in messages at about the same time to one instance of ODL, which packet distribution algorithm (e.g., round-robin, etc) OpenDaylight uses? 

Thanks,
Mohamad​


Re: bug 5173 - "deserialization failed" WARN messages

Bertrand Low <Bertrand.Low@...>
 

Hi Michal,

 

Thanks for your help on this. We have since tested the CSIT jobs with ODL distributions carrying the Bug 7071 fix and have not observed the “deserialization failed” WARN messages. Bug 7071 appears to have resolved the issue for us.

 

If we observe the “msgType: 300” or other similar erros in the future, we will let you know.

 

Cheers,

Bertrand

 

From: Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES at Cisco) [mailto:mirehak@...]
Sent: January 13, 2017 9:25 AM
To: Bertrand Low <Bertrand.Low@...>; michal.polkorab@...
Cc: Venkatrangan G - ERS, HCL Tech <venkatrangang@...>; openflowjava-dev@...; Vinh Nguyen <Vinh.Nguyen@...>
Subject: Re: [openflowjava-dev] bug 5173 - "deserialization failed" WARN messages

 

Hi Bertrand,

this is weird - in logs of odl1 machine I found:

 

2017-01-11 05:19:45,301 | INFO  | entLoopGroup-9-1 | ActionDeserializer               | 284 - org.opendaylight.openflowplugin.openflowjava-extension-nicira-api - 0.3.3.SNAPSHOT | No deserializer was found for key NiciraActionDeserializerKey [version=4, subtype=19]

 

This refers to FinTimeoutCodec which should be registered (unless you did not install feature odl-openflowplugin-nxm-extensions).

Nevertheless right after this we got:

2017-01-11 05:19:45,301 | WARN  | entLoopGroup-9-1 | OFDecoder                        | 270 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.3.SNAPSHOT | Message deserialization failed
 java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.common.action.rev150203.actions.grouping.Action msgType: 300 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly

 

I am not sure if those 2 issues are connected but the second one claims there is type=300 which is not defined in OF-spec.

 

That's why I believe we need at least definitions of those flows (xml/json dump of config, if possible).  Traffic captures would help to detect which side is spoiling payload. If you could cover just flow installation phase it would greatly simplify reading. Honestly I went into real dark wood by crawling through the robot logs. Timestamps are shifted and source is huge. So finding pairs is hard for me.

 

 

Regards,

Michal

 


From: Bertrand Low <Bertrand.Low@...>
Sent: Friday, January 13, 2017 01:16
To: Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES at Cisco); michal.polkorab@...
Cc: Venkatrangan G - ERS, HCL Tech; openflowjava-dev@...; Vinh Nguyen
Subject: RE: [openflowjava-dev] bug 5173 - "deserialization failed" WARN messages

 

Hi Michal,

 

I’ve attached the dump-flow. This is actually from the following CSIT run:

https://logs.opendaylight.org/sandbox/jenkins091/netvirt-legacy-csit-3node-openstack-mitaka-upstream-boron/6/archives/log.html.gz

In that run, we observe the “deserialization failed” WARN messages.

 

Please let me know if the attached dump-flow helps to give you some clues as to which flows may be causing the WARN messages. If you need us to grab the corresponding traffic capture of the statistics reply and the xml/json and FLOW_MOD message for a particular flow, let me know and we will try and get those.

 

Thanks,

Bertrand

 

From: Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES at Cisco) [mailto:mirehak@...]
Sent: January 11, 2017 5:40 AM
To: Bertrand Low <Bertrand.Low@...>; michal.polkorab@...
Cc: Venkatrangan G - ERS, HCL Tech <venkatrangang@...>; openflowjava-dev@...; Vinh Nguyen <Vinh.Nguyen@...>
Subject: Re: [openflowjava-dev] bug 5173 - "deserialization failed" WARN messages

 

Hi Bertrand,

the ovs-ofctl dump would be helpful. Also traffic capture of statistics reply containing that flow would shed some light.

 

And if you have xml/json of configured flow (or rpc input) and capture of corresponding flow-mod message then the circle would be complete I guess.

 

 

Regards,

Michal

 

 


From: Bertrand Low <Bertrand.Low@...>
Sent: Wednesday, January 11, 2017 05:52
To: Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES at Cisco); michal.polkorab@...
Cc: Venkatrangan G - ERS, HCL Tech; openflowjava-dev@...; Vinh Nguyen
Subject: RE: [openflowjava-dev] bug 5173 - "deserialization failed" WARN messages

 

Hi Michal,

 

What kind of wireshark traffic or flow information on the device would help?

For the flow information, would a simple ovs-ofctl dump-flows do?

 

Thanks,

Bertrand

 

From: Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES at Cisco) [mailto:mirehak@...]
Sent: January 10, 2017 6:07 AM
To: Bertrand Low <Bertrand.Low@...>; michal.polkorab@...
Cc: Venkatrangan G - ERS, HCL Tech <venkatrangang@...>; openflowjava-dev@...; Vinh Nguyen <Vinh.Nguyen@...>
Subject: Re: [openflowjava-dev] bug 5173 - "deserialization failed" WARN messages

 

Hi Bertrand,

my bad - experimenterId == null is normal state for standard action (declared in OFP spec). Logs you provided contain action of type 60. Standard action types go up to 27 (OF-1.3).

 

Could you inspect your traffic with wireshark?

Could you inspect directly flows on your device?

 

 

Regards,

Michal

 


From: Bertrand Low <Bertrand.Low@...>
Sent: Tuesday, January 10, 2017 07:38
To: Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES at Cisco); michal.polkorab@...
Cc: Venkatrangan G - ERS, HCL Tech; openflowjava-dev@...; Vinh Nguyen
Subject: RE: [openflowjava-dev] bug 5173 - "deserialization failed" WARN messages

 

Thanks for the response, Michal.

 

I’ve copied another similar, but different message below (longer snippet of the log); this is also an example of the type of messages we would like to remove/reduce from the log:

 

2016-11-07 19:46:32,794 | WARN  | entLoopGroup-9-2 | OFDecoder                        | 294 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.1.Boron-SR1 | Message deserialization failed

java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.common.action.rev150203.actions.grouping.Action msgType: 60 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly

        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializerRegistryImpl.getDeserializer(DeserializerRegistryImpl.java:69)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.util.ListDeserializer.deserializeList(ListDeserializer.java:52)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.AbstractActionInstructionDeserializer.deserializeActions(AbstractActionInstructionDeserializer.java:38)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.ApplyActionsInstructionDeserializer.deserialize(ApplyActionsInstructionDeserializer.java:36)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.ApplyActionsInstructionDeserializer.deserialize(ApplyActionsInstructionDeserializer.java:25)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.util.ListDeserializer.deserializeList(ListDeserializer.java:53)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.setFlow(MultipartReplyMessageFactory.java:304)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.deserialize(MultipartReplyMessageFactory.java:198)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.deserialize(MultipartReplyMessageFactory.java:148)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializationFactory.deserialize(DeserializationFactory.java:59)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:55)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:28)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)[140:io.netty.codec:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)[140:io.netty.codec:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.timeout.ReadTimeoutHandler.channelRead(ReadTimeoutHandler.java:152)[142:io.netty.handler:4.0.37.Final]

        at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:39)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:879)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:387)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:296)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[138:io.netty.common:4.0.37.Final]

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)[138:io.netty.common:4.0.37.Final]

        at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]

2016-11-07 19:46:33,291 | INFO  | entLoopGroup-9-2 | MultipartRequestOnTheFlyCallback | 299 - org.opendaylight.openflowplugin.impl - 0.3.1.Boron-SR1 | Ofheader was null.

2016-11-07 19:46:34,349 | WARN  | entLoopGroup-9-5 | StatisticsGatheringUtils         | 299 - org.opendaylight.openflowplugin.impl - 0.3.1.Boron-SR1 | Skip write statistics. Flow hash: 1127177399 not present in DeviceFlowRegistry

2016-11-07 19:46:34,349 | WARN  | entLoopGroup-9-5 | StatisticsGatheringUtils         | 299 - org.opendaylight.openflowplugin.impl - 0.3.1.Boron-SR1 | Skip write statistics. Flow hash: -355999137 not present in DeviceFlowRegistry

2016-11-07 19:46:36,304 | INFO  | entLoopGroup-9-2 | ActionDeserializer               | 308 - org.opendaylight.openflowplugin.openflowjava-extension-nicira-api - 0.3.1.Boron-SR1 | No deserializer was found for key NiciraActionDeserializerKey [version=4, subtype=19]

2016-11-07 19:46:36,304 | WARN  | entLoopGroup-9-2 | OFDecoder                        | 294 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.1.Boron-SR1 | Message deserialization failed

java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.common.action.rev150203.actions.grouping.Action msgType: 60 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly

        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializerRegistryImpl.getDeserializer(DeserializerRegistryImpl.java:69)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.util.ListDeserializer.deserializeList(ListDeserializer.java:52)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.AbstractActionInstructionDeserializer.deserializeActions(AbstractActionInstructionDeserializer.java:38)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.ApplyActionsInstructionDeserializer.deserialize(ApplyActionsInstructionDeserializer.java:36)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.ApplyActionsInstructionDeserializer.deserialize(ApplyActionsInstructionDeserializer.java:25)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.util.ListDeserializer.deserializeList(ListDeserializer.java:53)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.setFlow(MultipartReplyMessageFactory.java:304)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.deserialize(MultipartReplyMessageFactory.java:198)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.deserialize(MultipartReplyMessageFactory.java:148)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializationFactory.deserialize(DeserializationFactory.java:59)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:55)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:28)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)[140:io.netty.codec:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)[140:io.netty.codec:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.timeout.ReadTimeoutHandler.channelRead(ReadTimeoutHandler.java:152)[142:io.netty.handler:4.0.37.Final]

        at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:39)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:879)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:387)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:296)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[138:io.netty.common:4.0.37.Final]

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)[138:io.netty.common:4.0.37.Final]

        at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]

2016-11-07 19:46:36,801 | INFO  | entLoopGroup-9-2 | MultipartRequestOnTheFlyCallback | 299 - org.opendaylight.openflowplugin.impl - 0.3.1.Boron-SR1 | Ofheader was null.

2016-11-07 19:46:37,369 | WARN  | entLoopGroup-9-5 | StatisticsGatheringUtils         | 299 - org.opendaylight.openflowplugin.impl - 0.3.1.Boron-SR1 | Skip write statistics. Flow hash: 1127177399 not present in DeviceFlowRegistry

2016-11-07 19:46:37,369 | WARN  | entLoopGroup-9-5 | StatisticsGatheringUtils         | 299 - org.opendaylight.openflowplugin.impl - 0.3.1.Boron-SR1 | Skip write statistics. Flow hash: -355999137 not present in DeviceFlowRegistry

2016-11-07 19:46:39,815 | INFO  | entLoopGroup-9-2 | ActionDeserializer               | 308 - org.opendaylight.openflowplugin.openflowjava-extension-nicira-api - 0.3.1.Boron-SR1 | No deserializer was found for key NiciraActionDeserializerKey [version=4, subtype=19]

2016-11-07 19:46:39,815 | WARN  | entLoopGroup-9-2 | OFDecoder                        | 294 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.1.Boron-SR1 | Message deserialization failed

java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.common.action.rev150203.actions.grouping.Action msgType: 60 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly

        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializerRegistryImpl.getDeserializer(DeserializerRegistryImpl.java:69)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.util.ListDeserializer.deserializeList(ListDeserializer.java:52)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.AbstractActionInstructionDeserializer.deserializeActions(AbstractActionInstructionDeserializer.java:38)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.ApplyActionsInstructionDeserializer.deserialize(ApplyActionsInstructionDeserializer.java:36)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.ApplyActionsInstructionDeserializer.deserialize(ApplyActionsInstructionDeserializer.java:25)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.util.ListDeserializer.deserializeList(ListDeserializer.java:53)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.setFlow(MultipartReplyMessageFactory.java:304)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.deserialize(MultipartReplyMessageFactory.java:198)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.deserialize(MultipartReplyMessageFactory.java:148)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializationFactory.deserialize(DeserializationFactory.java:59)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:55)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:28)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)[140:io.netty.codec:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)[140:io.netty.codec:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.timeout.ReadTimeoutHandler.channelRead(ReadTimeoutHandler.java:152)[142:io.netty.handler:4.0.37.Final]

        at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:39)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:879)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:387)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:296)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[138:io.netty.common:4.0.37.Final]

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)[138:io.netty.common:4.0.37.Final]

        at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]

2016-11-07 19:46:40,311 | INFO  | entLoopGroup-9-2 | MultipartRequestOnTheFlyCallback | 299 - org.opendaylight.openflowplugin.impl - 0.3.1.Boron-SR1 | Ofheader was null.

2016-11-07 19:46:40,388 | WARN  | entLoopGroup-9-5 | StatisticsGatheringUtils         | 299 - org.opendaylight.openflowplugin.impl - 0.3.1.Boron-SR1 | Skip write statistics. Flow hash: 1127177399 not present in DeviceFlowRegistry

2016-11-07 19:46:40,389 | WARN  | entLoopGroup-9-5 | StatisticsGatheringUtils         | 299 - org.opendaylight.openflowplugin.impl - 0.3.1.Boron-SR1 | Skip write statistics. Flow hash: -355999137 not present in DeviceFlowRegistry

2016-11-07 19:46:43,324 | INFO  | entLoopGroup-9-2 | ActionDeserializer               | 308 - org.opendaylight.openflowplugin.openflowjava-extension-nicira-api - 0.3.1.Boron-SR1 | No deserializer was found for key NiciraActionDeserializerKey [version=4, subtype=19]

2016-11-07 19:46:43,324 | WARN  | entLoopGroup-9-2 | OFDecoder                        | 294 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.1.Boron-SR1 | Message deserialization failed

java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.common.action.rev150203.actions.grouping.Action msgType: 60 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly

        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializerRegistryImpl.getDeserializer(DeserializerRegistryImpl.java:69)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.util.ListDeserializer.deserializeList(ListDeserializer.java:52)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.AbstractActionInstructionDeserializer.deserializeActions(AbstractActionInstructionDeserializer.java:38)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.ApplyActionsInstructionDeserializer.deserialize(ApplyActionsInstructionDeserializer.java:36)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.instruction.ApplyActionsInstructionDeserializer.deserialize(ApplyActionsInstructionDeserializer.java:25)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.util.ListDeserializer.deserializeList(ListDeserializer.java:53)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.setFlow(MultipartReplyMessageFactory.java:304)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.deserialize(MultipartReplyMessageFactory.java:198)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.MultipartReplyMessageFactory.deserialize(MultipartReplyMessageFactory.java:148)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializationFactory.deserialize(DeserializationFactory.java:59)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:55)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:28)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)[140:io.netty.codec:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)[140:io.netty.codec:4.0.37.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)[140:io.netty.codec:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.handler.timeout.ReadTimeoutHandler.channelRead(ReadTimeoutHandler.java:152)[142:io.netty.handler:4.0.37.Final]

        at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:39)[294:org.opendaylight.openflowjava.openflow-protocol-impl:0.8.1.Boron-SR1]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)[139:io.netty.transport:4.0.37.Final]

        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:879)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:387)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:296)[143:io.netty.transport-native-epoll:4.0.37.Final]

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[138:io.netty.common:4.0.37.Final]

        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)[138:io.netty.common:4.0.37.Final]

        at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]

 

Thanks,

Bertrand

 

From: Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES at Cisco) [mailto:mirehak@...]
Sent: January 9, 2017 5:27 AM
To: Bertrand Low <Bertrand.Low@...>; michal.polkorab@...
Cc: Venkatrangan G - ERS, HCL Tech <venkatrangang@...>; openflowjava-dev@...; Vinh Nguyen <Vinh.Nguyen@...>
Subject: Re: [openflowjava-dev] bug 5173 - "deserialization failed" WARN messages

 

Hi Bertrand,

unfortunately I would need more details on this. Bug-5173 is about table-features update where actions and matches are encoded a bit differently than in a flow. This produces plenty of warnings after handshake when table-features are processed. And this is of no issue for devices where table-features is not supported.

 

From here on I am just guessing:

if you sent an experimenter action from device to controller (e.g.: packed inside of flow/group statistics) then this message complains about missing deserializer. Finding appropriate deserializer takes 2 steps:

 - compute lookup key (based on first bytes of experimenter element - action in this case) and

 - find deserializer in map using the key

 

First thing - in the message key.toString() shows that experimenterId is null. This is presumably a bug. Every experimenter element shall be owned by a particular vendor with assigned experimenterId. Could you provide stacktrace?

If you have registered appropriate custom deserializer then it is not found because of that null value in key. If you have not registered one then ofJava got no clue on how to handle your action.

 

Another complication ahead: deserializers of experimenter elements in ofJava consists of 2 steps - first the experimenterId is parsed out of byte-stream and then custom amount of next bytes is parsed in order to detect type and pick appropriate final deserializer.

 

Could you please provide more details on your message and stacktrace and eventually deserializer?

 

 

Regards,

Michal

 

 


From: openflowjava-dev-bounces@... <openflowjava-dev-bounces@...> on behalf of Bertrand Low <Bertrand.Low@...>
Sent: Thursday, December 29, 2016 21:36
To: michal.polkorab@...
Cc: Venkatrangan G - ERS, HCL Tech; openflowjava-dev@...; Vinh Nguyen
Subject: Re: [openflowjava-dev] bug 5173 - "deserialization failed" WARN messages

 

Hi Michal and openflowjava team,

 

Just wanted to follow-up with you on the email below in case it was missed due to the holidays. We would appreciate your guidance to know that we are taking the right approach to fix the bug. Thank you.

 

Bertrand

 

From: Bertrand Low
Sent: December 24, 2016 6:24 PM
To: michal.polkorab@...
Cc: Venkatrangan G - ERS, HCL Tech <venkatrangang@...>; openflowjava-dev@...
Subject: bug 5173 - "deserialization failed" WARN messages

 

Hi Michal and openflowjava team,

 

Regarding bug 5173, my team has found that the following kind of WARN log message creates unnecessary confusion to the end user and makes it hard to debug other issues in the log (due to the large number of such messages):

 

2016-12-18 01:35:18,356 | WARN  | entLoopGroup-7-1 | OFDecoder                        | 296 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.2.SNAPSHOT | Message deserialization failed

java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.common.action.rev150203.actions.grouping.Action msgType: 300 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctl

 

Are you or someone from the team able to help us with the removal of these messages?

 

If no one is able to help us at this time, any implementation guidance you can provide would be much-appreciated. My understanding is that we need to implement the respective serializer and deserializer for the ovs nicira extensions for all those objectClasses in order to remove these WARN messages – do you have any alternative suggestions?

 

Thanks,

Bertrand



::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.

----------------------------------------------------------------------------------------------------------------------------------------------------


OpenFlowJava M3 Status

Michal Rehak -X (mirehak - PANTHEON TECHNOLOGIES@Cisco) <mirehak@...>
 

Hello,

please see Openflow Protocol Library (Openflowjava) M3 status below:
 

Functionality Freeze:

1. Final list of externally consumable APIs defined: Yes
2. Are all your inter-project dependencies resolved (i.e., have the other projects you were counting on given you what you needed)? Yes
3. Were there any project-specific deliverables planned for this milestone? No
 
Karaf Features Defined:

4. Are all your project's features that are intended for release added to the features.xml and checked into the integration git repository? Yes
    https://git.opendaylight.org/gerrit/gitweb?p=openflowjava.git;a=blob;f=features/src/main/features/features.xml;h=63be931333918c387006083880d9940dc6317960;hb=HEAD
    ​features.xml are pulled by openflowplugin

5. List all top-level, user-facing, and stable Karaf features for your project.
    No user-facing features.
    Top-level feature: odl-openflowjava-protocol​ - whole openflowjava functionality
    Top-level features are usually installed by user-facing features (e.g.: in openflowplugin).
 
Documentation:

6. List the kinds of documentation you will provide including at least:
   - developer guide
     https://wiki.opendaylight.org/view/Openflow_Protocol_Library:Documentation

7. Have you checked in the AsciiDoc outlines to the docs repository? No
   - no feature / functionality updates since Lithium
 
Integration and Test:

8. Have you started automated system testing for your top-level features. No
​   - system test waiver approved
 
9. Have you filled out basic system test plan template for each top-level feature (karaf and not karaf) and a comprehensive system test plan template including functionality, cluster, scalability, performance, longevity/stability for each stable feature? No
   - system test waiver approved

Project Specific:

10. Have you updated your project facts with the project type category? Yes

11. Do you acknowledge the changes to the RC Blocking Bug Policy for Carbon Release [2]? Yes




Regards,

Michal




[0] https://wiki.opendaylight.org/view/Simultaneous_Release:Carbon_Release_Plan

[1] https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Feature_Integration_System_Test_Template

[2] https://lists.opendaylight.org/pipermail/tsc/2016-December/006468.html

 

 



Re: [ovsdb-dev] OVSDB scale

Sela, Guy <guy.sela@...>
 

+netvirt-dev

 

Sure, I just created one:

https://bugs.opendaylight.org/show_bug.cgi?id=7591

 

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Wednesday, January 18, 2017 11:52 PM
To: Anil Vishnoi <vishnoianil@...>
Cc: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Guy,

 

Could you raise a bug for Netvirt too?

 

OVSDB patch: https://git.opendaylight.org/gerrit/50643

 

Regards,

Vishal.

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 18 January 2017 23:34
To: Vishal Thapar <vishal.thapar@...>
Cc: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Sound good to me Vishal. Anybody planning to push a patch for this ?

 

On Wed, Jan 18, 2017 at 12:45 AM, Vishal Thapar <vishal.thapar@...> wrote:

Hi Guy,

 

Yes, you’re right. I was under wrong impression that inactivity_probe is configured as other_config so will be covered with existing yang. Just confirmed, we *will* need yang changes in OVSDB too. While at it, will also add max_backoff. I think we will want this in boron too?

 

Anil,

Any objections to the plan?

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 18 January 2017 14:02


To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I mean the API that OVSDB is exposing to its clients (In this case Netvirt).

It seems to me that OVSDB didn’t model the inactivity_probe in the YANG model itself.

After it will be modeled, OVSDB should exposed a way to define in while calling addBridge.

From the Netvirt perspective, we need an ability to configure it in a configuration file.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 5:22 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha, you mean the API that OVSDB is using? That is more of a convenience and we can add without that too. I’ll add code for it. If needed, we can add util accordingly to OVSDB. Either ways, the fix will be going in Netvirt, OVSDB change may or may not be needed.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:54
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi,

 

So it seems there should be 2 bugs?

OVSDB needs to expose this in its addBridge interface and NETVIRT should allow to configure it.

I couldn’t find how to configure it in OvsdbBridgeAugmentationBuilder and not in SouthboundUtils.addBridge

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 4:10 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

HI Guy,

 

If this knob is needed for Controller’s probe timer bug should be on netvirt as netvirt is adding controller. In case of Manager, it should be set by whosoever configures manager, which never comes from OVSDB.

 

Any configuration that goes into OVSDB in switch should ideally come from consumers of OVSDB, not plugin itself.

 

I can change this to netvirt bug, but want to make sure we are in agreement on nature of change coming in ie.e. a knob to set default inactivity for Controller that gets created by autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:30
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

https://bugs.opendaylight.org/show_bug.cgi?id=7563

 

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 3:27 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller, we were also creating controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 16:23
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

When you’re saying initial configuration do you mean before the OVS established an openflow connection?

Configuration for Controller Table in OVSDB is being set by ODL.

The CLI configuration for the inactivity probe looks like this for example:

sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000

 

So it looks like this can only be set after the connection

 

 

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:34 PM
To: Pearl, Tomer <tomer.pearl@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration. We were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.

 

Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time, you can tweak a flag captured in this: https://git.opendaylight.org/gerrit/#/c/49068/

 

If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have functional impact, so be careful.

 

Regards,

Vishal.

 

From: Pearl, Tomer [mailto:tomer.pearl@...]
Sent: 17 January 2017 15:56
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?

Thanks

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:20 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Close, inactivity_probe. Sorry for delay, had to fish in old logs.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 15:43
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

max_backoff ?

 

 

From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:10 PM
To: 'Vishal Thapar' <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:07 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many echo messages.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 14:57
To: Anil Vishnoi <vishnoianil@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Full GC in a 8G-16G Heap takes about 10+ seconds

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection won't happen atleast because of GC.

 

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Hi guys,

 

So a couple of questions:

 

1)      Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?

2)      Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 11:07 AM
To: Muthukumaran K <muthukumaran.k@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>


Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Muthu,

 

Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t recall what exactly were those.

 

Regards,

Vishal.

 

From: Muthukumaran K
Sent: 17 January 2017 14:33
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Vishal,

 

We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.

 

For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug

https://bugs.opendaylight.org/show_bug.cgi?id=4823

 

There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)

 

Regards

Muthu

 

 

 

From: Vishal Thapar
Sent: Tuesday, January 17, 2017 2:23 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>; Muthukumaran K <muthukumaran.k@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Adding Muthu.

 

Muthu,

I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?

 

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 13:49
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Thanks,

 

Did you manage to survive Full GCs at all ?

If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 8:50 AM
To: Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?

 

If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.

 

Regards,

Vishal.

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron. 

 

@vishal : do you have any number from your ovsdb testing ? 

 

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Hi Tomer,

back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.

Thanks,
JamO


[0] https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf



On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>

> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
> https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev



 

--

Thanks

Anil



 

--

Thanks

Anil



 

--

Thanks

Anil


Re: [ovsdb-dev] OVSDB scale

Vishal Thapar <vishal.thapar@...>
 

Guy,

 

Could you raise a bug for Netvirt too?

 

OVSDB patch: https://git.opendaylight.org/gerrit/50643

 

Regards,

Vishal.

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 18 January 2017 23:34
To: Vishal Thapar <vishal.thapar@...>
Cc: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Sound good to me Vishal. Anybody planning to push a patch for this ?

 

On Wed, Jan 18, 2017 at 12:45 AM, Vishal Thapar <vishal.thapar@...> wrote:

Hi Guy,

 

Yes, you’re right. I was under wrong impression that inactivity_probe is configured as other_config so will be covered with existing yang. Just confirmed, we *will* need yang changes in OVSDB too. While at it, will also add max_backoff. I think we will want this in boron too?

 

Anil,

Any objections to the plan?

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 18 January 2017 14:02


To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I mean the API that OVSDB is exposing to its clients (In this case Netvirt).

It seems to me that OVSDB didn’t model the inactivity_probe in the YANG model itself.

After it will be modeled, OVSDB should exposed a way to define in while calling addBridge.

From the Netvirt perspective, we need an ability to configure it in a configuration file.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 5:22 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha, you mean the API that OVSDB is using? That is more of a convenience and we can add without that too. I’ll add code for it. If needed, we can add util accordingly to OVSDB. Either ways, the fix will be going in Netvirt, OVSDB change may or may not be needed.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:54
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi,

 

So it seems there should be 2 bugs?

OVSDB needs to expose this in its addBridge interface and NETVIRT should allow to configure it.

I couldn’t find how to configure it in OvsdbBridgeAugmentationBuilder and not in SouthboundUtils.addBridge

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 4:10 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

HI Guy,

 

If this knob is needed for Controller’s probe timer bug should be on netvirt as netvirt is adding controller. In case of Manager, it should be set by whosoever configures manager, which never comes from OVSDB.

 

Any configuration that goes into OVSDB in switch should ideally come from consumers of OVSDB, not plugin itself.

 

I can change this to netvirt bug, but want to make sure we are in agreement on nature of change coming in ie.e. a knob to set default inactivity for Controller that gets created by autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:30
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

https://bugs.opendaylight.org/show_bug.cgi?id=7563

 

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 3:27 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller, we were also creating controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 16:23
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

When you’re saying initial configuration do you mean before the OVS established an openflow connection?

Configuration for Controller Table in OVSDB is being set by ODL.

The CLI configuration for the inactivity probe looks like this for example:

sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000

 

So it looks like this can only be set after the connection

 

 

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:34 PM
To: Pearl, Tomer <tomer.pearl@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration. We were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.

 

Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time, you can tweak a flag captured in this: https://git.opendaylight.org/gerrit/#/c/49068/

 

If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have functional impact, so be careful.

 

Regards,

Vishal.

 

From: Pearl, Tomer [mailto:tomer.pearl@...]
Sent: 17 January 2017 15:56
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?

Thanks

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:20 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Close, inactivity_probe. Sorry for delay, had to fish in old logs.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 15:43
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

max_backoff ?

 

 

From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:10 PM
To: 'Vishal Thapar' <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:07 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many echo messages.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 14:57
To: Anil Vishnoi <vishnoianil@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Full GC in a 8G-16G Heap takes about 10+ seconds

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection won't happen atleast because of GC.

 

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Hi guys,

 

So a couple of questions:

 

1)      Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?

2)      Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 11:07 AM
To: Muthukumaran K <muthukumaran.k@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>


Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Muthu,

 

Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t recall what exactly were those.

 

Regards,

Vishal.

 

From: Muthukumaran K
Sent: 17 January 2017 14:33
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Vishal,

 

We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.

 

For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug

https://bugs.opendaylight.org/show_bug.cgi?id=4823

 

There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)

 

Regards

Muthu

 

 

 

From: Vishal Thapar
Sent: Tuesday, January 17, 2017 2:23 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>; Muthukumaran K <muthukumaran.k@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Adding Muthu.

 

Muthu,

I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?

 

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 13:49
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Thanks,

 

Did you manage to survive Full GCs at all ?

If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 8:50 AM
To: Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?

 

If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.

 

Regards,

Vishal.

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron. 

 

@vishal : do you have any number from your ovsdb testing ? 

 

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Hi Tomer,

back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.

Thanks,
JamO


[0] https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf



On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>

> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
> https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev



 

--

Thanks

Anil



 

--

Thanks

Anil



 

--

Thanks

Anil


Re: [ovsdb-dev] OVSDB scale

Vishal Thapar <vishal.thapar@...>
 

I will J was just waiting for your confirmation.

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 18 January 2017 23:34
To: Vishal Thapar <vishal.thapar@...>
Cc: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Sound good to me Vishal. Anybody planning to push a patch for this ?

 

On Wed, Jan 18, 2017 at 12:45 AM, Vishal Thapar <vishal.thapar@...> wrote:

Hi Guy,

 

Yes, you’re right. I was under wrong impression that inactivity_probe is configured as other_config so will be covered with existing yang. Just confirmed, we *will* need yang changes in OVSDB too. While at it, will also add max_backoff. I think we will want this in boron too?

 

Anil,

Any objections to the plan?

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 18 January 2017 14:02


To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I mean the API that OVSDB is exposing to its clients (In this case Netvirt).

It seems to me that OVSDB didn’t model the inactivity_probe in the YANG model itself.

After it will be modeled, OVSDB should exposed a way to define in while calling addBridge.

From the Netvirt perspective, we need an ability to configure it in a configuration file.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 5:22 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha, you mean the API that OVSDB is using? That is more of a convenience and we can add without that too. I’ll add code for it. If needed, we can add util accordingly to OVSDB. Either ways, the fix will be going in Netvirt, OVSDB change may or may not be needed.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:54
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi,

 

So it seems there should be 2 bugs?

OVSDB needs to expose this in its addBridge interface and NETVIRT should allow to configure it.

I couldn’t find how to configure it in OvsdbBridgeAugmentationBuilder and not in SouthboundUtils.addBridge

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 4:10 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

HI Guy,

 

If this knob is needed for Controller’s probe timer bug should be on netvirt as netvirt is adding controller. In case of Manager, it should be set by whosoever configures manager, which never comes from OVSDB.

 

Any configuration that goes into OVSDB in switch should ideally come from consumers of OVSDB, not plugin itself.

 

I can change this to netvirt bug, but want to make sure we are in agreement on nature of change coming in ie.e. a knob to set default inactivity for Controller that gets created by autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:30
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

https://bugs.opendaylight.org/show_bug.cgi?id=7563

 

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 3:27 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller, we were also creating controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 16:23
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

When you’re saying initial configuration do you mean before the OVS established an openflow connection?

Configuration for Controller Table in OVSDB is being set by ODL.

The CLI configuration for the inactivity probe looks like this for example:

sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000

 

So it looks like this can only be set after the connection

 

 

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:34 PM
To: Pearl, Tomer <tomer.pearl@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration. We were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.

 

Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time, you can tweak a flag captured in this: https://git.opendaylight.org/gerrit/#/c/49068/

 

If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have functional impact, so be careful.

 

Regards,

Vishal.

 

From: Pearl, Tomer [mailto:tomer.pearl@...]
Sent: 17 January 2017 15:56
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?

Thanks

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:20 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Close, inactivity_probe. Sorry for delay, had to fish in old logs.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 15:43
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

max_backoff ?

 

 

From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:10 PM
To: 'Vishal Thapar' <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:07 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many echo messages.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 14:57
To: Anil Vishnoi <vishnoianil@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Full GC in a 8G-16G Heap takes about 10+ seconds

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection won't happen atleast because of GC.

 

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Hi guys,

 

So a couple of questions:

 

1)      Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?

2)      Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 11:07 AM
To: Muthukumaran K <muthukumaran.k@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>


Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Muthu,

 

Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t recall what exactly were those.

 

Regards,

Vishal.

 

From: Muthukumaran K
Sent: 17 January 2017 14:33
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Vishal,

 

We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.

 

For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug

https://bugs.opendaylight.org/show_bug.cgi?id=4823

 

There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)

 

Regards

Muthu

 

 

 

From: Vishal Thapar
Sent: Tuesday, January 17, 2017 2:23 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>; Muthukumaran K <muthukumaran.k@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Adding Muthu.

 

Muthu,

I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?

 

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 13:49
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Thanks,

 

Did you manage to survive Full GCs at all ?

If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 8:50 AM
To: Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?

 

If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.

 

Regards,

Vishal.

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron. 

 

@vishal : do you have any number from your ovsdb testing ? 

 

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Hi Tomer,

back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.

Thanks,
JamO


[0] https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf



On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>

> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
> https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev



 

--

Thanks

Anil



 

--

Thanks

Anil



 

--

Thanks

Anil


Re: [ovsdb-dev] OVSDB scale

Anil Vishnoi
 

Sound good to me Vishal. Anybody planning to push a patch for this ?

On Wed, Jan 18, 2017 at 12:45 AM, Vishal Thapar <vishal.thapar@...> wrote:

Hi Guy,

 

Yes, you’re right. I was under wrong impression that inactivity_probe is configured as other_config so will be covered with existing yang. Just confirmed, we *will* need yang changes in OVSDB too. While at it, will also add max_backoff. I think we will want this in boron too?

 

Anil,

Any objections to the plan?

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 18 January 2017 14:02


To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I mean the API that OVSDB is exposing to its clients (In this case Netvirt).

It seems to me that OVSDB didn’t model the inactivity_probe in the YANG model itself.

After it will be modeled, OVSDB should exposed a way to define in while calling addBridge.

From the Netvirt perspective, we need an ability to configure it in a configuration file.

 

From: Vishal Thapar [mailto:vishal.thapar@ericsson.com]
Sent: Tuesday, January 17, 2017 5:22 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha, you mean the API that OVSDB is using? That is more of a convenience and we can add without that too. I’ll add code for it. If needed, we can add util accordingly to OVSDB. Either ways, the fix will be going in Netvirt, OVSDB change may or may not be needed.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:54
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi,

 

So it seems there should be 2 bugs?

OVSDB needs to expose this in its addBridge interface and NETVIRT should allow to configure it.

I couldn’t find how to configure it in OvsdbBridgeAugmentationBuilder and not in SouthboundUtils.addBridge

 

From: Vishal Thapar [mailto:vishal.thapar@ericsson.com]
Sent: Tuesday, January 17, 2017 4:10 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

HI Guy,

 

If this knob is needed for Controller’s probe timer bug should be on netvirt as netvirt is adding controller. In case of Manager, it should be set by whosoever configures manager, which never comes from OVSDB.

 

Any configuration that goes into OVSDB in switch should ideally come from consumers of OVSDB, not plugin itself.

 

I can change this to netvirt bug, but want to make sure we are in agreement on nature of change coming in ie.e. a knob to set default inactivity for Controller that gets created by autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:30
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

https://bugs.opendaylight.org/show_bug.cgi?id=7563

 

 

From: Vishal Thapar [mailto:vishal.thapar@ericsson.com]
Sent: Tuesday, January 17, 2017 3:27 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller, we were also creating controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 16:23
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

When you’re saying initial configuration do you mean before the OVS established an openflow connection?

Configuration for Controller Table in OVSDB is being set by ODL.

The CLI configuration for the inactivity probe looks like this for example:

sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000

 

So it looks like this can only be set after the connection

 

 

 

From: Vishal Thapar [mailto:vishal.thapar@ericsson.com]
Sent: Tuesday, January 17, 2017 12:34 PM
To: Pearl, Tomer <tomer.pearl@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration. We were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.

 

Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time, you can tweak a flag captured in this: https://git.opendaylight.org/gerrit/#/c/49068/

 

If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have functional impact, so be careful.

 

Regards,

Vishal.

 

From: Pearl, Tomer [mailto:tomer.pearl@...]
Sent: 17 January 2017 15:56
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?

Thanks

 

From: Vishal Thapar [mailto:vishal.thapar@ericsson.com]
Sent: Tuesday, January 17, 2017 12:20 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Close, inactivity_probe. Sorry for delay, had to fish in old logs.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 15:43
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

max_backoff ?

 

 

From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:10 PM
To: 'Vishal Thapar' <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

 

From: Vishal Thapar [mailto:vishal.thapar@ericsson.com]
Sent: Tuesday, January 17, 2017 12:07 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many echo messages.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 14:57
To: Anil Vishnoi <vishnoianil@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Full GC in a 8G-16G Heap takes about 10+ seconds

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection won't happen atleast because of GC.

 

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Hi guys,

 

So a couple of questions:

 

1)      Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?

2)      Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?

 

From: Vishal Thapar [mailto:vishal.thapar@ericsson.com]
Sent: Tuesday, January 17, 2017 11:07 AM
To: Muthukumaran K <muthukumaran.k@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>


Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Muthu,

 

Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t recall what exactly were those.

 

Regards,

Vishal.

 

From: Muthukumaran K
Sent: 17 January 2017 14:33
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Vishal,

 

We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.

 

For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug

https://bugs.opendaylight.org/show_bug.cgi?id=4823

 

There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)

 

Regards

Muthu

 

 

 

From: Vishal Thapar
Sent: Tuesday, January 17, 2017 2:23 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>; Muthukumaran K <muthukumaran.k@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Adding Muthu.

 

Muthu,

I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?

 

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 13:49
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Thanks,

 

Did you manage to survive Full GCs at all ?

If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.

 

From: Vishal Thapar [mailto:vishal.thapar@ericsson.com]
Sent: Tuesday, January 17, 2017 8:50 AM
To: Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org; Sela, Guy <guy.sela@...>
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?

 

If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.

 

Regards,

Vishal.

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@lists.opendaylight.org; ovsdb-dev@....org; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron. 

 

@vishal : do you have any number from your ovsdb testing ? 

 

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Hi Tomer,

back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.

Thanks,
JamO


[0] https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf



On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>

> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@....org
> https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev



 

--

Thanks

Anil



 

--

Thanks

Anil




--
Thanks
Anil


Re: [ovsdb-dev] OVSDB scale

Vishal Thapar <vishal.thapar@...>
 

Hi Guy,

 

Yes, you’re right. I was under wrong impression that inactivity_probe is configured as other_config so will be covered with existing yang. Just confirmed, we *will* need yang changes in OVSDB too. While at it, will also add max_backoff. I think we will want this in boron too?

 

Anil,

Any objections to the plan?

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 18 January 2017 14:02
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I mean the API that OVSDB is exposing to its clients (In this case Netvirt).

It seems to me that OVSDB didn’t model the inactivity_probe in the YANG model itself.

After it will be modeled, OVSDB should exposed a way to define in while calling addBridge.

From the Netvirt perspective, we need an ability to configure it in a configuration file.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 5:22 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha, you mean the API that OVSDB is using? That is more of a convenience and we can add without that too. I’ll add code for it. If needed, we can add util accordingly to OVSDB. Either ways, the fix will be going in Netvirt, OVSDB change may or may not be needed.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:54
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi,

 

So it seems there should be 2 bugs?

OVSDB needs to expose this in its addBridge interface and NETVIRT should allow to configure it.

I couldn’t find how to configure it in OvsdbBridgeAugmentationBuilder and not in SouthboundUtils.addBridge

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 4:10 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

HI Guy,

 

If this knob is needed for Controller’s probe timer bug should be on netvirt as netvirt is adding controller. In case of Manager, it should be set by whosoever configures manager, which never comes from OVSDB.

 

Any configuration that goes into OVSDB in switch should ideally come from consumers of OVSDB, not plugin itself.

 

I can change this to netvirt bug, but want to make sure we are in agreement on nature of change coming in ie.e. a knob to set default inactivity for Controller that gets created by autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:30
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

https://bugs.opendaylight.org/show_bug.cgi?id=7563

 

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 3:27 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller, we were also creating controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 16:23
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

When you’re saying initial configuration do you mean before the OVS established an openflow connection?

Configuration for Controller Table in OVSDB is being set by ODL.

The CLI configuration for the inactivity probe looks like this for example:

sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000

 

So it looks like this can only be set after the connection

 

 

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:34 PM
To: Pearl, Tomer <tomer.pearl@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration. We were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.

 

Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time, you can tweak a flag captured in this: https://git.opendaylight.org/gerrit/#/c/49068/

 

If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have functional impact, so be careful.

 

Regards,

Vishal.

 

From: Pearl, Tomer [mailto:tomer.pearl@...]
Sent: 17 January 2017 15:56
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?

Thanks

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:20 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Close, inactivity_probe. Sorry for delay, had to fish in old logs.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 15:43
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

max_backoff ?

 

 

From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:10 PM
To: 'Vishal Thapar' <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:07 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many echo messages.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 14:57
To: Anil Vishnoi <vishnoianil@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Full GC in a 8G-16G Heap takes about 10+ seconds

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection won't happen atleast because of GC.

 

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Hi guys,

 

So a couple of questions:

 

1)      Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?

2)      Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 11:07 AM
To: Muthukumaran K <muthukumaran.k@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>


Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Muthu,

 

Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t recall what exactly were those.

 

Regards,

Vishal.

 

From: Muthukumaran K
Sent: 17 January 2017 14:33
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Vishal,

 

We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.

 

For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug

https://bugs.opendaylight.org/show_bug.cgi?id=4823

 

There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)

 

Regards

Muthu

 

 

 

From: Vishal Thapar
Sent: Tuesday, January 17, 2017 2:23 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>; Muthukumaran K <muthukumaran.k@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Adding Muthu.

 

Muthu,

I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?

 

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 13:49
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Thanks,

 

Did you manage to survive Full GCs at all ?

If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 8:50 AM
To: Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?

 

If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.

 

Regards,

Vishal.

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron. 

 

@vishal : do you have any number from your ovsdb testing ? 

 

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Hi Tomer,

back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.

Thanks,
JamO


[0] https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf



On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>

> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
> https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev



 

--

Thanks

Anil



 

--

Thanks

Anil


Re: [ovsdb-dev] OVSDB scale

Sela, Guy <guy.sela@...>
 

I mean the API that OVSDB is exposing to its clients (In this case Netvirt).

It seems to me that OVSDB didn’t model the inactivity_probe in the YANG model itself.

After it will be modeled, OVSDB should exposed a way to define in while calling addBridge.

From the Netvirt perspective, we need an ability to configure it in a configuration file.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 5:22 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha, you mean the API that OVSDB is using? That is more of a convenience and we can add without that too. I’ll add code for it. If needed, we can add util accordingly to OVSDB. Either ways, the fix will be going in Netvirt, OVSDB change may or may not be needed.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:54
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi,

 

So it seems there should be 2 bugs?

OVSDB needs to expose this in its addBridge interface and NETVIRT should allow to configure it.

I couldn’t find how to configure it in OvsdbBridgeAugmentationBuilder and not in SouthboundUtils.addBridge

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 4:10 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

HI Guy,

 

If this knob is needed for Controller’s probe timer bug should be on netvirt as netvirt is adding controller. In case of Manager, it should be set by whosoever configures manager, which never comes from OVSDB.

 

Any configuration that goes into OVSDB in switch should ideally come from consumers of OVSDB, not plugin itself.

 

I can change this to netvirt bug, but want to make sure we are in agreement on nature of change coming in ie.e. a knob to set default inactivity for Controller that gets created by autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:30
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

https://bugs.opendaylight.org/show_bug.cgi?id=7563

 

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 3:27 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller, we were also creating controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 16:23
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

When you’re saying initial configuration do you mean before the OVS established an openflow connection?

Configuration for Controller Table in OVSDB is being set by ODL.

The CLI configuration for the inactivity probe looks like this for example:

sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000

 

So it looks like this can only be set after the connection

 

 

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:34 PM
To: Pearl, Tomer <tomer.pearl@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration. We were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.

 

Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time, you can tweak a flag captured in this: https://git.opendaylight.org/gerrit/#/c/49068/

 

If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have functional impact, so be careful.

 

Regards,

Vishal.

 

From: Pearl, Tomer [mailto:tomer.pearl@...]
Sent: 17 January 2017 15:56
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?

Thanks

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:20 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Close, inactivity_probe. Sorry for delay, had to fish in old logs.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 15:43
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

max_backoff ?

 

 

From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:10 PM
To: 'Vishal Thapar' <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:07 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many echo messages.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 14:57
To: Anil Vishnoi <vishnoianil@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Full GC in a 8G-16G Heap takes about 10+ seconds

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection won't happen atleast because of GC.

 

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Hi guys,

 

So a couple of questions:

 

1)      Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?

2)      Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 11:07 AM
To: Muthukumaran K <muthukumaran.k@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>


Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Muthu,

 

Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t recall what exactly were those.

 

Regards,

Vishal.

 

From: Muthukumaran K
Sent: 17 January 2017 14:33
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Vishal,

 

We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.

 

For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug

https://bugs.opendaylight.org/show_bug.cgi?id=4823

 

There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)

 

Regards

Muthu

 

 

 

From: Vishal Thapar
Sent: Tuesday, January 17, 2017 2:23 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>; Muthukumaran K <muthukumaran.k@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Adding Muthu.

 

Muthu,

I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?

 

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 13:49
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Thanks,

 

Did you manage to survive Full GCs at all ?

If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 8:50 AM
To: Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?

 

If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.

 

Regards,

Vishal.

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron. 

 

@vishal : do you have any number from your ovsdb testing ? 

 

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Hi Tomer,

back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.

Thanks,
JamO


[0] https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf



On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>

> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
> https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev



 

--

Thanks

Anil



 

--

Thanks

Anil

101 - 120 of 861