z.archive.openflowjava-dev@lists.opendaylight.org

Home Messages Hashtags Subgroups

This group is locked. No changes can be made to the group while it is locked.

Date Date 1 - 20 of 30

[ovsdb-dev] OVSDB scale

Jamo Luhrsen <jluhrsen@...>

Hi Tomer,

back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.

Thanks,
JamO


[0] https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf

Show quoted text

On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
Hi,

I’m trying to bring up a setup with one ODL controller and 200+ OVSs.

I’m testing with Boron SR1 code

Are there any reports about ODL scale tests that I can look at ?

Is 200 OVSs an amount that supposed to work?

Thanks,

Tomer P.

_______________________________________________
ovsdb-dev mailing list
ovsdb-dev@...
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev

Anil Vishnoi

#719

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.

@vishal : do you have any number from your ovsdb testing ?

Show quoted text

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Thanks

Anil

Vishal Thapar <vishal.thapar@...>

#722

We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?

If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.

Regards,

Vishal.

Show quoted text

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.

@vishal : do you have any number from your ovsdb testing ?

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Thanks

Anil

Sela, Guy <guy.sela@...>

#723

Thanks,

Did you manage to survive Full GCs at all ?

If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.

Show quoted text

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 8:50 AM
To: Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

Regards,

Vishal.

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.

@vishal : do you have any number from your ovsdb testing ?

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Thanks

Anil

Vishal Thapar <vishal.thapar@...>

#724

Adding Muthu.

Muthu,

I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?

Regards,

Vishal.

Show quoted text

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 13:49
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

Thanks,

Did you manage to survive Full GCs at all ?

Regards,

Vishal.

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.

@vishal : do you have any number from your ovsdb testing ?

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Thanks

Anil

Muthukumaran K <muthukumaran.k@...>

#725

Hi Vishal,

We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.

For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug

https://bugs.opendaylight.org/show_bug.cgi?id=4823

There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)

Regards

Muthu

Show quoted text

From: Vishal Thapar
Sent: Tuesday, January 17, 2017 2:23 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>; Muthukumaran K <muthukumaran.k@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

Adding Muthu.

Muthu,

I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?

Regards,

Vishal.

Thanks,

Did you manage to survive Full GCs at all ?

Regards,

Vishal.

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.

@vishal : do you have any number from your ovsdb testing ?

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Thanks

Anil

Vishal Thapar <vishal.thapar@...>

#726

Hi Muthu,

Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t recall what exactly were those.

Regards,

Vishal.

Show quoted text

From: Muthukumaran K
Sent: 17 January 2017 14:33
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

Hi Vishal,

For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug

https://bugs.opendaylight.org/show_bug.cgi?id=4823

There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)

Regards

Muthu

Adding Muthu.

Muthu,

I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?

Regards,

Vishal.

Thanks,

Did you manage to survive Full GCs at all ?

Regards,

Vishal.

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.

@vishal : do you have any number from your ovsdb testing ?

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Thanks

Anil

Sela, Guy <guy.sela@...>

#727

Hi guys,

So a couple of questions:

1) Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?

2) Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?

Show quoted text

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 11:07 AM
To: Muthukumaran K <muthukumaran.k@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

Hi Muthu,

Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t recall what exactly were those.

Regards,

Vishal.

Hi Vishal,

For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug

https://bugs.opendaylight.org/show_bug.cgi?id=4823

There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)

Regards

Muthu

Adding Muthu.

Muthu,

I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?

Regards,

Vishal.

Thanks,

Did you manage to survive Full GCs at all ?

Regards,

Vishal.

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.

@vishal : do you have any number from your ovsdb testing ?

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Thanks

Anil

Anil Vishnoi

#720

I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection won't happen atleast because of GC.

Show quoted text

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

Sela, Guy <guy.sela@...>

#728

Full GC in a 8G-16G Heap takes about 10+ seconds

Show quoted text

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

Muthukumaran K <muthukumaran.k@...>

#729

Hi Anil,

In context of Openflowplugin (Helium) tests, we have observed this behavior when there is heavy GC induced due to immutables created and garbaged as part of flow provisioning combined with statistics-gathering. To reduce the pressure, we used to disable statistics which had a huge improvement in terms of garbage-generation. Side effect of GCs will be of course Echo Timeout.

So, as you suggest if we bump the echo timeouts, it will prevent the disconnects and we can get a more clearer picture of garbage-generation pattern. The also if we see huge garbage generation and rapid FCG count raise, we can then switch off statistics.

We can also monitor netstat -anp on the the ports connecting to OVS to see if there is a fast-producer slow-consumer type of issues (huge pile-up of send-q and recv-q can provide a hint)

Another important part (I assume this is already taken care of) is to run the system at ERROR log level instead of default INFO level

Regards

Muthu

Show quoted text

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 2:55 PM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

Vishal Thapar <vishal.thapar@...>

#730

Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many echo messages.

Regards,

Vishal.

Show quoted text

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 14:57
To: Anil Vishnoi <vishnoianil@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

Full GC in a 8G-16G Heap takes about 10+ seconds

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

Sela, Guy <guy.sela@...>

#731

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

Show quoted text

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:07 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

Regards,

Vishal.

Full GC in a 8G-16G Heap takes about 10+ seconds

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

Sela, Guy <guy.sela@...>

#732

max_backoff ?

Show quoted text

From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:10 PM
To: 'Vishal Thapar' <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

Regards,

Vishal.

Full GC in a 8G-16G Heap takes about 10+ seconds

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

Sela, Guy <guy.sela@...>

#733

Never mind, I found it, inactivity_probe

Explanation here: http://openvswitch.org/ovs-vswitchd.conf.db.5.pdf

Show quoted text

From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:13 PM
To: 'Vishal Thapar' <vishal.thapar@...>; 'Anil Vishnoi' <vishnoianil@...>
Cc: 'Muthukumaran K' <muthukumaran.k@...>; 'Jamo Luhrsen' <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; 'marcus.g.williams@...' <marcus.g.williams@...>; 'openflowjava-dev@...' <openflowjava-dev@...>; 'ovsdb-dev@...' <ovsdb-dev@...>
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

max_backoff ?

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

Regards,

Vishal.

Full GC in a 8G-16G Heap takes about 10+ seconds

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

Vishal Thapar <vishal.thapar@...>

#734

Close, inactivity_probe. Sorry for delay, had to fish in old logs.

Show quoted text

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 15:43
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

max_backoff ?

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

Regards,

Vishal.

Full GC in a 8G-16G Heap takes about 10+ seconds

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

Pearl, Tomer <tomer.pearl@...>

#721

Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?

Thanks

Show quoted text

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:20 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

Close, inactivity_probe. Sorry for delay, had to fish in old logs.

max_backoff ?

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

Regards,

Vishal.

Full GC in a 8G-16G Heap takes about 10+ seconds

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

Vishal Thapar <vishal.thapar@...>

#735

We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration. We were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.

Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time, you can tweak a flag captured in this: https://git.opendaylight.org/gerrit/#/c/49068/

If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have functional impact, so be careful.

Regards,

Vishal.

Show quoted text

From: Pearl, Tomer [mailto:tomer.pearl@...]
Sent: 17 January 2017 15:56
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?

Thanks

Close, inactivity_probe. Sorry for delay, had to fish in old logs.

max_backoff ?

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

Regards,

Vishal.

Full GC in a 8G-16G Heap takes about 10+ seconds

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

Sela, Guy <guy.sela@...>

#736

When you’re saying initial configuration do you mean before the OVS established an openflow connection?

Configuration for Controller Table in OVSDB is being set by ODL.

The CLI configuration for the inactivity probe looks like this for example:

sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000

So it looks like this can only be set after the connection

Show quoted text

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:34 PM
To: Pearl, Tomer <tomer.pearl@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have functional impact, so be careful.

Regards,

Vishal.

Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?

Thanks

Close, inactivity_probe. Sorry for delay, had to fish in old logs.

max_backoff ?

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

Regards,

Vishal.

Full GC in a 8G-16G Heap takes about 10+ seconds

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

Vishal Thapar <vishal.thapar@...>

#737

Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller, we were also creating controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.

Regards,

Vishal.

Show quoted text

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 16:23
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

When you’re saying initial configuration do you mean before the OVS established an openflow connection?

Configuration for Controller Table in OVSDB is being set by ODL.

The CLI configuration for the inactivity probe looks like this for example:

sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000

So it looks like this can only be set after the connection

If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have functional impact, so be careful.

Regards,

Vishal.

Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?

Thanks

Close, inactivity_probe. Sorry for delay, had to fish in old logs.

max_backoff ?

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

Regards,

Vishal.

Full GC in a 8G-16G Heap takes about 10+ seconds

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Thanks

Anil

1 - 20 of 30

Previous Topic Next Topic

Home Hashtags Subgroups Terms