Re: [ovsdb-dev] OVSDB scale


Sela, Guy <guy.sela@...>
 

Never mind, I found it, inactivity_probe

Explanation here: http://openvswitch.org/ovs-vswitchd.conf.db.5.pdf

 

 

From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:13 PM
To: 'Vishal Thapar' <vishal.thapar@...>; 'Anil Vishnoi' <vishnoianil@...>
Cc: 'Muthukumaran K' <muthukumaran.k@...>; 'Jamo Luhrsen' <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; 'marcus.g.williams@...' <marcus.g.williams@...>; 'openflowjava-dev@...' <openflowjava-dev@...>; 'ovsdb-dev@...' <ovsdb-dev@...>
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

max_backoff ?

 

 

From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:10 PM
To: 'Vishal Thapar' <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Great thanks.

Do you recall where you tweak this configuration ? A quick google search didn’t help me.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:07 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many echo messages.

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 14:57
To: Anil Vishnoi <vishnoianil@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Full GC in a 8G-16G Heap takes about 10+ seconds

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection won't happen atleast because of GC.

 

On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:

Hi guys,

 

So a couple of questions:

 

1)      Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?

2)      Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 11:07 AM
To: Muthukumaran K <muthukumaran.k@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>


Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Muthu,

 

Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t recall what exactly were those.

 

Regards,

Vishal.

 

From: Muthukumaran K
Sent: 17 January 2017 14:33
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Hi Vishal,

 

We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.

 

For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug

https://bugs.opendaylight.org/show_bug.cgi?id=4823

 

There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)

 

Regards

Muthu

 

 

 

From: Vishal Thapar
Sent: Tuesday, January 17, 2017 2:23 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>; Muthukumaran K <muthukumaran.k@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Adding Muthu.

 

Muthu,

I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?

 

 

Regards,

Vishal.

 

From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 13:49
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

Thanks,

 

Did you manage to survive Full GCs at all ?

If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.

 

From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 8:50 AM
To: Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?

 

If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.

 

Regards,

Vishal.

 

From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale

 

I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron. 

 

@vishal : do you have any number from your ovsdb testing ? 

 

On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:

Hi Tomer,

back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.

Thanks,
JamO


[0] https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf



On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>

> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
> https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev



 

--

Thanks

Anil



 

--

Thanks

Anil

Join z.archive.openflowjava-dev@lists.opendaylight.org to automatically receive all group messages.