|
Re: [ovsdb-dev] OVSDB scale
Vishal Thapar <vishal.thapar@...>
Aha, you mean the API that OVSDB is using? That is more of a convenience and we can add without that too. I’ll add code for it. If
needed, we can add util accordingly to OVSDB. Either ways, the fix will be going in Netvirt, OVSDB change may or may not be needed.
Regards,
Vishal.
From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:54
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
Hi,
So it seems there should be 2 bugs?
OVSDB needs to expose this in its addBridge interface and NETVIRT should allow to configure it.
I couldn’t find how to configure it in OvsdbBridgeAugmentationBuilder and not in SouthboundUtils.addBridge
HI Guy,
If this knob is needed for Controller’s probe timer bug should be on netvirt as netvirt is adding controller. In case of Manager, it should be set by whosoever
configures manager, which never comes from OVSDB.
Any configuration that goes into OVSDB in switch should ideally come from consumers of OVSDB, not plugin itself.
I can change this to netvirt bug, but want to make sure we are in agreement on nature of change coming in ie.e. a knob to set default inactivity for Controller
that gets created by autobridge.
Regards,
Vishal.
https://bugs.opendaylight.org/show_bug.cgi?id=7563
Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller, we were also creating
controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.
Regards,
Vishal.
When you’re saying initial configuration do you mean before the OVS established an openflow connection?
Configuration for Controller Table in OVSDB is being set by ODL.
The CLI configuration for the inactivity probe looks like this for example:
sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000
So it looks like this can only be set after the connection
We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration. We
were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.
Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time, you can tweak
a flag captured in this:
https://git.opendaylight.org/gerrit/#/c/49068/
If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have
functional impact, so be careful.
Regards,
Vishal.
Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?
Thanks
Close, inactivity_probe. Sorry for delay, had to fish in old logs.
max_backoff ?
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no.
of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many
echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts,
probably disconnection won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt
models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test
may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used
the settings mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results
in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full
mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory usage. While
provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil
Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Hi,
So it seems there should be 2 bugs?
OVSDB needs to expose this in its addBridge interface and NETVIRT should allow to configure it.
I couldn’t find how to configure it in OvsdbBridgeAugmentationBuilder and not in SouthboundUtils.addBridge
From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 4:10 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
HI Guy,
If this knob is needed for Controller’s probe timer bug should be on netvirt as netvirt is adding controller. In case of Manager, it should be set
by whosoever configures manager, which never comes from OVSDB.
Any configuration that goes into OVSDB in switch should ideally come from consumers of OVSDB, not plugin itself.
I can change this to netvirt bug, but want to make sure we are in agreement on nature of change coming in ie.e. a knob to set default inactivity
for Controller that gets created by autobridge.
Regards,
Vishal.
https://bugs.opendaylight.org/show_bug.cgi?id=7563
Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller, we were
also creating controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.
Regards,
Vishal.
When you’re saying initial configuration do you mean before the OVS established an openflow connection?
Configuration for Controller Table in OVSDB is being set by ODL.
The CLI configuration for the inactivity probe looks like this for example:
sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000
So it looks like this can only be set after the connection
We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration.
We were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.
Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time,
you can tweak a flag captured in this:
https://git.opendaylight.org/gerrit/#/c/49068/
If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag
can have functional impact, so be careful.
Regards,
Vishal.
Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?
Thanks
Close, inactivity_probe. Sorry for delay, had to fish in old logs.
max_backoff ?
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing
to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing
too many echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection
won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy < guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and
all the netvirt models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not
be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings
mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit
of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the
time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory
usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Vishal Thapar <vishal.thapar@...>
HI Guy,
If this knob is needed for Controller’s probe timer bug should be on netvirt as netvirt is adding controller. In case of Manager, it
should be set by whosoever configures manager, which never comes from OVSDB.
Any configuration that goes into OVSDB in switch should ideally come from consumers of OVSDB, not plugin itself.
I can change this to netvirt bug, but want to make sure we are in agreement on nature of change coming in ie.e. a knob to set default
inactivity for Controller that gets created by autobridge.
Regards,
Vishal.
From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 19:30
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
https://bugs.opendaylight.org/show_bug.cgi?id=7563
Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller, we were also creating
controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.
Regards,
Vishal.
When you’re saying initial configuration do you mean before the OVS established an openflow connection?
Configuration for Controller Table in OVSDB is being set by ODL.
The CLI configuration for the inactivity probe looks like this for example:
sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000
So it looks like this can only be set after the connection
We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration. We
were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.
Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time, you can tweak
a flag captured in this:
https://git.opendaylight.org/gerrit/#/c/49068/
If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have
functional impact, so be careful.
Regards,
Vishal.
Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?
Thanks
Close, inactivity_probe. Sorry for delay, had to fish in old logs.
max_backoff ?
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no.
of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many
echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts,
probably disconnection won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt
models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test
may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used
the settings mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results
in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full
mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory usage. While
provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil
Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
https://bugs.opendaylight.org/show_bug.cgi?id=7563
From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 3:27 PM
To: Sela, Guy <guy.sela@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller, we were
also creating controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.
Regards,
Vishal.
When you’re saying initial configuration do you mean before the OVS established an openflow connection?
Configuration for Controller Table in OVSDB is being set by ODL.
The CLI configuration for the inactivity probe looks like this for example:
sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000
So it looks like this can only be set after the connection
We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration.
We were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.
Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time,
you can tweak a flag captured in this:
https://git.opendaylight.org/gerrit/#/c/49068/
If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag
can have functional impact, so be careful.
Regards,
Vishal.
Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?
Thanks
Close, inactivity_probe. Sorry for delay, had to fish in old logs.
max_backoff ?
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing
to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing
too many echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection
won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy < guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and
all the netvirt models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not
be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings
mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit
of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the
time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory
usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Vishal Thapar <vishal.thapar@...>
Aha! I meant for Manager. This field is present in Manager and Controller both and I was specifically talking about Manager. For controller,
we were also creating controller manually, not using the autobridge code. I’d recommend creating an enhancement bug for this. We should add this knob and code to autobridge.
Regards,
Vishal.
From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 16:23
To: Vishal Thapar <vishal.thapar@...>; Pearl, Tomer <tomer.pearl@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
When you’re saying initial configuration do you mean before the OVS established an openflow connection?
Configuration for Controller Table in OVSDB is being set by ODL.
The CLI configuration for the inactivity probe looks like this for example:
sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000
So it looks like this can only be set after the connection
We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration. We
were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.
Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time, you can tweak
a flag captured in this:
https://git.opendaylight.org/gerrit/#/c/49068/
If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag can have
functional impact, so be careful.
Regards,
Vishal.
Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?
Thanks
Close, inactivity_probe. Sorry for delay, had to fish in old logs.
max_backoff ?
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no.
of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many
echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts,
probably disconnection won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt
models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test
may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used
the settings mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results
in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full
mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory usage. While
provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil
Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
When you’re saying initial configuration do you mean before the OVS established an openflow connection?
Configuration for Controller Table in OVSDB is being set by ODL.
The CLI configuration for the inactivity probe looks like this for example:
sudo ovs-vsctl add Controller 8383a19f-4899-4808-ba0b-c970af081c3e inactivity_probe 10000
So it looks like this can only be set after the connection
From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:34 PM
To: Pearl, Tomer <tomer.pearl@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part of initial configuration.
We were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by default.
Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager at a time,
you can tweak a flag captured in this:
https://git.opendaylight.org/gerrit/#/c/49068/
If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing this flag
can have functional impact, so be careful.
Regards,
Vishal.
Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?
Thanks
Close, inactivity_probe. Sorry for delay, had to fish in old logs.
max_backoff ?
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing
to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing
too many echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection
won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy < guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and
all the netvirt models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not
be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings
mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit
of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the
time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory
usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Vishal Thapar <vishal.thapar@...>
We don’t have configuration for this today. We can add it or change netvirt code to add API. We changed it directly on OVS as part
of initial configuration. We were using scripts to configure OVSes with manager, so just added one more command. Another parameter you may want to look at is stats_interval which governs how frequently stat updates come, though we later on disabled stats by
default.
Finally, if you’re using HA Proxy between OVS and ODL for manager connection or using single node i.e. OVS connects to only one manager
at a time, you can tweak a flag captured in this:
https://git.opendaylight.org/gerrit/#/c/49068/
If you change this for a deployment where each OVS connects to each ODL node in cluster [multiple manager connections to cluster] changing
this flag can have functional impact, so be careful.
Regards,
Vishal.
From: Pearl, Tomer [mailto:tomer.pearl@...]
Sent: 17 January 2017 15:56
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?
Thanks
Close, inactivity_probe. Sorry for delay, had to fish in old logs.
max_backoff ?
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no.
of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many
echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts,
probably disconnection won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt
models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test
may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used
the settings mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results
in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full
mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory usage. While
provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil
Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Pearl, Tomer <tomer.pearl@...>
Can we set this value via OVSDB configuration file? Or do we need to change code to use some api for this?
Thanks
From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:20 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
Close, inactivity_probe. Sorry for delay, had to fish in old logs.
max_backoff ?
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing
to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing
too many echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection
won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy < guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and
all the netvirt models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not
be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings
mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit
of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the
time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory
usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Vishal Thapar <vishal.thapar@...>
Close, inactivity_probe. Sorry for delay, had to fish in old logs.
From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 15:43
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
max_backoff ?
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing to reduce no.
of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing too many
echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts,
probably disconnection won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt
models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test
may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used
the settings mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results
in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full
mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory usage. While
provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil
Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Never mind, I found it, inactivity_probe
Explanation here:
http://openvswitch.org/ovs-vswitchd.conf.db.5.pdf
From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:13 PM
To: 'Vishal Thapar' <vishal.thapar@...>; 'Anil Vishnoi' <vishnoianil@...>
Cc: 'Muthukumaran K' <muthukumaran.k@...>; 'Jamo Luhrsen' <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; 'marcus.g.williams@...' <marcus.g.williams@...>; 'openflowjava-dev@...' <openflowjava-dev@...>;
'ovsdb-dev@...' <ovsdb-dev@...>
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
max_backoff ?
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing
to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing
too many echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection
won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy < guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and
all the netvirt models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not
be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings
mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit
of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the
time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory
usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
max_backoff ?
From: Sela, Guy
Sent: Tuesday, January 17, 2017 12:10 PM
To: 'Vishal Thapar' <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing
to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing
too many echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection
won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy < guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and
all the netvirt models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not
be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings
mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit
of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the
time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory
usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Great thanks.
Do you recall where you tweak this configuration ? A quick google search didn’t help me.
From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 12:07 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>
Cc: Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different testing
to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll be processing
too many echo messages.
Regards,
Vishal.
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection
won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy < guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and
all the netvirt models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not
be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings
mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit
of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the
time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory
usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Vishal Thapar <vishal.thapar@...>
Good catch Anil. I forgot that we *did* increased timeout to 30-60 seconds from default of 5. I say 30-60 because we did different
testing to reduce no. of echo messages going back and forth. But with 5 we used to see frequent disconnects, so yes, I’d agree with Anil that increasing timeouts should be a better solution. In fact default of 5 is terrible as you start scaling up, you’ll
be processing too many echo messages.
Regards,
Vishal.
From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 14:57
To: Anil Vishnoi <vishnoianil@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>;
Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts,
probably disconnection won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt
models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test
may not be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used
the settings mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results
in a bit of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full
mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory usage. While
provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil
Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Muthukumaran K <muthukumaran.k@...>
Hi Anil,
In context of Openflowplugin (Helium) tests, we have observed this behavior when there is heavy GC induced due to immutables created and garbaged as part of flow provisioning
combined with statistics-gathering. To reduce the pressure, we used to disable statistics which had a huge improvement in terms of garbage-generation. Side effect of GCs will be of course Echo Timeout.
So, as you suggest if we bump the echo timeouts, it will prevent the disconnects and we can get a more clearer picture of garbage-generation pattern. The also if we see huge
garbage generation and rapid FCG count raise, we can then switch off statistics.
We can also monitor netstat -anp on the the ports connecting to OVS to see if there is a fast-producer slow-consumer type of issues (huge pile-up of send-q and recv-q can provide
a hint)
Another important part (I assume this is already taken care of) is to run the system at ERROR log level instead of default INFO level
Regards
Muthu
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 2:55 PM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection
won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy < guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and
all the netvirt models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not
be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings
mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit
of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the
time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory
usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Full GC in a 8G-16G Heap takes about 10+ seconds
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: Tuesday, January 17, 2017 11:25 AM
To: Sela, Guy <guy.sela@...>
Cc: Vishal Thapar <vishal.thapar@...>; Muthukumaran K <muthukumaran.k@...>; Jamo Luhrsen <jluhrsen@...>; Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection
won't happen atleast because of GC.
toggle quoted messageShow quoted text
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy < guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and
all the netvirt models to the mix, I couldn’t recall what exactly were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not
be applicable for Guy’s case. However, increasing the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings
mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit
of chaos. Is there any way around this other than avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the
time with a full mesh. Scale limits come more from size of datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory
usage. While provisioning you may hit a high peak but will come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
I think we should look at why OVS is getting disconnected during the GC? Is it because of the Echo timeout? Tuning GC will help, but i don't think so it will fix the root cause. I think if we can increase the echo timeouts, probably disconnection won't happen atleast because of GC.
On Tue, Jan 17, 2017 at 1:15 AM, Sela, Guy <guy.sela@...> wrote:
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t recall what exactly
were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing the Heap
and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than avoiding
Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of
datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will
come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@lists.opendaylight.org;
ovsdb-dev@....org; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@....org
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Hi guys,
So a couple of questions:
1)
Did you reached Full GC? And if so, did the OVSs disconnected? And did everything continued working smoothly afterwards?
2)
Do you have some script or mechanism you can share that will allow to quickly count number of flows in the data store?
From: Vishal Thapar [mailto:vishal.thapar@...]
Sent: Tuesday, January 17, 2017 11:07 AM
To: Muthukumaran K <muthukumaran.k@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t recall what exactly
were those.
Regards,
Vishal.
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing the Heap
and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than avoiding
Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of
datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will
come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
toggle quoted messageShow quoted text
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Vishal Thapar <vishal.thapar@...>
Hi Muthu,
Yeah, we had used similar heap and GC settings when testing ITM which added OVSDB and all the netvirt models to the mix, I couldn’t
recall what exactly were those.
Regards,
Vishal.
From: Muthukumaran K
Sent: 17 January 2017 14:33
To: Vishal Thapar <vishal.thapar@...>; Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing
the Heap and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than
avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of datastore than
anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will come down
once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
toggle quoted messageShow quoted text
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen < jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Muthukumaran K <muthukumaran.k@...>
Hi Vishal,
We had focused mainly on baseline feature of OFPlugin so some of our changes specific to that drive-test may not be applicable for Guy’s case. However, increasing the Heap
and using G1GC is something he had already accounted.
For the scenario we were chasing (only openflowplugin + a load-driver app – bulk-o-matic), we had used the settings mentioned in the last reply of this bug
https://bugs.opendaylight.org/show_bug.cgi?id=4823
There are few more tweaks in Openflowplugin – but they are all related to specifics of OFPlugin (Helium)
Regards
Muthu
From: Vishal Thapar
Sent: Tuesday, January 17, 2017 2:23 PM
To: Sela, Guy <guy.sela@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>; Muthukumaran K <muthukumaran.k@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than avoiding
Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of
datastore than anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will
come down once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
toggle quoted messageShow quoted text
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
|
|
|
Re: [ovsdb-dev] OVSDB scale
Vishal Thapar <vishal.thapar@...>
Adding Muthu.
Muthu,
I believe we did have to do some tweaks with heapsize, GC settings etc.right? Do you recall?
Regards,
Vishal.
From: Sela, Guy [mailto:guy.sela@...]
Sent: 17 January 2017 13:49
To: Vishal Thapar <vishal.thapar@...>; Anil Vishnoi <vishnoianil@...>; Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>; marcus.g.williams@...; openflowjava-dev@...; ovsdb-dev@...
Subject: RE: [openflowjava-dev] [ovsdb-dev] OVSDB scale
Thanks,
Did you manage to survive Full GCs at all ?
If I don’t avoid it, a Full GC causes all OVSs to disconnect from the ODL, and it results in a bit of chaos. Is there any way around this other than
avoiding Full GC? I managed to avoid it in my testing using 16G heap size and G1 collector.
We tested not just OVSDB but OVSDB+Netvirt/VPNService at scale of about 80 OVS at the time with a full mesh. Scale limits come more from size of datastore than
anything else. So how many devices you can scale depends on extent of features you’re testing. Is it just OVSDB, or Netvit with multiple VMs per compute across multiple networks?
If you’re running into memory issues would be good to increase memory and capture memory usage. While provisioning you may hit a high peak but will come down
once it is done. I’ll check if I can get details of numbers we tested, should be lying somewhere in archived mails.
Regards,
Vishal.
From: Anil Vishnoi [mailto:vishnoianil@...]
Sent: 17 January 2017 12:09
To: Jamo Luhrsen <jluhrsen@...>
Cc: Pearl, Tomer <tomer.pearl@...>;
marcus.g.williams@...;
openflowjava-dev@...;
ovsdb-dev@...; Sela, Guy <guy.sela@...>; Vishal Thapar <vishal.thapar@...>
Subject: Re: [openflowjava-dev] [ovsdb-dev] OVSDB scale
I believe team from Ericsson also did some testing with it and we made some more performance improvement on boron.
@vishal : do you have any number from your ovsdb testing ?
toggle quoted messageShow quoted text
On Tue, Jan 3, 2017 at 10:05 PM, Jamo Luhrsen < jluhrsen@...> wrote:
Hi Tomer,
back in Beryllium there was a performance report released [0]. You can see on page 31 that we
saw OVSDB scale up to 1800 nodes. There may be more recent tests done, and I think Marcus
may have some idea. But, I think your 200 number should be achievable.
Thanks,
JamO
[0]
https://www.opendaylight.org/sites/www.opendaylight.org/files/odl_performancetechnicalreport_1-1_052716.pdf
On 01/02/2017 06:02 AM, Pearl, Tomer wrote:
> Hi,
>
>
>
> I’m trying to bring up a setup with one ODL controller and 200+ OVSs.
>
> I’m testing with Boron SR1 code
>
>
>
> Are there any reports about ODL scale tests that I can look at ?
>
> Is 200 OVSs an amount that supposed to work?
>
>
>
> Thanks,
>
>
>
> Tomer P.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> ovsdb-dev mailing list
> ovsdb-dev@...
>
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev
>
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
--
|
|
|
