[integration-dev] Inputs on cbench-like Java based tool that supports openflow 1.3


Daniel Farrell <dfarrell@...>
 

Hello Raksha!

This is a great project! Wonderful news to start a Saturday. :)

There have been discussions about building a CBench replacement for
quite some time. Ed and I talked about it at a Dev Design Summit
seriously enough that I created a "Better CBench" repo (but didn't
actually end up with time to build it).

There's little doubt that it needs to be replaced. WCBench makes
it more usable, but CBench is barely holding together and is becoming
more and more irrelevant as OF1.3 becomes normal.

One thing to keep in mind is that CBench-style tests aren't meant
to provide more than a small portion of the performance picture for
an SDN controller. Tests that uses OF southbound packet-in messages
to stress the controller are also very hard to compare controller to
controller, as it's easy to optimize that metric to amazing values
without actually getting much real-world benefit. It also of course
only applies to reactive OF SDN controllers. That said, CBench and
WCBench have found a bunch of bugs for us and we still use them in CI
to help watch for changes in performance (and verify it doesn't fall
off a cliff).

You could do this work under the umbrella of the newly created ODL
Performance Group (I'm on a wiki-creating sprint atm, link soon). The
idea is to provide a namespace under which we can organize perf work.

I'd be glad to help! as you can tell I've thought about this quite
a bit and am excited it's making more progress.

Thank you for your work!

Daniel Farrell
Software Engineer, Red Hat SDN Team
https://twitter.com/dfarrell07

----- Original Message -----
Hi all,

I am an intern in Intel Corporation with OpenDaylight team. As my intern
project, I have been thinking to build a new tool preferably in Java using
openflowjava library to benchmark SDN controllers on latency and throughput
to start with. The existing cbench benchmarking tool supports Openflow 1.0
only and the new tool would include support for the latest version of
Openflow (OF 1.3) and TLS.

I would appreciate your inputs/suggestions regarding the above project idea.
Also feel free to critic if this idea is already being
implemented/completed. If not and the idea sounds good, I would be happy to
work on this.

Thanks & Regards,
Raksha



_______________________________________________
integration-dev mailing list
integration-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/integration-dev


Madhava Bangera, Raksha <raksha.madhava.bangera@...>
 

Thanks, Daniel for your feedback and encouragement!

-----Original Message-----
From: Daniel Farrell [mailto:dfarrell@redhat.com]
Sent: Saturday, August 29, 2015 8:28 AM
To: Madhava Bangera, Raksha
Cc: integration-dev@lists.opendaylight.org; openflowjava-dev@lists.opendaylight.org
Subject: Re: [integration-dev] Inputs on cbench-like Java based tool that supports openflow 1.3

Hello Raksha!

This is a great project! Wonderful news to start a Saturday. :)

There have been discussions about building a CBench replacement for quite some time. Ed and I talked about it at a Dev Design Summit seriously enough that I created a "Better CBench" repo (but didn't actually end up with time to build it).

There's little doubt that it needs to be replaced. WCBench makes it more usable, but CBench is barely holding together and is becoming more and more irrelevant as OF1.3 becomes normal.

One thing to keep in mind is that CBench-style tests aren't meant to provide more than a small portion of the performance picture for an SDN controller. Tests that uses OF southbound packet-in messages to stress the controller are also very hard to compare controller to controller, as it's easy to optimize that metric to amazing values without actually getting much real-world benefit. It also of course only applies to reactive OF SDN controllers. That said, CBench and WCBench have found a bunch of bugs for us and we still use them in CI to help watch for changes in performance (and verify it doesn't fall off a cliff).

You could do this work under the umbrella of the newly created ODL Performance Group (I'm on a wiki-creating sprint atm, link soon). The idea is to provide a namespace under which we can organize perf work.

I'd be glad to help! as you can tell I've thought about this quite a bit and am excited it's making more progress.

Thank you for your work!

Daniel Farrell
Software Engineer, Red Hat SDN Team
https://twitter.com/dfarrell07

----- Original Message -----
Hi all,

I am an intern in Intel Corporation with OpenDaylight team. As my
intern project, I have been thinking to build a new tool preferably in
Java using openflowjava library to benchmark SDN controllers on
latency and throughput to start with. The existing cbench benchmarking
tool supports Openflow 1.0 only and the new tool would include support
for the latest version of Openflow (OF 1.3) and TLS.

I would appreciate your inputs/suggestions regarding the above project idea.
Also feel free to critic if this idea is already being
implemented/completed. If not and the idea sounds good, I would be
happy to work on this.

Thanks & Regards,
Raksha



_______________________________________________
integration-dev mailing list
integration-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/integration-dev


Jamo Luhrsen <jluhrsen@...>
 

Madhava,

Since this tool should be controller agnostic, Daniel's idea of hosting it in the Perf Group
umbrella makes perfect sense.

There are a lot of *features* we'd all probably like to have but no need to eat the whole
elephant now. If we just had a tool that replicated the current cbench test with OF1.3,
we would start using it immediately :)

Feel free to reach out for help anytime. Someone always seems ready to help.

JamO

On 08/31/2015 03:22 PM, Madhava Bangera, Raksha wrote:
Thanks, Daniel for your feedback and encouragement!

-----Original Message-----
From: Daniel Farrell [mailto:dfarrell@redhat.com]
Sent: Saturday, August 29, 2015 8:28 AM
To: Madhava Bangera, Raksha
Cc: integration-dev@lists.opendaylight.org; openflowjava-dev@lists.opendaylight.org
Subject: Re: [integration-dev] Inputs on cbench-like Java based tool that supports openflow 1.3

Hello Raksha!

This is a great project! Wonderful news to start a Saturday. :)

There have been discussions about building a CBench replacement for quite some time. Ed and I talked about it at a Dev Design Summit seriously enough that I created a "Better CBench" repo (but didn't actually end up with time to build it).

There's little doubt that it needs to be replaced. WCBench makes it more usable, but CBench is barely holding together and is becoming more and more irrelevant as OF1.3 becomes normal.

One thing to keep in mind is that CBench-style tests aren't meant to provide more than a small portion of the performance picture for an SDN controller. Tests that uses OF southbound packet-in messages to stress the controller are also very hard to compare controller to controller, as it's easy to optimize that metric to amazing values without actually getting much real-world benefit. It also of course only applies to reactive OF SDN controllers. That said, CBench and WCBench have found a bunch of bugs for us and we still use them in CI to help watch for changes in performance (and verify it doesn't fall off a cliff).

You could do this work under the umbrella of the newly created ODL Performance Group (I'm on a wiki-creating sprint atm, link soon). The idea is to provide a namespace under which we can organize perf work.

I'd be glad to help! as you can tell I've thought about this quite a bit and am excited it's making more progress.

Thank you for your work!

Daniel Farrell
Software Engineer, Red Hat SDN Team
https://twitter.com/dfarrell07

----- Original Message -----
Hi all,

I am an intern in Intel Corporation with OpenDaylight team. As my
intern project, I have been thinking to build a new tool preferably in
Java using openflowjava library to benchmark SDN controllers on
latency and throughput to start with. The existing cbench benchmarking
tool supports Openflow 1.0 only and the new tool would include support
for the latest version of Openflow (OF 1.3) and TLS.

I would appreciate your inputs/suggestions regarding the above project idea.
Also feel free to critic if this idea is already being
implemented/completed. If not and the idea sounds good, I would be
happy to work on this.

Thanks & Regards,
Raksha



_______________________________________________
integration-dev mailing list
integration-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/integration-dev
_______________________________________________
integration-dev mailing list
integration-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/integration-dev


Madhava Bangera, Raksha <raksha.madhava.bangera@...>
 

Thanks, Jamo.

-----Original Message-----
From: Jamo Luhrsen [mailto:jluhrsen@gmail.com]
Sent: Monday, August 31, 2015 5:44 PM
To: Madhava Bangera, Raksha; Daniel Farrell
Cc: openflowjava-dev@lists.opendaylight.org; integration-dev@lists.opendaylight.org
Subject: Re: [integration-dev] Inputs on cbench-like Java based tool that supports openflow 1.3

Madhava,

Since this tool should be controller agnostic, Daniel's idea of hosting it in the Perf Group umbrella makes perfect sense.

There are a lot of *features* we'd all probably like to have but no need to eat the whole elephant now. If we just had a tool that replicated the current cbench test with OF1.3, we would start using it immediately :)

Feel free to reach out for help anytime. Someone always seems ready to help.

JamO

On 08/31/2015 03:22 PM, Madhava Bangera, Raksha wrote:
Thanks, Daniel for your feedback and encouragement!

-----Original Message-----
From: Daniel Farrell [mailto:dfarrell@redhat.com]
Sent: Saturday, August 29, 2015 8:28 AM
To: Madhava Bangera, Raksha
Cc: integration-dev@lists.opendaylight.org;
openflowjava-dev@lists.opendaylight.org
Subject: Re: [integration-dev] Inputs on cbench-like Java based tool
that supports openflow 1.3

Hello Raksha!

This is a great project! Wonderful news to start a Saturday. :)

There have been discussions about building a CBench replacement for quite some time. Ed and I talked about it at a Dev Design Summit seriously enough that I created a "Better CBench" repo (but didn't actually end up with time to build it).

There's little doubt that it needs to be replaced. WCBench makes it more usable, but CBench is barely holding together and is becoming more and more irrelevant as OF1.3 becomes normal.

One thing to keep in mind is that CBench-style tests aren't meant to provide more than a small portion of the performance picture for an SDN controller. Tests that uses OF southbound packet-in messages to stress the controller are also very hard to compare controller to controller, as it's easy to optimize that metric to amazing values without actually getting much real-world benefit. It also of course only applies to reactive OF SDN controllers. That said, CBench and WCBench have found a bunch of bugs for us and we still use them in CI to help watch for changes in performance (and verify it doesn't fall off a cliff).

You could do this work under the umbrella of the newly created ODL Performance Group (I'm on a wiki-creating sprint atm, link soon). The idea is to provide a namespace under which we can organize perf work.

I'd be glad to help! as you can tell I've thought about this quite a bit and am excited it's making more progress.

Thank you for your work!

Daniel Farrell
Software Engineer, Red Hat SDN Team
https://twitter.com/dfarrell07

----- Original Message -----
Hi all,

I am an intern in Intel Corporation with OpenDaylight team. As my
intern project, I have been thinking to build a new tool preferably
in Java using openflowjava library to benchmark SDN controllers on
latency and throughput to start with. The existing cbench
benchmarking tool supports Openflow 1.0 only and the new tool would
include support for the latest version of Openflow (OF 1.3) and TLS.

I would appreciate your inputs/suggestions regarding the above project idea.
Also feel free to critic if this idea is already being
implemented/completed. If not and the idea sounds good, I would be
happy to work on this.

Thanks & Regards,
Raksha



_______________________________________________
integration-dev mailing list
integration-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/integration-dev
_______________________________________________
integration-dev mailing list
integration-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/integration-dev