Re: [integration-dev] [opendaylight-dev] openflow performance regression


Abhijit Kumbhare
 

Luis,

In today's OF Plugin meeting Tony provided the info on this & the fix:

 

[09:08:50] <ttkacik1> whole issue is in droptest itself

[09:09:40] <abhijitkumbhare> Oh - but have there been any changes to droptest?

[09:09:54] <ttkacik1> droptest was using MD-SAL threads even if it is explicitly docuemnted in javadocs for MD-SAL Notifcation APIs to do not any work which may take time

[09:10:48] <abhijitkumbhare> so someone needs to change drop test

[09:10:53] <ttkacik1> patch which dropped performance was announced to mailing lists before: https://lists.opendaylight.org/pipermail/openflowplugin-dev/2015-May/003022.html

[09:11:08] <ttkacik1> even points to patch

[09:11:30] <abhijitkumbhare> OK

[09:11:54] <ttkacik1> which changed bit behavior of notif. broker impl but not API contractz

[09:12:11] <ttkacik1> which showed that droptest was stealing MD-SAL threads to do its work

[09:12:19] <ttkacik1> that is why perf. dropped

[09:12:40] <ttkacik1> now notification broker makes sure none of the listener is invoked from multiple threads at same time

[09:12:51] <abhijitkumbhare> so someone needs to change drop test to work better with the lithium notification broker

[09:13:10] <abhijitkumbhare> right?

[09:13:29] <ttkacik1> and that was implementation specific behaviour on which droptest relies

[09:13:38] <ttkacik1> I pushed initial fix to droptest as of now

[09:14:12] <abhijitkumbhare> OK - good

[09:14:37] <jamoluhrsen> #info https://git.opendaylight.org/gerrit/#/c/19717/ is Tony's drop-test patch

[09:14:39] <ttkacik1> https://git.opendaylight.org/gerrit/#/c/19717/

[09:14:46] <michal_rehak> testing the patch right now

[09:15:21] <abhijitkumbhare> OK

[09:15:22] <michal_rehak> looks like it improves throughput in He-codebase of ofPlugin 3 times

[09:16:03] <michal_rehak> merged


Thanks,

Abhijit


On Wed, May 6, 2015 at 9:20 AM, Luis Gomez <ecelgp@...> wrote:
OK, you are right Tony, looking at the exact patches that went in the distribution that reduced performance:

openflowplugin:

controller:

It seems the openflowplugin patch is more candidate than the other 2 from controller.

BR/Luis


On May 6, 2015, at 8:17 AM, Tony Tkacik -X (ttkacik - Pantheon Technologies SRO at Cisco) <ttkacik@...> wrote:

is not even in the code-path of throughput testing, since change was to clustered datastore
And for throughput testing suite is using dropallpacketsrpc on
Which  is based on RPCs and Notifications only.
 
Tony
 
From: openflowjava-dev-bounces@... [mailto:openflowjava-dev-bounces@...] On Behalf Of Luis Gomez
Sent: Wednesday, May 06, 2015 5:01 PM
To: Luhrsen, Jamo
Cc: openflowplugin-dev@...; integration-dev@...; openflowjava-dev@...; Mathieu Lemay
Subject: Re: [openflowjava-dev] [integration-dev] [opendaylight-dev] openflow performance regression
 
Hi Jamo. The patch that reduced the performance is:
 
 
You could not see this clearly in the CI because projects that have branched to stable/lithium like controller or yangtools are not triggering any test in CI.
 
BR/Luis
 
 
 
On May 6, 2015, at 7:07 AM, Luhrsen, Jamo <james.luhrsen@...> wrote:
 
I notified openflowplugin-dev about this yesterday.  I was assuming this might have
been the patch [1]  that reduced the performance, as it was what looked like the
patch that triggered the first test that saw the drop.  But, that was not a commit
from Robert, so now I’m not sure.

also, please see the Lithium re-design performance plots [2], just to be able to
compare.

Thanks,
JamO

[1]  https://git.opendaylight.org/gerrit/#/c/19638/
[2]  https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cds-cbench-performance-lithium-redesign-only-master/plot/


On May 6, 2015, at 6:15 AM, Colin Dixon <colin@...<mailto:colin@...>> wrote:

I just heard that from Tony. Sorry for that.

--Colin


On Wed, May 6, 2015 at 9:11 AM, Mathieu Lemay <mlemay@...<mailto:mlemay@...>> wrote:
Yes I think Robert sent out an e-mail on re-enabling notifications...

Cheers
Mathieu

On Wed, May 6, 2015 at 9:06 AM, Colin Dixon <colin@...<mailto:colin@...>> wrote:
https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cds-cbench-performance-only-master/plot/

Sometime around 19 hours ago it looks like our performance went off a cliff from ~100k flows per second to ~20k flows per second...

--Colin


_______________________________________________
dev mailing list
dev@...<mailto:dev@...>
https://lists.opendaylight.org/mailman/listinfo/dev




--
[http://www.inocybe.com/wp-content/uploads/2014/09/default-login-image.png]

Mathieu Lemay
President & CEO
Inocybe Technologies
1-888-445-7505<tel:1-888-445-7505>
www.inocybe.com<http://www.inocybe.com/>

_______________________________________________
integration-dev mailing list
integration-dev@...<mailto:integration-dev@...>
https://lists.opendaylight.org/mailman/listinfo/integration-dev

_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev


Join {z.archive.openflowjava-dev@lists.opendaylight.org to automatically receive all group messages.