> On May 30, 2019, at 2:26 PM, Robert Varga <nite@...> wrote:
>
> On 30/05/2019 20:58, Luis Gomez wrote:
>>
>> 2) Some perf regression in controller:
>>
>> https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-ask-all-neon/247/robot/controller-clustering-ask.txt/Chasing%20The%20Leader/Unregister_Candidates_And_Validate_Criteria/
>> <https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-ask-all-neon/247/robot/controller-clustering-ask.txt/Chasing
>> The Leader/Unregister_Candidates_And_Validate_Criteria/>
>
> #2​44
> 27-May-2019 09:46
>
> Hmm... what is the usual achieved rate?

Around 50 and now we get less than 5 so we are talking about 1 magnitude perf regression.

The test itself does:

1) Start a singleton registration flap on every controller instance with this RPC: /restconf/operations/odl-mdsal-lowlevel-control:register-flapping-singleton

2) Maintain the flap for 60 secs.

3) Stop the flap on every controller instance: /restconf/operations/odl-mdsal-lowlevel-control:unregister-flapping-singleton

4) Get flap count from above RPC response: <output xmlns="tag:opendaylight.org,2017:controller:yang:lowlevel:control"><flap-count>83</flap-count></output>

5) Add all the flaps for the 3 controller instances and divide the total by 60 secs.

So in the original controller handled ~50 flaps/sec, after the regression is less than 5 flaps/sec.

I hope this helps.


>
> Jam0, we've been messing with those numbers last, but I do not remember
> the specifics...
>
>> https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-fluorine/220/robot/controller-clustering-tell.txt/Chasing%20The%20Leader/Unregister_Candidates_And_Validate_Criteria/
>> <https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-fluorine/220/robot/controller-clustering-tell.txt/Chasing
>> The Leader/Unregister_Candidates_And_Validate_Criteria/>
>>
>> I am not sure how critical is this perf test.
>
> Well, regressions needs to be investigated. On the other hand, we have
> seen problems with Nexus, so this could (I am not saying it is) an env
> issue.
>
> Let's see what the usual rates were, but I think these should actually
> improve in this release :)
>
> Also, let's make this a blocker until we understand more. It is the last
> SR after all...
>
> Regards,
> Robert
>

_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release

OK, just opened the ticket here: https://jira.opendaylight.org/browse/CONTROLLER-1900

On Jun 5, 2019, at 8:16 PM, Daniel De La Rosa <ddelarosa@...> wrote:

Robert

It seems that this regression still happening so Luis is going to raise a ticket to get this fixed and be able to release Fluorine SR3... Any updates from your end?

Thanks

On Tue, Jun 4, 2019 at 1:43 PM Luis Gomez <ecelgp@...> wrote:

> On May 30, 2019, at 2:26 PM, Robert Varga <nite@...> wrote:
>
> On 30/05/2019 20:58, Luis Gomez wrote:
>>
>> 2) Some perf regression in controller:
>>
>> https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-ask-all-neon/247/robot/controller-clustering-ask.txt/Chasing%20The%20Leader/Unregister_Candidates_And_Validate_Criteria/
>> <https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-ask-all-neon/247/robot/controller-clustering-ask.txt/Chasing
>> The Leader/Unregister_Candidates_And_Validate_Criteria/>
>
> #2​44
> 27-May-2019 09:46
>
> Hmm... what is the usual achieved rate?

Around 50 and now we get less than 5 so we are talking about 1 magnitude perf regression.

The test itself does:

1) Start a singleton registration flap on every controller instance with this RPC: /restconf/operations/odl-mdsal-lowlevel-control:register-flapping-singleton

2) Maintain the flap for 60 secs.

3) Stop the flap on every controller instance: /restconf/operations/odl-mdsal-lowlevel-control:unregister-flapping-singleton

4) Get flap count from above RPC response: <output xmlns="tag:opendaylight.org,2017:controller:yang:lowlevel:control"><flap-count>83</flap-count></output>

5) Add all the flaps for the 3 controller instances and divide the total by 60 secs.

So in the original controller handled ~50 flaps/sec, after the regression is less than 5 flaps/sec.

I hope this helps.


>
> Jam0, we've been messing with those numbers last, but I do not remember
> the specifics...
>
>> https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-fluorine/220/robot/controller-clustering-tell.txt/Chasing%20The%20Leader/Unregister_Candidates_And_Validate_Criteria/
>> <https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-fluorine/220/robot/controller-clustering-tell.txt/Chasing
>> The Leader/Unregister_Candidates_And_Validate_Criteria/>
>>
>> I am not sure how critical is this perf test.
>
> Well, regressions needs to be investigated. On the other hand, we have
> seen problems with Nexus, so this could (I am not saying it is) an env
> issue.
>
> Let's see what the usual rates were, but I think these should actually
> improve in this release :)
>
> Also, let's make this a blocker until we understand more. It is the last
> SR after all...
>
> Regards,
> Robert
>

_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release

--

Daniel de la Rosa
Customer Support Manager
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120

I think we have to wait for next AR build and then ask Anil to create the spreadsheet for the new build as usual:

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-fluorine-mvn35-openjdk8/

BR/Luis

On Jun 12, 2019, at 9:24 AM, Daniel De La Rosa <ddelarosa@...> wrote:

Since this is fixed now and the other two blockers are not real blockers for fluorine sr3, 

https://jira.opendaylight.org/browse/AAA-185

https://jira.opendaylight.org/browse/BGPCEP-873

The spreadsheet can be completed @Luis Gomez ? and Fluorine SR3 released?

Thanks

On Wed, Jun 5, 2019 at 9:28 PM Luis Gomez <ecelgp@...> wrote:

OK, just opened the ticket here: https://jira.opendaylight.org/browse/CONTROLLER-1900

On Jun 5, 2019, at 8:16 PM, Daniel De La Rosa <ddelarosa@...> wrote:

Robert

It seems that this regression still happening so Luis is going to raise a ticket to get this fixed and be able to release Fluorine SR3... Any updates from your end?

Thanks

On Tue, Jun 4, 2019 at 1:43 PM Luis Gomez <ecelgp@...> wrote:

> On May 30, 2019, at 2:26 PM, Robert Varga <nite@...> wrote:
>
> On 30/05/2019 20:58, Luis Gomez wrote:
>>
>> 2) Some perf regression in controller:
>>
>> https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-ask-all-neon/247/robot/controller-clustering-ask.txt/Chasing%20The%20Leader/Unregister_Candidates_And_Validate_Criteria/
>> <https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-ask-all-neon/247/robot/controller-clustering-ask.txt/Chasing
>> The Leader/Unregister_Candidates_And_Validate_Criteria/>
>
> #2​44
> 27-May-2019 09:46
>
> Hmm... what is the usual achieved rate?

Around 50 and now we get less than 5 so we are talking about 1 magnitude perf regression.

The test itself does:

1) Start a singleton registration flap on every controller instance with this RPC: /restconf/operations/odl-mdsal-lowlevel-control:register-flapping-singleton

2) Maintain the flap for 60 secs.

3) Stop the flap on every controller instance: /restconf/operations/odl-mdsal-lowlevel-control:unregister-flapping-singleton

4) Get flap count from above RPC response: <output xmlns="tag:opendaylight.org,2017:controller:yang:lowlevel:control"><flap-count>83</flap-count></output>

5) Add all the flaps for the 3 controller instances and divide the total by 60 secs.

So in the original controller handled ~50 flaps/sec, after the regression is less than 5 flaps/sec.

I hope this helps.


>
> Jam0, we've been messing with those numbers last, but I do not remember
> the specifics...
>
>> https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-fluorine/220/robot/controller-clustering-tell.txt/Chasing%20The%20Leader/Unregister_Candidates_And_Validate_Criteria/
>> <https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-fluorine/220/robot/controller-clustering-tell.txt/Chasing
>> The Leader/Unregister_Candidates_And_Validate_Criteria/>
>>
>> I am not sure how critical is this perf test.
>
> Well, regressions needs to be investigated. On the other hand, we have
> seen problems with Nexus, so this could (I am not saying it is) an env
> issue.
>
> Let's see what the usual rates were, but I think these should actually
> improve in this release :)
>
> Also, let's make this a blocker until we understand more. It is the last
> SR after all...
>
> Regards,
> Robert
>

_______________________________________________
release mailing list
release@...
https://lists.opendaylight.org/mailman/listinfo/release

--

Daniel de la Rosa
Customer Support Manager
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120

--

Daniel de la Rosa
Customer Support Manager
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120