1. Topics
  2. OF13 NSF test stability

OF13 NSF test stability

Luis Gomez
 

Hi Abhijit,

One thing we need to work once we get all these enhancement in the OF13 plugin is the stability of the NSF test.

If you see old OF10 plugin NSF test:

https://jenkins.opendaylight.org/integration/job/integration-csit-base/

compared to new OF13 plugin NSF test (~same test cases for OF1.0 and OF1.3 switch simulation):

https://jenkins.opendaylight.org/integration/job/integration-csit-base-of13/

They are far to be similar in terms of stability. Also important to note, most of NSF we are testing are AD-SAL apps like topology mgr, switch mgr, FRM, etc... so in this case it is hard to say if the issue is in the OF13 plugin, the SAL adapters or both.

BR/Luis


On 4/28/2014 3:00 PM, Abhijit Kumbhare wrote:

Thanks Luis! Yes - we really would like these 2 things get tested a bit to make sure we do not break things.

I am hoping both these are done by Wednesday. However the first one (datastore) needs a fix by Vaclav - and I do not have a date as to when that will happen. The second (basic thread model changes) - Muthu said he will push these Tuesday his time - so we should have this tomorrow.


On Mon, Apr 28, 2014 at 2:54 PM, Luis Gomez <ecelgp@...> wrote:
Hi Abhijt, sorry I could not make the meeting this morning, I had a conflicting appointment.

Here are a few comments:


1) Making the new data store as the default in the OpenFlow plugin repository - for testing more

Do you know when this is going to happen? it would be good if we can pass the existing test suites before so we make sure we do not break everything after the change.


2) The basic thread model changes (almost done)

Same thing here, we would like to test this in Ericsson Lab once it is available, Chris O'Shea is continuously collecting CBench results in the Lab.

Thanks/Luis



On 4/28/2014 11:49 AM, Abhijit Kumbhare wrote:
Status of the action items:

1) Making the new data store as the default in the OpenFlow plugin repository - for testing more. - Vaclav/Michal

This was gated by data store bug (being worked by controller team)

Datastore bug partially fixed - Vaclav working on this. Vaclav currently heavily loaded - we will need a rough timeline from him (copied). 

2) The basic thread model changes (almost done) - Muthu/Prasanna (need to involve Luis/integration test team after changes merged to test OpenFlow 1.0/1.3 continue to work as before)

Muthu's local env failure fixed. One IT test case failing - should be fixed shortly - outlook April 29. Will be merged into a topic branch & then merged into the master. Additional commit will be done later to make it configurable (queuekeeper pool size - main connection, auxiliary connection; thread pool for the ticket finisher). For now the config parameters will be in config.ini. The wiki diagrams will need to be updated with the changes. Once Muthu has merged his changes back - we will need to have the integration test team test to see if the existing OpenFlow 1.0/1.3 functionality is not broken.

Prasanna (one person from Ericsson will be assigned) to help in making this configurable & help in performance measurements/tuning.

Prasanna, Michal, Tony & Muthu to meet Tuesday April 29 morning (Slovakia/India time) - to discuss the performance measurements/tuning.

Side Note:
Prasanna not seeing the performance improvements with the new datastore (with the new datastore seeing less performance than with the old datastore). He will need to sync up with Tony/Robert. Config of the machine - high end - 8 core CPU.

3) Critical bug "Cbench in throughput mode causes the controller to run out of memory":

https://bugs.opendaylight.org/show_bug.cgi?id=803 

 Michal Rehak investigating.

4) Michal was going to be pulling the cherry picked gerrits into the stable branch.

Remove the previously created stable branch - and cherry pick gerrits into the stable branch. Since Michal Rehak was working on this and he is heavily loaded - we will reach out to Surekha Bejgam - for help in this release engineering activity.
 
Kamal will be taking this up. 

Kamal's team will also take up https://bugs.opendaylight.org/show_bug.cgi?id=275. Prasanna advised that Kamal's team build up upon an unmerged gerrit http://git.opendaylight.org/gerrit/4720 for this bug.


- Abhijit



_______________________________________________
openflowplugin-dev mailing list
openflowplugin-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev



Abhijit Kumbhare
 

Would you happen to know where exactly is the code for these tests? Were these tests working before (at Hydrogen release timeframe)?


On Mon, Apr 28, 2014 at 7:58 PM, Luis Gomez <ecelgp@...> wrote:
Hi Abhijit,

One thing we need to work once we get all these enhancement in the OF13 plugin is the stability of the NSF test.

If you see old OF10 plugin NSF test:

https://jenkins.opendaylight.org/integration/job/integration-csit-base/

compared to new OF13 plugin NSF test (~same test cases for OF1.0 and OF1.3 switch simulation):

https://jenkins.opendaylight.org/integration/job/integration-csit-base-of13/

They are far to be similar in terms of stability. Also important to note, most of NSF we are testing are AD-SAL apps like topology mgr, switch mgr, FRM, etc... so in this case it is hard to say if the issue is in the OF13 plugin, the SAL adapters or both.

BR/Luis


On 4/28/2014 3:00 PM, Abhijit Kumbhare wrote:
Thanks Luis! Yes - we really would like these 2 things get tested a bit to make sure we do not break things.

I am hoping both these are done by Wednesday. However the first one (datastore) needs a fix by Vaclav - and I do not have a date as to when that will happen. The second (basic thread model changes) - Muthu said he will push these Tuesday his time - so we should have this tomorrow.


On Mon, Apr 28, 2014 at 2:54 PM, Luis Gomez <ecelgp@...> wrote:
Hi Abhijt, sorry I could not make the meeting this morning, I had a conflicting appointment.

Here are a few comments:


1) Making the new data store as the default in the OpenFlow plugin repository - for testing more

Do you know when this is going to happen? it would be good if we can pass the existing test suites before so we make sure we do not break everything after the change.


2) The basic thread model changes (almost done)

Same thing here, we would like to test this in Ericsson Lab once it is available, Chris O'Shea is continuously collecting CBench results in the Lab.

Thanks/Luis



On 4/28/2014 11:49 AM, Abhijit Kumbhare wrote:
Status of the action items:

1) Making the new data store as the default in the OpenFlow plugin repository - for testing more. - Vaclav/Michal

This was gated by data store bug (being worked by controller team)

Datastore bug partially fixed - Vaclav working on this. Vaclav currently heavily loaded - we will need a rough timeline from him (copied). 

2) The basic thread model changes (almost done) - Muthu/Prasanna (need to involve Luis/integration test team after changes merged to test OpenFlow 1.0/1.3 continue to work as before)

Muthu's local env failure fixed. One IT test case failing - should be fixed shortly - outlook April 29. Will be merged into a topic branch & then merged into the master. Additional commit will be done later to make it configurable (queuekeeper pool size - main connection, auxiliary connection; thread pool for the ticket finisher). For now the config parameters will be in config.ini. The wiki diagrams will need to be updated with the changes. Once Muthu has merged his changes back - we will need to have the integration test team test to see if the existing OpenFlow 1.0/1.3 functionality is not broken.

Prasanna (one person from Ericsson will be assigned) to help in making this configurable & help in performance measurements/tuning.

Prasanna, Michal, Tony & Muthu to meet Tuesday April 29 morning (Slovakia/India time) - to discuss the performance measurements/tuning.

Side Note:
Prasanna not seeing the performance improvements with the new datastore (with the new datastore seeing less performance than with the old datastore). He will need to sync up with Tony/Robert. Config of the machine - high end - 8 core CPU.

3) Critical bug "Cbench in throughput mode causes the controller to run out of memory":

https://bugs.opendaylight.org/show_bug.cgi?id=803 

 Michal Rehak investigating.

4) Michal was going to be pulling the cherry picked gerrits into the stable branch.

Remove the previously created stable branch - and cherry pick gerrits into the stable branch. Since Michal Rehak was working on this and he is heavily loaded - we will reach out to Surekha Bejgam - for help in this release engineering activity.
 
Kamal will be taking this up. 

Kamal's team will also take up https://bugs.opendaylight.org/show_bug.cgi?id=275. Prasanna advised that Kamal's team build up upon an unmerged gerrit http://git.opendaylight.org/gerrit/4720 for this bug.


- Abhijit



_______________________________________________
openflowplugin-dev mailing list
openflowplugin-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev




Luis Gomez
 

Hi Abhijit,

The test cases were much worse by the time of Hydrogen. The best way to run these tests locally is to download the test VM here:

https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Test_VMs

There is a link to TWS recording at the bottom of the wiki on how to use the VM. I can also explain it in one of the OF calls or directly to the developer is going to look at this.

BR/Luis



On 4/28/2014 8:18 PM, Abhijit Kumbhare wrote:

Would you happen to know where exactly is the code for these tests? Were these tests working before (at Hydrogen release timeframe)?


On Mon, Apr 28, 2014 at 7:58 PM, Luis Gomez <ecelgp@...> wrote:
Hi Abhijit,

One thing we need to work once we get all these enhancement in the OF13 plugin is the stability of the NSF test.

If you see old OF10 plugin NSF test:

https://jenkins.opendaylight.org/integration/job/integration-csit-base/

compared to new OF13 plugin NSF test (~same test cases for OF1.0 and OF1.3 switch simulation):

https://jenkins.opendaylight.org/integration/job/integration-csit-base-of13/

They are far to be similar in terms of stability. Also important to note, most of NSF we are testing are AD-SAL apps like topology mgr, switch mgr, FRM, etc... so in this case it is hard to say if the issue is in the OF13 plugin, the SAL adapters or both.

BR/Luis


On 4/28/2014 3:00 PM, Abhijit Kumbhare wrote:
Thanks Luis! Yes - we really would like these 2 things get tested a bit to make sure we do not break things.

I am hoping both these are done by Wednesday. However the first one (datastore) needs a fix by Vaclav - and I do not have a date as to when that will happen. The second (basic thread model changes) - Muthu said he will push these Tuesday his time - so we should have this tomorrow.


On Mon, Apr 28, 2014 at 2:54 PM, Luis Gomez <ecelgp@...> wrote:
Hi Abhijt, sorry I could not make the meeting this morning, I had a conflicting appointment.

Here are a few comments:


1) Making the new data store as the default in the OpenFlow plugin repository - for testing more

Do you know when this is going to happen? it would be good if we can pass the existing test suites before so we make sure we do not break everything after the change.


2) The basic thread model changes (almost done)

Same thing here, we would like to test this in Ericsson Lab once it is available, Chris O'Shea is continuously collecting CBench results in the Lab.

Thanks/Luis



On 4/28/2014 11:49 AM, Abhijit Kumbhare wrote:
Status of the action items:

1) Making the new data store as the default in the OpenFlow plugin repository - for testing more. - Vaclav/Michal

This was gated by data store bug (being worked by controller team)

Datastore bug partially fixed - Vaclav working on this. Vaclav currently heavily loaded - we will need a rough timeline from him (copied). 

2) The basic thread model changes (almost done) - Muthu/Prasanna (need to involve Luis/integration test team after changes merged to test OpenFlow 1.0/1.3 continue to work as before)

Muthu's local env failure fixed. One IT test case failing - should be fixed shortly - outlook April 29. Will be merged into a topic branch & then merged into the master. Additional commit will be done later to make it configurable (queuekeeper pool size - main connection, auxiliary connection; thread pool for the ticket finisher). For now the config parameters will be in config.ini. The wiki diagrams will need to be updated with the changes. Once Muthu has merged his changes back - we will need to have the integration test team test to see if the existing OpenFlow 1.0/1.3 functionality is not broken.

Prasanna (one person from Ericsson will be assigned) to help in making this configurable & help in performance measurements/tuning.

Prasanna, Michal, Tony & Muthu to meet Tuesday April 29 morning (Slovakia/India time) - to discuss the performance measurements/tuning.

Side Note:
Prasanna not seeing the performance improvements with the new datastore (with the new datastore seeing less performance than with the old datastore). He will need to sync up with Tony/Robert. Config of the machine - high end - 8 core CPU.

3) Critical bug "Cbench in throughput mode causes the controller to run out of memory":

https://bugs.opendaylight.org/show_bug.cgi?id=803 

 Michal Rehak investigating.

4) Michal was going to be pulling the cherry picked gerrits into the stable branch.

Remove the previously created stable branch - and cherry pick gerrits into the stable branch. Since Michal Rehak was working on this and he is heavily loaded - we will reach out to Surekha Bejgam - for help in this release engineering activity.
 
Kamal will be taking this up. 

Kamal's team will also take up https://bugs.opendaylight.org/show_bug.cgi?id=275. Prasanna advised that Kamal's team build up upon an unmerged gerrit http://git.opendaylight.org/gerrit/4720 for this bug.


- Abhijit



_______________________________________________
openflowplugin-dev mailing list
openflowplugin-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev





Abhijit Kumbhare
 



On Mon, Apr 28, 2014 at 8:35 PM, Luis Gomez <ecelgp@...> wrote:
Hi Abhijit,

The test cases were much worse by the time of Hydrogen. The best way to run these tests locally is to download the test VM here:

https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Test_VMs

There is a link to TWS recording at the bottom of the wiki on how to use the VM. I can also explain it in one of the OF calls or directly to the developer is going to look at this.

BR/Luis




On 4/28/2014 8:18 PM, Abhijit Kumbhare wrote:
Would you happen to know where exactly is the code for these tests? Were these tests working before (at Hydrogen release timeframe)?


On Mon, Apr 28, 2014 at 7:58 PM, Luis Gomez <ecelgp@...> wrote:
Hi Abhijit,

One thing we need to work once we get all these enhancement in the OF13 plugin is the stability of the NSF test.

If you see old OF10 plugin NSF test:

https://jenkins.opendaylight.org/integration/job/integration-csit-base/

compared to new OF13 plugin NSF test (~same test cases for OF1.0 and OF1.3 switch simulation):

https://jenkins.opendaylight.org/integration/job/integration-csit-base-of13/

They are far to be similar in terms of stability. Also important to note, most of NSF we are testing are AD-SAL apps like topology mgr, switch mgr, FRM, etc... so in this case it is hard to say if the issue is in the OF13 plugin, the SAL adapters or both.

BR/Luis


On 4/28/2014 3:00 PM, Abhijit Kumbhare wrote:
Thanks Luis! Yes - we really would like these 2 things get tested a bit to make sure we do not break things.

I am hoping both these are done by Wednesday. However the first one (datastore) needs a fix by Vaclav - and I do not have a date as to when that will happen. The second (basic thread model changes) - Muthu said he will push these Tuesday his time - so we should have this tomorrow.


On Mon, Apr 28, 2014 at 2:54 PM, Luis Gomez <ecelgp@...> wrote:
Hi Abhijt, sorry I could not make the meeting this morning, I had a conflicting appointment.

Here are a few comments:


1) Making the new data store as the default in the OpenFlow plugin repository - for testing more

Do you know when this is going to happen? it would be good if we can pass the existing test suites before so we make sure we do not break everything after the change.


2) The basic thread model changes (almost done)

Same thing here, we would like to test this in Ericsson Lab once it is available, Chris O'Shea is continuously collecting CBench results in the Lab.

Thanks/Luis



On 4/28/2014 11:49 AM, Abhijit Kumbhare wrote:
Status of the action items:

1) Making the new data store as the default in the OpenFlow plugin repository - for testing more. - Vaclav/Michal

This was gated by data store bug (being worked by controller team)

Datastore bug partially fixed - Vaclav working on this. Vaclav currently heavily loaded - we will need a rough timeline from him (copied). 

2) The basic thread model changes (almost done) - Muthu/Prasanna (need to involve Luis/integration test team after changes merged to test OpenFlow 1.0/1.3 continue to work as before)

Muthu's local env failure fixed. One IT test case failing - should be fixed shortly - outlook April 29. Will be merged into a topic branch & then merged into the master. Additional commit will be done later to make it configurable (queuekeeper pool size - main connection, auxiliary connection; thread pool for the ticket finisher). For now the config parameters will be in config.ini. The wiki diagrams will need to be updated with the changes. Once Muthu has merged his changes back - we will need to have the integration test team test to see if the existing OpenFlow 1.0/1.3 functionality is not broken.

Prasanna (one person from Ericsson will be assigned) to help in making this configurable & help in performance measurements/tuning.

Prasanna, Michal, Tony & Muthu to meet Tuesday April 29 morning (Slovakia/India time) - to discuss the performance measurements/tuning.

Side Note:
Prasanna not seeing the performance improvements with the new datastore (with the new datastore seeing less performance than with the old datastore). He will need to sync up with Tony/Robert. Config of the machine - high end - 8 core CPU.

3) Critical bug "Cbench in throughput mode causes the controller to run out of memory":

https://bugs.opendaylight.org/show_bug.cgi?id=803 

 Michal Rehak investigating.

4) Michal was going to be pulling the cherry picked gerrits into the stable branch.

Remove the previously created stable branch - and cherry pick gerrits into the stable branch. Since Michal Rehak was working on this and he is heavily loaded - we will reach out to Surekha Bejgam - for help in this release engineering activity.
 
Kamal will be taking this up. 

Kamal's team will also take up https://bugs.opendaylight.org/show_bug.cgi?id=275. Prasanna advised that Kamal's team build up upon an unmerged gerrit http://git.opendaylight.org/gerrit/4720 for this bug.


- Abhijit



_______________________________________________
openflowplugin-dev mailing list
openflowplugin-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev






Luis Gomez
 

Hi Abhijit,

Before filing bugs I would like someone from integration to run these particular TCs in his local environment to discard an issue in the LF automation.

In addition Madhusudhan has already found some issues related to our OF13 test suite:

https://bugs.opendaylight.org/show_bug.cgi?id=782

This is we see a lot of errors in the OSGi console even when the tests pass.

BR/Luis


On 4/28/2014 8:46 PM, Abhijit Kumbhare wrote:



On Mon, Apr 28, 2014 at 8:35 PM, Luis Gomez <ecelgp@...> wrote:
Hi Abhijit,

The test cases were much worse by the time of Hydrogen. The best way to run these tests locally is to download the test VM here:

https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Test_VMs

There is a link to TWS recording at the bottom of the wiki on how to use the VM. I can also explain it in one of the OF calls or directly to the developer is going to look at this.

BR/Luis




On 4/28/2014 8:18 PM, Abhijit Kumbhare wrote:
Would you happen to know where exactly is the code for these tests? Were these tests working before (at Hydrogen release timeframe)?


On Mon, Apr 28, 2014 at 7:58 PM, Luis Gomez <ecelgp@...> wrote:
Hi Abhijit,

One thing we need to work once we get all these enhancement in the OF13 plugin is the stability of the NSF test.

If you see old OF10 plugin NSF test:

https://jenkins.opendaylight.org/integration/job/integration-csit-base/

compared to new OF13 plugin NSF test (~same test cases for OF1.0 and OF1.3 switch simulation):

https://jenkins.opendaylight.org/integration/job/integration-csit-base-of13/

They are far to be similar in terms of stability. Also important to note, most of NSF we are testing are AD-SAL apps like topology mgr, switch mgr, FRM, etc... so in this case it is hard to say if the issue is in the OF13 plugin, the SAL adapters or both.

BR/Luis


On 4/28/2014 3:00 PM, Abhijit Kumbhare wrote:
Thanks Luis! Yes - we really would like these 2 things get tested a bit to make sure we do not break things.

I am hoping both these are done by Wednesday. However the first one (datastore) needs a fix by Vaclav - and I do not have a date as to when that will happen. The second (basic thread model changes) - Muthu said he will push these Tuesday his time - so we should have this tomorrow.


On Mon, Apr 28, 2014 at 2:54 PM, Luis Gomez <ecelgp@...> wrote:
Hi Abhijt, sorry I could not make the meeting this morning, I had a conflicting appointment.

Here are a few comments:


1) Making the new data store as the default in the OpenFlow plugin repository - for testing more

Do you know when this is going to happen? it would be good if we can pass the existing test suites before so we make sure we do not break everything after the change.


2) The basic thread model changes (almost done)

Same thing here, we would like to test this in Ericsson Lab once it is available, Chris O'Shea is continuously collecting CBench results in the Lab.

Thanks/Luis



On 4/28/2014 11:49 AM, Abhijit Kumbhare wrote:
Status of the action items:

1) Making the new data store as the default in the OpenFlow plugin repository - for testing more. - Vaclav/Michal

This was gated by data store bug (being worked by controller team)

Datastore bug partially fixed - Vaclav working on this. Vaclav currently heavily loaded - we will need a rough timeline from him (copied). 

2) The basic thread model changes (almost done) - Muthu/Prasanna (need to involve Luis/integration test team after changes merged to test OpenFlow 1.0/1.3 continue to work as before)

Muthu's local env failure fixed. One IT test case failing - should be fixed shortly - outlook April 29. Will be merged into a topic branch & then merged into the master. Additional commit will be done later to make it configurable (queuekeeper pool size - main connection, auxiliary connection; thread pool for the ticket finisher). For now the config parameters will be in config.ini. The wiki diagrams will need to be updated with the changes. Once Muthu has merged his changes back - we will need to have the integration test team test to see if the existing OpenFlow 1.0/1.3 functionality is not broken.

Prasanna (one person from Ericsson will be assigned) to help in making this configurable & help in performance measurements/tuning.

Prasanna, Michal, Tony & Muthu to meet Tuesday April 29 morning (Slovakia/India time) - to discuss the performance measurements/tuning.

Side Note:
Prasanna not seeing the performance improvements with the new datastore (with the new datastore seeing less performance than with the old datastore). He will need to sync up with Tony/Robert. Config of the machine - high end - 8 core CPU.

3) Critical bug "Cbench in throughput mode causes the controller to run out of memory":

https://bugs.opendaylight.org/show_bug.cgi?id=803 

 Michal Rehak investigating.

4) Michal was going to be pulling the cherry picked gerrits into the stable branch.

Remove the previously created stable branch - and cherry pick gerrits into the stable branch. Since Michal Rehak was working on this and he is heavily loaded - we will reach out to Surekha Bejgam - for help in this release engineering activity.
 
Kamal will be taking this up. 

Kamal's team will also take up https://bugs.opendaylight.org/show_bug.cgi?id=275. Prasanna advised that Kamal's team build up upon an unmerged gerrit http://git.opendaylight.org/gerrit/4720 for this bug.


- Abhijit



_______________________________________________
openflowplugin-dev mailing list
openflowplugin-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev







1 - 5 of 5