Test tools meeting yesterday


Luis Gomez <luis.gomez@...>
 

Hi all,

 

Some notes from test tools meeting yesterday:

 

- Top priority now is to get the Test VMs together with Jenkins working in OpenDaylight. We have Andy helping with this

- Robot Test cases: think about which global variables we want to pass to Robot (pybot) so that test can run on any test bed: “controller IP” is one for sure

- Robot libraries: It would be nice to have a robot library to control mininet actions: create topology, do ping, etc… (same as TestON has)

- REST API: Ask developers whether testing JSON or XML application is enough or we need to test both

 

Thanks/Luis

 


denghui huang
 

Hi all
 
    Please see my comment in line.


2013/11/16 Luis Gomez <luis.gomez@...>

Hi all,

 

Some notes from test tools meeting yesterday:

 

- Top priority now is to get the Test VMs together with Jenkins working in OpenDaylight. We have Andy helping with this

- Robot Test cases: think about which global variables we want to pass to Robot (pybot) so that test can run on any test bed: “controller IP” is one for sure

      I think Mininet VM IP and Mininet topology tree level also should be global variables for those scenario in which Mininet as Device emulator.

- Robot libraries: It would be nice to have a robot library to control mininet actions: create topology, do ping, etc… (same as TestON has)

     I think we have two option for this, one is using robotframework-sshlibrary, the other one is using MininetCLIDriver from TestON, in the following couple of days, i will try to evaluate this two option, which one is better for our project? Once this done, then we don't need to setup mininet in jenkins job. if you guys have suggestion for this, that would be great.
 

- REST API: Ask developers whether testing JSON or XML application is enough or we need to test both

 

Thanks/Luis

 


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev



Baohua Yang <yangbaohua@...>
 

Agree, the parameters  to startup mininet can also be passed when run the test suites (A suite can include several test cases, e.g., the base suite).


On Sun, Nov 17, 2013 at 11:11 PM, huang denghui <huangdenghui@...> wrote:

Hi all
 
    Please see my comment in line.


2013/11/16 Luis Gomez <luis.gomez@...>

Hi all,

 

Some notes from test tools meeting yesterday:

 

- Top priority now is to get the Test VMs together with Jenkins working in OpenDaylight. We have Andy helping with this

- Robot Test cases: think about which global variables we want to pass to Robot (pybot) so that test can run on any test bed: “controller IP” is one for sure

      I think Mininet VM IP and Mininet topology tree level also should be global variables for those scenario in which Mininet as Device emulator.

- Robot libraries: It would be nice to have a robot library to control mininet actions: create topology, do ping, etc… (same as TestON has)

     I think we have two option for this, one is using robotframework-sshlibrary, the other one is using MininetCLIDriver from TestON, in the following couple of days, i will try to evaluate this two option, which one is better for our project? Once this done, then we don't need to setup mininet in jenkins job. if you guys have suggestion for this, that would be great.
 

- REST API: Ask developers whether testing JSON or XML application is enough or we need to test both

 

Thanks/Luis

 


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev



_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev




--
Best wishes!
Baohua


Luis Gomez <luis.gomez@...>
 

IMHO, there are 2 things we should pass to the test suite:

 

- Controller IP, as our test code should run on any test bed regardless of its IP addressing.

 

- Mininet IP, same reason as above if we control mininet from Robot. Note that passing the control to Robot means that we will need to start and stop mininet on every test case (Topology Mgr, SW Mgr, etc…) as we agreed to have all test cases (written in the same test file) self-contained and independent. This should not be an issue anyway because start/stop mininet is very fast.

 

The mininet topology and other test parameter that do not change during the test suite can be passed as well, but I do not think it is a good idea because it adds extra complication and confusion at robot start, like if a test case/suite is written to run on a particular topology and we pass by accident another topology, it will not work. For all the constants that do not change across the test suite I would rather create a constant definition file in Robot.

 

BR/Luis

 

 

From: Baohua Yang [mailto:yangbaohua@...]
Sent: Sunday, November 17, 2013 5:21 PM
To: huang denghui
Cc: Luis Gomez; integration-dev@...
Subject: Re: [integration-dev] Test tools meeting yesterday

 

Agree, the parameters  to startup mininet can also be passed when run the test suites (A suite can include several test cases, e.g., the base suite).

 

On Sun, Nov 17, 2013 at 11:11 PM, huang denghui <huangdenghui@...> wrote:

Hi all
 

    Please see my comment in line.

 

2013/11/16 Luis Gomez <luis.gomez@...>

Hi all,

 

Some notes from test tools meeting yesterday:

 

- Top priority now is to get the Test VMs together with Jenkins working in OpenDaylight. We have Andy helping with this

- Robot Test cases: think about which global variables we want to pass to Robot (pybot) so that test can run on any test bed: “controller IP” is one for sure

      I think Mininet VM IP and Mininet topology tree level also should be global variables for those scenario in which Mininet as Device emulator.

- Robot libraries: It would be nice to have a robot library to control mininet actions: create topology, do ping, etc… (same as TestON has)

     I think we have two option for this, one is using robotframework-sshlibrary, the other one is using MininetCLIDriver from TestON, in the following couple of days, i will try to evaluate this two option, which one is better for our project? Once this done, then we don't need to setup mininet in jenkins job. if you guys have suggestion for this, that would be great.

 

- REST API: Ask developers whether testing JSON or XML application is enough or we need to test both

 

Thanks/Luis

 

 

_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev

 


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev



 

--
Best wishes!
Baohua


denghui huang
 

Hi all 
    My comments in line.


2013/11/18 Luis Gomez <luis.gomez@...>

IMHO, there are 2 things we should pass to the test suite:

 

- Controller IP, as our test code should run on any test bed regardless of its IP addressing.

 

- Mininet IP, same reason as above if we control mininet from Robot. Note that passing the control to Robot means that we will need to start and stop mininet on every test case (Topology Mgr, SW Mgr, etc…) as we agreed to have all test cases (written in the same test file) self-contained and independent. This should not be an issue anyway because start/stop mininet is very fast.

 
   I think we don't need to have all test cases written in the same test file. we just need to write one file for setup environment with sequence 01, and another file for cleanup environment  with the last sequence 0(x+2), x is the number of the test suite under base directory, such as, 01__suite_setup.txt, ...... 0(x+2)__suite_tear_down.txt, it means all test suites under base directory comprise one big suite called base suite.  

 

The mininet topology and other test parameter that do not change during the test suite can be passed as well, but I do not think it is a good idea because it adds extra complication and confusion at robot start, like if a test case/suite is written to run on a particular topology and we pass by accident another topology, it will not work. For all the constants that do not change across the test suite I would rather create a constant definition file in Robot.

 

 My intention to introduce topology tree level, is that base test suite can support many kind of topology, not just only support one, it means all test cases is independent with topology, we do not to change the topology during test suite executing.

 

BR/Luis

 

 

From: Baohua Yang [mailto:yangbaohua@...]
Sent: Sunday, November 17, 2013 5:21 PM
To: huang denghui
Cc: Luis Gomez; integration-dev@...
Subject: Re: [integration-dev] Test tools meeting yesterday

 

Agree, the parameters  to startup mininet can also be passed when run the test suites (A suite can include several test cases, e.g., the base suite).

 

On Sun, Nov 17, 2013 at 11:11 PM, huang denghui <huangdenghui@...> wrote:

Hi all
 

    Please see my comment in line.

 

2013/11/16 Luis Gomez <luis.gomez@...>

Hi all,

 

Some notes from test tools meeting yesterday:

 

- Top priority now is to get the Test VMs together with Jenkins working in OpenDaylight. We have Andy helping with this

- Robot Test cases: think about which global variables we want to pass to Robot (pybot) so that test can run on any test bed: “controller IP” is one for sure

      I think Mininet VM IP and Mininet topology tree level also should be global variables for those scenario in which Mininet as Device emulator.

- Robot libraries: It would be nice to have a robot library to control mininet actions: create topology, do ping, etc… (same as TestON has)

     I think we have two option for this, one is using robotframework-sshlibrary, the other one is using MininetCLIDriver from TestON, in the following couple of days, i will try to evaluate this two option, which one is better for our project? Once this done, then we don't need to setup mininet in jenkins job. if you guys have suggestion for this, that would be great.

 

- REST API: Ask developers whether testing JSON or XML application is enough or we need to test both

 

Thanks/Luis

 

 

_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev

 


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev



 

--
Best wishes!
Baohua



Baohua Yang <yangbaohua@...>
 

About the test suites organization, a possible idea:

Can we add a main script (e.g., bash) which will pick the robot scripts and run them in given order.
And the bash script can also help do some environment setup and assistant things (maybe it can help connect to the mininet vm or boot mininet environment, etc.).
An obvious advantage is that we can freely name the test cases files as we like, without mandatory number prefix.

Do you think it's worthy to do so? 

About the more topology supports:
It would be better that we can support tests on variable typologies, which is more convincing.
Maybe it is not a short achievable aim, but I think we do need put it on the development plan.




On Tue, Nov 19, 2013 at 4:34 PM, huang denghui <huangdenghui@...> wrote:
Hi all 
    My comments in line.


2013/11/18 Luis Gomez <luis.gomez@...>

IMHO, there are 2 things we should pass to the test suite:

 

- Controller IP, as our test code should run on any test bed regardless of its IP addressing.

 

- Mininet IP, same reason as above if we control mininet from Robot. Note that passing the control to Robot means that we will need to start and stop mininet on every test case (Topology Mgr, SW Mgr, etc…) as we agreed to have all test cases (written in the same test file) self-contained and independent. This should not be an issue anyway because start/stop mininet is very fast.

 
   I think we don't need to have all test cases written in the same test file. we just need to write one file for setup environment with sequence 01, and another file for cleanup environment  with the last sequence 0(x+2), x is the number of the test suite under base directory, such as, 01__suite_setup.txt, ...... 0(x+2)__suite_tear_down.txt, it means all test suites under base directory comprise one big suite called base suite.  

 

The mininet topology and other test parameter that do not change during the test suite can be passed as well, but I do not think it is a good idea because it adds extra complication and confusion at robot start, like if a test case/suite is written to run on a particular topology and we pass by accident another topology, it will not work. For all the constants that do not change across the test suite I would rather create a constant definition file in Robot.

 

 My intention to introduce topology tree level, is that base test suite can support many kind of topology, not just only support one, it means all test cases is independent with topology, we do not to change the topology during test suite executing.

 

BR/Luis

 

 

From: Baohua Yang [mailto:yangbaohua@...]
Sent: Sunday, November 17, 2013 5:21 PM
To: huang denghui
Cc: Luis Gomez; integration-dev@...
Subject: Re: [integration-dev] Test tools meeting yesterday

 

Agree, the parameters  to startup mininet can also be passed when run the test suites (A suite can include several test cases, e.g., the base suite).

 

On Sun, Nov 17, 2013 at 11:11 PM, huang denghui <huangdenghui@...> wrote:

Hi all
 

    Please see my comment in line.

 

2013/11/16 Luis Gomez <luis.gomez@...>

Hi all,

 

Some notes from test tools meeting yesterday:

 

- Top priority now is to get the Test VMs together with Jenkins working in OpenDaylight. We have Andy helping with this

- Robot Test cases: think about which global variables we want to pass to Robot (pybot) so that test can run on any test bed: “controller IP” is one for sure

      I think Mininet VM IP and Mininet topology tree level also should be global variables for those scenario in which Mininet as Device emulator.

- Robot libraries: It would be nice to have a robot library to control mininet actions: create topology, do ping, etc… (same as TestON has)

     I think we have two option for this, one is using robotframework-sshlibrary, the other one is using MininetCLIDriver from TestON, in the following couple of days, i will try to evaluate this two option, which one is better for our project? Once this done, then we don't need to setup mininet in jenkins job. if you guys have suggestion for this, that would be great.

 

- REST API: Ask developers whether testing JSON or XML application is enough or we need to test both

 

Thanks/Luis

 

 

_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev

 


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev



 

--
Best wishes!
Baohua





--
Best wishes!
Baohua