Re: CSIT test files


Luis Gomez <luis.gomez@...>
 

Hi Baohua,

 

See all the failures I get after re-running the suite when container manager file in included:

 

[odluser@odl-test-tools ~]$ pybot -v CONTROLLER:10.125.136.52 -v MININET:10.125.136.53 -v USER_HOME:. -v MININET_USER:mininet integration/test/csit/suites/base/

==============================================================================

Base :: Test suite for the OpenDaylight base edition                         

==============================================================================

Base.switch Manager :: Test suite for the switch manager bundle.             

==============================================================================

List all nodes :: List all nodes and their properties in the network. | PASS |

------------------------------------------------------------------------------

Get a node :: Get a node                                              | PASS |

------------------------------------------------------------------------------

Add property to node :: Add a property to node                        | PASS |

------------------------------------------------------------------------------

Remove property from node :: Remove a property from node              | PASS |

------------------------------------------------------------------------------

Add property to nodeconnector :: Add a property to nodeconnector      | PASS |

------------------------------------------------------------------------------

Remove property from nodeconnector :: Remove a property from nodec... | PASS |

------------------------------------------------------------------------------

Base.switch Manager :: Test suite for the switch manager bundle.      | PASS |

6 critical tests, 6 passed, 0 failed

6 tests total, 6 passed, 0 failed

==============================================================================

Base.010 Topology Manager :: Test suite for Topology Manager                 

==============================================================================

Get Topology :: Get Topology and validate the result.                 | FAIL |

'{"edgeProperties":[]}' does not contain 'NodeConnector":{"type":"OF","node":{"type":"OF","id":"00:00:00:00:00:00:00:01"},"id":"1"}' 2 times

------------------------------------------------------------------------------

Add a userlink :: Add a userlink, list to validate the result.        | PASS |

------------------------------------------------------------------------------

Remove a userlink :: Remove a userlink, list to validate the result.  | PASS |

------------------------------------------------------------------------------

Base.010 Topology Manager :: Test suite for Topology Manager          | FAIL |

3 critical tests, 2 passed, 1 failed

3 tests total, 2 passed, 1 failed

==============================================================================

Base.015 Forwarding Rules Manager :: Test suite for the forwarding rule man...

==============================================================================

Add a flow :: Add a flow, list to validate the result.                | FAIL |

Response status code error: 500 != 201

------------------------------------------------------------------------------

Remove a flow :: Remove a flow, list to validate the result.          | FAIL |

Response status code error: 404 != 204

------------------------------------------------------------------------------

Base.015 Forwarding Rules Manager :: Test suite for the forwarding... | FAIL |

2 critical tests, 0 passed, 2 failed

2 tests total, 0 passed, 2 failed

==============================================================================

Base.020 Host Tracker :: Test suite for the host tracker module.             

==============================================================================

Add a host :: Add a host, list to validate the result.                | PASS |

------------------------------------------------------------------------------

Remove a host :: Remove a host, list to validate the result.          | PASS |

------------------------------------------------------------------------------

Base.020 Host Tracker :: Test suite for the host tracker module.      | PASS |

2 critical tests, 2 passed, 0 failed

2 tests total, 2 passed, 0 failed

==============================================================================

Base.025 Arp Handler :: Test suite for the arp handler module.               

==============================================================================

Add a subnet :: Add a subnet, list to validate the result.            | PASS |

------------------------------------------------------------------------------

Remove a subnet :: Remove a subnet, list to validate the result.      | PASS |

------------------------------------------------------------------------------

Base.025 Arp Handler :: Test suite for the arp handler module.        | PASS |

2 critical tests, 2 passed, 0 failed

2 tests total, 2 passed, 0 failed

==============================================================================

Base.030 Forwarding Manager :: Test suite for the forwarding manager module. 

==============================================================================

Add a static route :: Add a static route, list to validate the res... | FAIL |

Response status code error: 409 != 201

------------------------------------------------------------------------------

Remove a static route :: Remove a static route, list to validate t... | PASS |

------------------------------------------------------------------------------

Base.030 Forwarding Manager :: Test suite for the forwarding manag... | FAIL |

2 critical tests, 1 passed, 1 failed

2 tests total, 1 passed, 1 failed

==============================================================================

Base.035 Container Manager :: Test suite for the container manager module.   

==============================================================================

Add a container :: Add a container, list to validate the result.      | PASS |

------------------------------------------------------------------------------

Remove a container :: Remove a container, list to validate the res... | PASS |

------------------------------------------------------------------------------

Base.035 Container Manager :: Test suite for the container manager... | PASS |

2 critical tests, 2 passed, 0 failed

2 tests total, 2 passed, 0 failed

==============================================================================

Base :: Test suite for the OpenDaylight base edition                  | FAIL |

19 critical tests, 15 passed, 4 failed

19 tests total, 15 passed, 4 failed

==============================================================================

Output:  /home/odluser/output.xml

Log:     /home/odluser/log.html

Report:  /home/odluser/report.html

 

 

From: Baohua Yang [mailto:yangbaohua@...]
Sent: Friday, November 22, 2013 11:06 PM
To: Luis Gomez
Cc: denghuihuang@...; integration-dev@...
Subject: Re: CSIT test files

 

hi luis

    What's the problem about the container manager test now?

    I think it will restore the container configuration to 'default' after the test passes.

    Review the code and +1.

 

On Sat, Nov 23, 2013 at 1:12 PM, Luis Gomez <luis.gomez@...> wrote:

Hi Bahoua and Denghui,

 

I have submitted patch for your cases below with 2 exceptions:

 

- statistics_manager.txt because it is incomplete.

- container_manager.txt because even if it works, everything got screwed up in the controller after enabling containers so we can either add it at the very end of all or wait until we understand how to restore the controller from container mode.

 

Also please merge the code if you agree with the TCs.

 

http://git.opendaylight.org/gerrit/3030

 

BR/Luis

 

 

From: Baohua Yang [mailto:yangbaohua@...]
Sent: Thursday, November 21, 2013 5:43 PM
To: Luis Gomez
Cc: integration-dev@...; denghuihuang@...
Subject: Re: [integration-dev] Robot test case order

 

Currently, the passed test modules include

arp_handler.txt

container_manager.txt

forwarding_manager.txt

forwarding_rule_manager.txt

host_tracker.txt

statistics_manager.txt

topology_manager.txt

 

Wish you can help start with and test them first.  

 

Denghui  is working on the switch manager module.

Thanks!

 

 

On Fri, Nov 22, 2013 at 8:38 AM, Luis Gomez <luis.gomez@...> wrote:

Hi Baohua,

I am very close to have mininet part of Robot framework. After that the next step wll be to put some test files into the csit folder to start doing system test. How are you and Dengui doing with your system test files? Are they ready to be moved from robot_tools to csit?

BR/Luis



-----Original Message-----
From: Baohua Yang [mailto:yangbaohua@...]
Sent: Thursday, November 21, 2013 4:21 PM
To: Luis Gomez; integration-dev@...
Cc: denghuihuang@...
Subject: Re: [integration-dev] Robot test case order

sure!
i guess it would be the best solution currently.
sorry cannot attend the csit test tool meeting today.
denghui, please help backup if possible, thanks!


On 11/22/13, Luis Gomez <luis.gomez@...> wrote:
> Hi people,
>
> Time is ticking for CSIT release and I have not seen a clear proposal to run
> test files order so here is mine:
>
> We use the Robot recommended way of adding a number in the text file but we
> leave some space in the middle so we can easily insert some test file in
> case it is needed:
>
> 005__Switch_Manager.txt
> 010__Topology_Manager.txt
> 015__Forwarding_Rules_Manager.txt
> 020__Statistics_Manager.txt
> ....
>
> We can use this evening call to discuss more if someone thinks differently.
>
> BR/Luis
>
>
>
> From: Carol Sanders [mailto:carol.sanders@...]
> Sent: Thursday, November 21, 2013 11:13 AM
> To: Luis Gomez
> Subject: RE: CSIT wiki
>
> Looks good.
>
> From: Luis Gomez [mailto:luis.gomez@...]
> Sent: Thursday, November 21, 2013 10:56 AM
> To: Carol Sanders
> Subject: RE: CSIT wiki
>
> OK, then I am going to propose to use XXX__ format but leaving some space in
> the middle so we can easily insert test files later if needed, something
> like:
>
> 005__Switch_Manager.txt
> 010__Topology_Manager.txt
> 015__Forwarding_Manager.txt
> 020__Clustering_Manager.txt
> ....
>
> What do you think?
>
>
>
> From: Carol Sanders [mailto:carol.sanders@...]
> Sent: Thursday, November 21, 2013 10:36 AM
> To: Luis Gomez
> Subject: RE: CSIT wiki
>
> Hi Luis,
>
> My apologies..
>
> Robot will execute .txt or .html files by normal alphabetical order in a
> directory. So if you want order I would number them.
>
> Carol
>
> From: Luis Gomez [mailto:luis.gomez@...]
> Sent: Thursday, November 21, 2013 10:09 AM
> To: Carol Sanders
> Subject: FW: CSIT wiki
>
> Hi Carol,
>
> How do you normally implement test order in Robot? The test number XX__ on
> the file name or a file calling the different test files? I see second is
> more flexible but can have some issues on reporting and also on suite
> initialization file.
> Please let me know what is your view here as this is blocking the CSIT
> release.
>
> BR/Luis
>
>
> From: Luis Gomez
> Sent: Wednesday, November 20, 2013 9:27 PM
> To: 'integration-dev@...';
> 'infrastructure@...'
> Subject: CSIT wiki
>
> Hi team,
>
> I have updated our wiki with all the information around CSIT we have
> discussed lately:
>
> https://wiki.opendaylight.org/view/CrossProject:Integration_Group:CSIT
>
> What I would like you to do is:
>
> - Review the wiki above and let me know if something is missing or wrong
> - Decide on which way we are going to setup the test case order (this is the
> only thing missing in the wiki)
>
> Remember this is the very last step before we release the system test (CSIT)
> to the community so please respond to this mail asap.
>
> Thanks/Luis
>
>

--
Sent from my mobile device

Best wishes!
Baohua



 

--
Best wishes!
Baohua



 

--
Best wishes!
Baohua

Join {integration-dev@lists.opendaylight.org to automatically receive all group messages.