Hi Colin, sorry for my late answer. See my answers in-line:

On Jan 9, 2015, at 11:04 AM, Colin Scott <cs@...> wrote:

Hey all,

I'm a graduate student at UC Berkeley, focusing on testing and troubleshooting SDN control software.

I'd like to understand what sorts of QA testing you folks do at ODL. In particular:

  - Are the QA tests primarily intended to catch performance bugs, correctness bugs, or integration bugs?

So far feature bugs and integration bugs, we will have something for performance bugs this release cycle

  - Are there programmatically generated tests (e.g. fuzz tests), or are they all generated by hand?

They are by hand today but there was an intern last year that worked on a fuzz framework for NB API:

https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Fuzzy_Test

  - Do the tests trigger node failures?

You mean resilience test? taking link downs and so on? we do not have that yet but we are considering doing a PoC/demo and corresponding testing this release cycle

  - What scale do the tests run at?

Most of tests today are low scale, 3-7 devices, we have a few scalability tests that go to 31 and 63 switches

  - When a failing test is found, how do you debug it?

We use Robot frameworks that helps a lot debugging test cases issues. After that we use manual reproduction of the test case to confirm the issue and file the corresponding bug 

  - What diagnostic information is collected beyond console/logging output?

So far we collect robot test reports and controller log output

For what it's worth, I'm interested because we've developed a tool for troubleshooting control software, and we'd like to understand how applicable it would be to a project like ODL. 

Next week we will gather Tue 27th and Wed 28th in Brocade office in San Jose. It would be awesome if you can join to learn more how we do test and share your tool:

https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Lithium_Test_Strategy#Topics_for_the_27th-28th_meetup

Thanks!

-Colin

_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev

_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev