ODL Integration Testing


Luis Gomez <luis.gomez@...>
 

Hi Thomas,

 

If I understand correctly you are having the same issue as I heard on some other projects, this is: the integration tests you do locally using real or simulated devices do not work at OpenDaylight because there you have only the controller instance, so in the end you have to disable these tests when committing code in OpenDaylight. I can think on a couple of things to avoid this:

 

1) We already have a test bed with a few VMs for system test including OVS. So any integration test you can put into system test will have the device you are looking for. However system test does not work the same as project build test, here are some differences:

 

- system test is triggered after code has been merged to master not when it is pushed in the system like the project verify Jenkins job

- Once a new code is merged to master, the system test will build a release vehicle, download the artifact to the controller VM and start the test case suite for the release under test. So system test runs on release vehicles instead of individual project code

- The system test is kind of black-box test so we only use controller external interfaces: NB APIs and SB plugins, so no java interface test is available here

 

2) I also believe it is possible to setup some VMs to work in conjunction with the project build so that integration tests can run during verify and merge jobs. However to implement this we will need to follow some steps like: 1. consult other projects to see who else is interested (the more support the better), 2. involve the TSC as they have to approve the new VMs, 3. Check with Linux Foundation how they will install and support the new VMs.

 

Let me know what are your thoughts and also with regards to item 2), although I do not think this can be done in very short time, it is better to start asap if we think it is a good idea.

 

I am also ccing integration and infrastructure because this conversation might be of their interest as well.

 

BR/Luis

 

 

 

From: Thomas Bachman [mailto:tbachman@...]
Sent: Monday, December 09, 2013 9:33 AM
To: Luis Gomez
Cc: mavenugo@...
Subject: ODL Integration Testing

 

Hi Luis,

 

I realize that you and your team are probably super-busy making the 1st ODL release happen, but I'm hoping you have a second to help me with something.


I'm working with the OVSDB team to improve our test coverage. I guess someone on the team already wrote some ITs, but they currently don't have a tie-in to Jenkins. As you can imagine, we'd like to have this integration so that developers can have quicker feedback if/when they've broken something. However, there is a slight complication -- the tests, as currently written, talk to an actual OVS instance, and therefore change the OVS instance's state (e.g. create bridges, add ports, etc.). Our concern is that if this were connected to Jenkins, it could cause false failures in other IT tests that might be dependent on an OVS (e.g. an unexpected bridge is present/configured).

 

I was wondering if you had any ideas on how to address this, or if you know of a better way of doing it?  I asked tykeal about the possibility of spinning up a VM instance specifically for this type of testing, but it appears that we aren't able to do this yet.

 

Thanks again, and let me know if  you have any questions.

 

cheers,

 

-Thomas Bachman

 

 


Thomas Bachman <tbachman@...>
 

Hi Luis,


Thanks for the reply!  I agree that #2 is where we want to go. I'll follow-up with the other projects, and then take the result of those conversations up with the TSC.

In parallel, I'm going to see what other paths I can get going, just so that we can have this level of coverage for Hydrogen. I share some of the same concerns with with #1, in addition to the state modification ramifications I mentioned, so I'm less inclined to go this route.

Also -- just so I don't step on any toes here -- do you view this level of testing as something maintained by you and your team, or something maintained by each development team?

cheers,

-Thomas Bachman


On Monday, December 9, 2013 3:30 PM, Luis Gomez <luis.gomez@...> wrote:
Hi Thomas,
 
If I understand correctly you are having the same issue as I heard on some other projects, this is: the integration tests you do locally using real or simulated devices do not work at OpenDaylight because there you have only the controller instance, so in the end you have to disable these tests when committing code in OpenDaylight. I can think on a couple of things to avoid this:
 
1) We already have a test bed with a few VMs for system test including OVS. So any integration test you can put into system test will have the device you are looking for. However system test does not work the same as project build test, here are some differences:
 
- system test is triggered after code has been merged to master not when it is pushed in the system like the project verify Jenkins job
- Once a new code is merged to master, the system test will build a release vehicle, download the artifact to the controller VM and start the test case suite for the release under test. So system test runs on release vehicles instead of individual project code
- The system test is kind of black-box test so we only use controller external interfaces: NB APIs and SB plugins, so no java interface test is available here
 
2) I also believe it is possible to setup some VMs to work in conjunction with the project build so that integration tests can run during verify and merge jobs. However to implement this we will need to follow some steps like: 1. consult other projects to see who else is interested (the more support the better), 2. involve the TSC as they have to approve the new VMs, 3. Check with Linux Foundation how they will install and support the new VMs.
 
Let me know what are your thoughts and also with regards to item 2), although I do not think this can be done in very short time, it is better to start asap if we think it is a good idea.
 
I am also ccing integration and infrastructure because this conversation might be of their interest as well.
 
BR/Luis
 
 
 
From: Thomas Bachman [mailto:tbachman@...]
Sent: Monday, December 09, 2013 9:33 AM
To: Luis Gomez
Cc: mavenugo@...
Subject: ODL Integration Testing
 
Hi Luis,
 
I realize that you and your team are probably super-busy making the 1st ODL release happen, but I'm hoping you have a second to help me with something.

I'm working with the OVSDB team to improve our test coverage. I guess someone on the team already wrote some ITs, but they currently don't have a tie-in to Jenkins. As you can imagine, we'd like to have this integration so that developers can have quicker feedback if/when they've broken something. However, there is a slight complication -- the tests, as currently written, talk to an actual OVS instance, and therefore change the OVS instance's state (e.g. create bridges, add ports, etc.). Our concern is that if this were connected to Jenkins, it could cause false failures in other IT tests that might be dependent on an OVS (e.g. an unexpected bridge is present/configured).
 
I was wondering if you had any ideas on how to address this, or if you know of a better way of doing it?  I asked tykeal about the possibility of spinning up a VM instance specifically for this type of testing, but it appears that we aren't able to do this yet.
 
Thanks again, and let me know if  you have any questions.
 
cheers,
 
-Thomas Bachman
 
 



Luis Gomez <luis.gomez@...>
 

Hi Thomas,

 

This is a very good question and actually to get also a good answer we will need to know more about the resources to be implemented. Normally all that is around one project development is project responsibility, while all that is common or project integration development is our responsibility. So if the resources (the VM in this case) are dedicated to one particular project I would say the project together with the LF support can maintain them, however if the resources are to be shared across different projects, then I think we (the integration team) together with the LF support should maintain them.

 

BR/Luis

 

 

From: Thomas Bachman [mailto:tbachman@...]
Sent: Monday, December 09, 2013 2:03 PM
To: Luis Gomez
Cc: mavenugo@...; 'integration-dev@...'; 'infrastructure@...'
Subject: Re: ODL Integration Testing

 

Hi Luis,

 


Thanks for the reply!  I agree that #2 is where we want to go. I'll follow-up with the other projects, and then take the result of those conversations up with the TSC.

 

In parallel, I'm going to see what other paths I can get going, just so that we can have this level of coverage for Hydrogen. I share some of the same concerns with with #1, in addition to the state modification ramifications I mentioned, so I'm less inclined to go this route.

 

Also -- just so I don't step on any toes here -- do you view this level of testing as something maintained by you and your team, or something maintained by each development team?

 

cheers,

 

-Thomas Bachman

 

On Monday, December 9, 2013 3:30 PM, Luis Gomez <luis.gomez@...> wrote:

Hi Thomas,

 

If I understand correctly you are having the same issue as I heard on some other projects, this is: the integration tests you do locally using real or simulated devices do not work at OpenDaylight because there you have only the controller instance, so in the end you have to disable these tests when committing code in OpenDaylight. I can think on a couple of things to avoid this:

 

1) We already have a test bed with a few VMs for system test including OVS. So any integration test you can put into system test will have the device you are looking for. However system test does not work the same as project build test, here are some differences:

 

- system test is triggered after code has been merged to master not when it is pushed in the system like the project verify Jenkins job

- Once a new code is merged to master, the system test will build a release vehicle, download the artifact to the controller VM and start the test case suite for the release under test. So system test runs on release vehicles instead of individual project code

- The system test is kind of black-box test so we only use controller external interfaces: NB APIs and SB plugins, so no java interface test is available here

 

2) I also believe it is possible to setup some VMs to work in conjunction with the project build so that integration tests can run during verify and merge jobs. However to implement this we will need to follow some steps like: 1. consult other projects to see who else is interested (the more support the better), 2. involve the TSC as they have to approve the new VMs, 3. Check with Linux Foundation how they will install and support the new VMs.

 

Let me know what are your thoughts and also with regards to item 2), although I do not think this can be done in very short time, it is better to start asap if we think it is a good idea.

 

I am also ccing integration and infrastructure because this conversation might be of their interest as well.

 

BR/Luis

 

 

 

From: Thomas Bachman [mailto:tbachman@...]
Sent: Monday, December 09, 2013 9:33 AM
To: Luis Gomez
Cc: mavenugo@...
Subject: ODL Integration Testing

 

Hi Luis,

 

I realize that you and your team are probably super-busy making the 1st ODL release happen, but I'm hoping you have a second to help me with something.


I'm working with the OVSDB team to improve our test coverage. I guess someone on the team already wrote some ITs, but they currently don't have a tie-in to Jenkins. As you can imagine, we'd like to have this integration so that developers can have quicker feedback if/when they've broken something. However, there is a slight complication -- the tests, as currently written, talk to an actual OVS instance, and therefore change the OVS instance's state (e.g. create bridges, add ports, etc.). Our concern is that if this were connected to Jenkins, it could cause false failures in other IT tests that might be dependent on an OVS (e.g. an unexpected bridge is present/configured).

 

I was wondering if you had any ideas on how to address this, or if you know of a better way of doing it?  I asked tykeal about the possibility of spinning up a VM instance specifically for this type of testing, but it appears that we aren't able to do this yet.

 

Thanks again, and let me know if  you have any questions.

 

cheers,

 

-Thomas Bachman