Date   

breakages in CSIT

JamO Luhrsen
 

All,

There was some breaking changes in CSIT/infra dependencies late last week that has
put us on a path to upgrade (finally) to python3. There are multiple activities happening
around this to bring CSIT back to a healthy state.

This is just FYI.

Please reach out if you have a specific issue that needs to be resolved and we can
hopefully point you in the right direction. IRC still works, btw :)

Thanks,
JamO


Re: CSIT jobs breaking on python2 trouble

Thanh ha <zxiiro@...>
 

I think its probably time for us to default to Python 3 considering Python 2 is EOL at this point. More and more packages will become Python 3 supported only so we'll likely to continue hitting issues in the future if we keep on Python 2.

Another patch I started looking into is consolidating all the Python requirements into a single requirements file (https://git.opendaylight.org/gerrit/c/releng/builder/+/90435). One of the problems I find with Python dependencies is that when you run "pip install" for packages separately, Python's dependency manager does not take into account previously installed packages so later pip installs could pull in incompatible versions of similar dependencies thus breaking previous pip installed packages. Having all of the packages in a single requirements file and pip installing all of them together in 1 command will resolve this kind of issue.

I'm happy to help with a Python 3 migration but would need assistance from integration team to test such an effort.

Regards,
Thanh


On Fri, Jun 19, 2020 at 3:16 AM JamO Luhrsen <jluhrsen@...> wrote:

ugly hack:

https://git.opendaylight.org/gerrit/c/releng/builder/+/90504


On 6/18/20 7:01 PM, JamO Luhrsen via lists.opendaylight.org wrote:

https://jira.opendaylight.org/browse/INTTEST-103

On 6/18/20 6:57 PM, JamO Luhrsen via lists.opendaylight.org wrote:
Example:

https://jenkins.opendaylight.org/releng/job/netconf-csit-1node-userfeatures-all-magnesium/283/console

We still have some pip install (which is python2) going on in our shell scripts to run our robot
jobs. Some package somewhere is probably recently upgraded that is no longer going to
allow python2. I have not looked very deep yet, just fyi for now.

JamO




    


Re: CSIT jobs breaking on python2 trouble

JamO Luhrsen
 

ugly hack:

https://git.opendaylight.org/gerrit/c/releng/builder/+/90504


On 6/18/20 7:01 PM, JamO Luhrsen via lists.opendaylight.org wrote:

https://jira.opendaylight.org/browse/INTTEST-103

On 6/18/20 6:57 PM, JamO Luhrsen via lists.opendaylight.org wrote:
Example:

https://jenkins.opendaylight.org/releng/job/netconf-csit-1node-userfeatures-all-magnesium/283/console

We still have some pip install (which is python2) going on in our shell scripts to run our robot
jobs. Some package somewhere is probably recently upgraded that is no longer going to
allow python2. I have not looked very deep yet, just fyi for now.

JamO




    


Re: CSIT jobs breaking on python2 trouble

JamO Luhrsen
 

https://jira.opendaylight.org/browse/INTTEST-103

On 6/18/20 6:57 PM, JamO Luhrsen via lists.opendaylight.org wrote:

Example:

https://jenkins.opendaylight.org/releng/job/netconf-csit-1node-userfeatures-all-magnesium/283/console

We still have some pip install (which is python2) going on in our shell scripts to run our robot
jobs. Some package somewhere is probably recently upgraded that is no longer going to
allow python2. I have not looked very deep yet, just fyi for now.

JamO



    


CSIT jobs breaking on python2 trouble

JamO Luhrsen
 

Example:

https://jenkins.opendaylight.org/releng/job/netconf-csit-1node-userfeatures-all-magnesium/283/console

We still have some pip install (which is python2) going on in our shell scripts to run our robot
jobs. Some package somewhere is probably recently upgraded that is no longer going to
allow python2. I have not looked very deep yet, just fyi for now.

JamO


Re: builder[master]: Reduce Netvirt jobs costs

Srinivas <srinivas.rachakonda@...>
 

Hi Anil,

 

Let me discuss with Team and get back to you.

 

Thanks,

Srinivas

+91-9243478719

 

From: integration-dev@... <integration-dev@...> On Behalf Of Anil Belur
Sent: 18 June 2020 12:01
To: Jamo Luhrsen <jluhrsen@...>
Cc: Luis Gomez <ecelgp@...>; Srinivas Rachakonda <srinivas.rachakonda@...>; app-dev@...; Bengt Thuree <bthuree@...>; integration-dev@...
Subject: Re: [integration-dev] builder[master]: Reduce Netvirt jobs costs

 

Hi Srinivas: 

 

I think the Netvirt job count increased from ~400 to 570 with the table provided by Srinivas/Netvirt team, we need to make sure this reduces the frequency of jobs or there is very little scope for the job costs being saved here. 

 

Thanks,

Anil

 

On Thu, Jun 18, 2020 at 7:37 AM Jamo Luhrsen <jluhrsen@...> wrote:

Yeah, this needs to be dialed in a little tighter. There is a lot that could be
confusing. Some history:

1) there is a list of jobs that we skip in the daily AR run, like any tempest
    job, so in those got marked in the netvirt yaml with daily

2) we used to have a special high-frequency job that covered some netvirt
    jobs and ran multiple times a day. that was removed though:
    https://git.opendaylight.org/gerrit/c/releng/builder/+/81858

 

Srinivas, can you please dig through the definitions and remove the daily
trigger from anything that is not in the list of jobs to skip. like here:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-test-jobs.yaml#L65

If a job is in the list of jobs to skip (it's just matching on job name regex),
then it will still need a trigger like daily or weekly.

 

Good catch, Luis!

 

JamO

 

 

On 6/16/20 12:58 PM, Luis Gomez wrote:

Guys, I just realized in this patch there is a new daily trigger:

 

schedule: '{schedule-daily}'

 

This has never been required before because the AR already runs daily and executes netvirt tests.

 

So now I see the same test job runs twice a day (because AR + this new trigger):

 

 

Is this intended or not?

 

BR/Luis

 



On Jun 15, 2020, at 10:49 AM, Jamo Luhrsen <jluhrsen@...> wrote:

 

ok, patch is merged.

Thanks,
JamO

On 6/13/20 8:26 PM, Srinivas Rachakonda wrote:

Hi Jamo,

 

I have reviewed the scripts and it looks fine.

Marked +1 in the review.

 

R.Srinivas

+91-9243478719

 

 

 

On Sat, 13 Jun 2020 at 01:49, Jamo Luhrsen <jluhrsen@...> wrote:

Srinivas,

Can you please check this patch to see if it's in line with what you were expecting?

Thanks,
JamO

#netvirt

On 6/11/20 1:13 AM, Gerrit Code Review wrote:

Patch set 2:Code-Review +1

View Change

To view, visit change 90385. To unsubscribe, or for help writing mail filters, visit settings.

Gerrit-Project: releng/builder

Gerrit-Branch: master

Gerrit-Change-Id: Ice7702c633e8f49b2b02f0828d130bb48f762e81

Gerrit-Change-Number: 90385

Gerrit-PatchSet: 2

Gerrit-Owner: Anil Belur <abelur@...>

Gerrit-Reviewer: Bengt Thuree <bthuree@...>

Gerrit-Reviewer: Jamo Luhrsen <jluhrsen@...>

Gerrit-Reviewer: Luis Gomez <ecelgp@...>

Gerrit-Reviewer: Srinivas Rachakonda <srinivas.rachakonda@...>

Gerrit-Reviewer: Thanh Ha (zxiiro) <zxiiro@...>

Gerrit-Reviewer: jenkins-releng <jenkins-releng@...>

Gerrit-Comment-Date: Thu, 11 Jun 2020 08:13:11 +0000

Gerrit-HasComments: No

Gerrit-Has-Labels: Yes

Gerrit-MessageType: comment

 


Re: builder[master]: Reduce Netvirt jobs costs

Anil Belur
 

Hi Srinivas: 

I think the Netvirt job count increased from ~400 to 570 with the table provided by Srinivas/Netvirt team, we need to make sure this reduces the frequency of jobs or there is very little scope for the job costs being saved here. 

Thanks,
Anil


On Thu, Jun 18, 2020 at 7:37 AM Jamo Luhrsen <jluhrsen@...> wrote:

Yeah, this needs to be dialed in a little tighter. There is a lot that could be
confusing. Some history:

1) there is a list of jobs that we skip in the daily AR run, like any tempest
    job, so in those got marked in the netvirt yaml with daily

2) we used to have a special high-frequency job that covered some netvirt
    jobs and ran multiple times a day. that was removed though:
    https://git.opendaylight.org/gerrit/c/releng/builder/+/81858


Srinivas, can you please dig through the definitions and remove the daily
trigger from anything that is not in the list of jobs to skip. like here:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-test-jobs.yaml#L65

If a job is in the list of jobs to skip (it's just matching on job name regex),
then it will still need a trigger like daily or weekly.


Good catch, Luis!


JamO



On 6/16/20 12:58 PM, Luis Gomez wrote:
Guys, I just realized in this patch there is a new daily trigger:

schedule: '{schedule-daily}'

This has never been required before because the AR already runs daily and executes netvirt tests.

So now I see the same test job runs twice a day (because AR + this new trigger):


Is this intended or not?

BR/Luis


On Jun 15, 2020, at 10:49 AM, Jamo Luhrsen <jluhrsen@...> wrote:

ok, patch is merged.

Thanks,
JamO

On 6/13/20 8:26 PM, Srinivas Rachakonda wrote:
Hi Jamo,

I have reviewed the scripts and it looks fine.
Marked +1 in the review.

R.Srinivas
+91-9243478719



On Sat, 13 Jun 2020 at 01:49, Jamo Luhrsen <jluhrsen@...> wrote:

Srinivas,

Can you please check this patch to see if it's in line with what you were expecting?

Thanks,
JamO

#netvirt

On 6/11/20 1:13 AM, Gerrit Code Review wrote:

Patch set 2:Code-Review +1

View Change

To view, visit change 90385. To unsubscribe, or for help writing mail filters, visit settings.

Gerrit-Project: releng/builder
Gerrit-Branch: master
Gerrit-Change-Id: Ice7702c633e8f49b2b02f0828d130bb48f762e81
Gerrit-Change-Number: 90385
Gerrit-PatchSet: 2
Gerrit-Owner: Anil Belur <abelur@...>
Gerrit-Reviewer: Bengt Thuree <bthuree@...>
Gerrit-Reviewer: Jamo Luhrsen <jluhrsen@...>
Gerrit-Reviewer: Luis Gomez <ecelgp@...>
Gerrit-Reviewer: Srinivas Rachakonda <srinivas.rachakonda@...>
Gerrit-Reviewer: Thanh Ha (zxiiro) <zxiiro@...>
Gerrit-Reviewer: jenkins-releng <jenkins-releng@...>
Gerrit-Comment-Date: Thu, 11 Jun 2020 08:13:11 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment


Re: builder[master]: Reduce Netvirt jobs costs

Srinivas <srinivas.rachakonda@...>
 

Hi Jamo,

 

Raised the below review:

https://git.opendaylight.org/gerrit/c/releng/builder/+/90493

 

 

Thanks,

Srinivas

+91-9243478719

 

From: integration-dev@... <integration-dev@...> On Behalf Of JamO Luhrsen
Sent: 18 June 2020 03:07
To: Luis Gomez <ecelgp@...>; Anil Belur <abelur@...>
Cc: Srinivas Rachakonda <srinivas.rachakonda@...>; app-dev@...; bthuree@...; integration-dev@...
Subject: Re: [integration-dev] builder[master]: Reduce Netvirt jobs costs

 

Yeah, this needs to be dialed in a little tighter. There is a lot that could be
confusing. Some history:

1) there is a list of jobs that we skip in the daily AR run, like any tempest
    job, so in those got marked in the netvirt yaml with daily

2) we used to have a special high-frequency job that covered some netvirt
    jobs and ran multiple times a day. that was removed though:
    https://git.opendaylight.org/gerrit/c/releng/builder/+/81858

 

Srinivas, can you please dig through the definitions and remove the daily
trigger from anything that is not in the list of jobs to skip. like here:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-test-jobs.yaml#L65

If a job is in the list of jobs to skip (it's just matching on job name regex),
then it will still need a trigger like daily or weekly.

 

Good catch, Luis!

 

JamO

 

 

On 6/16/20 12:58 PM, Luis Gomez wrote:

Guys, I just realized in this patch there is a new daily trigger:

 

schedule: '{schedule-daily}'

 

This has never been required before because the AR already runs daily and executes netvirt tests.

 

So now I see the same test job runs twice a day (because AR + this new trigger):

 

 

Is this intended or not?

 

BR/Luis

 



On Jun 15, 2020, at 10:49 AM, Jamo Luhrsen <jluhrsen@...> wrote:

 

ok, patch is merged.

Thanks,
JamO

On 6/13/20 8:26 PM, Srinivas Rachakonda wrote:

Hi Jamo,

 

I have reviewed the scripts and it looks fine.

Marked +1 in the review.

 

R.Srinivas

+91-9243478719

 

 

 

On Sat, 13 Jun 2020 at 01:49, Jamo Luhrsen <jluhrsen@...> wrote:

Srinivas,

Can you please check this patch to see if it's in line with what you were expecting?

Thanks,
JamO

#netvirt

On 6/11/20 1:13 AM, Gerrit Code Review wrote:

Patch set 2:Code-Review +1

View Change

To view, visit change 90385. To unsubscribe, or for help writing mail filters, visit settings.

Gerrit-Project: releng/builder

Gerrit-Branch: master

Gerrit-Change-Id: Ice7702c633e8f49b2b02f0828d130bb48f762e81

Gerrit-Change-Number: 90385

Gerrit-PatchSet: 2

Gerrit-Owner: Anil Belur <abelur@...>

Gerrit-Reviewer: Bengt Thuree <bthuree@...>

Gerrit-Reviewer: Jamo Luhrsen <jluhrsen@...>

Gerrit-Reviewer: Luis Gomez <ecelgp@...>

Gerrit-Reviewer: Srinivas Rachakonda <srinivas.rachakonda@...>

Gerrit-Reviewer: Thanh Ha (zxiiro) <zxiiro@...>

Gerrit-Reviewer: jenkins-releng <jenkins-releng@...>

Gerrit-Comment-Date: Thu, 11 Jun 2020 08:13:11 +0000

Gerrit-HasComments: No

Gerrit-Has-Labels: Yes

Gerrit-MessageType: comment

 


Re: builder[master]: Reduce Netvirt jobs costs

JamO Luhrsen
 

Yeah, this needs to be dialed in a little tighter. There is a lot that could be
confusing. Some history:

1) there is a list of jobs that we skip in the daily AR run, like any tempest
    job, so in those got marked in the netvirt yaml with daily

2) we used to have a special high-frequency job that covered some netvirt
    jobs and ran multiple times a day. that was removed though:
    https://git.opendaylight.org/gerrit/c/releng/builder/+/81858


Srinivas, can you please dig through the definitions and remove the daily
trigger from anything that is not in the list of jobs to skip. like here:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-test-jobs.yaml#L65

If a job is in the list of jobs to skip (it's just matching on job name regex),
then it will still need a trigger like daily or weekly.


Good catch, Luis!


JamO



On 6/16/20 12:58 PM, Luis Gomez wrote:

Guys, I just realized in this patch there is a new daily trigger:

schedule: '{schedule-daily}'

This has never been required before because the AR already runs daily and executes netvirt tests.

So now I see the same test job runs twice a day (because AR + this new trigger):


Is this intended or not?

BR/Luis


On Jun 15, 2020, at 10:49 AM, Jamo Luhrsen <jluhrsen@...> wrote:

ok, patch is merged.

Thanks,
JamO

On 6/13/20 8:26 PM, Srinivas Rachakonda wrote:
Hi Jamo,

I have reviewed the scripts and it looks fine.
Marked +1 in the review.

R.Srinivas
+91-9243478719



On Sat, 13 Jun 2020 at 01:49, Jamo Luhrsen <jluhrsen@...> wrote:

Srinivas,

Can you please check this patch to see if it's in line with what you were expecting?

Thanks,
JamO

#netvirt

On 6/11/20 1:13 AM, Gerrit Code Review wrote:

Patch set 2:Code-Review +1

View Change

To view, visit change 90385. To unsubscribe, or for help writing mail filters, visit settings.

Gerrit-Project: releng/builder
Gerrit-Branch: master
Gerrit-Change-Id: Ice7702c633e8f49b2b02f0828d130bb48f762e81
Gerrit-Change-Number: 90385
Gerrit-PatchSet: 2
Gerrit-Owner: Anil Belur <abelur@...>
Gerrit-Reviewer: Bengt Thuree <bthuree@...>
Gerrit-Reviewer: Jamo Luhrsen <jluhrsen@...>
Gerrit-Reviewer: Luis Gomez <ecelgp@...>
Gerrit-Reviewer: Srinivas Rachakonda <srinivas.rachakonda@...>
Gerrit-Reviewer: Thanh Ha (zxiiro) <zxiiro@...>
Gerrit-Reviewer: jenkins-releng <jenkins-releng@...>
Gerrit-Comment-Date: Thu, 11 Jun 2020 08:13:11 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment


no integration meeting tomorrow (06/16/2020)

JamO Luhrsen
 

I cannot make our normal 9pm meeting time, so let's cancel this week.

Thanks,
JamO


Re: Netvirt jobs are failing with stack error: uwsgi directory no found

JamO Luhrsen
 

Thanks. I gave a review just now.

JamO

On 6/16/20 5:15 AM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

Raised the below review:

https://git.opendaylight.org/gerrit/c/releng/builder/+/90459

 

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 16 June 2020 00:04
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'Thanh Ha' <zxiiro@...>; 'Dayavanti Gopal Kamath' <dayavanti.gopal.kamath@...>; 'P S Ravikanth' <ravikanth.ps@...>; 'odl netvirt dev' <netvirt-dev@...>; 'Karthikeyan Krishnan' <karthikeyan.k@...>; integration-dev@...
Subject: Re: Netvirt jobs are failing with stack error: uwsgi directory no found

 

The last passing job is installing uWSGI v2.0.18 so maybe v2.0.19 is broken? Maybe you can
try pinning it back to v2.0.18 to see if it helps?

 

Thanks,
JamO

 

On 6/15/20 8:31 AM, srinivas.rachakonda@... wrote:

Hi Jamo/Anil,

 

Netvirt CSIT jobs are failing with the below error since morning.

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upgrade-snat-conntrack-magnesium/274/control_1/stack.log.gz

 

 

2020-06-15 10:54:16.154 |   Downloading https://files.pythonhosted.org/packages/6f/e9/729f0eb15f4847c729c1a841bcbb5e327d08d208de0cbe661224955c9297/uWSGI-2.0.19.tar.gz (804kB)

2020-06-15 10:54:17.017 |   Saved ./uWSGI-2.0.19.tar.gz

2020-06-15 10:54:19.063 | Successfully downloaded uwsgi

2020-06-15 10:54:19.808 | You are using pip version 9.0.3, however version 20.1.1 is available.

2020-06-15 10:54:19.808 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-06-15 10:54:19.853 | + lib/apache:install_apache_uwsgi:97       :   local uwsgi

2020-06-15 10:54:19.858 | ++ lib/apache:install_apache_uwsgi:98       :   ls 'uwsgi*'

2020-06-15 10:54:19.861 | ls: cannot access uwsgi*: No such file or directory

2020-06-15 10:54:19.865 | + lib/apache:install_apache_uwsgi:98       :   uwsgi=

2020-06-15 10:54:19.869 | + lib/apache:install_apache_uwsgi:1        :   exit_trap

2020-06-15 10:54:19.873 | + ./stack.sh:exit_trap:516                 :   local r=2

2020-06-15 10:54:19.877 | ++ ./stack.sh:exit_trap:517                 :   jobs -p

2020-06-15 10:54:19.879 | + ./stack.sh:exit_trap:517                 :   jobs=

2020-06-15 10:54:19.881 | + ./stack.sh:exit_trap:520                 :   [[ -n '' ]]

2020-06-15 10:54:19.883 | + ./stack.sh:exit_trap:526                 :   '[' -f '' ']'

2020-06-15 10:54:19.885 | + ./stack.sh:exit_trap:531                 :   kill_spinner

2020-06-15 10:54:19.887 | + ./stack.sh:kill_spinner:430              :   '[' '!' -z '' ']'

2020-06-15 10:54:19.889 | + ./stack.sh:exit_trap:533                 :   [[ 2 -ne 0 ]]

2020-06-15 10:54:19.891 | + ./stack.sh:exit_trap:534                 :   echo 'Error on exit'

2020-06-15 10:54:19.891 | Error on exit

2020-06-15 10:54:19.894 | + ./stack.sh:exit_trap:536                 :   type -p generate-subunit

2020-06-15 10:54:19.896 | + ./stack.sh:exit_trap:537                 :   generate-subunit 1592218203 256 fail

2020-06-15 10:54:21.128 | + ./stack.sh:exit_trap:539                 :   [[ -z /opt/stack/logs ]]

2020-06-15 10:54:21.132 | + ./stack.sh:exit_trap:542                 :   /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs

2020-06-15 10:54:21.513 | ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

2020-06-15 10:54:21.616 | ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

2020-06-15 10:54:22.462 | + ./stack.sh:exit_trap:551                 :   exit 2

 

Could you please look into the issue.

 

 

Thanks,

Srinivas

+91-9243478719

 


Re: Netvirt jobs are failing with stack error: uwsgi directory no found

Srinivas <srinivas.rachakonda@...>
 

Hi Jamo,

 

Raised the below review:

https://git.opendaylight.org/gerrit/c/releng/builder/+/90459

 

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 16 June 2020 00:04
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'Thanh Ha' <zxiiro@...>; 'Dayavanti Gopal Kamath' <dayavanti.gopal.kamath@...>; 'P S Ravikanth' <ravikanth.ps@...>; 'odl netvirt dev' <netvirt-dev@...>; 'Karthikeyan Krishnan' <karthikeyan.k@...>; integration-dev@...
Subject: Re: Netvirt jobs are failing with stack error: uwsgi directory no found

 

The last passing job is installing uWSGI v2.0.18 so maybe v2.0.19 is broken? Maybe you can
try pinning it back to v2.0.18 to see if it helps?

 

Thanks,
JamO

 

On 6/15/20 8:31 AM, srinivas.rachakonda@... wrote:

Hi Jamo/Anil,

 

Netvirt CSIT jobs are failing with the below error since morning.

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upgrade-snat-conntrack-magnesium/274/control_1/stack.log.gz

 

 

2020-06-15 10:54:16.154 |   Downloading https://files.pythonhosted.org/packages/6f/e9/729f0eb15f4847c729c1a841bcbb5e327d08d208de0cbe661224955c9297/uWSGI-2.0.19.tar.gz (804kB)

2020-06-15 10:54:17.017 |   Saved ./uWSGI-2.0.19.tar.gz

2020-06-15 10:54:19.063 | Successfully downloaded uwsgi

2020-06-15 10:54:19.808 | You are using pip version 9.0.3, however version 20.1.1 is available.

2020-06-15 10:54:19.808 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-06-15 10:54:19.853 | + lib/apache:install_apache_uwsgi:97       :   local uwsgi

2020-06-15 10:54:19.858 | ++ lib/apache:install_apache_uwsgi:98       :   ls 'uwsgi*'

2020-06-15 10:54:19.861 | ls: cannot access uwsgi*: No such file or directory

2020-06-15 10:54:19.865 | + lib/apache:install_apache_uwsgi:98       :   uwsgi=

2020-06-15 10:54:19.869 | + lib/apache:install_apache_uwsgi:1        :   exit_trap

2020-06-15 10:54:19.873 | + ./stack.sh:exit_trap:516                 :   local r=2

2020-06-15 10:54:19.877 | ++ ./stack.sh:exit_trap:517                 :   jobs -p

2020-06-15 10:54:19.879 | + ./stack.sh:exit_trap:517                 :   jobs=

2020-06-15 10:54:19.881 | + ./stack.sh:exit_trap:520                 :   [[ -n '' ]]

2020-06-15 10:54:19.883 | + ./stack.sh:exit_trap:526                 :   '[' -f '' ']'

2020-06-15 10:54:19.885 | + ./stack.sh:exit_trap:531                 :   kill_spinner

2020-06-15 10:54:19.887 | + ./stack.sh:kill_spinner:430              :   '[' '!' -z '' ']'

2020-06-15 10:54:19.889 | + ./stack.sh:exit_trap:533                 :   [[ 2 -ne 0 ]]

2020-06-15 10:54:19.891 | + ./stack.sh:exit_trap:534                 :   echo 'Error on exit'

2020-06-15 10:54:19.891 | Error on exit

2020-06-15 10:54:19.894 | + ./stack.sh:exit_trap:536                 :   type -p generate-subunit

2020-06-15 10:54:19.896 | + ./stack.sh:exit_trap:537                 :   generate-subunit 1592218203 256 fail

2020-06-15 10:54:21.128 | + ./stack.sh:exit_trap:539                 :   [[ -z /opt/stack/logs ]]

2020-06-15 10:54:21.132 | + ./stack.sh:exit_trap:542                 :   /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs

2020-06-15 10:54:21.513 | ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

2020-06-15 10:54:21.616 | ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

2020-06-15 10:54:22.462 | + ./stack.sh:exit_trap:551                 :   exit 2

 

Could you please look into the issue.

 

 

Thanks,

Srinivas

+91-9243478719

 


Re: Netvirt jobs are failing with stack error: uwsgi directory no found

JamO Luhrsen
 

The last passing job is installing uWSGI v2.0.18 so maybe v2.0.19 is broken? Maybe you can
try pinning it back to v2.0.18 to see if it helps?


Thanks,
JamO


On 6/15/20 8:31 AM, srinivas.rachakonda@... wrote:

Hi Jamo/Anil,

 

Netvirt CSIT jobs are failing with the below error since morning.

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upgrade-snat-conntrack-magnesium/274/control_1/stack.log.gz

 

 

2020-06-15 10:54:16.154 |   Downloading https://files.pythonhosted.org/packages/6f/e9/729f0eb15f4847c729c1a841bcbb5e327d08d208de0cbe661224955c9297/uWSGI-2.0.19.tar.gz (804kB)

2020-06-15 10:54:17.017 |   Saved ./uWSGI-2.0.19.tar.gz

2020-06-15 10:54:19.063 | Successfully downloaded uwsgi

2020-06-15 10:54:19.808 | You are using pip version 9.0.3, however version 20.1.1 is available.

2020-06-15 10:54:19.808 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-06-15 10:54:19.853 | + lib/apache:install_apache_uwsgi:97       :   local uwsgi

2020-06-15 10:54:19.858 | ++ lib/apache:install_apache_uwsgi:98       :   ls 'uwsgi*'

2020-06-15 10:54:19.861 | ls: cannot access uwsgi*: No such file or directory

2020-06-15 10:54:19.865 | + lib/apache:install_apache_uwsgi:98       :   uwsgi=

2020-06-15 10:54:19.869 | + lib/apache:install_apache_uwsgi:1        :   exit_trap

2020-06-15 10:54:19.873 | + ./stack.sh:exit_trap:516                 :   local r=2

2020-06-15 10:54:19.877 | ++ ./stack.sh:exit_trap:517                 :   jobs -p

2020-06-15 10:54:19.879 | + ./stack.sh:exit_trap:517                 :   jobs=

2020-06-15 10:54:19.881 | + ./stack.sh:exit_trap:520                 :   [[ -n '' ]]

2020-06-15 10:54:19.883 | + ./stack.sh:exit_trap:526                 :   '[' -f '' ']'

2020-06-15 10:54:19.885 | + ./stack.sh:exit_trap:531                 :   kill_spinner

2020-06-15 10:54:19.887 | + ./stack.sh:kill_spinner:430              :   '[' '!' -z '' ']'

2020-06-15 10:54:19.889 | + ./stack.sh:exit_trap:533                 :   [[ 2 -ne 0 ]]

2020-06-15 10:54:19.891 | + ./stack.sh:exit_trap:534                 :   echo 'Error on exit'

2020-06-15 10:54:19.891 | Error on exit

2020-06-15 10:54:19.894 | + ./stack.sh:exit_trap:536                 :   type -p generate-subunit

2020-06-15 10:54:19.896 | + ./stack.sh:exit_trap:537                 :   generate-subunit 1592218203 256 fail

2020-06-15 10:54:21.128 | + ./stack.sh:exit_trap:539                 :   [[ -z /opt/stack/logs ]]

2020-06-15 10:54:21.132 | + ./stack.sh:exit_trap:542                 :   /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs

2020-06-15 10:54:21.513 | ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

2020-06-15 10:54:21.616 | ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

2020-06-15 10:54:22.462 | + ./stack.sh:exit_trap:551                 :   exit 2

 

Could you please look into the issue.

 

 

Thanks,

Srinivas

+91-9243478719

 


Re: [release] controller CSIT for Aluminium

Thanh ha <zxiiro@...>
 

On Fri, Jun 12, 2020 at 2:02 PM Robert Varga <nite@...> wrote:
On 12/06/2020 19:52, Robert Varga wrote:
> On 09/06/2020 02:49, JamO Luhrsen wrote:
>> Hi all,
>>
>> I've had an action item for a while to dig through Aluminium controller
>> CSIT to
>> try to get a handle on where we stand after the MRI bumps. This has turned
>> up some jira tickets:
>
> [snip]
>
>> Trouble on the test side:
>>
>> https://jira.opendaylight.org/browse/INTTEST-100
>
> This is across all releases, I just updated the ticket and raised it to
> blocker, as it does block any attempt to release anything. I have
> attached known-good and known-bad console.log and the 'diff -U 0'
> between them.
>
> It seems to be pointing at
> https://git.opendaylight.org/gerrit/c/releng/builder/+/90196/2/jjb/integration/integration-templates.yaml
>
> which would seem to indicate trouble with python setup in the new image.
>
> Thanh, any idea on how to debug this further?

This patch https://git.opendaylight.org/gerrit/c/releng/builder/+/90433 will at least make mininet available to be used on the sandbox directly. Maybe we can run a few commands and try to reproduce on the sandbox with a stripped down job that just tries to run the script. I'll work with JamO to see if we can find anything useful to debug.

Regards,
Thanh


Re: [release] controller CSIT for Aluminium

Robert Varga
 

On 12/06/2020 19:52, Robert Varga wrote:
On 09/06/2020 02:49, JamO Luhrsen wrote:
Hi all,

I've had an action item for a while to dig through Aluminium controller
CSIT to
try to get a handle on where we stand after the MRI bumps. This has turned
up some jira tickets:
[snip]

Trouble on the test side:

https://jira.opendaylight.org/browse/INTTEST-100
This is across all releases, I just updated the ticket and raised it to
blocker, as it does block any attempt to release anything. I have
attached known-good and known-bad console.log and the 'diff -U 0'
between them.

It seems to be pointing at
https://git.opendaylight.org/gerrit/c/releng/builder/+/90196/2/jjb/integration/integration-templates.yaml

which would seem to indicate trouble with python setup in the new image.

Thanh, any idea on how to debug this further?
+correct email


Thanks,
Robert


Re: [release] controller CSIT for Aluminium

Robert Varga
 

On 09/06/2020 02:49, JamO Luhrsen wrote:
Hi all,

I've had an action item for a while to dig through Aluminium controller
CSIT to
try to get a handle on where we stand after the MRI bumps. This has turned
up some jira tickets:
[snip]

Trouble on the test side:

https://jira.opendaylight.org/browse/INTTEST-100
This is across all releases, I just updated the ticket and raised it to
blocker, as it does block any attempt to release anything. I have
attached known-good and known-bad console.log and the 'diff -U 0'
between them.

It seems to be pointing at
https://git.opendaylight.org/gerrit/c/releng/builder/+/90196/2/jjb/integration/integration-templates.yaml

which would seem to indicate trouble with python setup in the new image.

Thanh, any idea on how to debug this further?

Thanks,
Robert


Re: validate-autorelease-aluminium failing

Luis Gomez
 

The check conditions have changed after we introduced MRI projects, so now it is possible to have projects in distribution but not in AR. Because of this, we can either rework the script to check something different (not sure what) or just remove as I did.

BR/Luis

On Jun 12, 2020, at 10:06 AM, Thanh Ha <zxiiro@...> wrote:

This just turns off the comparison entirely so not really fixing but I think it's the right approach until someone can fix the script. Did a quick look at the script and it looks like it parses pom.xml files dynamically to figure out what to compare which means we definitely need to adjust the script to allow exceptions for MRI projects.

Regards,
Thanh

On Fri, Jun 12, 2020 at 12:22 PM Luis Gomez <ecelgp@...> wrote:
This should fix: https://git.opendaylight.org/gerrit/c/releng/builder/+/90432

On Jun 12, 2020, at 9:13 AM, Robert Varga <nite@...> wrote:

Well, we fixed feature packaging in distribution (i.e. we now include
features-controller-testing and others).

Also, validate-autorelease triggers only when pom.xmls are modified I
think, hence it is not typically seen.

Regards,
Robert


On 12/06/2020 18:11, Luis Gomez wrote:
I think you are right, but why this starts to fail now?


On Jun 11, 2020, at 11:41 PM, Robert Varga <nite@...> wrote:

Hello,

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/infrautils-validate-autorelease-aluminium/2/console.log.gz
failed with:

[infrautils-validate-autorelease-aluminium] $ /bin/bash -l /tmp/jenkins5280574077411919720.sh
---> compare-autorelease-projects.sh
ERROR: List of projects in integration/distribution but NOT in releng/autorelease: mdsal yangtools controller
Build step 'Execute shell' marked build as failure

I think this is wrong, since MRI projects need to be in the distribution
and not be present in autorelease.

Perhaps compare-autorelease-projects.sh needs to be updated with list of
MRI projects?

Regards,
Robert







Re: validate-autorelease-aluminium failing

Thanh ha <zxiiro@...>
 

This just turns off the comparison entirely so not really fixing but I think it's the right approach until someone can fix the script. Did a quick look at the script and it looks like it parses pom.xml files dynamically to figure out what to compare which means we definitely need to adjust the script to allow exceptions for MRI projects.

Regards,
Thanh


On Fri, Jun 12, 2020 at 12:22 PM Luis Gomez <ecelgp@...> wrote:
This should fix: https://git.opendaylight.org/gerrit/c/releng/builder/+/90432

On Jun 12, 2020, at 9:13 AM, Robert Varga <nite@...> wrote:

Well, we fixed feature packaging in distribution (i.e. we now include
features-controller-testing and others).

Also, validate-autorelease triggers only when pom.xmls are modified I
think, hence it is not typically seen.

Regards,
Robert


On 12/06/2020 18:11, Luis Gomez wrote:
I think you are right, but why this starts to fail now?


On Jun 11, 2020, at 11:41 PM, Robert Varga <nite@...> wrote:

Hello,

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/infrautils-validate-autorelease-aluminium/2/console.log.gz
failed with:

[infrautils-validate-autorelease-aluminium] $ /bin/bash -l /tmp/jenkins5280574077411919720.sh
---> compare-autorelease-projects.sh
ERROR: List of projects in integration/distribution but NOT in releng/autorelease: mdsal yangtools controller
Build step 'Execute shell' marked build as failure

I think this is wrong, since MRI projects need to be in the distribution
and not be present in autorelease.

Perhaps compare-autorelease-projects.sh needs to be updated with list of
MRI projects?

Regards,
Robert







Re: validate-autorelease-aluminium failing

Luis Gomez
 

On Jun 12, 2020, at 9:13 AM, Robert Varga <nite@...> wrote:

Well, we fixed feature packaging in distribution (i.e. we now include
features-controller-testing and others).

Also, validate-autorelease triggers only when pom.xmls are modified I
think, hence it is not typically seen.

Regards,
Robert


On 12/06/2020 18:11, Luis Gomez wrote:
I think you are right, but why this starts to fail now?


On Jun 11, 2020, at 11:41 PM, Robert Varga <nite@...> wrote:

Hello,

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/infrautils-validate-autorelease-aluminium/2/console.log.gz
failed with:

[infrautils-validate-autorelease-aluminium] $ /bin/bash -l /tmp/jenkins5280574077411919720.sh
---> compare-autorelease-projects.sh
ERROR: List of projects in integration/distribution but NOT in releng/autorelease: mdsal yangtools controller
Build step 'Execute shell' marked build as failure

I think this is wrong, since MRI projects need to be in the distribution
and not be present in autorelease.

Perhaps compare-autorelease-projects.sh needs to be updated with list of
MRI projects?

Regards,
Robert






Re: validate-autorelease-aluminium failing

Robert Varga
 

Well, we fixed feature packaging in distribution (i.e. we now include
features-controller-testing and others).

Also, validate-autorelease triggers only when pom.xmls are modified I
think, hence it is not typically seen.

Regards,
Robert

On 12/06/2020 18:11, Luis Gomez wrote:
I think you are right, but why this starts to fail now?


On Jun 11, 2020, at 11:41 PM, Robert Varga <nite@...> wrote:

Hello,

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/infrautils-validate-autorelease-aluminium/2/console.log.gz
failed with:

[infrautils-validate-autorelease-aluminium] $ /bin/bash -l /tmp/jenkins5280574077411919720.sh
---> compare-autorelease-projects.sh
ERROR: List of projects in integration/distribution but NOT in releng/autorelease: mdsal yangtools controller
Build step 'Execute shell' marked build as failure
I think this is wrong, since MRI projects need to be in the distribution
and not be present in autorelease.

Perhaps compare-autorelease-projects.sh needs to be updated with list of
MRI projects?

Regards,
Robert