1. Integration-Dev
  2. Topics

Latest CSIT job is failing

Srinivas <srinivas.rachakonda@...>
 

Hi Anil,

 

The latest CSIT jobs are failing with the below error:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/

 

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/160/control_1/stack.log.gz

 

2020-02-04 14:14:45.501 | Collecting oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 |   Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.7.0, 2.8.0, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, 3.13.0, 3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.17.0, 3.18.0, 3.18.1, 3.19.0, 3.20.0, 3.21.0, 3.21.1, 3.21.2, 3.22.0, 3.23.0, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.27.0, 3.28.0, 3.28.1, 3.29.0, 3.29.1, 3.30.0, 3.31.0)

2020-02-04 14:14:45.501 | No matching distribution found for oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 | You are using pip version 9.0.1, however version 20.0.2 is available.

2020-02-04 14:14:45.501 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-02-04 14:14:45.501 |

2020-02-04 14:14:45.501 | =================================== log end ====================================

2020-02-04 14:14:45.501 | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.501 | ___________________________________ summary ____________________________________

2020-02-04 14:14:45.501 | ERROR:   full: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.520 | + lib/tempest:install_tempest:1            :   exit_trap

 

 

Thanks,

Srinivas

+91-9243478719

 

smime.p7s

smime.p7s

JamO Luhrsen
 

Hi Srinivas,

This is a netvirt project problem. I have some experience there so can
try to give some help.

oslo.concurrency was updated in openstack/requirements/upper-constraints.txt
recently (yesterday):

https://github.com/openstack/requirements/commit/0c5ace0ec8ac670b6cbeb64fb8577d11adc90bc5#diff-0bdd949ed8a7fdd4f95240bd951779c8

As you pointed out, devstack failed to find v4.0.0 so stacking failed.

locally, when I do a pip search on oslo.concurrencty, I see v4.0.0 available so not sure
why it's not found in the job. I do see that it's there on pypi:

https://pypi.org/project/oslo.concurrency/#history

Until you can figure out why it can't be found, you could try to pin it to the older
version that was working before yesterday. We do that for other things I think. Here's
one example:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-openstack-run-test.sh#L159

Hope it helps,
JamO


On 2/4/20 7:20 AM, srinivas.rachakonda@... wrote:

Hi Anil,

 

The latest CSIT jobs are failing with the below error:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/

 

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/160/control_1/stack.log.gz

 

2020-02-04 14:14:45.501 | Collecting oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 |   Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.7.0, 2.8.0, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, 3.13.0, 3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.17.0, 3.18.0, 3.18.1, 3.19.0, 3.20.0, 3.21.0, 3.21.1, 3.21.2, 3.22.0, 3.23.0, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.27.0, 3.28.0, 3.28.1, 3.29.0, 3.29.1, 3.30.0, 3.31.0)

2020-02-04 14:14:45.501 | No matching distribution found for oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 | You are using pip version 9.0.1, however version 20.0.2 is available.

2020-02-04 14:14:45.501 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-02-04 14:14:45.501 |

2020-02-04 14:14:45.501 | =================================== log end ====================================

2020-02-04 14:14:45.501 | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.501 | ___________________________________ summary ____________________________________

2020-02-04 14:14:45.501 | ERROR:   full: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.520 | + lib/tempest:install_tempest:1            :   exit_trap

 

 

Thanks,

Srinivas

+91-9243478719

 
Srinivas <srinivas.rachakonda@...>
 

Hi Jamo,

 

I have added the below line and then scheduled the job.

 

/jjb/integration/integration-deploy-openstack-run-test.sh

@@ -156,6 +156,7 @@ function fix_libvirt_python_build() {

         git clone https://git.openstack.org/openstack/requirements;

         cd requirements;

         git checkout ${ODL_ML2_BRANCH};

        sed -i s/oslo.concurrency===4.0.0/oslo.concurrency===3.25.1/ upper-constraints.txt;

         sed -i s/libvirt-python===3.10.0/libvirt-python===4.10.0/ upper-constraints.txt

         "

}

 

But still the job picked oslo.concurrency==4.0.0

On further debugging it occurs with tempest clone /opt/stack/tempest/requirements.txt

 

Please let me know where should I be looking for tempest changes.

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 00:31
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...
Subject: Re: Latest CSIT job is failing

 

Hi Srinivas,

This is a netvirt project problem. I have some experience there so can
try to give some help.

oslo.concurrency was updated in openstack/requirements/upper-constraints.txt
recently (yesterday):

https://github.com/openstack/requirements/commit/0c5ace0ec8ac670b6cbeb64fb8577d11adc90bc5#diff-0bdd949ed8a7fdd4f95240bd951779c8

As you pointed out, devstack failed to find v4.0.0 so stacking failed.

locally, when I do a pip search on oslo.concurrencty, I see v4.0.0 available so not sure
why it's not found in the job. I do see that it's there on pypi:

https://pypi.org/project/oslo.concurrency/#history

Until you can figure out why it can't be found, you could try to pin it to the older
version that was working before yesterday. We do that for other things I think. Here's
one example:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-openstack-run-test.sh#L159

Hope it helps,
JamO

On 2/4/20 7:20 AM, srinivas.rachakonda@... wrote:

Hi Anil,

 

The latest CSIT jobs are failing with the below error:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/

 

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/160/control_1/stack.log.gz

 

2020-02-04 14:14:45.501 | Collecting oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 |   Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.7.0, 2.8.0, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, 3.13.0, 3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.17.0, 3.18.0, 3.18.1, 3.19.0, 3.20.0, 3.21.0, 3.21.1, 3.21.2, 3.22.0, 3.23.0, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.27.0, 3.28.0, 3.28.1, 3.29.0, 3.29.1, 3.30.0, 3.31.0)

2020-02-04 14:14:45.501 | No matching distribution found for oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 | You are using pip version 9.0.1, however version 20.0.2 is available.

2020-02-04 14:14:45.501 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-02-04 14:14:45.501 |

2020-02-04 14:14:45.501 | =================================== log end ====================================

2020-02-04 14:14:45.501 | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.501 | ___________________________________ summary ____________________________________

2020-02-04 14:14:45.501 | ERROR:   full: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.520 | + lib/tempest:install_tempest:1            :   exit_trap

 

 

Thanks,

Srinivas

+91-9243478719

 

 

smime.p7s

smime.p7s

JamO Luhrsen
 

Maybe you are hitting this bug?
  https://bugs.launchpad.net/tripleo/+bug/1861803

But that was found and fixed in tripleo, not tempest which you
found is your problem.

somethings to try:

1) your job does not need tempest, iirc, so you can remove that
   plugin from being install in the jjb .yaml

2) take a look at moving things to python3? That might be a little
   bit of an undertaking but python2 is dead/dying and this probably
   wont be the last problem related to that.

3) push a similar fix to the tempest project, although getting
   that merged may be hard.


Thanks,
JamO
On 2/5/20 8:44 AM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

I have added the below line and then scheduled the job.

 

/jjb/integration/integration-deploy-openstack-run-test.sh

@@ -156,6 +156,7 @@ function fix_libvirt_python_build() {

         git clone https://git.openstack.org/openstack/requirements;

         cd requirements;

         git checkout ${ODL_ML2_BRANCH};

        sed -i s/oslo.concurrency===4.0.0/oslo.concurrency===3.25.1/ upper-constraints.txt;

         sed -i s/libvirt-python===3.10.0/libvirt-python===4.10.0/ upper-constraints.txt

         "

}

 

But still the job picked oslo.concurrency==4.0.0

On further debugging it occurs with tempest clone /opt/stack/tempest/requirements.txt

 

Please let me know where should I be looking for tempest changes.

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 00:31
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...
Subject: Re: Latest CSIT job is failing

 

Hi Srinivas,

This is a netvirt project problem. I have some experience there so can
try to give some help.

oslo.concurrency was updated in openstack/requirements/upper-constraints.txt
recently (yesterday):

https://github.com/openstack/requirements/commit/0c5ace0ec8ac670b6cbeb64fb8577d11adc90bc5#diff-0bdd949ed8a7fdd4f95240bd951779c8

As you pointed out, devstack failed to find v4.0.0 so stacking failed.

locally, when I do a pip search on oslo.concurrencty, I see v4.0.0 available so not sure
why it's not found in the job. I do see that it's there on pypi:

https://pypi.org/project/oslo.concurrency/#history

Until you can figure out why it can't be found, you could try to pin it to the older
version that was working before yesterday. We do that for other things I think. Here's
one example:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-openstack-run-test.sh#L159

Hope it helps,
JamO

On 2/4/20 7:20 AM, srinivas.rachakonda@... wrote:

Hi Anil,

 

The latest CSIT jobs are failing with the below error:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/

 

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/160/control_1/stack.log.gz

 

2020-02-04 14:14:45.501 | Collecting oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 |   Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.7.0, 2.8.0, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, 3.13.0, 3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.17.0, 3.18.0, 3.18.1, 3.19.0, 3.20.0, 3.21.0, 3.21.1, 3.21.2, 3.22.0, 3.23.0, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.27.0, 3.28.0, 3.28.1, 3.29.0, 3.29.1, 3.30.0, 3.31.0)

2020-02-04 14:14:45.501 | No matching distribution found for oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 | You are using pip version 9.0.1, however version 20.0.2 is available.

2020-02-04 14:14:45.501 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-02-04 14:14:45.501 |

2020-02-04 14:14:45.501 | =================================== log end ====================================

2020-02-04 14:14:45.501 | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.501 | ___________________________________ summary ____________________________________

2020-02-04 14:14:45.501 | ERROR:   full: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.520 | + lib/tempest:install_tempest:1            :   exit_trap

 

 

Thanks,

Srinivas

+91-9243478719

 

 
Srinivas <srinivas.rachakonda@...>
 

Hi Jamo,

 

I have removed tempest and raised the below review:

https://git.opendaylight.org/gerrit/c/releng/builder/+/87462

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 22:46
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...; 'Dayavanti Gopal Kamath' <dayavanti.gopal.kamath@...>
Subject: Re: Latest CSIT job is failing

 

Maybe you are hitting this bug?
  https://bugs.launchpad.net/tripleo/+bug/1861803

But that was found and fixed in tripleo, not tempest which you
found is your problem.

somethings to try:

1) your job does not need tempest, iirc, so you can remove that
   plugin from being install in the jjb .yaml

2) take a look at moving things to python3? That might be a little
   bit of an undertaking but python2 is dead/dying and this probably
   wont be the last problem related to that.

3) push a similar fix to the tempest project, although getting
   that merged may be hard.


Thanks,
JamO

On 2/5/20 8:44 AM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

I have added the below line and then scheduled the job.

 

/jjb/integration/integration-deploy-openstack-run-test.sh

@@ -156,6 +156,7 @@ function fix_libvirt_python_build() {

         git clone https://git.openstack.org/openstack/requirements;

         cd requirements;

         git checkout ${ODL_ML2_BRANCH};

        sed -i s/oslo.concurrency===4.0.0/oslo.concurrency===3.25.1/ upper-constraints.txt;

         sed -i s/libvirt-python===3.10.0/libvirt-python===4.10.0/ upper-constraints.txt

         "

}

 

But still the job picked oslo.concurrency==4.0.0

On further debugging it occurs with tempest clone /opt/stack/tempest/requirements.txt

 

Please let me know where should I be looking for tempest changes.

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 00:31
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...
Subject: Re: Latest CSIT job is failing

 

Hi Srinivas,

This is a netvirt project problem. I have some experience there so can
try to give some help.

oslo.concurrency was updated in openstack/requirements/upper-constraints.txt
recently (yesterday):

https://github.com/openstack/requirements/commit/0c5ace0ec8ac670b6cbeb64fb8577d11adc90bc5#diff-0bdd949ed8a7fdd4f95240bd951779c8

As you pointed out, devstack failed to find v4.0.0 so stacking failed.

locally, when I do a pip search on oslo.concurrencty, I see v4.0.0 available so not sure
why it's not found in the job. I do see that it's there on pypi:

https://pypi.org/project/oslo.concurrency/#history

Until you can figure out why it can't be found, you could try to pin it to the older
version that was working before yesterday. We do that for other things I think. Here's
one example:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-openstack-run-test.sh#L159

Hope it helps,
JamO


On 2/4/20 7:20 AM, srinivas.rachakonda@... wrote:

Hi Anil,

 

The latest CSIT jobs are failing with the below error:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/

 

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/160/control_1/stack.log.gz

 

2020-02-04 14:14:45.501 | Collecting oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 |   Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.7.0, 2.8.0, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, 3.13.0, 3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.17.0, 3.18.0, 3.18.1, 3.19.0, 3.20.0, 3.21.0, 3.21.1, 3.21.2, 3.22.0, 3.23.0, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.27.0, 3.28.0, 3.28.1, 3.29.0, 3.29.1, 3.30.0, 3.31.0)

2020-02-04 14:14:45.501 | No matching distribution found for oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 | You are using pip version 9.0.1, however version 20.0.2 is available.

2020-02-04 14:14:45.501 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-02-04 14:14:45.501 |

2020-02-04 14:14:45.501 | =================================== log end ====================================

2020-02-04 14:14:45.501 | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.501 | ___________________________________ summary ____________________________________

2020-02-04 14:14:45.501 | ERROR:   full: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.520 | + lib/tempest:install_tempest:1            :   exit_trap

 

 

Thanks,

Srinivas

+91-9243478719

 

 

 

smime.p7s

smime.p7s

JamO Luhrsen
 

ok. let me know how it goes.

one thing I thought of though, is that the "tempest" CSIT
jobs obviously cannot have tempest removed, so you'll need
to figure that out eventually I guess.

JamO

On 2/5/20 9:07 PM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

I have removed tempest and raised the below review:

https://git.opendaylight.org/gerrit/c/releng/builder/+/87462

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 22:46
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...; 'Dayavanti Gopal Kamath' <dayavanti.gopal.kamath@...>
Subject: Re: Latest CSIT job is failing

 

Maybe you are hitting this bug?
  https://bugs.launchpad.net/tripleo/+bug/1861803

But that was found and fixed in tripleo, not tempest which you
found is your problem.

somethings to try:

1) your job does not need tempest, iirc, so you can remove that
   plugin from being install in the jjb .yaml

2) take a look at moving things to python3? That might be a little
   bit of an undertaking but python2 is dead/dying and this probably
   wont be the last problem related to that.

3) push a similar fix to the tempest project, although getting
   that merged may be hard.


Thanks,
JamO

On 2/5/20 8:44 AM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

I have added the below line and then scheduled the job.

 

/jjb/integration/integration-deploy-openstack-run-test.sh

@@ -156,6 +156,7 @@ function fix_libvirt_python_build() {

         git clone https://git.openstack.org/openstack/requirements;

         cd requirements;

         git checkout ${ODL_ML2_BRANCH};

        sed -i s/oslo.concurrency===4.0.0/oslo.concurrency===3.25.1/ upper-constraints.txt;

         sed -i s/libvirt-python===3.10.0/libvirt-python===4.10.0/ upper-constraints.txt

         "

}

 

But still the job picked oslo.concurrency==4.0.0

On further debugging it occurs with tempest clone /opt/stack/tempest/requirements.txt

 

Please let me know where should I be looking for tempest changes.

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 00:31
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...
Subject: Re: Latest CSIT job is failing

 

Hi Srinivas,

This is a netvirt project problem. I have some experience there so can
try to give some help.

oslo.concurrency was updated in openstack/requirements/upper-constraints.txt
recently (yesterday):

https://github.com/openstack/requirements/commit/0c5ace0ec8ac670b6cbeb64fb8577d11adc90bc5#diff-0bdd949ed8a7fdd4f95240bd951779c8

As you pointed out, devstack failed to find v4.0.0 so stacking failed.

locally, when I do a pip search on oslo.concurrencty, I see v4.0.0 available so not sure
why it's not found in the job. I do see that it's there on pypi:

https://pypi.org/project/oslo.concurrency/#history

Until you can figure out why it can't be found, you could try to pin it to the older
version that was working before yesterday. We do that for other things I think. Here's
one example:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-openstack-run-test.sh#L159

Hope it helps,
JamO


On 2/4/20 7:20 AM, srinivas.rachakonda@... wrote:

Hi Anil,

 

The latest CSIT jobs are failing with the below error:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/

 

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/160/control_1/stack.log.gz

 

2020-02-04 14:14:45.501 | Collecting oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 |   Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.7.0, 2.8.0, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, 3.13.0, 3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.17.0, 3.18.0, 3.18.1, 3.19.0, 3.20.0, 3.21.0, 3.21.1, 3.21.2, 3.22.0, 3.23.0, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.27.0, 3.28.0, 3.28.1, 3.29.0, 3.29.1, 3.30.0, 3.31.0)

2020-02-04 14:14:45.501 | No matching distribution found for oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 | You are using pip version 9.0.1, however version 20.0.2 is available.

2020-02-04 14:14:45.501 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-02-04 14:14:45.501 |

2020-02-04 14:14:45.501 | =================================== log end ====================================

2020-02-04 14:14:45.501 | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.501 | ___________________________________ summary ____________________________________

2020-02-04 14:14:45.501 | ERROR:   full: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.520 | + lib/tempest:install_tempest:1            :   exit_trap

 

 

Thanks,

Srinivas

+91-9243478719

 

 

 
Srinivas <srinivas.rachakonda@...>
 

Hi Jamo,

 

When I remove tempest, the scripts are failing with flavour not present.

Its seen that while bringing up tempest m1.nova flavour is created.

This flavour is used by many suites.

 

So we cannot ignore tempest service installation and need to bring up oslo.cocurrency==4.0.0 into the stack.

I need help regarding this.

 

This is blocking all the CSIT jobs.

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 06 February 2020 11:02
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...; 'Dayavanti Gopal Kamath' <dayavanti.gopal.kamath@...>; karthikeyan.k@...
Subject: Re: Latest CSIT job is failing

 

ok. let me know how it goes.

one thing I thought of though, is that the "tempest" CSIT
jobs obviously cannot have tempest removed, so you'll need
to figure that out eventually I guess.

JamO

On 2/5/20 9:07 PM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

I have removed tempest and raised the below review:

https://git.opendaylight.org/gerrit/c/releng/builder/+/87462

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 22:46
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...; 'Dayavanti Gopal Kamath' <dayavanti.gopal.kamath@...>
Subject: Re: Latest CSIT job is failing

 

Maybe you are hitting this bug?
  https://bugs.launchpad.net/tripleo/+bug/1861803

But that was found and fixed in tripleo, not tempest which you
found is your problem.

somethings to try:

1) your job does not need tempest, iirc, so you can remove that
   plugin from being install in the jjb .yaml

2) take a look at moving things to python3? That might be a little
   bit of an undertaking but python2 is dead/dying and this probably
   wont be the last problem related to that.

3) push a similar fix to the tempest project, although getting
   that merged may be hard.


Thanks,
JamO

On 2/5/20 8:44 AM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

I have added the below line and then scheduled the job.

 

/jjb/integration/integration-deploy-openstack-run-test.sh

@@ -156,6 +156,7 @@ function fix_libvirt_python_build() {

         git clone https://git.openstack.org/openstack/requirements;

         cd requirements;

         git checkout ${ODL_ML2_BRANCH};

        sed -i s/oslo.concurrency===4.0.0/oslo.concurrency===3.25.1/ upper-constraints.txt;

         sed -i s/libvirt-python===3.10.0/libvirt-python===4.10.0/ upper-constraints.txt

         "

}

 

But still the job picked oslo.concurrency==4.0.0

On further debugging it occurs with tempest clone /opt/stack/tempest/requirements.txt

 

Please let me know where should I be looking for tempest changes.

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 00:31
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...
Subject: Re: Latest CSIT job is failing

 

Hi Srinivas,

This is a netvirt project problem. I have some experience there so can
try to give some help.

oslo.concurrency was updated in openstack/requirements/upper-constraints.txt
recently (yesterday):

https://github.com/openstack/requirements/commit/0c5ace0ec8ac670b6cbeb64fb8577d11adc90bc5#diff-0bdd949ed8a7fdd4f95240bd951779c8

As you pointed out, devstack failed to find v4.0.0 so stacking failed.

locally, when I do a pip search on oslo.concurrencty, I see v4.0.0 available so not sure
why it's not found in the job. I do see that it's there on pypi:

https://pypi.org/project/oslo.concurrency/#history

Until you can figure out why it can't be found, you could try to pin it to the older
version that was working before yesterday. We do that for other things I think. Here's
one example:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-openstack-run-test.sh#L159

Hope it helps,
JamO



On 2/4/20 7:20 AM, srinivas.rachakonda@... wrote:

Hi Anil,

 

The latest CSIT jobs are failing with the below error:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/

 

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/160/control_1/stack.log.gz

 

2020-02-04 14:14:45.501 | Collecting oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 |   Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.7.0, 2.8.0, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, 3.13.0, 3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.17.0, 3.18.0, 3.18.1, 3.19.0, 3.20.0, 3.21.0, 3.21.1, 3.21.2, 3.22.0, 3.23.0, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.27.0, 3.28.0, 3.28.1, 3.29.0, 3.29.1, 3.30.0, 3.31.0)

2020-02-04 14:14:45.501 | No matching distribution found for oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 | You are using pip version 9.0.1, however version 20.0.2 is available.

2020-02-04 14:14:45.501 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-02-04 14:14:45.501 |

2020-02-04 14:14:45.501 | =================================== log end ====================================

2020-02-04 14:14:45.501 | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.501 | ___________________________________ summary ____________________________________

2020-02-04 14:14:45.501 | ERROR:   full: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.520 | + lib/tempest:install_tempest:1            :   exit_trap

 

 

Thanks,

Srinivas

+91-9243478719

 

 

 

 

smime.p7s

smime.p7s

JamO Luhrsen
 

Srinivas,

Looks like the devstack community also noticed the oslo.concurrency
problem and patched it, which is why the jobs are now working again:

https://opendev.org/openstack/devstack/commit/7d43b099c5f353a0


But, at this point I think it's just a matter of time before another
python2 problem bites. There is a flag in devstack to tell it to
use python3:

https://git.opendaylight.org/gerrit/c/releng/builder/+/87587


That actually gets past the oslo.concurrency issue (I tested it
before the above devstack fix), but it ends up with some weird
keystone issue that I don't know how to fix. I tried uninstalling
all keystone packages prior to stacking, but it still fails.
You can see a log here:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/735/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/15/control_1/stack.log.gz


but here is the failure snippet:

2020-02-11 18:41:35.549 | + lib/keystone:init_keystone:469           :   /usr/bin/keystone-manage --config-file /etc/keystone/keystone.conf db_sync
2020-02-11 18:41:35.810 | Traceback (most recent call last):
2020-02-11 18:41:35.810 |   File "/usr/bin/keystone-manage", line 6, in <module>
2020-02-11 18:41:35.810 |     from keystone.cmd.manage import main
2020-02-11 18:41:35.810 | ImportError: No module named keystone.cmd.manage

If you have the time, I would suggest trying to find a way past
this keystone error so the netvirt devstack can use python3
going forward.

Thanks,
JamO
On 2/7/20 6:45 PM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

When I remove tempest, the scripts are failing with flavour not present.

Its seen that while bringing up tempest m1.nova flavour is created.

This flavour is used by many suites.

 

So we cannot ignore tempest service installation and need to bring up oslo.cocurrency==4.0.0 into the stack.

I need help regarding this.

 

This is blocking all the CSIT jobs.

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 06 February 2020 11:02
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...; 'Dayavanti Gopal Kamath' <dayavanti.gopal.kamath@...>; karthikeyan.k@...
Subject: Re: Latest CSIT job is failing

 

ok. let me know how it goes.

one thing I thought of though, is that the "tempest" CSIT
jobs obviously cannot have tempest removed, so you'll need
to figure that out eventually I guess.

JamO

On 2/5/20 9:07 PM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

I have removed tempest and raised the below review:

https://git.opendaylight.org/gerrit/c/releng/builder/+/87462

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 22:46
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...; 'Dayavanti Gopal Kamath' <dayavanti.gopal.kamath@...>
Subject: Re: Latest CSIT job is failing

 

Maybe you are hitting this bug?
  https://bugs.launchpad.net/tripleo/+bug/1861803

But that was found and fixed in tripleo, not tempest which you
found is your problem.

somethings to try:

1) your job does not need tempest, iirc, so you can remove that
   plugin from being install in the jjb .yaml

2) take a look at moving things to python3? That might be a little
   bit of an undertaking but python2 is dead/dying and this probably
   wont be the last problem related to that.

3) push a similar fix to the tempest project, although getting
   that merged may be hard.


Thanks,
JamO

On 2/5/20 8:44 AM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

I have added the below line and then scheduled the job.

 

/jjb/integration/integration-deploy-openstack-run-test.sh

@@ -156,6 +156,7 @@ function fix_libvirt_python_build() {

         git clone https://git.openstack.org/openstack/requirements;

         cd requirements;

         git checkout ${ODL_ML2_BRANCH};

        sed -i s/oslo.concurrency===4.0.0/oslo.concurrency===3.25.1/ upper-constraints.txt;

         sed -i s/libvirt-python===3.10.0/libvirt-python===4.10.0/ upper-constraints.txt

         "

}

 

But still the job picked oslo.concurrency==4.0.0

On further debugging it occurs with tempest clone /opt/stack/tempest/requirements.txt

 

Please let me know where should I be looking for tempest changes.

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 00:31
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...
Subject: Re: Latest CSIT job is failing

 

Hi Srinivas,

This is a netvirt project problem. I have some experience there so can
try to give some help.

oslo.concurrency was updated in openstack/requirements/upper-constraints.txt
recently (yesterday):

https://github.com/openstack/requirements/commit/0c5ace0ec8ac670b6cbeb64fb8577d11adc90bc5#diff-0bdd949ed8a7fdd4f95240bd951779c8

As you pointed out, devstack failed to find v4.0.0 so stacking failed.

locally, when I do a pip search on oslo.concurrencty, I see v4.0.0 available so not sure
why it's not found in the job. I do see that it's there on pypi:

https://pypi.org/project/oslo.concurrency/#history

Until you can figure out why it can't be found, you could try to pin it to the older
version that was working before yesterday. We do that for other things I think. Here's
one example:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-openstack-run-test.sh#L159

Hope it helps,
JamO



On 2/4/20 7:20 AM, srinivas.rachakonda@... wrote:

Hi Anil,

 

The latest CSIT jobs are failing with the below error:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/

 

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/160/control_1/stack.log.gz

 

2020-02-04 14:14:45.501 | Collecting oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 |   Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.7.0, 2.8.0, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, 3.13.0, 3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.17.0, 3.18.0, 3.18.1, 3.19.0, 3.20.0, 3.21.0, 3.21.1, 3.21.2, 3.22.0, 3.23.0, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.27.0, 3.28.0, 3.28.1, 3.29.0, 3.29.1, 3.30.0, 3.31.0)

2020-02-04 14:14:45.501 | No matching distribution found for oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 | You are using pip version 9.0.1, however version 20.0.2 is available.

2020-02-04 14:14:45.501 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-02-04 14:14:45.501 |

2020-02-04 14:14:45.501 | =================================== log end ====================================

2020-02-04 14:14:45.501 | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.501 | ___________________________________ summary ____________________________________

2020-02-04 14:14:45.501 | ERROR:   full: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.520 | + lib/tempest:install_tempest:1            :   exit_trap

 

 

Thanks,

Srinivas

+91-9243478719

 

 

 

 
Srinivas <srinivas.rachakonda@...>
 

Hi Jamo,

 

Thanks for the looking into the issue.

I will work on the keystone issue and see if fix can be provided.

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 12 February 2020 00:37
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...; 'Dayavanti Gopal Kamath' <dayavanti.gopal.kamath@...>; karthikeyan.k@...; ravikanth.ps@...; abhishek.nagori@...
Subject: Re: Latest CSIT job is failing

 

Srinivas,

Looks like the devstack community also noticed the oslo.concurrency
problem and patched it, which is why the jobs are now working again:

https://opendev.org/openstack/devstack/commit/7d43b099c5f353a0


But, at this point I think it's just a matter of time before another
python2 problem bites. There is a flag in devstack to tell it to
use python3:

https://git.opendaylight.org/gerrit/c/releng/builder/+/87587


That actually gets past the oslo.concurrency issue (I tested it
before the above devstack fix), but it ends up with some weird
keystone issue that I don't know how to fix. I tried uninstalling
all keystone packages prior to stacking, but it still fails.
You can see a log here:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/735/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/15/control_1/stack.log.gz


but here is the failure snippet:


2020-02-11 18:41:35.549 | + lib/keystone:init_keystone:469           :   /usr/bin/keystone-manage --config-file /etc/keystone/keystone.conf db_sync
2020-02-11 18:41:35.810 | Traceback (most recent call last):
2020-02-11 18:41:35.810 |   File "/usr/bin/keystone-manage", line 6, in <module>
2020-02-11 18:41:35.810 |     from keystone.cmd.manage import main
2020-02-11 18:41:35.810 | ImportError: No module named keystone.cmd.manage
 
If you have the time, I would suggest trying to find a way past
this keystone error so the netvirt devstack can use python3
going forward.
 
Thanks,
JamO
 
 
 
 
 
 

On 2/7/20 6:45 PM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

When I remove tempest, the scripts are failing with flavour not present.

Its seen that while bringing up tempest m1.nova flavour is created.

This flavour is used by many suites.

 

So we cannot ignore tempest service installation and need to bring up oslo.cocurrency==4.0.0 into the stack.

I need help regarding this.

 

This is blocking all the CSIT jobs.

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 06 February 2020 11:02
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...; 'Dayavanti Gopal Kamath' <dayavanti.gopal.kamath@...>; karthikeyan.k@...
Subject: Re: Latest CSIT job is failing

 

ok. let me know how it goes.

one thing I thought of though, is that the "tempest" CSIT
jobs obviously cannot have tempest removed, so you'll need
to figure that out eventually I guess.

JamO

On 2/5/20 9:07 PM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

I have removed tempest and raised the below review:

https://git.opendaylight.org/gerrit/c/releng/builder/+/87462

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 22:46
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...; 'Dayavanti Gopal Kamath' <dayavanti.gopal.kamath@...>
Subject: Re: Latest CSIT job is failing

 

Maybe you are hitting this bug?
  https://bugs.launchpad.net/tripleo/+bug/1861803

But that was found and fixed in tripleo, not tempest which you
found is your problem.

somethings to try:

1) your job does not need tempest, iirc, so you can remove that
   plugin from being install in the jjb .yaml

2) take a look at moving things to python3? That might be a little
   bit of an undertaking but python2 is dead/dying and this probably
   wont be the last problem related to that.

3) push a similar fix to the tempest project, although getting
   that merged may be hard.


Thanks,
JamO

On 2/5/20 8:44 AM, srinivas.rachakonda@... wrote:

Hi Jamo,

 

I have added the below line and then scheduled the job.

 

/jjb/integration/integration-deploy-openstack-run-test.sh

@@ -156,6 +156,7 @@ function fix_libvirt_python_build() {

         git clone https://git.openstack.org/openstack/requirements;

         cd requirements;

         git checkout ${ODL_ML2_BRANCH};

        sed -i s/oslo.concurrency===4.0.0/oslo.concurrency===3.25.1/ upper-constraints.txt;

         sed -i s/libvirt-python===3.10.0/libvirt-python===4.10.0/ upper-constraints.txt

         "

}

 

But still the job picked oslo.concurrency==4.0.0

On further debugging it occurs with tempest clone /opt/stack/tempest/requirements.txt

 

Please let me know where should I be looking for tempest changes.

 

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 05 February 2020 00:31
To: srinivas.rachakonda@...; 'Anil Belur' <abelur@...>
Cc: 'netvirt-dev' <netvirt-dev@...>; integration-dev@...
Subject: Re: Latest CSIT job is failing

 

Hi Srinivas,

This is a netvirt project problem. I have some experience there so can
try to give some help.

oslo.concurrency was updated in openstack/requirements/upper-constraints.txt
recently (yesterday):

https://github.com/openstack/requirements/commit/0c5ace0ec8ac670b6cbeb64fb8577d11adc90bc5#diff-0bdd949ed8a7fdd4f95240bd951779c8

As you pointed out, devstack failed to find v4.0.0 so stacking failed.

locally, when I do a pip search on oslo.concurrencty, I see v4.0.0 available so not sure
why it's not found in the job. I do see that it's there on pypi:

https://pypi.org/project/oslo.concurrency/#history

Until you can figure out why it can't be found, you could try to pin it to the older
version that was working before yesterday. We do that for other things I think. Here's
one example:

https://github.com/opendaylight/releng-builder/blob/master/jjb/integration/integration-deploy-openstack-run-test.sh#L159

Hope it helps,
JamO




On 2/4/20 7:20 AM, srinivas.rachakonda@... wrote:

Hi Anil,

 

The latest CSIT jobs are failing with the below error:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/

 

 

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-magnesium/160/control_1/stack.log.gz

 

2020-02-04 14:14:45.501 | Collecting oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 |   Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.10.0, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.6.1, 2.7.0, 2.8.0, 3.0.0, 3.1.0, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.8.0, 3.9.0, 3.10.0, 3.11.0, 3.12.0, 3.13.0, 3.14.0, 3.14.1, 3.15.0, 3.16.0, 3.17.0, 3.18.0, 3.18.1, 3.19.0, 3.20.0, 3.21.0, 3.21.1, 3.21.2, 3.22.0, 3.23.0, 3.24.0, 3.25.0, 3.25.1, 3.26.0, 3.27.0, 3.28.0, 3.28.1, 3.29.0, 3.29.1, 3.30.0, 3.31.0)

2020-02-04 14:14:45.501 | No matching distribution found for oslo.concurrency===4.0.0 (from -c https://releases.openstack.org/constraints/upper/master (line 29))

2020-02-04 14:14:45.501 | You are using pip version 9.0.1, however version 20.0.2 is available.

2020-02-04 14:14:45.501 | You should consider upgrading via the 'pip install --upgrade pip' command.

2020-02-04 14:14:45.501 |

2020-02-04 14:14:45.501 | =================================== log end ====================================

2020-02-04 14:14:45.501 | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.501 | ___________________________________ summary ____________________________________

2020-02-04 14:14:45.501 | ERROR:   full: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/tempest/requirements.txt]; v = InvocationError(u'/opt/stack/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/tempest/requirements.txt', 1)

2020-02-04 14:14:45.520 | + lib/tempest:install_tempest:1            :   exit_trap

 

 

Thanks,

Srinivas

+91-9243478719

 

 

 

 

 

smime.p7s

smime.p7s

1 - 9 of 9