From: Faseela K
Sent: Thursday, March 21, 2019 4:51 PM
To: integration-dev@...; Jaya Priyadarshini
Cc: netvirt-dev@...
Subject: Netvirt patch tests failing on Sodium

From: Jaya Priyadarshini
Sent: Friday, March 22, 2019 2:40 PM
To: Faseela K; integration-dev@...
Cc: netvirt-dev@...
Subject: RE: Netvirt patch tests failing on Sodium

From: Faseela K
Sent: Thursday, March 21, 2019 4:51 PM
To: integration-dev@...; Jaya Priyadarshini
Cc: netvirt-dev@...
Subject: Netvirt patch tests failing on Sodium

On Sat, Mar 23, 2019 at 1:12 PM Jamo Luhrsen <jluhrsen@...> wrote:

On Sat, Mar 23, 2019 at 1:12 PM Jamo Luhrsen <jluhrsen@...> wrote:

Hi Faseela,

 

Trying option 2, have pushed sandbox jobs, monitoring the result, will update asap.

 

Regards

Jaya

 

From: Faseela K
Sent: Monday, March 25, 2019 3:04 PM
To: Sam Hague; Jamo Luhrsen
Cc: Jaya Priyadarshini; Jamo Luhrsen; Dayavanti Gopal Kamath; Venkatrangan Govindarajan; netvirt-dev@...; integration-dev@...
Subject: RE: [integration-dev] Netvirt patch tests failing on Sodium

 

Looks like the issues are not yet resolved.

 

@Jaya : Could you please confirm? Can we try out the approach-2 whatever Sam has suggested?

 

Thanks,

Faseela

 

From: Sam Hague <shague@...>
Sent: Sunday, March 24, 2019 4:26 AM
To: Jamo Luhrsen <jluhrsen@...>
Cc: Jaya Priyadarshini <jaya.priyadarshini@...>; Jamo Luhrsen <jluhrsen@...>; Dayavanti Gopal Kamath <dayavanti.gopal.kamath@...>; Faseela K <faseela.k@...>; Venkatrangan Govindarajan <vgovindarajan@...>; netvirt-dev@...; integration-dev@...
Subject: Re: [integration-dev] Netvirt patch tests failing on Sodium

 

There is a problem with the polkit package. [1] at the bottom has the log below and hows polkit is failing to update. The stack fails. The other error you see about rabbitmq failing is a byproduct of that since the stack failed, rabbitmq never starts and the deployment script sits there for an hour waiting for rabbitmq. (side note, I started a patch to not do the rabbitmq if there was s stack failure but never finished it).

 

So need to figure out how to get polkit working again. [2] is as thread showing others having issues with it. They mention doing a "yum clean all" or yum remove polkit. It's possible we don't need it.

 

[3] mentions simply grabbing the polkit package and rpm installing it. That seems fairly easy. I would do that in the openstack-run.sh script before we stack, yum install http://centos.mirror.garr.it/centos/7.6.1810/updates/x86_64/Packages/polkit-0.112-18.el7_6.1.x86_64.rpm. The reason I say easier, is because the polkit update is happening during the stacking which is a pain to modify. So either remove the package or try to install the updated package in the deploy shell script.

 

Also possible this may go away soon. They say the source of this issue is some delta rpm thing. they mention adding this "deltarpm=0 to /etc/yum.conf". So you could try adding that also before stacking. Just ssh over to the node and add that to the yum.conf. So I would try this solution first. Then try the rpm install.

2019-03-23 19:04:23.739 | Error downloading packages:

2019-03-23 19:04:23.739 |   polkit-0.112-18.el7_6.1.x86_64: [Errno 256] No more mirrors to try.

[1] https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/1303/control_1/stack.log.gz

 

[2] https://www.centos.org/forums/viewtopic.php?f=48&t=70041

 

[3] https://www.linuxquestions.org/questions/showthread.php?s=d0cebe91cc12b6b19aedd3c9cdf88b02&p=5976710#post5976710

 

On Sat, Mar 23, 2019 at 1:12 PM Jamo Luhrsen <jluhrsen@...> wrote:

are things still broken? I see this job is fine now:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/

JamO

On 3/23/19 5:17 AM, Jaya Priyadarshini wrote:
> Yea, but still everything is failing, how do we fix it as of now?
>
> Regards
>
> Jaya
>
> -----Original Message-----
> From: Jamo Luhrsen [mailto:jluhrsen@...]
> Sent: Friday, March 22, 2019 10:59 PM
> To: Dayavanti Gopal Kamath; Jaya Priyadarshini; Faseela K; sam Hague; Venkatrangan Govindarajan
> Cc: netvirt-dev@...; integration-dev@...
> Subject: Re: Netvirt patch tests failing on Sodium
>
> This is exactly the idea of the apex snapshots. It works. It saves time and gives
>
> us a tested and stable VM to just bring up and then test whatever ODL version
>
> we want. We have automatic gate jobs running, like this [0], which are stable.
>
> problem is nobody is working on it any more. It needs people for maintenance,
>
> improvements, monitoring.
>
> As for doing this same thing with devstack, I've been told in the past that
>
> devstack VMs don't work as expected after a reboot. So saving a snapshot and
>
> bringing it up would be the same thing. It's something we've discussed long
>
> back and dismissed. I may be remembering wrong.
>
> JamO
>
> [0] https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-neon/
>
> On 3/22/19 6:02 AM, Dayavanti Gopal Kamath wrote:
>
>  > We really need to change our csits to work with a stable working snapshot of the devstack vm with all stacking done. Can
>
>  > we not just stack, store the vm image somewhere, if no other storage is available, neutron git repository, and pull it
>
>  > from there for every run? It will help cut down test execution time by 30 minutes and shield us from this unnecessary
>
>  > pain. We can choose when to update the vm image, after testing.
>
>  >
>
>  > Thanks,
>
>  >
>
>  > daya
>
>  >
>
>  > *From:* netvirt-dev-bounces@... [mailto:netvirt-dev-bounces@...] *On Behalf Of
>
>  > *Jaya Priyadarshini
>
>  > *Sent:* Friday, March 22, 2019 3:18 PM
>
>  > *To:* Faseela K <faseela.k@...>; sam Hague <shague@...>; Jamo Luhrsen <jluhrsen@...>;
>
>  > Venkatrangan Govindarajan <vgovindarajan@...>
>
>  > *Cc:* netvirt-dev@...; integration-dev@...
>
>  > *Subject:* Re: [netvirt-dev] Netvirt patch tests failing on Sodium
>
>  >
>
>  > Hi Faseela,
>
>  >
>
>  > Few more observations,
>
>  >
>
>  > (i)Same jobs passes for sodium but fails for neon and fluorine .
>
>  >
>
>  > (ii)Some jobs have different failure reason as well, apart from rabbitmq issue. They are as below:
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/1158/compute_1/stack.log.gz
>
>  >
>
>  > Stack stops abruptly at
> Downloadinghttps://files.pythonhosted.org/packages/e7/1e/3dcca007f974fe4eb369bf1b8629d5e342bb3055e2001b2e5340aaefae7a/uwsgi-2.0.18.tar.gz 
> (
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-queens-tempest-stateful-snat-conntrack-neon/187/console.log.gz
>
>  >
>
>  > stack fails and there is no stack.log also (no compute folder itself).
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-sfc-neon/187/console.log.gz
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-neon/221/console.log.gz
>
>  >
>
>  > Resolving nexus.opendaylight.org (nexus.opendaylight.org)... 10.30.168.3, 2604:e100:1:0:f816:3eff:fe45:48d6
>
>  >
>
>  > Connecting to nexus.opendaylight.org (nexus.opendaylight.org)|10.30.168.3|:443... connected.
>
>  >
>
>  > HTTP request sent, awaiting response... Extracting the new controller...
>
>  >
>
>  > 404 Not Found
>
>  >
>
>  > Can anyone please help?...
>
>  >
>
>  > Regards
>
>  >
>
>  > Jaya
>
>  >
>
>  > *From:*Jaya Priyadarshini
>
>  > *Sent:* Friday, March 22, 2019 2:40 PM
>
>  > *To:* Faseela K; integration-dev@... <mailto:integration-dev@...>
>
>  > *Cc:* netvirt-dev@... <mailto:netvirt-dev@...>
>
>  > *Subject:* RE: Netvirt patch tests failing on Sodium
>
>  >
>
>  > Hi Faseela,
>
>  >
>
>  > All the jobs in netvirt and sandbox are failing due to this
>
>  >
>
>  > + ssh -t -t 10.30.170.79 'sudo rabbitmqctl list_vhosts'
>
>  >
>
>  > *10:34:52*Warning: Permanently added '10.30.170.79' (ECDSA) to the list of known hosts.
>
>  >
>
>  > *10:34:52*Connection to 10.30.170.79 closed.
>
>  >
>
>  > *10:34:52*+ grep nova_cell1 rabbit.txt
>
>  >
>
>  > *10:34:52*+ rc=1
>
>  >
>
>  > *10:34:52*+ (( 1 == 0 ))
>
>  >
>
>  > *10:34:52*+ (( 60 == 60 ))
>
>  >
>
>  > *10:34:52*+ break
>
>  >
>
>  > *10:34:52*+ return 1
>
>  >
>
>  > It is trying for 60 attempts and fails, not sure which change could have done this.
>
>  >
>
>  > Stack log says this change has gone in.
>
>  >
>
>  > DevStack Version: queens
>
>  >
>
>  > Change: e98e7d5ae3df3ba1ccb2b1a2af649a853d05e56b Merge "Enable nova VNC configuration on the subnode" into stable/queens
>
>  > 2019-02-27 14:00:34 +0000
>
>  >
>
>  > Can anyone please help resolving this.
>
>  >
>
>  > Regards
>
>  >
>
>  > Jaya
>
>  >
>
>  > *From:*Faseela K
>
>  > *Sent:* Thursday, March 21, 2019 4:51 PM
>
>  > *To:* integration-dev@... <mailto:integration-dev@...>; Jaya Priyadarshini
>
>  > *Cc:* netvirt-dev@... <mailto:netvirt-dev@...>
>
>  > *Subject:* Netvirt patch tests failing on Sodium
>
>  >
>
>  > Jaya/Venkat,
>
>  >
>
>  >     Anybody has an idea why the patch tests are failing?
>
>  >
>
>  > https://git.opendaylight.org/gerrit/#/c/75427/
>
>  >
>
>  >    See below errors in console :
>
>  >
>
>  >      *16:05:03*tar: Removing leading `/' from member names
>
>  >
>
>  > *16:05:03*tar: /tmp/zrpcd.init.log: Cannot stat: No such file or directory
>
>  >
>
>  > *16:05:03*tar: Exiting with failure status due to previous errors
>
>  >
>
>  > *16:05:03*Connection to 10.30.170.37 closed.
>
>  >
>
>  > Thanks,
>
>  >
>
>  > Faseela
>
>  >
>
>  >
> https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/11/console
>
>  >
>
>
> _______________________________________________
> integration-dev mailing list
> integration-dev@...
> https://lists.opendaylight.org/mailman/listinfo/integration-dev
>

Hi Faseela,

 

Trying option 2, have pushed sandbox jobs, monitoring the result, will update asap.

 

Regards

Jaya

 

From: Faseela K
Sent: Monday, March 25, 2019 3:04 PM
To: Sam Hague; Jamo Luhrsen
Cc: Jaya Priyadarshini; Jamo Luhrsen; Dayavanti Gopal Kamath; Venkatrangan Govindarajan; netvirt-dev@...; integration-dev@...
Subject: RE: [integration-dev] Netvirt patch tests failing on Sodium

 

Looks like the issues are not yet resolved.

 

@Jaya : Could you please confirm? Can we try out the approach-2 whatever Sam has suggested?

 

Thanks,

Faseela

 

From: Sam Hague <shague@...>
Sent: Sunday, March 24, 2019 4:26 AM
To: Jamo Luhrsen <jluhrsen@...>
Cc: Jaya Priyadarshini <jaya.priyadarshini@...>; Jamo Luhrsen <jluhrsen@...>; Dayavanti Gopal Kamath <dayavanti.gopal.kamath@...>; Faseela K <faseela.k@...>; Venkatrangan Govindarajan <vgovindarajan@...>; netvirt-dev@...; integration-dev@...
Subject: Re: [integration-dev] Netvirt patch tests failing on Sodium

 

There is a problem with the polkit package. [1] at the bottom has the log below and hows polkit is failing to update. The stack fails. The other error you see about rabbitmq failing is a byproduct of that since the stack failed, rabbitmq never starts and the deployment script sits there for an hour waiting for rabbitmq. (side note, I started a patch to not do the rabbitmq if there was s stack failure but never finished it).

 

So need to figure out how to get polkit working again. [2] is as thread showing others having issues with it. They mention doing a "yum clean all" or yum remove polkit. It's possible we don't need it.

 

[3] mentions simply grabbing the polkit package and rpm installing it. That seems fairly easy. I would do that in the openstack-run.sh script before we stack, yum install http://centos.mirror.garr.it/centos/7.6.1810/updates/x86_64/Packages/polkit-0.112-18.el7_6.1.x86_64.rpm. The reason I say easier, is because the polkit update is happening during the stacking which is a pain to modify. So either remove the package or try to install the updated package in the deploy shell script.

 

Also possible this may go away soon. They say the source of this issue is some delta rpm thing. they mention adding this "deltarpm=0 to /etc/yum.conf". So you could try adding that also before stacking. Just ssh over to the node and add that to the yum.conf. So I would try this solution first. Then try the rpm install.

2019-03-23 19:04:23.739 | Error downloading packages:

2019-03-23 19:04:23.739 |   polkit-0.112-18.el7_6.1.x86_64: [Errno 256] No more mirrors to try.

[1] https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/1303/control_1/stack.log.gz

 

[2] https://www.centos.org/forums/viewtopic.php?f=48&t=70041

 

[3] https://www.linuxquestions.org/questions/showthread.php?s=d0cebe91cc12b6b19aedd3c9cdf88b02&p=5976710#post5976710

 

On Sat, Mar 23, 2019 at 1:12 PM Jamo Luhrsen <jluhrsen@...> wrote:

are things still broken? I see this job is fine now:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/

JamO

On 3/23/19 5:17 AM, Jaya Priyadarshini wrote:
> Yea, but still everything is failing, how do we fix it as of now?
>
> Regards
>
> Jaya
>
> -----Original Message-----
> From: Jamo Luhrsen [mailto:jluhrsen@...]
> Sent: Friday, March 22, 2019 10:59 PM
> To: Dayavanti Gopal Kamath; Jaya Priyadarshini; Faseela K; sam Hague; Venkatrangan Govindarajan
> Cc: netvirt-dev@...; integration-dev@...
> Subject: Re: Netvirt patch tests failing on Sodium
>
> This is exactly the idea of the apex snapshots. It works. It saves time and gives
>
> us a tested and stable VM to just bring up and then test whatever ODL version
>
> we want. We have automatic gate jobs running, like this [0], which are stable.
>
> problem is nobody is working on it any more. It needs people for maintenance,
>
> improvements, monitoring.
>
> As for doing this same thing with devstack, I've been told in the past that
>
> devstack VMs don't work as expected after a reboot. So saving a snapshot and
>
> bringing it up would be the same thing. It's something we've discussed long
>
> back and dismissed. I may be remembering wrong.
>
> JamO
>
> [0] https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-neon/
>
> On 3/22/19 6:02 AM, Dayavanti Gopal Kamath wrote:
>
>  > We really need to change our csits to work with a stable working snapshot of the devstack vm with all stacking done. Can
>
>  > we not just stack, store the vm image somewhere, if no other storage is available, neutron git repository, and pull it
>
>  > from there for every run? It will help cut down test execution time by 30 minutes and shield us from this unnecessary
>
>  > pain. We can choose when to update the vm image, after testing.
>
>  >
>
>  > Thanks,
>
>  >
>
>  > daya
>
>  >
>
>  > *From:* netvirt-dev-bounces@... [mailto:netvirt-dev-bounces@...] *On Behalf Of
>
>  > *Jaya Priyadarshini
>
>  > *Sent:* Friday, March 22, 2019 3:18 PM
>
>  > *To:* Faseela K <faseela.k@...>; sam Hague <shague@...>; Jamo Luhrsen <jluhrsen@...>;
>
>  > Venkatrangan Govindarajan <vgovindarajan@...>
>
>  > *Cc:* netvirt-dev@...; integration-dev@...
>
>  > *Subject:* Re: [netvirt-dev] Netvirt patch tests failing on Sodium
>
>  >
>
>  > Hi Faseela,
>
>  >
>
>  > Few more observations,
>
>  >
>
>  > (i)Same jobs passes for sodium but fails for neon and fluorine .
>
>  >
>
>  > (ii)Some jobs have different failure reason as well, apart from rabbitmq issue. They are as below:
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/1158/compute_1/stack.log.gz
>
>  >
>
>  > Stack stops abruptly at
> Downloadinghttps://files.pythonhosted.org/packages/e7/1e/3dcca007f974fe4eb369bf1b8629d5e342bb3055e2001b2e5340aaefae7a/uwsgi-2.0.18.tar.gz 
> (
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-queens-tempest-stateful-snat-conntrack-neon/187/console.log.gz
>
>  >
>
>  > stack fails and there is no stack.log also (no compute folder itself).
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-sfc-neon/187/console.log.gz
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-neon/221/console.log.gz
>
>  >
>
>  > Resolving nexus.opendaylight.org (nexus.opendaylight.org)... 10.30.168.3, 2604:e100:1:0:f816:3eff:fe45:48d6
>
>  >
>
>  > Connecting to nexus.opendaylight.org (nexus.opendaylight.org)|10.30.168.3|:443... connected.
>
>  >
>
>  > HTTP request sent, awaiting response... Extracting the new controller...
>
>  >
>
>  > 404 Not Found
>
>  >
>
>  > Can anyone please help?...
>
>  >
>
>  > Regards
>
>  >
>
>  > Jaya
>
>  >
>
>  > *From:*Jaya Priyadarshini
>
>  > *Sent:* Friday, March 22, 2019 2:40 PM
>
>  > *To:* Faseela K; integration-dev@... <mailto:integration-dev@...>
>
>  > *Cc:* netvirt-dev@... <mailto:netvirt-dev@...>
>
>  > *Subject:* RE: Netvirt patch tests failing on Sodium
>
>  >
>
>  > Hi Faseela,
>
>  >
>
>  > All the jobs in netvirt and sandbox are failing due to this
>
>  >
>
>  > + ssh -t -t 10.30.170.79 'sudo rabbitmqctl list_vhosts'
>
>  >
>
>  > *10:34:52*Warning: Permanently added '10.30.170.79' (ECDSA) to the list of known hosts.
>
>  >
>
>  > *10:34:52*Connection to 10.30.170.79 closed.
>
>  >
>
>  > *10:34:52*+ grep nova_cell1 rabbit.txt
>
>  >
>
>  > *10:34:52*+ rc=1
>
>  >
>
>  > *10:34:52*+ (( 1 == 0 ))
>
>  >
>
>  > *10:34:52*+ (( 60 == 60 ))
>
>  >
>
>  > *10:34:52*+ break
>
>  >
>
>  > *10:34:52*+ return 1
>
>  >
>
>  > It is trying for 60 attempts and fails, not sure which change could have done this.
>
>  >
>
>  > Stack log says this change has gone in.
>
>  >
>
>  > DevStack Version: queens
>
>  >
>
>  > Change: e98e7d5ae3df3ba1ccb2b1a2af649a853d05e56b Merge "Enable nova VNC configuration on the subnode" into stable/queens
>
>  > 2019-02-27 14:00:34 +0000
>
>  >
>
>  > Can anyone please help resolving this.
>
>  >
>
>  > Regards
>
>  >
>
>  > Jaya
>
>  >
>
>  > *From:*Faseela K
>
>  > *Sent:* Thursday, March 21, 2019 4:51 PM
>
>  > *To:* integration-dev@... <mailto:integration-dev@...>; Jaya Priyadarshini
>
>  > *Cc:* netvirt-dev@... <mailto:netvirt-dev@...>
>
>  > *Subject:* Netvirt patch tests failing on Sodium
>
>  >
>
>  > Jaya/Venkat,
>
>  >
>
>  >     Anybody has an idea why the patch tests are failing?
>
>  >
>
>  > https://git.opendaylight.org/gerrit/#/c/75427/
>
>  >
>
>  >    See below errors in console :
>
>  >
>
>  >      *16:05:03*tar: Removing leading `/' from member names
>
>  >
>
>  > *16:05:03*tar: /tmp/zrpcd.init.log: Cannot stat: No such file or directory
>
>  >
>
>  > *16:05:03*tar: Exiting with failure status due to previous errors
>
>  >
>
>  > *16:05:03*Connection to 10.30.170.37 closed.
>
>  >
>
>  > Thanks,
>
>  >
>
>  > Faseela
>
>  >
>
>  >
> https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/11/console
>
>  >
>
>
> _______________________________________________
> integration-dev mailing list
> integration-dev@...
> https://lists.opendaylight.org/mailman/listinfo/integration-dev
>

From: Jaya Priyadarshini
Sent: Monday, March 25, 2019 11:33 PM
To: 'Sam Hague'
Cc: Faseela K; Jamo Luhrsen; Jamo Luhrsen; Dayavanti Gopal Kamath; Venkatrangan Govindarajan; netvirt-dev@...; integration-dev@...
Subject: RE: [integration-dev] Netvirt patch tests failing on Sodium

Hi Faseela,

 

Trying option 2, have pushed sandbox jobs, monitoring the result, will update asap.

 

Regards

Jaya

 

From: Faseela K
Sent: Monday, March 25, 2019 3:04 PM
To: Sam Hague; Jamo Luhrsen
Cc: Jaya Priyadarshini; Jamo Luhrsen; Dayavanti Gopal Kamath; Venkatrangan Govindarajan; netvirt-dev@...; integration-dev@...
Subject: RE: [integration-dev] Netvirt patch tests failing on Sodium

 

Looks like the issues are not yet resolved.

 

@Jaya : Could you please confirm? Can we try out the approach-2 whatever Sam has suggested?

 

Thanks,

Faseela

 

From: Sam Hague <shague@...>
Sent: Sunday, March 24, 2019 4:26 AM
To: Jamo Luhrsen <jluhrsen@...>
Cc: Jaya Priyadarshini <jaya.priyadarshini@...>; Jamo Luhrsen <jluhrsen@...>; Dayavanti Gopal Kamath <dayavanti.gopal.kamath@...>; Faseela K <faseela.k@...>; Venkatrangan Govindarajan <vgovindarajan@...>; netvirt-dev@...; integration-dev@...
Subject: Re: [integration-dev] Netvirt patch tests failing on Sodium

 

There is a problem with the polkit package. [1] at the bottom has the log below and hows polkit is failing to update. The stack fails. The other error you see about rabbitmq failing is a byproduct of that since the stack failed, rabbitmq never starts and the deployment script sits there for an hour waiting for rabbitmq. (side note, I started a patch to not do the rabbitmq if there was s stack failure but never finished it).

 

So need to figure out how to get polkit working again. [2] is as thread showing others having issues with it. They mention doing a "yum clean all" or yum remove polkit. It's possible we don't need it.

 

[3] mentions simply grabbing the polkit package and rpm installing it. That seems fairly easy. I would do that in the openstack-run.sh script before we stack, yum install http://centos.mirror.garr.it/centos/7.6.1810/updates/x86_64/Packages/polkit-0.112-18.el7_6.1.x86_64.rpm. The reason I say easier, is because the polkit update is happening during the stacking which is a pain to modify. So either remove the package or try to install the updated package in the deploy shell script.

 

Also possible this may go away soon. They say the source of this issue is some delta rpm thing. they mention adding this "deltarpm=0 to /etc/yum.conf". So you could try adding that also before stacking. Just ssh over to the node and add that to the yum.conf. So I would try this solution first. Then try the rpm install.

2019-03-23 19:04:23.739 | Error downloading packages:

2019-03-23 19:04:23.739 |   polkit-0.112-18.el7_6.1.x86_64: [Errno 256] No more mirrors to try.

[1] https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/1303/control_1/stack.log.gz

 

[2] https://www.centos.org/forums/viewtopic.php?f=48&t=70041

 

[3] https://www.linuxquestions.org/questions/showthread.php?s=d0cebe91cc12b6b19aedd3c9cdf88b02&p=5976710#post5976710

 

On Sat, Mar 23, 2019 at 1:12 PM Jamo Luhrsen <jluhrsen@...> wrote:

are things still broken? I see this job is fine now:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-fluorine/

JamO

On 3/23/19 5:17 AM, Jaya Priyadarshini wrote:
> Yea, but still everything is failing, how do we fix it as of now?
>
> Regards
>
> Jaya
>
> -----Original Message-----
> From: Jamo Luhrsen [mailto:jluhrsen@...]
> Sent: Friday, March 22, 2019 10:59 PM
> To: Dayavanti Gopal Kamath; Jaya Priyadarshini; Faseela K; sam Hague; Venkatrangan Govindarajan
> Cc: netvirt-dev@...; integration-dev@...
> Subject: Re: Netvirt patch tests failing on Sodium
>
> This is exactly the idea of the apex snapshots. It works. It saves time and gives
>
> us a tested and stable VM to just bring up and then test whatever ODL version
>
> we want. We have automatic gate jobs running, like this [0], which are stable.
>
> problem is nobody is working on it any more. It needs people for maintenance,
>
> improvements, monitoring.
>
> As for doing this same thing with devstack, I've been told in the past that
>
> devstack VMs don't work as expected after a reboot. So saving a snapshot and
>
> bringing it up would be the same thing. It's something we've discussed long
>
> back and dismissed. I may be remembering wrong.
>
> JamO
>
> [0] https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-neon/
>
> On 3/22/19 6:02 AM, Dayavanti Gopal Kamath wrote:
>
>  > We really need to change our csits to work with a stable working snapshot of the devstack vm with all stacking done. Can
>
>  > we not just stack, store the vm image somewhere, if no other storage is available, neutron git repository, and pull it
>
>  > from there for every run? It will help cut down test execution time by 30 minutes and shield us from this unnecessary
>
>  > pain. We can choose when to update the vm image, after testing.
>
>  >
>
>  > Thanks,
>
>  >
>
>  > daya
>
>  >
>
>  > *From:* netvirt-dev-bounces@... [mailto:netvirt-dev-bounces@...] *On Behalf Of
>
>  > *Jaya Priyadarshini
>
>  > *Sent:* Friday, March 22, 2019 3:18 PM
>
>  > *To:* Faseela K <faseela.k@...>; sam Hague <shague@...>; Jamo Luhrsen <jluhrsen@...>;
>
>  > Venkatrangan Govindarajan <vgovindarajan@...>
>
>  > *Cc:* netvirt-dev@...; integration-dev@...
>
>  > *Subject:* Re: [netvirt-dev] Netvirt patch tests failing on Sodium
>
>  >
>
>  > Hi Faseela,
>
>  >
>
>  > Few more observations,
>
>  >
>
>  > (i)Same jobs passes for sodium but fails for neon and fluorine .
>
>  >
>
>  > (ii)Some jobs have different failure reason as well, apart from rabbitmq issue. They are as below:
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/1158/compute_1/stack.log.gz
>
>  >
>
>  > Stack stops abruptly at
> Downloadinghttps://files.pythonhosted.org/packages/e7/1e/3dcca007f974fe4eb369bf1b8629d5e342bb3055e2001b2e5340aaefae7a/uwsgi-2.0.18.tar.gz 
> (
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-queens-tempest-stateful-snat-conntrack-neon/187/console.log.gz
>
>  >
>
>  > stack fails and there is no stack.log also (no compute folder itself).
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-sfc-neon/187/console.log.gz
>
>  >
>
>  >
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-neon/221/console.log.gz
>
>  >
>
>  > Resolving nexus.opendaylight.org (nexus.opendaylight.org)... 10.30.168.3, 2604:e100:1:0:f816:3eff:fe45:48d6
>
>  >
>
>  > Connecting to nexus.opendaylight.org (nexus.opendaylight.org)|10.30.168.3|:443... connected.
>
>  >
>
>  > HTTP request sent, awaiting response... Extracting the new controller...
>
>  >
>
>  > 404 Not Found
>
>  >
>
>  > Can anyone please help?...
>
>  >
>
>  > Regards
>
>  >
>
>  > Jaya
>
>  >
>
>  > *From:*Jaya Priyadarshini
>
>  > *Sent:* Friday, March 22, 2019 2:40 PM
>
>  > *To:* Faseela K; integration-dev@... <mailto:integration-dev@...>
>
>  > *Cc:* netvirt-dev@... <mailto:netvirt-dev@...>
>
>  > *Subject:* RE: Netvirt patch tests failing on Sodium
>
>  >
>
>  > Hi Faseela,
>
>  >
>
>  > All the jobs in netvirt and sandbox are failing due to this
>
>  >
>
>  > + ssh -t -t 10.30.170.79 'sudo rabbitmqctl list_vhosts'
>
>  >
>
>  > *10:34:52*Warning: Permanently added '10.30.170.79' (ECDSA) to the list of known hosts.
>
>  >
>
>  > *10:34:52*Connection to 10.30.170.79 closed.
>
>  >
>
>  > *10:34:52*+ grep nova_cell1 rabbit.txt
>
>  >
>
>  > *10:34:52*+ rc=1
>
>  >
>
>  > *10:34:52*+ (( 1 == 0 ))
>
>  >
>
>  > *10:34:52*+ (( 60 == 60 ))
>
>  >
>
>  > *10:34:52*+ break
>
>  >
>
>  > *10:34:52*+ return 1
>
>  >
>
>  > It is trying for 60 attempts and fails, not sure which change could have done this.
>
>  >
>
>  > Stack log says this change has gone in.
>
>  >
>
>  > DevStack Version: queens
>
>  >
>
>  > Change: e98e7d5ae3df3ba1ccb2b1a2af649a853d05e56b Merge "Enable nova VNC configuration on the subnode" into stable/queens
>
>  > 2019-02-27 14:00:34 +0000
>
>  >
>
>  > Can anyone please help resolving this.
>
>  >
>
>  > Regards
>
>  >
>
>  > Jaya
>
>  >
>
>  > *From:*Faseela K
>
>  > *Sent:* Thursday, March 21, 2019 4:51 PM
>
>  > *To:* integration-dev@... <mailto:integration-dev@...>; Jaya Priyadarshini
>
>  > *Cc:* netvirt-dev@... <mailto:netvirt-dev@...>
>
>  > *Subject:* Netvirt patch tests failing on Sodium
>
>  >
>
>  > Jaya/Venkat,
>
>  >
>
>  >     Anybody has an idea why the patch tests are failing?
>
>  >
>
>  > https://git.opendaylight.org/gerrit/#/c/75427/
>
>  >
>
>  >    See below errors in console :
>
>  >
>
>  >      *16:05:03*tar: Removing leading `/' from member names
>
>  >
>
>  > *16:05:03*tar: /tmp/zrpcd.init.log: Cannot stat: No such file or directory
>
>  >
>
>  > *16:05:03*tar: Exiting with failure status due to previous errors
>
>  >
>
>  > *16:05:03*Connection to 10.30.170.37 closed.
>
>  >
>
>  > Thanks,
>
>  >
>
>  > Faseela
>
>  >
>
>  >
> https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/11/console
>
>  >
>
>
> _______________________________________________
> integration-dev mailing list
> integration-dev@...
> https://lists.opendaylight.org/mailman/listinfo/integration-dev
>