netvirt-csit gone all red


JayaPr
 

Hi All,

 

Almost all last build have failed with below error in netvirt

 

+ ssh 10.30.170.83 'sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron'

Warning: Permanently added '10.30.170.83' (ECDSA) to the list of known hosts.

sed: can't read /opt/stack/devstack/lib/neutron: No such file or directory

++ trap_handler 833 15569

++ local prog=/tmp/jenkins3447639189848961768.sh

++ local lastline=833

++ local lasterr=15569

++ echo 'trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569'

trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569

++ echo 'trap_handler: command: ssh ${!CONTROLIP} "sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron"'

trap_handler: command: ssh ${!CONTROLIP} "sed -i 's/flat_networks public/flat_networks public,physnet1/' /opt/stack/devstack/lib/neutron"

++ exit 1

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

 

Trying to see if I can fix it, Anyone has any idea what could have gone wrong?

 

Regards

Jaya


Faseela K
 

Hi everyone,

   Please stop triggering patch tests for netvirt until we fix the issue.

Thanks,

Faseela

 

From: netvirt-dev <netvirt-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 12:55 PM
To: netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: [netvirt-dev] netvirt-csit gone all red

 

Hi All,

 

Almost all last build have failed with below error in netvirt

 

+ ssh 10.30.170.83 'sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron'

Warning: Permanently added '10.30.170.83' (ECDSA) to the list of known hosts.

sed: can't read /opt/stack/devstack/lib/neutron: No such file or directory

++ trap_handler 833 15569

++ local prog=/tmp/jenkins3447639189848961768.sh

++ local lastline=833

++ local lasterr=15569

++ echo 'trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569'

trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569

++ echo 'trap_handler: command: ssh ${!CONTROLIP} "sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron"'

trap_handler: command: ssh ${!CONTROLIP} "sed -i 's/flat_networks public/flat_networks public,physnet1/' /opt/stack/devstack/lib/neutron"

++ exit 1

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

 

Trying to see if I can fix it, Anyone has any idea what could have gone wrong?

 

Regards

Jaya


JayaPr
 

Well, found the reason behind this:

 

git clone https://git.openstack.org/openstack-dev/devstack --branch stable/rocky

Cloning into 'devstack'...

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Encountered end of file

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.136.133

Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.136.133|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 2156 (2.1K) [text/plain]

 

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt by Sam is not working anymore.

And this problem is existing for all devstack branches , be it queens or rocky.

Will be fixing it.

 

Regards

Jaya

 

 

From: Faseela K <faseela.k@...>
Sent: Tuesday, July 23, 2019 3:08 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: RE: netvirt-csit gone all red

 

Hi everyone,

   Please stop triggering patch tests for netvirt until we fix the issue.

Thanks,

Faseela

 

From: netvirt-dev <netvirt-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 12:55 PM
To: netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: [netvirt-dev] netvirt-csit gone all red

 

Hi All,

 

Almost all last build have failed with below error in netvirt

 

+ ssh 10.30.170.83 'sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron'

Warning: Permanently added '10.30.170.83' (ECDSA) to the list of known hosts.

sed: can't read /opt/stack/devstack/lib/neutron: No such file or directory

++ trap_handler 833 15569

++ local prog=/tmp/jenkins3447639189848961768.sh

++ local lastline=833

++ local lasterr=15569

++ echo 'trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569'

trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569

++ echo 'trap_handler: command: ssh ${!CONTROLIP} "sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron"'

trap_handler: command: ssh ${!CONTROLIP} "sed -i 's/flat_networks public/flat_networks public,physnet1/' /opt/stack/devstack/lib/neutron"

++ exit 1

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

 

Trying to see if I can fix it, Anyone has any idea what could have gone wrong?

 

Regards

Jaya


JayaPr
 

Anil,

 

Can u check this, this is seen in this job.

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/109/

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/111/control_1/get_devstack.sh.txt.gz

 

All jobs of this link has this issue.

 

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Empty reply from server

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

 

But I am able to access the same from all my local machines, only cloud computes are having this issue.

 

Regards

Jaya

From: integration-dev <integration-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 4:16 PM
To: Faseela K <faseela.k@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: Re: [integration-dev] netvirt-csit gone all red

 

Well, found the reason behind this:

 

git clone https://git.openstack.org/openstack-dev/devstack --branch stable/rocky

Cloning into 'devstack'...

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Encountered end of file

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.136.133

Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.136.133|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 2156 (2.1K) [text/plain]

 

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt by Sam is not working anymore.

And this problem is existing for all devstack branches , be it queens or rocky.

Will be fixing it.

 

Regards

Jaya

 

 

From: Faseela K <faseela.k@...>
Sent: Tuesday, July 23, 2019 3:08 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: RE: netvirt-csit gone all red

 

Hi everyone,

   Please stop triggering patch tests for netvirt until we fix the issue.

Thanks,

Faseela

 

From: netvirt-dev <netvirt-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 12:55 PM
To: netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: [netvirt-dev] netvirt-csit gone all red

 

Hi All,

 

Almost all last build have failed with below error in netvirt

 

+ ssh 10.30.170.83 'sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron'

Warning: Permanently added '10.30.170.83' (ECDSA) to the list of known hosts.

sed: can't read /opt/stack/devstack/lib/neutron: No such file or directory

++ trap_handler 833 15569

++ local prog=/tmp/jenkins3447639189848961768.sh

++ local lastline=833

++ local lasterr=15569

++ echo 'trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569'

trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569

++ echo 'trap_handler: command: ssh ${!CONTROLIP} "sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron"'

trap_handler: command: ssh ${!CONTROLIP} "sed -i 's/flat_networks public/flat_networks public,physnet1/' /opt/stack/devstack/lib/neutron"

++ exit 1

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

 

Trying to see if I can fix it, Anyone has any idea what could have gone wrong?

 

Regards

Jaya


Anil Belur
 

Greetings Jaya:

Can you raise an issue at https://support.linuxfoundation.org about it please?

Thanks,
Anil

On Thu, Jul 25, 2019 at 5:29 PM Jaya Priyadarshini <jaya.priyadarshini@...> wrote:

Anil,

 

Can u check this, this is seen in this job.

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/109/

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/111/control_1/get_devstack.sh.txt.gz

 

All jobs of this link has this issue.

 

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Empty reply from server

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

 

But I am able to access the same from all my local machines, only cloud computes are having this issue.

 

Regards

Jaya

From: integration-dev <integration-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 4:16 PM
To: Faseela K <faseela.k@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: Re: [integration-dev] netvirt-csit gone all red

 

Well, found the reason behind this:

 

git clone https://git.openstack.org/openstack-dev/devstack --branch stable/rocky

Cloning into 'devstack'...

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Encountered end of file

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.136.133

Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.136.133|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 2156 (2.1K) [text/plain]

 

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt by Sam is not working anymore.

And this problem is existing for all devstack branches , be it queens or rocky.

Will be fixing it.

 

Regards

Jaya

 

 

From: Faseela K <faseela.k@...>
Sent: Tuesday, July 23, 2019 3:08 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: RE: netvirt-csit gone all red

 

Hi everyone,

   Please stop triggering patch tests for netvirt until we fix the issue.

Thanks,

Faseela

 

From: netvirt-dev <netvirt-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 12:55 PM
To: netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: [netvirt-dev] netvirt-csit gone all red

 

Hi All,

 

Almost all last build have failed with below error in netvirt

 

+ ssh 10.30.170.83 'sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron'

Warning: Permanently added '10.30.170.83' (ECDSA) to the list of known hosts.

sed: can't read /opt/stack/devstack/lib/neutron: No such file or directory

++ trap_handler 833 15569

++ local prog=/tmp/jenkins3447639189848961768.sh

++ local lastline=833

++ local lasterr=15569

++ echo 'trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569'

trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569

++ echo 'trap_handler: command: ssh ${!CONTROLIP} "sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron"'

trap_handler: command: ssh ${!CONTROLIP} "sed -i 's/flat_networks public/flat_networks public,physnet1/' /opt/stack/devstack/lib/neutron"

++ exit 1

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

 

Trying to see if I can fix it, Anyone has any idea what could have gone wrong?

 

Regards

Jaya

_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev


Anil Belur
 

I think this might be an issue with the openstack infra itself, and not on our infra. I'm seeing the git clone issue locally.

Cloning into 'devstack'...
fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Failed to connect to opendev.org port 443: No route to host

- Anil 

On Thu, Jul 25, 2019 at 5:51 PM Anil Belur <abelur@...> wrote:
Greetings Jaya:

Can you raise an issue at https://support.linuxfoundation.org about it please?

Thanks,
Anil

On Thu, Jul 25, 2019 at 5:29 PM Jaya Priyadarshini <jaya.priyadarshini@...> wrote:

Anil,

 

Can u check this, this is seen in this job.

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/109/

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/111/control_1/get_devstack.sh.txt.gz

 

All jobs of this link has this issue.

 

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Empty reply from server

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

 

But I am able to access the same from all my local machines, only cloud computes are having this issue.

 

Regards

Jaya

From: integration-dev <integration-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 4:16 PM
To: Faseela K <faseela.k@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: Re: [integration-dev] netvirt-csit gone all red

 

Well, found the reason behind this:

 

git clone https://git.openstack.org/openstack-dev/devstack --branch stable/rocky

Cloning into 'devstack'...

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Encountered end of file

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.136.133

Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.136.133|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 2156 (2.1K) [text/plain]

 

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt by Sam is not working anymore.

And this problem is existing for all devstack branches , be it queens or rocky.

Will be fixing it.

 

Regards

Jaya

 

 

From: Faseela K <faseela.k@...>
Sent: Tuesday, July 23, 2019 3:08 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: RE: netvirt-csit gone all red

 

Hi everyone,

   Please stop triggering patch tests for netvirt until we fix the issue.

Thanks,

Faseela

 

From: netvirt-dev <netvirt-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 12:55 PM
To: netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: [netvirt-dev] netvirt-csit gone all red

 

Hi All,

 

Almost all last build have failed with below error in netvirt

 

+ ssh 10.30.170.83 'sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron'

Warning: Permanently added '10.30.170.83' (ECDSA) to the list of known hosts.

sed: can't read /opt/stack/devstack/lib/neutron: No such file or directory

++ trap_handler 833 15569

++ local prog=/tmp/jenkins3447639189848961768.sh

++ local lastline=833

++ local lasterr=15569

++ echo 'trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569'

trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569

++ echo 'trap_handler: command: ssh ${!CONTROLIP} "sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron"'

trap_handler: command: ssh ${!CONTROLIP} "sed -i 's/flat_networks public/flat_networks public,physnet1/' /opt/stack/devstack/lib/neutron"

++ exit 1

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

 

Trying to see if I can fix it, Anyone has any idea what could have gone wrong?

 

Regards

Jaya

_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev


JayaPr
 

Hi Anil,

 

I am able to do git clone locally.

 

root@ubuntu:~# git clone https://git.openstack.org/openstack-dev/devstack --branch stable/queens

Cloning into 'devstack'...

remote: Enumerating objects: 43869, done.

remote: Counting objects: 100% (43869/43869), done.

remote: Compressing objects: 100% (12754/12754), done.

remote: Total 43869 (delta 31336), reused 42513 (delta 30435)

Receiving objects: 100% (43869/43869), 8.34 MiB | 2.56 MiB/s, done.

Resolving deltas: 100% (31336/31336), done.

Checking connectivity... done.

root@ubuntu:~#

 

Regards

Jaya

 

From: Anil Belur <abelur@...>
Sent: Thursday, July 25, 2019 3:05 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>
Cc: Faseela K <faseela.k@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: Re: [integration-dev] netvirt-csit gone all red

 

I think this might be an issue with the openstack infra itself, and not on our infra. I'm seeing the git clone issue locally.

 

Cloning into 'devstack'...
fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Failed to connect to opendev.org port 443: No route to host

 

- Anil 

 

On Thu, Jul 25, 2019 at 5:51 PM Anil Belur <abelur@...> wrote:

Greetings Jaya:

 

Can you raise an issue at https://support.linuxfoundation.org about it please?

 

Thanks,

Anil

 

On Thu, Jul 25, 2019 at 5:29 PM Jaya Priyadarshini <jaya.priyadarshini@...> wrote:

Anil,

 

Can u check this, this is seen in this job.

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/109/

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/111/control_1/get_devstack.sh.txt.gz

 

All jobs of this link has this issue.

 

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Empty reply from server

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

 

But I am able to access the same from all my local machines, only cloud computes are having this issue.

 

Regards

Jaya

From: integration-dev <integration-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 4:16 PM
To: Faseela K <faseela.k@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: Re: [integration-dev] netvirt-csit gone all red

 

Well, found the reason behind this:

 

git clone https://git.openstack.org/openstack-dev/devstack --branch stable/rocky

Cloning into 'devstack'...

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Encountered end of file

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.136.133

Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.136.133|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 2156 (2.1K) [text/plain]

 

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt by Sam is not working anymore.

And this problem is existing for all devstack branches , be it queens or rocky.

Will be fixing it.

 

Regards

Jaya

 

 

From: Faseela K <faseela.k@...>
Sent: Tuesday, July 23, 2019 3:08 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: RE: netvirt-csit gone all red

 

Hi everyone,

   Please stop triggering patch tests for netvirt until we fix the issue.

Thanks,

Faseela

 

From: netvirt-dev <netvirt-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 12:55 PM
To: netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: [netvirt-dev] netvirt-csit gone all red

 

Hi All,

 

Almost all last build have failed with below error in netvirt

 

+ ssh 10.30.170.83 'sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron'

Warning: Permanently added '10.30.170.83' (ECDSA) to the list of known hosts.

sed: can't read /opt/stack/devstack/lib/neutron: No such file or directory

++ trap_handler 833 15569

++ local prog=/tmp/jenkins3447639189848961768.sh

++ local lastline=833

++ local lasterr=15569

++ echo 'trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569'

trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569

++ echo 'trap_handler: command: ssh ${!CONTROLIP} "sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron"'

trap_handler: command: ssh ${!CONTROLIP} "sed -i 's/flat_networks public/flat_networks public,physnet1/' /opt/stack/devstack/lib/neutron"

++ exit 1

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

 

Trying to see if I can fix it, Anyone has any idea what could have gone wrong?

 

Regards

Jaya

_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev


Faseela K
 

Anil,

 

      Looks like the connectivity towards openstack infra from our CSIT VMs have some issues.

      When we try from our local servers, we are not hitting the issue.

      We have been blocked because of this for quite sometime now.

      How do we take this forward now?

     

      

Thanks,

Faseela

 

From: Jaya Priyadarshini <jaya.priyadarshini@...>
Sent: Thursday, July 25, 2019 3:27 PM
To: Anil Belur <abelur@...>
Cc: Faseela K <faseela.k@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: RE: [integration-dev] netvirt-csit gone all red

 

Hi Anil,

 

I am able to do git clone locally.

 

root@ubuntu:~# git clone https://git.openstack.org/openstack-dev/devstack --branch stable/queens

Cloning into 'devstack'...

remote: Enumerating objects: 43869, done.

remote: Counting objects: 100% (43869/43869), done.

remote: Compressing objects: 100% (12754/12754), done.

remote: Total 43869 (delta 31336), reused 42513 (delta 30435)

Receiving objects: 100% (43869/43869), 8.34 MiB | 2.56 MiB/s, done.

Resolving deltas: 100% (31336/31336), done.

Checking connectivity... done.

root@ubuntu:~#

 

Regards

Jaya

 

From: Anil Belur <abelur@...>
Sent: Thursday, July 25, 2019 3:05 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>
Cc: Faseela K <faseela.k@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: Re: [integration-dev] netvirt-csit gone all red

 

I think this might be an issue with the openstack infra itself, and not on our infra. I'm seeing the git clone issue locally.

 

Cloning into 'devstack'...
fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Failed to connect to opendev.org port 443: No route to host

 

- Anil 

 

On Thu, Jul 25, 2019 at 5:51 PM Anil Belur <abelur@...> wrote:

Greetings Jaya:

 

Can you raise an issue at https://support.linuxfoundation.org about it please?

 

Thanks,

Anil

 

On Thu, Jul 25, 2019 at 5:29 PM Jaya Priyadarshini <jaya.priyadarshini@...> wrote:

Anil,

 

Can u check this, this is seen in this job.

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/109/

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/111/control_1/get_devstack.sh.txt.gz

 

All jobs of this link has this issue.

 

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Empty reply from server

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

 

But I am able to access the same from all my local machines, only cloud computes are having this issue.

 

Regards

Jaya

From: integration-dev <integration-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 4:16 PM
To: Faseela K <faseela.k@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: Re: [integration-dev] netvirt-csit gone all red

 

Well, found the reason behind this:

 

git clone https://git.openstack.org/openstack-dev/devstack --branch stable/rocky

Cloning into 'devstack'...

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Encountered end of file

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.136.133

Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.136.133|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 2156 (2.1K) [text/plain]

 

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt by Sam is not working anymore.

And this problem is existing for all devstack branches , be it queens or rocky.

Will be fixing it.

 

Regards

Jaya

 

 

From: Faseela K <faseela.k@...>
Sent: Tuesday, July 23, 2019 3:08 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: RE: netvirt-csit gone all red

 

Hi everyone,

   Please stop triggering patch tests for netvirt until we fix the issue.

Thanks,

Faseela

 

From: netvirt-dev <netvirt-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 12:55 PM
To: netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: [netvirt-dev] netvirt-csit gone all red

 

Hi All,

 

Almost all last build have failed with below error in netvirt

 

+ ssh 10.30.170.83 'sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron'

Warning: Permanently added '10.30.170.83' (ECDSA) to the list of known hosts.

sed: can't read /opt/stack/devstack/lib/neutron: No such file or directory

++ trap_handler 833 15569

++ local prog=/tmp/jenkins3447639189848961768.sh

++ local lastline=833

++ local lasterr=15569

++ echo 'trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569'

trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569

++ echo 'trap_handler: command: ssh ${!CONTROLIP} "sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron"'

trap_handler: command: ssh ${!CONTROLIP} "sed -i 's/flat_networks public/flat_networks public,physnet1/' /opt/stack/devstack/lib/neutron"

++ exit 1

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

 

Trying to see if I can fix it, Anyone has any idea what could have gone wrong?

 

Regards

Jaya

_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev


JamO Luhrsen
 

Faseela, Jaya,

Are we only talking about a single problem now, or multiple?

The problem I replied to yesterday was a legit packaging version issue that we
had no control over and was fixed by openstack.

The problem in the link Jaya gave below is the failure to clone a devstack
repo. How often is that happening. I ran the clone in a loop on my laptop
for over 30m (so some hundreds of times) and I never got a failure. If it's
happening a lot in our infra (still) then we will need Anil's help to
figure it out.

Or are there other random problems?

Thanks,
JamO

On 7/25/19 4:51 AM, Faseela K wrote:

Anil,

 

      Looks like the connectivity towards openstack infra from our CSIT VMs have some issues.

      When we try from our local servers, we are not hitting the issue.

      We have been blocked because of this for quite sometime now.

      How do we take this forward now?

     

      

Thanks,

Faseela

 

From: Jaya Priyadarshini <jaya.priyadarshini@...>
Sent: Thursday, July 25, 2019 3:27 PM
To: Anil Belur <abelur@...>
Cc: Faseela K <faseela.k@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: RE: [integration-dev] netvirt-csit gone all red

 

Hi Anil,

 

I am able to do git clone locally.

 

root@ubuntu:~# git clone https://git.openstack.org/openstack-dev/devstack --branch stable/queens

Cloning into 'devstack'...

remote: Enumerating objects: 43869, done.

remote: Counting objects: 100% (43869/43869), done.

remote: Compressing objects: 100% (12754/12754), done.

remote: Total 43869 (delta 31336), reused 42513 (delta 30435)

Receiving objects: 100% (43869/43869), 8.34 MiB | 2.56 MiB/s, done.

Resolving deltas: 100% (31336/31336), done.

Checking connectivity... done.

root@ubuntu:~#

 

Regards

Jaya

 

From: Anil Belur <abelur@...>
Sent: Thursday, July 25, 2019 3:05 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>
Cc: Faseela K <faseela.k@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: Re: [integration-dev] netvirt-csit gone all red

 

I think this might be an issue with the openstack infra itself, and not on our infra. I'm seeing the git clone issue locally.

 

Cloning into 'devstack'...
fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Failed to connect to opendev.org port 443: No route to host

 

- Anil 

 

On Thu, Jul 25, 2019 at 5:51 PM Anil Belur <abelur@...> wrote:

Greetings Jaya:

 

Can you raise an issue at https://support.linuxfoundation.org about it please?

 

Thanks,

Anil

 

On Thu, Jul 25, 2019 at 5:29 PM Jaya Priyadarshini <jaya.priyadarshini@...> wrote:

Anil,

 

Can u check this, this is seen in this job.

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/109/

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/111/control_1/get_devstack.sh.txt.gz

 

All jobs of this link has this issue.

 

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Empty reply from server

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

 

But I am able to access the same from all my local machines, only cloud computes are having this issue.

 

Regards

Jaya

From: integration-dev <integration-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 4:16 PM
To: Faseela K <faseela.k@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: Re: [integration-dev] netvirt-csit gone all red

 

Well, found the reason behind this:

 

git clone https://git.openstack.org/openstack-dev/devstack --branch stable/rocky

Cloning into 'devstack'...

fatal: unable to access 'https://git.openstack.org/openstack-dev/devstack/': Encountered end of file

/tmp/get_devstack.sh: line 19: cd: devstack: No such file or directory

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.136.133

Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.136.133|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 2156 (2.1K) [text/plain]

 

--2019-07-23 00:58:49--  https://raw.githubusercontent.com/shague/odl_tools/master/fix-logging.patch.txt by Sam is not working anymore.

And this problem is existing for all devstack branches , be it queens or rocky.

Will be fixing it.

 

Regards

Jaya

 

 

From: Faseela K <faseela.k@...>
Sent: Tuesday, July 23, 2019 3:08 PM
To: Jaya Priyadarshini <jaya.priyadarshini@...>; netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: RE: netvirt-csit gone all red

 

Hi everyone,

   Please stop triggering patch tests for netvirt until we fix the issue.

Thanks,

Faseela

 

From: netvirt-dev <netvirt-dev-bounces@...> On Behalf Of Jaya Priyadarshini
Sent: Tuesday, July 23, 2019 12:55 PM
To: netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: [netvirt-dev] netvirt-csit gone all red

 

Hi All,

 

Almost all last build have failed with below error in netvirt

 

+ ssh 10.30.170.83 'sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron'

Warning: Permanently added '10.30.170.83' (ECDSA) to the list of known hosts.

sed: can't read /opt/stack/devstack/lib/neutron: No such file or directory

++ trap_handler 833 15569

++ local prog=/tmp/jenkins3447639189848961768.sh

++ local lastline=833

++ local lasterr=15569

++ echo 'trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569'

trap_handler: /tmp/jenkins3447639189848961768.sh: line 833: exit status of last command: 15569

++ echo 'trap_handler: command: ssh ${!CONTROLIP} "sed -i '\''s/flat_networks public/flat_networks public,physnet1/'\'' /opt/stack/devstack/lib/neutron"'

trap_handler: command: ssh ${!CONTROLIP} "sed -i 's/flat_networks public/flat_networks public,physnet1/' /opt/stack/devstack/lib/neutron"

++ exit 1

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

 

Trying to see if I can fix it, Anyone has any idea what could have gone wrong?

 

Regards

Jaya

_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev


_______________________________________________
integration-dev mailing list
integration-dev@...
https://lists.opendaylight.org/mailman/listinfo/integration-dev


Robert Varga
 

On 25/07/2019 11:34, Anil Belur wrote:
I think this might be an issue with the openstack infra itself, and not
on our infra. I'm seeing the git clone issue locally.
I have this:

$ git clone https://git.openstack.org/openstack-dev/devstack/
Cloning into 'devstack'...
warning: redirecting to https://opendev.org/openstack/devstack/
remote: Enumerating objects: 43869, done.
remote: Counting objects: 100% (43869/43869), done.
remote: Compressing objects: 100% (12752/12752), done.
remote: Total 43869 (delta 31336), reused 42516 (delta 30437


Cloning into 'devstack'...
fatal: unable to access
'https://git.openstack.org/openstack-dev/devstack/': Failed to connect
to opendev.org <http://opendev.org> port 443: No route to host
That does look like a routing problem. Since people can run the command
fine, it has to be something between our workers and the Internet...

Regards,
Robert


Anil Belur
 



On Fri, Jul 26, 2019 at 3:34 AM Robert Varga <nite@...> wrote:
On 25/07/2019 11:34, Anil Belur wrote:
> I think this might be an issue with the openstack infra itself, and not
> on our infra. I'm seeing the git clone issue locally.

I have this:

> $ git clone https://git.openstack.org/openstack-dev/devstack/
> Cloning into 'devstack'...
> warning: redirecting to https://opendev.org/openstack/devstack/
> remote: Enumerating objects: 43869, done.
> remote: Counting objects: 100% (43869/43869), done.
> remote: Compressing objects: 100% (12752/12752), done.
> remote: Total 43869 (delta 31336), reused 42516 (delta 30437


>
> Cloning into 'devstack'...
> fatal: unable to access
> 'https://git.openstack.org/openstack-dev/devstack/': Failed to connect
> to opendev.org <http://opendev.org> port 443: No route to host

That does look like a routing problem. Since people can run the command
fine, it has to be something between our workers and the Internet...


Hello,

I reached out to the cloud provider to make sure this is not an issue with the infra, 
this looks like an issue with git.openstack.org.


Thanks,
Anil


Faseela K
 

Thanks Anil!

From the link below, I learn that the issues with openstack infra are solved now.

We will run CSIT now and confirm!

 

Thanks,

Faseela

 

From: integration-dev <integration-dev-bounces@...> On Behalf Of Anil Belur
Sent: Friday, July 26, 2019 2:22 AM
To: Robert Varga <nite@...>
Cc: netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: Re: [integration-dev] netvirt-csit gone all red

 

 

 

On Fri, Jul 26, 2019 at 3:34 AM Robert Varga <nite@...> wrote:

On 25/07/2019 11:34, Anil Belur wrote:
> I think this might be an issue with the openstack infra itself, and not
> on our infra. I'm seeing the git clone issue locally.

I have this:

> $ git clone https://git.openstack.org/openstack-dev/devstack/
> Cloning into 'devstack'...
> warning: redirecting to https://opendev.org/openstack/devstack/
> remote: Enumerating objects: 43869, done.
> remote: Counting objects: 100% (43869/43869), done.
> remote: Compressing objects: 100% (12752/12752), done.
> remote: Total 43869 (delta 31336), reused 42516 (delta 30437


>
> Cloning into 'devstack'...
> fatal: unable to access
> 'https://git.openstack.org/openstack-dev/devstack/': Failed to connect
> to opendev.org <http://opendev.org> port 443: No route to host

That does look like a routing problem. Since people can run the command
fine, it has to be something between our workers and the Internet...

 

Hello,

 

I reached out to the cloud provider to make sure this is not an issue with the infra, 

this looks like an issue with git.openstack.org.

 

 

Thanks,

Anil


Faseela K
 

The infra issues are fixed now.

We had one more issue with the actual CSIT, since the yang cleanup patch got merged.

A change in odltools is needed for the same, that is being analyzed. [2]

Till that [1] should take care of the issue, and CSIT should run fine now for developers to resume their activities on Sodium.

 

Thanks,

Faseela

 

[1] https://git.opendaylight.org/gerrit/#/c/integration/test/+/83266/

[2] https://git.opendaylight.org/gerrit/#/c/odltools/+/83267/

 

From: Faseela K
Sent: Friday, July 26, 2019 11:08 AM
To: Anil Belur <abelur@...>; Robert Varga <nite@...>
Cc: netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: RE: [integration-dev] netvirt-csit gone all red

 

Thanks Anil!

From the link below, I learn that the issues with openstack infra are solved now.

We will run CSIT now and confirm!

 

Thanks,

Faseela

 

From: integration-dev <integration-dev-bounces@...> On Behalf Of Anil Belur
Sent: Friday, July 26, 2019 2:22 AM
To: Robert Varga <nite@...>
Cc: netvirt-dev@...; 'integration-dev@...' (integration-dev@...) (integration-dev@...) <integration-dev@...>
Subject: Re: [integration-dev] netvirt-csit gone all red

 

 

 

On Fri, Jul 26, 2019 at 3:34 AM Robert Varga <nite@...> wrote:

On 25/07/2019 11:34, Anil Belur wrote:
> I think this might be an issue with the openstack infra itself, and not
> on our infra. I'm seeing the git clone issue locally.

I have this:

> $ git clone https://git.openstack.org/openstack-dev/devstack/
> Cloning into 'devstack'...
> warning: redirecting to https://opendev.org/openstack/devstack/
> remote: Enumerating objects: 43869, done.
> remote: Counting objects: 100% (43869/43869), done.
> remote: Compressing objects: 100% (12752/12752), done.
> remote: Total 43869 (delta 31336), reused 42516 (delta 30437


>
> Cloning into 'devstack'...
> fatal: unable to access
> 'https://git.openstack.org/openstack-dev/devstack/': Failed to connect
> to opendev.org <http://opendev.org> port 443: No route to host

That does look like a routing problem. Since people can run the command
fine, it has to be something between our workers and the Internet...

 

Hello,

 

I reached out to the cloud provider to make sure this is not an issue with the infra, 

this looks like an issue with git.openstack.org.

 

 

Thanks,

Anil