Need help - CSIT Jobs is not creating logs


JamO Luhrsen
 

Srinivas,

look at your console log:

https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/3/consoleFull

robot is indeed running. it's started at timestamp 21:27:11 and
lots of PASS test cases happen (150+) before jenkins kills it for
the job running past 6 hours at timestamp: 2:59:17.

I do see a recent job has passed though and robot tests took 4.5
hours to complete.

Something must be making the tests run slow enough in the aborted
jobs that we hit the 6 hour limit.

Originally, since this is a -gate- job I was thinking it could be
a bad netvirt patch with a bug creating slowness, but I think it's
happening periodically on the -upstream- job which means the
problem is not in the patch.

here's a build time trend page:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-sodium/buildTimeTrend

you can see it happened 4 times in the past 30 jobs.

some options:

  - see if we can get the 6 hour timeout increased
  - speed up the tests
  - split the job up so the suites run in different jobs


hope it helps.

Thanks,
JamO





On 1/28/20 11:50 PM, Anil Belur wrote:

Hi Srinivas: 
I can see that the build is timing out "10:59:17 Build timed out (after 360 minutes). Marking the build as failed."How long does this job take to complete? Can you please increase the timeout to check if it works?
Thanks,Anil

On Wed, Jan 29, 2020 at 4:43 PM <srinivas.rachakonda@...> wrote:

++

Hi Anil,

 

We are not able to see robot logs in sandbox for the below specific job.

https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/3/

 

Could please help us if it’s an infra issue.

 

Thanks,

Srinivas

+91-9243478719

 

From: R P Karthika . <r.p.karthika@...>
Sent: 29 January 2020 12:10
To: Srinivas Rachakonda <srinivas.rachakonda@...>; Yenuganti Vasudha <yenuganti.vasudha@...>
Cc: Karthikeyan Krishnan <karthikeyan.k@...>; P S Ravikanth <ravikanth.ps@...>; Hema Gopalkrishnan <hema.gopalkrishnan@...>; R Srinivasan E <r.e.srinivasan@...>
Subject: RE: Need help - CSIT Jobs is not creating logs

 

Hi Srinivas,

Robot logs are not getting generated because the job is failing in infra level itself. i.e., it still did not reach the position where robot testcases will start executing.

 

Regards,

R P Karthika

From: srinivas.rachakonda@... <srinivas.rachakonda@...>
Sent: Wednesday, January 29, 2020 10:07 AM
To: R P Karthika . <r.p.karthika@...>; Yenuganti Vasudha <yenuganti.vasudha@...>
Cc: Karthikeyan Krishnan <karthikeyan.k@...>; P S Ravikanth <ravikanth.ps@...>
Subject: Need help - CSIT Jobs is not creating logs

 

Hi Karthika,

 

Below jobs is not creating robot logs even though the scripts all have passed.:

https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/3/

 

Can you please look into it and let me know what could be the issue.

 

Thanks,

Srinivas

+91-9243478719

 



Srinivas <srinivas.rachakonda@...>
 

Hi Jamo/Anil,

 

Thanks for looking into it

I will be working on the below mentioned suggestions:

 

  - see if we can get the 6 hour timeout increased
  - speed up the tests
  - split the job up so the suites run in different jobs

 

I will keep you updated on the working approach.

 

Thanks,

Srinivas

+91-9243478719

 

From: Jamo Luhrsen <jluhrsen@...>
Sent: 31 January 2020 08:14
To: Anil Belur <abelur@...>; srinivas.rachakonda@...
Cc: R P Karthika . <r.p.karthika@...>; Yenuganti Vasudha <yenuganti.vasudha@...>; Karthikeyan Krishnan <karthikeyan.k@...>; P S Ravikanth <ravikanth.ps@...>; Hema Gopalkrishnan <hema.gopalkrishnan@...>; netvirt-dev <netvirt-dev@...>; R Srinivasan E <r.e.srinivasan@...>; integration-dev@...
Subject: Re: Need help - CSIT Jobs is not creating logs

 

Srinivas,

look at your console log:

https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/3/consoleFull

robot is indeed running. it's started at timestamp 21:27:11 and
lots of PASS test cases happen (150+) before jenkins kills it for
the job running past 6 hours at timestamp: 2:59:17.

I do see a recent job has passed though and robot tests took 4.5
hours to complete.

Something must be making the tests run slow enough in the aborted
jobs that we hit the 6 hour limit.

Originally, since this is a -gate- job I was thinking it could be
a bad netvirt patch with a bug creating slowness, but I think it's
happening periodically on the -upstream- job which means the
problem is not in the patch.

here's a build time trend page:

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-sodium/buildTimeTrend

you can see it happened 4 times in the past 30 jobs.

some options:

  - see if we can get the 6 hour timeout increased
  - speed up the tests
  - split the job up so the suites run in different jobs


hope it helps.

Thanks,
JamO




On 1/28/20 11:50 PM, Anil Belur wrote:

Hi Srinivas: 
I can see that the build is timing out "10:59:17 Build timed out (after 360 minutes). Marking the build as failed."How long does this job take to complete? Can you please increase the timeout to check if it works?
Thanks,Anil

 

On Wed, Jan 29, 2020 at 4:43 PM <srinivas.rachakonda@...> wrote:

++

Hi Anil,

 

We are not able to see robot logs in sandbox for the below specific job.

https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/3/

 

Could please help us if it’s an infra issue.

 

Thanks,

Srinivas

+91-9243478719

 

From: R P Karthika . <r.p.karthika@...>
Sent: 29 January 2020 12:10
To: Srinivas Rachakonda <srinivas.rachakonda@...>; Yenuganti Vasudha <yenuganti.vasudha@...>
Cc: Karthikeyan Krishnan <karthikeyan.k@...>; P S Ravikanth <ravikanth.ps@...>; Hema Gopalkrishnan <hema.gopalkrishnan@...>; R Srinivasan E <r.e.srinivasan@...>
Subject: RE: Need help - CSIT Jobs is not creating logs

 

Hi Srinivas,

Robot logs are not getting generated because the job is failing in infra level itself. i.e., it still did not reach the position where robot testcases will start executing.

 

Regards,

R P Karthika

From: srinivas.rachakonda@... <srinivas.rachakonda@...>
Sent: Wednesday, January 29, 2020 10:07 AM
To: R P Karthika . <r.p.karthika@...>; Yenuganti Vasudha <yenuganti.vasudha@...>
Cc: Karthikeyan Krishnan <karthikeyan.k@...>; P S Ravikanth <ravikanth.ps@...>
Subject: Need help - CSIT Jobs is not creating logs

 

Hi Karthika,

 

Below jobs is not creating robot logs even though the scripts all have passed.:

https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/3/

 

Can you please look into it and let me know what could be the issue.

 

Thanks,

Srinivas

+91-9243478719