Need help - CSIT Jobs is not creating logs
look at your console log:
https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/3/consoleFull
robot is indeed running. it's started at timestamp 21:27:11 and
lots of PASS test cases happen (150+) before jenkins kills it for
the job running past 6 hours at timestamp: 2:59:17.
I do see a recent job has passed though and robot tests took 4.5
hours to complete.
Something must be making the tests run slow enough in the aborted
jobs that we hit the 6 hour limit.
Originally, since this is a -gate- job I was thinking it could be
a bad netvirt patch with a bug creating slowness, but I think it's
happening periodically on the -upstream- job which means the
problem is not in the patch.
here's a build time trend page:
https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-sodium/buildTimeTrend
you can see it happened 4 times in the past 30 jobs.
some options:
- see if we can get the 6 hour timeout increased
- speed up the tests
- split the job up so the suites run in different jobs
hope it helps.
Thanks,
JamO
Hi Srinivas:
I can see that the build is timing out "10:59:17 Build timed out (after 360 minutes). Marking the build as failed."How long does this job take to complete? Can you please increase the timeout to check if it works?
Thanks,Anil
On Wed, Jan 29, 2020 at 4:43 PM <srinivas.rachakonda@...> wrote:
++
Hi Anil,
We are not able to see robot logs in sandbox for the below specific job.
Could please help us if it’s an infra issue.
Thanks,
Srinivas
+91-9243478719
From: R P Karthika . <r.p.karthika@...>
Sent: 29 January 2020 12:10
To: Srinivas Rachakonda <srinivas.rachakonda@...>; Yenuganti Vasudha <yenuganti.vasudha@...>
Cc: Karthikeyan Krishnan <karthikeyan.k@...>; P S Ravikanth <ravikanth.ps@...>; Hema Gopalkrishnan <hema.gopalkrishnan@...>; R Srinivasan E <r.e.srinivasan@...>
Subject: RE: Need help - CSIT Jobs is not creating logs
Hi Srinivas,
Robot logs are not getting generated because the job is failing in infra level itself. i.e., it still did not reach the position where robot testcases will start executing.
Regards,
R P Karthika
From: srinivas.rachakonda@... <srinivas.rachakonda@...>
Sent: Wednesday, January 29, 2020 10:07 AM
To: R P Karthika . <r.p.karthika@...>; Yenuganti Vasudha <yenuganti.vasudha@...>
Cc: Karthikeyan Krishnan <karthikeyan.k@...>; P S Ravikanth <ravikanth.ps@...>
Subject: Need help - CSIT Jobs is not creating logs
Hi Karthika,
Below jobs is not creating robot logs even though the scripts all have passed.:
Can you please look into it and let me know what could be the issue.
Thanks,
Srinivas
+91-9243478719
Hi Jamo/Anil,
Thanks for looking into it
I will be working on the below mentioned suggestions:
- see if we can get the 6 hour timeout increased
- speed up the tests
- split the job up so the suites run in different jobs
I will keep you updated on the working approach.
Thanks,
Srinivas
+91-9243478719
Sent: 31 January 2020 08:14
To: Anil Belur <abelur@...>; srinivas.rachakonda@...
Cc: R P Karthika . <r.p.karthika@...>; Yenuganti Vasudha <yenuganti.vasudha@...>; Karthikeyan Krishnan <karthikeyan.k@...>; P S Ravikanth <ravikanth.ps@...>; Hema Gopalkrishnan <hema.gopalkrishnan@...>; netvirt-dev <netvirt-dev@...>; R Srinivasan E <r.e.srinivasan@...>; integration-dev@...
Subject: Re: Need help - CSIT Jobs is not creating logs
Srinivas,
look at your console log:
https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-sodium/3/consoleFull
robot is indeed running. it's started at timestamp 21:27:11 and
lots of PASS test cases happen (150+) before jenkins kills it for
the job running past 6 hours at timestamp: 2:59:17.
I do see a recent job has passed though and robot tests took 4.5
hours to complete.
Something must be making the tests run slow enough in the aborted
jobs that we hit the 6 hour limit.
Originally, since this is a -gate- job I was thinking it could be
a bad netvirt patch with a bug creating slowness, but I think it's
happening periodically on the -upstream- job which means the
problem is not in the patch.
here's a build time trend page:
https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-sodium/buildTimeTrend
you can see it happened 4 times in the past 30 jobs.
some options:
- see if we can get the 6 hour timeout increased
- speed up the tests
- split the job up so the suites run in different jobs
hope it helps.
Thanks,
JamO
On 1/28/20 11:50 PM, Anil Belur wrote:
Hi Srinivas:
I can see that the build is timing out "10:59:17 Build timed out (after 360 minutes). Marking the build as failed."How long does this job take to complete? Can you please increase the timeout to check if it works?
Thanks,Anil
On Wed, Jan 29, 2020 at 4:43 PM <srinivas.rachakonda@...> wrote:
++
Hi Anil,
We are not able to see robot logs in sandbox for the below specific job.
Could please help us if it’s an infra issue.
Thanks,
Srinivas
+91-9243478719
From: R P Karthika . <r.p.karthika@...>
Sent: 29 January 2020 12:10
To: Srinivas Rachakonda <srinivas.rachakonda@...>; Yenuganti Vasudha <yenuganti.vasudha@...>
Cc: Karthikeyan Krishnan <karthikeyan.k@...>; P S Ravikanth <ravikanth.ps@...>; Hema Gopalkrishnan <hema.gopalkrishnan@...>; R Srinivasan E <r.e.srinivasan@...>
Subject: RE: Need help - CSIT Jobs is not creating logs
Hi Srinivas,
Robot logs are not getting generated because the job is failing in infra level itself. i.e., it still did not reach the position where robot testcases will start executing.
Regards,
R P Karthika
From: srinivas.rachakonda@... <srinivas.rachakonda@...>
Sent: Wednesday, January 29, 2020 10:07 AM
To: R P Karthika . <r.p.karthika@...>; Yenuganti Vasudha <yenuganti.vasudha@...>
Cc: Karthikeyan Krishnan <karthikeyan.k@...>; P S Ravikanth <ravikanth.ps@...>
Subject: Need help - CSIT Jobs is not creating logs
Hi Karthika,
Below jobs is not creating robot logs even though the scripts all have passed.:
Can you please look into it and let me know what could be the issue.
Thanks,
Srinivas
+91-9243478719