Dug into this a bit more. I think the latest change in lftools (#63173) for copy archives that use copy2 might be causing an issue with copying all the files/subfolders in the deploy log subcommand. With the latest release of lftools v0.31.0 all files in the archives are not being copied which is impacting the CSIT jobs. I've not had enough time to test my theory.

However from the job logs the before the change in question got released the deploy logs worked as expected (https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-tell-all-sodium/499/) and after v0.31.0 was released, deploy logs has some of the required files missing (https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-tell-all-sodium/500).
Cheers,Anil

On Tue, Mar 3, 2020 at 11:50 AM JamO Luhrsen <jluhrsen@...> wrote:

Looks like our CSIT jobs are not collecting and archiving the
robot or controller logs. I have not looked any deeper than
noticing it, at this point.

This will be a release blocker until we can get it fixed. I'll
try to find some time to look, but wanted to get this out as
an FYI.

Thanks,
JamO

https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-19138

If you hack a jenkins job config to install lftools==0.30.1,
things will work. I have done that in the sandbox.

I was trying to find a hack that I could push in releng/builder
as a workaround:

https://git.opendaylight.org/gerrit/c/releng/builder/+/88227


Unfortunately, it chokes at the end with a 401 error. Not sure how to get around
that so stuck manually configuring sandbox jobs to keep working.

JamO

On 3/3/20 5:22 AM, Anil Belur wrote:

Dug into this a bit more. I think the latest change in lftools (#63173) for copy archives that use copy2 might be causing an issue with copying all the files/subfolders in the deploy log subcommand. With the latest release of lftools v0.31.0 all files in the archives are not being copied which is impacting the CSIT jobs. I've not had enough time to test my theory.

However from the job logs the before the change in question got released the deploy logs worked as expected (https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-tell-all-sodium/499/) and after v0.31.0 was released, deploy logs has some of the required files missing (https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-tell-all-sodium/500).
Cheers,Anil

On Tue, Mar 3, 2020 at 11:50 AM JamO Luhrsen <jluhrsen@...> wrote:

Looks like our CSIT jobs are not collecting and archiving the
robot or controller logs. I have not looked any deeper than
noticing it, at this point.

This will be a release blocker until we can get it fixed. I'll
try to find some time to look, but wanted to get this out as
an FYI.

Thanks,
JamO

Jamo, We have reverted the change in lftools and release v0.31.1, now the logs are getting upload as before. 

Thanks,

Anil

On Wed, Mar 4, 2020 at 6:44 AM Jamo Luhrsen <jluhrsen@...> wrote:

Oh, and global-jjb patch to pin, but that will require another
global-jjb release and update in releng/builder:

https://gerrit.linuxfoundation.org/infra/c/releng/global-jjb/+/63268

On 3/3/20 12:35 PM, JamO Luhrsen via Lists.Opendaylight.Org wrote:

https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-19138

If you hack a jenkins job config to install lftools==0.30.1,
things will work. I have done that in the sandbox.

I was trying to find a hack that I could push in releng/builder
as a workaround:

https://git.opendaylight.org/gerrit/c/releng/builder/+/88227


Unfortunately, it chokes at the end with a 401 error. Not sure how to get around
that so stuck manually configuring sandbox jobs to keep working.

JamO

On 3/3/20 5:22 AM, Anil Belur wrote:

Dug into this a bit more. I think the latest change in lftools (#63173) for copy archives that use copy2 might be causing an issue with copying all the files/subfolders in the deploy log subcommand. With the latest release of lftools v0.31.0 all files in the archives are not being copied which is impacting the CSIT jobs. I've not had enough time to test my theory.

However from the job logs the before the change in question got released the deploy logs worked as expected (https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-tell-all-sodium/499/) and after v0.31.0 was released, deploy logs has some of the required files missing (https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-tell-all-sodium/500).
Cheers,Anil

On Tue, Mar 3, 2020 at 11:50 AM JamO Luhrsen <jluhrsen@...> wrote:

Looks like our CSIT jobs are not collecting and archiving the
robot or controller logs. I have not looked any deeper than
noticing it, at this point.

This will be a release blocker until we can get it fixed. I'll
try to find some time to look, but wanted to get this out as
an FYI.

Thanks,
JamO