[release] [it-infrastructure-alerts][notice] ODL Jenkins maintenance window - 1600-1800 PST, Mar 01, 2020.


Anil Belur
 

With the Jenkins plugin updates on production, we are seeing some dependencies errors between plugins is causing Jenkins to throw up errors on boot.
Working on resolving the dependencies ATM.

Dependency errors:
Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins.
Pipeline: Groovy version 2.80
Script Security Plugin version 1.58 is older than required. To fix, install version 1.63 or later.
Command Agent Launcher Plugin version 1.4
Script Security Plugin version 1.58 is older than required. To fix, install version 1.68 or later.
Jenkins Git plugin version 4.1.1
Script Security Plugin version 1.58 is older than required. To fix, install version 1.66 or later.
JMS Messaging Plugin version 1.1.13
Script Security Plugin version 1.58 is older than required. To fix, install version 1.68 or later.
Downstream dependency errors:
These plugins failed to load because of one or more of the errors above. Fix those and these plugins will load again.
Pipeline: Declarative Extension Points API version 1.5.1
Pipeline: Groovy version 2.80 failed to load. Fix this plugin first.
Pipeline Graph Analysis Plugin version 1.10
Pipeline: Groovy version 2.80 failed to load. Fix this plugin first.
Pipeline version 2.6
Pipeline: Stage View Plugin version 2.13 failed to load. Fix this plugin first.
Pipeline: Stage View Plugin version 2.13
Pipeline: REST API Plugin version 2.13 failed to load. Fix this plugin first.
Forensics API Plugin version 0.7.0
Pipeline: Groovy version 2.80 failed to load. Fix this plugin first.
Maven Cascade Release Plugin version 1.3.2
Jenkins Git plugin version 4.1.1 failed to load. Fix this plugin first.
Pipeline: Declarative version 1.5.1
Pipeline: Shared Groovy Libraries version 2.15 failed to load. Fix this plugin first.
Pipeline: Shared Groovy Libraries version 2.15
Pipeline: Groovy version 2.80 failed to load. Fix this plugin first.
Warnings Next Generation Plugin version 8.0.0
Forensics API Plugin version 0.7.0 failed to load. Fix this plugin first.
Pipeline: Multibranch version 2.21
Pipeline: Groovy version 2.80 failed to load. Fix this plugin first.
Gradle Plugin version 1.36
Pipeline: Groovy version 2.80 failed to load. Fix this plugin first.
Docker Pipeline version 1.22
Pipeline: Groovy version 2.80 failed to load. Fix this plugin first.
Pipeline: REST API Plugin version 2.13
Pipeline Graph Analysis Plugin version 1.10 failed to load. Fix this plugin first.
Pipeline: Declarative Agent API version 1.1.1
Pipeline: Declarative Extension Points API version 1.5.1 failed to load. Fix this plugin first.


On Thu, Feb 27, 2020 at 1:35 PM Anil Belur via Lists.Opendaylight.Org <abelur=linuxfoundation.org@...> wrote:
What: LF will perform maintenance on ODL Jenkins servers to upgrade the plugins to the latest.

When: 1600-1800 PST, Sunday, Mar 01, 2020

Why: The latest OpenStack API requires OpenStack-clouds plugin updated and also many other plugins outdated and require to be updated. 
 
Impact: All ODL Jenkins (sandbox/releng) may be inaccessible during this time and
Jenkins will be put in shutdown mode before the window starts and any long-running Jenkins jobs _will_ be canceled if they don't complete before the start of the window.

Notices will be posted to the mailing lists and in #opendaylight on Freenode at the start and end of the maintenance.

Thanks,
Anil Belur


Anil Belur
 

Hello all,

The maintenance work is now complete. We've had to copy all plugins from sandbox over to releng, since resolving the dependencies manually is not straight forward and Jenkins plugin management does not handle dependencies as well as packaging. We have an existing backup of the plugin directory. Going forward, if we have any missing plugins on the production/releng that will need to be installed.

Note: we are seeing a metadata issue on the cloud, when the nodes are coming online, which is causing the Jenkins jobs to fail. I'm working with the cloud provider to get this resolved. 

Apologies for any inconvenience. 

Thanks,
Anil

On Thu, Feb 27, 2020 at 1:35 PM Anil Belur via Lists.Opendaylight.Org <abelur=linuxfoundation.org@...> wrote:
What: LF will perform maintenance on ODL Jenkins servers to upgrade the plugins to the latest.

When: 1600-1800 PST, Sunday, Mar 01, 2020

Why: The latest OpenStack API requires OpenStack-clouds plugin updated and also many other plugins outdated and require to be updated. 
 
Impact: All ODL Jenkins (sandbox/releng) may be inaccessible during this time and
Jenkins will be put in shutdown mode before the window starts and any long-running Jenkins jobs _will_ be canceled if they don't complete before the start of the window.

Notices will be posted to the mailing lists and in #opendaylight on Freenode at the start and end of the maintenance.

Thanks,
Anil Belur


Abhijit Kumbhare
 

Thanks for the update. Hopefully gets resolved soon.

On Mon, Mar 2, 2020 at 11:14 PM Anil Belur <abelur@...> wrote:
Hello all,

The maintenance work is now complete. We've had to copy all plugins from sandbox over to releng, since resolving the dependencies manually is not straight forward and Jenkins plugin management does not handle dependencies as well as packaging. We have an existing backup of the plugin directory. Going forward, if we have any missing plugins on the production/releng that will need to be installed.

Note: we are seeing a metadata issue on the cloud, when the nodes are coming online, which is causing the Jenkins jobs to fail. I'm working with the cloud provider to get this resolved. 

Apologies for any inconvenience. 

Thanks,
Anil

On Thu, Feb 27, 2020 at 1:35 PM Anil Belur via Lists.Opendaylight.Org <abelur=linuxfoundation.org@...> wrote:
What: LF will perform maintenance on ODL Jenkins servers to upgrade the plugins to the latest.

When: 1600-1800 PST, Sunday, Mar 01, 2020

Why: The latest OpenStack API requires OpenStack-clouds plugin updated and also many other plugins outdated and require to be updated. 
 
Impact: All ODL Jenkins (sandbox/releng) may be inaccessible during this time and
Jenkins will be put in shutdown mode before the window starts and any long-running Jenkins jobs _will_ be canceled if they don't complete before the start of the window.

Notices will be posted to the mailing lists and in #opendaylight on Freenode at the start and end of the maintenance.

Thanks,
Anil Belur



Andrew Grimberg
 

We're going to be taking another brief outage on the Jenkins node.

While we got things up and working again it all came to a halt when the
regularly schedule ci-management-cfg-merge job fired and wiped out the
cloud configuration.

It looks like the version of the OpenStack plugin we're running is not
compatible with the latest version of global-jjb. We're going to change
the plugin to the version that I just verified works on the ODL sandbox.

-Andy-

On 2020-03-02 23:55, Abhijit Kumbhare wrote:
Thanks for the update. Hopefully gets resolved soon.

On Mon, Mar 2, 2020 at 11:14 PM Anil Belur <abelur@...
<mailto:abelur@...>> wrote:

Hello all,

The maintenance work is now complete. We've had to copy all plugins
from sandbox over to releng, since resolving the dependencies
manually is not straight forward and Jenkins plugin management does
not handle dependencies as well as packaging. We have an existing
backup of the plugin directory. Going forward, if we have any
missing plugins on the production/releng that will need to be installed.

Note: we are seeing a metadata issue on the cloud, when the nodes
are coming online, which is causing the Jenkins jobs to fail. I'm
working with the cloud provider to get this resolved. 

Apologies for any inconvenience. 

Thanks,
Anil

On Thu, Feb 27, 2020 at 1:35 PM Anil Belur via
Lists.Opendaylight.Org <http://Lists.Opendaylight.Org>
<abelur=linuxfoundation.org@...
<mailto:linuxfoundation.org@...>> wrote:

What: LF will perform maintenance on ODL Jenkins servers to
upgrade the plugins to the latest.

When: 1600-1800 PST, Sunday, Mar 01, 2020

Why: The latest OpenStack API requires OpenStack-clouds plugin
updated and also many other plugins outdated and require to be
updated. 
 
Impact: All ODL Jenkins (sandbox/releng) may be inaccessible
during this time and
Jenkins will be put in shutdown mode before the window starts
and any long-running Jenkins jobs _will_ be canceled if they
don't complete before the start of the window.

Notices will be posted to the mailing lists and in #opendaylight
on Freenode at the start and end of the maintenance.

Thanks,
Anil Belur




--
Andrew J Grimberg
Manager Release Engineering
The Linux Foundation