On 05/02/2021 10:16, Robert Varga wrote:

On 03/02/2021 00:03, Anil Belur wrote:

Greetings Robert:

Hello Anil,

Hello again,

sorry, for self-reply, but as it happens ...

[snip]

2. All releng/global-jjb (templates) scripts do not require all of the
PyPi dependencies to be installed and are tied down to the $job or
$project, since this approach binding them all into the same env has a
risk of the deps being broken more frequently.
3. PyPi libs/modules are updated more frequently.

While that is true, this line of reasoning completely ignores the
failure mode and recovery.

As it stands any of:
- busted global-jjb
- PyPi package updates
- PyPi repository unavailability

As we have seen in these past weeks, any such failure immediately
propagates to all jobs and breaks them -- resulting in nothing working
anymore, with no real avenue for recovery without help of LF IT.

... we just got hit by this.

[snip]

I am sorry, but I fail to see how Python packages special enough to inflict:
- breakages occurring at completely random times

A case in point:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/yangtools-maven-verify-master-mvn35-openjdk11/3761/console.log.gz
just failed with:

[yangtools-maven-verify-master-mvn35-openjdk11] $ /bin/bash /tmp/jenkins2774821607907773560.sh
---> python-tools-install.sh
Generating Requirements File
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 /home/jenkins/.local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpyskb3tqv
cwd: /tmp/pip-install-ibprnn2v/cryptography_c4b114d9e7a14d488215cb74c3a04315
Complete output (144 lines):

[...]

writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
running build_ext
generating cffi module 'build/temp.linux-x86_64-3.6/_padding.c'
creating build/temp.linux-x86_64-3.6
generating cffi module 'build/temp.linux-x86_64-3.6/_openssl.c'
running build_rust

=============================DEBUG ASSISTANCE=============================
If you are seeing a compilation error please try the following steps to
successfully install cryptography:
1) Upgrade to the latest pip and try again. This will fix errors for most
users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
2) Read https://cryptography.io/en/latest/installation.html for specific
instructions for your platform.
3) Check our frequently asked questions for more information:
https://cryptography.io/en/latest/faq.html
4) Ensure you have a recent Rust toolchain installed.
=============================DEBUG ASSISTANCE=============================

error: Can not find Rust compiler
----------------------------------------
ERROR: Failed building wheel for cryptography
ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly
Build step 'Execute shell' marked build as failure

python-tools-install.sh comes from global-jjb. This means jobs are
currently broken because of global-jjb stopped working.

I have filed
https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-21509
and we are blocked on it. Let's see what sort KPIs that issue will have.

Bye,
Robert