On 05/02/2021 10:16, Robert Varga wrote:
> On 03/02/2021 00:03, Anil Belur wrote:
>> Greetings Robert:
>
> Hello Anil,

Hello again,

sorry, for self-reply, but as it happens ...

[snip]

>> 2. All releng/global-jjb (templates) scripts do not require all of the
>> PyPi dependencies to be installed and are tied down to the $job or
>> $project, since this approach binding them all into the same env has a
>> risk of the deps being broken more frequently.
>> 3. PyPi libs/modules are updated more frequently.
>
> While that is true, this line of reasoning completely ignores the
> failure mode and recovery.
>
> As it stands any of:
> - busted global-jjb
> - PyPi package updates
> - PyPi repository unavailability
>
> As we have seen in these past weeks, any such failure immediately
> propagates to all jobs and breaks them -- resulting in nothing working
> anymore, with no real avenue for recovery without help of LF IT.

... we just got hit by this.

[snip]

>
> I am sorry, but I fail to see how Python packages special enough to inflict:
> - breakages occurring at completely random times

A case in point:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/yangtools-maven-verify-master-mvn35-openjdk11/3761/console.log.gz
just failed with:

> [yangtools-maven-verify-master-mvn35-openjdk11] $ /bin/bash /tmp/jenkins2774821607907773560.sh
> ---> python-tools-install.sh
> Generating Requirements File
>   ERROR: Command errored out with exit status 1:
>    command: /usr/bin/python3 /home/jenkins/.local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpyskb3tqv
>        cwd: /tmp/pip-install-ibprnn2v/cryptography_c4b114d9e7a14d488215cb74c3a04315
>   Complete output (144 lines):

[...]

>   writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
>   running build_ext
>   generating cffi module 'build/temp.linux-x86_64-3.6/_padding.c'
>   creating build/temp.linux-x86_64-3.6
>   generating cffi module 'build/temp.linux-x86_64-3.6/_openssl.c'
>   running build_rust
>   
>       =============================DEBUG ASSISTANCE=============================
>       If you are seeing a compilation error please try the following steps to
>       successfully install cryptography:
>       1) Upgrade to the latest pip and try again. This will fix errors for most
>          users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
>       2) Read https://cryptography.io/en/latest/installation.html for specific
>          instructions for your platform.
>       3) Check our frequently asked questions for more information:
>          https://cryptography.io/en/latest/faq.html
>       4) Ensure you have a recent Rust toolchain installed.
>       =============================DEBUG ASSISTANCE=============================
>   
>   error: Can not find Rust compiler
>   ----------------------------------------
>   ERROR: Failed building wheel for cryptography
> ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly
> Build step 'Execute shell' marked build as failure

python-tools-install.sh comes from global-jjb. This means jobs are
currently broken because of global-jjb stopped working.

I have filed
https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-21509
and we are blocked on it. Let's see what sort KPIs that issue will have.

Bye,
Robert