Re: [integration-dev] global-jjb vs. packer vs. Jenkins jobs


Anil Belur
 

Greetings Robert:

lf-env.sh: Creates a virtual env and sets up the environment, while the python-tools-install.sh Installs the python tools/utils during Job runtime. Since releng/global-jjb is a repo of Generic JJB templates (can be used by any of the CI management repositories), its up to the $project/$job to install the dependencies required for running the job.  

We have discussed this in the past, installing PyPI dependencies during packer image build time, comes with its own set of problems and added costs:
1. This requires maintaining a large number of packer images (if the project needs to support multiple versions of python/PyPI deps).
2. All releng/global-jjb (templates) scripts do not require all of the PyPi dependencies to be installed and are tied down to the $job or $project, since this approach binding them all into the same env has a risk of the deps being broken more frequently.
3. PyPi libs/modules are updated more frequently. 

Thanks,
Anil


On Mon, Jan 25, 2021 at 7:44 PM Robert Varga <nite@...> wrote:
Hello everyone,

as the (still current) failure to start Jenkins jobs shows, our current
way of integrating with external dependencies (global-jjb) is beyond
fragile.

The way our jobs work is that:

1) we have a base image, created by builder-packer-* jobs on a regular
basis and roll up distro upgrades plus some other things (like mininet,
etc.) that we need

2) the Jenkins job launches on that base image and call two scripts from
global-jjb, both of which end up installing more things:
   a) python-tools-install.sh
   b) lf-env.sh

3) the actual job runs

4) some more stuff invoking lf-env.sh to setup another Python
environment runs.

Now, it is clear that everything in 1) is invariant and updated in a
controlled way.

The problem is with 2), where again, everything is supposed to be
invariant for a particular version of global-jjb -- yet we reinstall
these things on every single job run.

Not only is this subject to random breakage (like now, or when pip
repositories are unavailable), etc.

It also takes around 3 minutes of each job execution, which does not
sound like much, but it is full 30%(!) of runtime of
yangtools-release-merge (which takes around 10 minutes).

We obviously can and must do better: global-jjb's environment-impacting
scripts must all be executed during builder-packer, so that they become
proper invariants.

For that, global-jjb needs to grow two things:

1) a way to install *all* of its dependencies without doing anything
else, for use in packer jobs

2) compatibility checks on the environment to ensure it is uptodate
enough to run a particular global-jjb version's scripts

With that, our jobs should be both faster and more reliable.

Does anybody see a problem why this would not work?

If not, I will be filing LFIT issues to get this done.

Regards,
Robert




Join TSC@lists.opendaylight.org to automatically receive all group messages.