[release] [releng] releng/builder update and future changes to job templates

Thanh Ha thanh.ha at linuxfoundation.org
Tue Sep 20 00:29:25 UTC 2016

Hi Everyone,

Today we merged the final 2 patches "distribution" and "merge" jobs. All of
the releng templates are now freestyle jobs. We've noticed ever since we
started transitioning the Jenkins jobs to freestyle Jenkins reboot times
have decreased significantly from ~40+ minutes to < 5 minutes. I think
overall this has been a good change for our infra.


On Sat, Aug 13, 2016 at 4:03 PM, Thanh Ha <thanh.ha at linuxfoundation.org>

> Hi Everyone,
> Just wanted to send an update on this. We've now migrated the following
> job types to freestyle jobs:
> - clm
> - distribution-check
> - integration
> - periodic
> - validate-autorelease
> All cases we've found they run a bit faster as freestyle jobs vs the maven
> project type jobs, in some cases 5 - 10 minutes faster.
> At this point we only have following releng team managed jobs remaining to
> migrate:
> - distribution
> - merge
> - sonar
> I believe the distribution & merge jobs require us to change the Jenkins
> Deploy postbuild step as that's no longer a valid option in freestyle jobs.
> We'll likely have to configure a regular "mvn deploy" build step to push
> the artifacts at the end of a build.
> As for Sonar we need to confirm the freestyle version of the job works as
> expected before we can push. Anil is looking into confirming these
> remaining job types work as freestyle jobs before we start merging them.
> As for Logs we're working on building a new Jenkins plugin to inject the
> log location into the Gerrit Trigger build summary message so that at the
> end of a build you can link the direct logs.opendaylight.org URL
> conveniently into the build report comment that Gerrit Trigger produces so
> that it's all in one place. I hope to have this ready in the near future as
> it would be really handy to be able to link to the log server directly.
> Thanh
> On Thu, Jul 14, 2016 at 7:14 PM, Thanh Ha <thanh.ha at linuxfoundation.org>
> wrote:
>> Hi Everyone,
>> So for the past few weeks there's been quite a lot of changes to the JJB
>> management in releng/builder. The most significant one was merged this week
>> to completely get rid of our custom scripting layer on top of JJB so now we
>> are back on pure JJB as recent features added to JJB has eliminated the
>> need for us to have the extra scripting layer. This helps us simplify our
>> job management more and allows us to move forward with making a few more
>> big changes that we're planning to start rolling out in the coming weeks.
>> One of our plans is to finally allow projects to choose their build VM
>> size in cases where the project has good reason to require building on say
>> a 4 cpu system rather than our default 2 cpu systems that most jobs are
>> running against now. We'll be updating the templates to allow this in the
>> coming days.
>> We're also looking into larger changes discussed below.
>> *Logs*
>> The first one I want to mention is in regards to logs and archives. You
>> may have noticed that we now have a http://logs.opendaylight.org site
>> and many builds are setting their description to a link to where to find
>> the logs for that build. We plan to continue to migrate job archives out of
>> Jenkins and into this new logs.opendaylight.org. A few points about the
>> new service:
>>     * Plan is to make Jenkins only keep record of 7 days worth of job
>> data in Jenkins
>>     * Longer term archives in logs.opendaylight.org
>>     * Remove the archives publisher from all jobs in Jenkins in favour of
>> the point above
>>     * releng silo logs will be stored in logs.opendaylight.org for 6
>> months
>>     * sandbox silo logs will be stored in logs.opendaylight.org for 7
>> days
>> *Jenkins Job Builder Templates*
>> We are looking to replace all "matrix" and "maven project" job types and
>> replace them with Jenkins freestyle jobs as we keep running into issues
>> caused by these 2 job types and believe switching to the regular freestyle
>> job types will be better in the long term.
>> In the case of Matrix, the matrix master doesn't queue jobs but instead
>> launches all jobs as soon as they are queued so causes issues such as today
>> when we had a large job queue and an overloaded matrix master that crashed
>> Jenkins and caused most jobs to fail with "Too many open files".
>> In the case of the Maven Project type, we understand that many folks find
>> the module listing on the build page useful to identify where a build
>> failed but the issues we have on the server side is not worth maintaining
>> this job type. Some issues we identified are:
>> * It's a re-implementation of Maven's runtime to inject the event spy
>> code to enable module listing. But since it's not actively maintained it's
>> not keeping up to features available in the official Maven Binary and handy
>> things like parallel builds are broken.
>> * Fingerprint generation is slow and it seems to fingerprint every
>> artifact that Maven produces even if you don't  want to archive them in the
>> build. Fingerprints also cause our Jenkins Master to take a very long time
>> to boot, with no fingerprints in the system Jenkins Master can boot in
>> about 2 minutes but as more and more jobs store fingerprint data Jenkins
>> Master can take 10s of minutes to boot.
>> * Archives all Maven artifacts regardless of if you need it to archive
>> them or not in your build. For example periodic jobs that don't need to
>> archive them. If you ever wondered why when a build is nearly done Jenkins
>> tends to hang for a seemingly long while. It's because of this archiving
>> and it's copying files to the Jenkins master that you may not have asked it
>> to archive.
>> We plan on rolling out these changes over the next few weeks. We will try
>> to maintain build history if possible but in some cases the history might
>> not be kept. If the jobs push it's data to logs.opendaylight.org though
>> the data available there won't be cleared.
>> Let us know if you have any questions.
>> Regards,
>> Thanh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opendaylight.org/pipermail/release/attachments/20160919/d8183a64/attachment.html>

More information about the release mailing list