[OpenDaylight Infrastructure] [release] Autorelease carbon failed to build packetcable-policy-karaf from packetcable


Thanh Ha <thanh.ha@...>
 

On Thu, Apr 5, 2018 at 6:55 AM, Robert Varga <nite@...> wrote:
On 05/04/18 11:26, Stephen Kitt wrote:
> On Thu, 5 Apr 2018 19:16:08 +1000
> Anil Belur <abelur@...> wrote:
>>>> According to
>>>> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/
>>> autorelease-release-carbon/552/_sys-info.log.gz
>>>>
>>>> we are maxing out the 150 GB disk. Did something change recently
>>>> that is causing significantly more logging to take place?
>>>
>>> This might be related to https://git.opendaylight.org/gerrit/69529
>>> — we keep Pax Exam runtime trees so we can extract the Karaf logs...
>>
>> I see we may only be copying the karaf logs and not runtime generated
>> files, while these logs are preserved into `$WORKSPACE/archives` and
>> published into log server.
>> It would be good to remove Pax runtime trees/files after the logs are
>> copied.
>
> I don’t think that would be sufficient. We’d really need to move the
> Karaf logs out of the way and delete Pax runtime trees during the
> build.

Or perhaps have SFT configure karaf to log outside of the pax tree and
not keep it?


Okay I think this confirms the culprit. I used ncdu on a running build system after only 3 hrs of a carbon build and can confirm it's the pax stuff that's taking all the disk.

Here's a small snippet of 3 pax directories for controller, alto, and mdsal:

controller/target/pax
    2.4 GiB [##########] /pax


alto/target/pax
    4.4 GiB [##########] /pax                                                                                                                                     

mdsal/target/pax
    3.1 GiB [##########] /pax          


As far as I can tell anything that is a feature has one of these directories and they are always the largest directory in every project. I guess next steps would be to remove these directories after grabbing the log? Where does this need to be done?

Regards,
Thanh


Stephen Kitt <skitt@...>
 

On Thu, 5 Apr 2018 23:33:37 -0400
Thanh Ha <thanh.ha@...> wrote:
As far as I can tell anything that is a feature has one of these
directories and they are always the largest directory in every
project. I guess next steps would be to remove these directories
after grabbing the log? Where does this need to be done?
https://git.opendaylight.org/gerrit/70415

Regards,

Stephen