canonical-ci-engineering team mailing list archive

[Francis Ginther <francis.ginther@xxxxxxxxxxxxx>]

Mathieu suggested using 'pbuilder --clean' [1], which in my local
testing appears to do the trick (and shouldn't require IS intervention
if pbuilder is on the sudo list). The big thing to note is that you
don't want to run this while a build is in progress (I don't think
pbuilder checks for builds in progress).

I've created a 'pbuilder-clean' job to perform this, but have set it
to disabled and have not executed it. If you decide this is the right
action, all you need to do is enable the job and run it.

[1] - pbuilder man page:
       --clean
              Cleans  up  the  directory specified by the configuration BUILD‐
              PLACE and APTCACHE specified in /etc/pbuilderrc

Francis

On Mon, Apr 28, 2014 at 4:23 PM, Łukasz 'sil2100' Zemczak
<lukasz.zemczak@xxxxxxxxxxxxx> wrote:
> Hi Francis
>
> On 28.04.2014 22:19, Francis Ginther wrote:
>> Builds are now failing due to insufficient disk space [1]. I've
>> determined that 60% of / is consumed by /var/cache/pbuilder/build [2]
>> and most of those directories have older timestamps. My theory is that
>> these were created by failed or aborted builds and were not cleaned
>> up.
>
> Argh, that again... Yeah, we had something similar on the previous
> jenkins on canonistack. We had to clean up the pbuilder cache then as
> well then through ssh, I can't remember if we thought up some long-term
> remedy back then.
>
>> I would like to request that any directory with a timestamp > 1 day be
>> purged by IS. However, I don't want to request deletion from IS
>> without someone to confirm this proposal. If someone with a little
>> more ci-train knowledge can confirm, I'll send the rt request.
>>
>> Long term I recommend a cleanup job to run at least daily to purge the
>> old build dirs. It will need sudo permissions as these directories are
>> owned by root.
>>
>> For the short term, I recommend only executing one build at a time (at
>> least through the cow build portion of the build).
>>
>> [1] - https://ci-train.ubuntu.com/job/landing-002-1-build/41/console
>>  - mktemp: failed to create directory via template
>> '/tmp/debsign.XXXXXXXX': No space left on device
>> [2] - https://ci-train.ubuntu.com/job/fginther-test/9/console
>>
>> Francis
>>
>
> I wouldn't try to make any changes without Didier around - he should be
> back on Wednesday. Especially that we don't have ssh access to the
> machine anymore so we can't really see how the directory structure looks
> like there right now.
>
> But let's talk tomorrow some more.
>
> Best regards,
>
> --
> Łukasz 'sil2100' Zemczak
>  lukasz.zemczak@xxxxxxxxxxxxx
>  www.canonical.com



-- 
Francis Ginther
Canonical - Ubuntu Engineering - Continuous Integration Team