canonical-ci-engineering team mailing list archive

[Timo Jyrinki <timo.jyrinki@xxxxxxxxxxxxx>]

pbuilder --clean should work but it's indeed the biggest weapon
available. I'd rather go for removing the older directories based on
timestamps.

Another place that will eat all the disk space eventually is
pbuilder's apt-cache, and I used the following to clean that the last
time:
mkdir /var/cache/pbuilder/trusty-amd64/aptcache/partial
apt-get -o Dir::Cache::archives=/var/cache/pbuilder/trusty-amd64/aptcache
autoclean

-Timo


On Tue, Apr 29, 2014 at 12:44 AM, Francis Ginther
<francis.ginther@xxxxxxxxxxxxx> wrote:
> Mathieu suggested using 'pbuilder --clean' [1], which in my local
> testing appears to do the trick (and shouldn't require IS intervention
> if pbuilder is on the sudo list). The big thing to note is that you
> don't want to run this while a build is in progress (I don't think
> pbuilder checks for builds in progress).
>
> I've created a 'pbuilder-clean' job to perform this, but have set it
> to disabled and have not executed it. If you decide this is the right
> action, all you need to do is enable the job and run it.
>
> [1] - pbuilder man page:
>        --clean
>               Cleans  up  the  directory specified by the configuration BUILD‐
>               PLACE and APTCACHE specified in /etc/pbuilderrc
>
> Francis
>
> On Mon, Apr 28, 2014 at 4:23 PM, Łukasz 'sil2100' Zemczak
> <lukasz.zemczak@xxxxxxxxxxxxx> wrote:
>> Hi Francis
>>
>> On 28.04.2014 22:19, Francis Ginther wrote:
>>> Builds are now failing due to insufficient disk space [1]. I've
>>> determined that 60% of / is consumed by /var/cache/pbuilder/build [2]
>>> and most of those directories have older timestamps. My theory is that
>>> these were created by failed or aborted builds and were not cleaned
>>> up.
>>
>> Argh, that again... Yeah, we had something similar on the previous
>> jenkins on canonistack. We had to clean up the pbuilder cache then as
>> well then through ssh, I can't remember if we thought up some long-term
>> remedy back then.
>>
>>> I would like to request that any directory with a timestamp > 1 day be
>>> purged by IS. However, I don't want to request deletion from IS
>>> without someone to confirm this proposal. If someone with a little
>>> more ci-train knowledge can confirm, I'll send the rt request.
>>>
>>> Long term I recommend a cleanup job to run at least daily to purge the
>>> old build dirs. It will need sudo permissions as these directories are
>>> owned by root.
>>>
>>> For the short term, I recommend only executing one build at a time (at
>>> least through the cow build portion of the build).
>>>
>>> [1] - https://ci-train.ubuntu.com/job/landing-002-1-build/41/console
>>>  - mktemp: failed to create directory via template
>>> '/tmp/debsign.XXXXXXXX': No space left on device
>>> [2] - https://ci-train.ubuntu.com/job/fginther-test/9/console
>>>
>>> Francis
>>>
>>
>> I wouldn't try to make any changes without Didier around - he should be
>> back on Wednesday. Especially that we don't have ssh access to the
>> machine anymore so we can't really see how the directory structure looks
>> like there right now.
>>
>> But let's talk tomorrow some more.
>>
>> Best regards,
>>
>> --
>> Łukasz 'sil2100' Zemczak
>>  lukasz.zemczak@xxxxxxxxxxxxx
>>  www.canonical.com
>
>
>
> --
> Francis Ginther
> Canonical - Ubuntu Engineering - Continuous Integration Team