canonical-ci-engineering team mailing list archive

[Łukasz 'sil2100' Zemczak <lukasz.zemczak@xxxxxxxxxxxxx>]

Hi Francis

On 28.04.2014 22:19, Francis Ginther wrote:
> Builds are now failing due to insufficient disk space [1]. I've
> determined that 60% of / is consumed by /var/cache/pbuilder/build [2]
> and most of those directories have older timestamps. My theory is that
> these were created by failed or aborted builds and were not cleaned
> up.

Argh, that again... Yeah, we had something similar on the previous
jenkins on canonistack. We had to clean up the pbuilder cache then as
well then through ssh, I can't remember if we thought up some long-term
remedy back then.

> I would like to request that any directory with a timestamp > 1 day be
> purged by IS. However, I don't want to request deletion from IS
> without someone to confirm this proposal. If someone with a little
> more ci-train knowledge can confirm, I'll send the rt request.
> 
> Long term I recommend a cleanup job to run at least daily to purge the
> old build dirs. It will need sudo permissions as these directories are
> owned by root.
> 
> For the short term, I recommend only executing one build at a time (at
> least through the cow build portion of the build).
> 
> [1] - https://ci-train.ubuntu.com/job/landing-002-1-build/41/console
>  - mktemp: failed to create directory via template
> '/tmp/debsign.XXXXXXXX': No space left on device
> [2] - https://ci-train.ubuntu.com/job/fginther-test/9/console
> 
> Francis
> 

I wouldn't try to make any changes without Didier around - he should be
back on Wednesday. Especially that we don't have ssh access to the
machine anymore so we can't really see how the directory structure looks
like there right now.

But let's talk tomorrow some more.

Best regards,

-- 
Łukasz 'sil2100' Zemczak
 lukasz.zemczak@xxxxxxxxxxxxx
 www.canonical.com