canonical-ci-engineering team mailing list archive

[Łukasz 'sil2100' Zemczak <lukasz.zemczak@xxxxxxxxxxxxx>]

Hi Timo,

Right, that wouldn't work in the current settings - I had the same
problem when trying to do a different job that required usage of sudo.
Currently the only thing we have the permissions to use sudo for is
executing cowbuilder.

I guess what we could do is ask for the same for at least pbuilder
execution. Would be nice to have the same for apt-get as well, but it
might be a security threat (?). At least I see no additional risk for
running pbuilder as sudo.

btw. I also think it would be nice if this machine could get more
storage than the one before. Since currently it only has downsides, it
seems we were better off on canonistack even...

On 29.04.2014 07:24, Timo Jyrinki wrote:
> Ok, that job doesn't/wouldn't work, since 'sudo' doesn't work from a
> job. It asks for a password and fails.
> 
> I tried modifying that job to do other cleaning, but I've not
> succeeded with any variation like su jenkins -c apt-get ... With the
> default permissions nothing sensible can be done.
> 
> Zoltan was blocked on his SDK building on this, so I went to #webops
> and luckily found pjdc active as there is no vanguard in my morning.
> He "twiddled the reserved block counts to match policy", and
> additionally ran both the commands I suggested and pbuilder --clean
> (since nothing was building). 5.2G free on / -> we should live another
> day (or week). All seems good currently.
> 
> -Timo
> 
> On Tue, Apr 29, 2014 at 7:04 AM, Timo Jyrinki
> <timo.jyrinki@xxxxxxxxxxxxx> wrote:
>> pbuilder --clean should work but it's indeed the biggest weapon
>> available. I'd rather go for removing the older directories based on
>> timestamps.
>>
>> Another place that will eat all the disk space eventually is
>> pbuilder's apt-cache, and I used the following to clean that the last
>> time:
>> mkdir /var/cache/pbuilder/trusty-amd64/aptcache/partial
>> apt-get -o Dir::Cache::archives=/var/cache/pbuilder/trusty-amd64/aptcache
>> autoclean
>>
>> -Timo
>>
>>
>> On Tue, Apr 29, 2014 at 12:44 AM, Francis Ginther
>> <francis.ginther@xxxxxxxxxxxxx> wrote:
>>> Mathieu suggested using 'pbuilder --clean' [1], which in my local
>>> testing appears to do the trick (and shouldn't require IS intervention
>>> if pbuilder is on the sudo list). The big thing to note is that you
>>> don't want to run this while a build is in progress (I don't think
>>> pbuilder checks for builds in progress).
>>>
>>> I've created a 'pbuilder-clean' job to perform this, but have set it
>>> to disabled and have not executed it. If you decide this is the right
>>> action, all you need to do is enable the job and run it.
>>>
>>> [1] - pbuilder man page:
>>>        --clean
>>>               Cleans  up  the  directory specified by the configuration BUILD‐
>>>               PLACE and APTCACHE specified in /etc/pbuilderrc
>>>
>>> Francis
>>>
>>> On Mon, Apr 28, 2014 at 4:23 PM, Łukasz 'sil2100' Zemczak
>>> <lukasz.zemczak@xxxxxxxxxxxxx> wrote:
>>>> Hi Francis
>>>>
>>>> On 28.04.2014 22:19, Francis Ginther wrote:
>>>>> Builds are now failing due to insufficient disk space [1]. I've
>>>>> determined that 60% of / is consumed by /var/cache/pbuilder/build [2]
>>>>> and most of those directories have older timestamps. My theory is that
>>>>> these were created by failed or aborted builds and were not cleaned
>>>>> up.
>>>>
>>>> Argh, that again... Yeah, we had something similar on the previous
>>>> jenkins on canonistack. We had to clean up the pbuilder cache then as
>>>> well then through ssh, I can't remember if we thought up some long-term
>>>> remedy back then.
>>>>
>>>>> I would like to request that any directory with a timestamp > 1 day be
>>>>> purged by IS. However, I don't want to request deletion from IS
>>>>> without someone to confirm this proposal. If someone with a little
>>>>> more ci-train knowledge can confirm, I'll send the rt request.
>>>>>
>>>>> Long term I recommend a cleanup job to run at least daily to purge the
>>>>> old build dirs. It will need sudo permissions as these directories are
>>>>> owned by root.
>>>>>
>>>>> For the short term, I recommend only executing one build at a time (at
>>>>> least through the cow build portion of the build).
>>>>>
>>>>> [1] - https://ci-train.ubuntu.com/job/landing-002-1-build/41/console
>>>>>  - mktemp: failed to create directory via template
>>>>> '/tmp/debsign.XXXXXXXX': No space left on device
>>>>> [2] - https://ci-train.ubuntu.com/job/fginther-test/9/console
>>>>>
>>>>> Francis
>>>>>
>>>>
>>>> I wouldn't try to make any changes without Didier around - he should be
>>>> back on Wednesday. Especially that we don't have ssh access to the
>>>> machine anymore so we can't really see how the directory structure looks
>>>> like there right now.
>>>>
>>>> But let's talk tomorrow some more.
>>>>
>>>> Best regards,
>>>>
>>>> --
>>>> Łukasz 'sil2100' Zemczak
>>>>  lukasz.zemczak@xxxxxxxxxxxxx
>>>>  www.canonical.com
>>>
>>>
>>>
>>> --
>>> Francis Ginther
>>> Canonical - Ubuntu Engineering - Continuous Integration Team


-- 
Łukasz 'sil2100' Zemczak
 lukasz.zemczak@xxxxxxxxxxxxx
 www.canonical.com