← Back to team overview

canonical-ci-engineering team mailing list archive

Re: Autorestarting jenkins slaves

 

I second the motion for upstart jobs for each individual node. I also
would't mind seeing us get away from using SSH to restart remote nodes
since that will allow us to eliminate another plugin (or three).

~w

On 12/09/2013 09:21 AM, Francis Ginther wrote:
> Vila,
>
> My recommendation is to deprecate /usr/local/bin/start-jenkins-slaves
> and rely on individual upstart jobs, one for each slave. This is
> mainly a personal preference to handle all our slaves that use jnlp to
> do so with the same method. It also provides some logging for free.
>
> Francis
>
> On Mon, Dec 9, 2013 at 7:38 AM, Vincent Ladeuil <vila+ci@xxxxxxxxxxxxx> wrote:
>> Hi,
>>
>> I've discussed that with jamespage and came up with the following
>> workaround:
>>
>>    modified      debian/jenkins-slave.upstart
>>
>>
>> === modified file 'debian/jenkins-slave.upstart'
>> --- debian/jenkins-slave.upstart        2013-02-17 17:11:13 +0000
>> +++ debian/jenkins-slave.upstart        2013-12-09 10:29:01 +0000
>> @@ -17,3 +17,6 @@
>>      exec start-stop-daemon --start -c $JENKINS_USER --exec $JAVA --name jenkins-slave \
>>          -- $JAVA_ARGS -jar $JENKINS_RUN/slave.jar $JENKINS_ARGS
>>  end script
>> +
>> +# respawn if the slave crash
>> +respawn
>>
>>
>> I've deployed that on jatayu by adding 'respawn' to
>> /etc/init/jenkins-slave.conf so daily-release-executor should now
>> restart automatically (I've restarted the jenkins-slave service).
>>
>> ....
>>
>>
>> And then I noticed that per my own documented instructions at
>> https://wiki.canonical.com/UbuntuEngineering/CI/Playbook/q-jenkins#daily-release-executor
>> (wanting to update them...) that this is not the Right Thing :-/
>>
>> First, patching /etc/init/jenkins-slave.conf will cause issues when
>> upgrading to a new version. This can/should be fixed by proposing that
>> upstream though.
>>
>> Second, we use /usr/local/bin/start-jenkins-slaves which ignores the
>> upstart jobs :-/
>>
>> I also realized then that we rely on start-jenkins-slaves to start
>> multiple slaves on jatayu... daily-release-executor indeed but also the
>> acer-veriton-0[123] slaves via /etc/default/jenkins-slave.
>>
>> Feedback welcome first and then I'll probably revert that change until
>> we figure a better solution for which I'll file some asana 'Post 1SS' or
>> 'CI Vanguard' task ;)
>>
>>     Vincent
>>
>> --
>> Mailing list: https://launchpad.net/~canonical-ci-engineering
>> Post to     : canonical-ci-engineering@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~canonical-ci-engineering
>> More help   : https://help.launchpad.net/ListHelp
>
>



References