canonical-ci-engineering team mailing list archive

[Martin Pitt <martin.pitt@xxxxxxxxxx>]

Steve Langasek [2014-12-10  0:57 -0800]:
> Well, except that still doesn't help us at all, because the specific tests
> we're trying to implement here are system-image upgrade tests - which means
> we need to have control over the base version of OS that's installed on the
> device.  And this has been specifically called out as something that's not
> supported.

Right. I discussed this with Barry a few weeks ago. In the "old"
Jenkins world, these would be custom Jenkins jobs which install the
desired old image version, and then call the test. TTBOMK the current
UCI airline does not support any of that yet, it only offers
pre-allocated cloud instances.

Setting up the testbed can happen totally outside of adt-run, or we
capture that logic into an ssh setup script (see man adt-virt-ssh) so
that one has a single adt-run command to call.

But in either way, the infrastructure you run this on obviously needs
to support this kind of operation, i. e. that at some level something
can call ubuntu-device-flash.

> So I think that still leaves us at square one regarding automation of the
> tests.  Provisioning the devices is the time-consuming part anyway; wrapping
> a call to 'system-image-cli' on the phone inside a call to 'adt-run', when a
> QA person is still going to have to manually flash the device between each
> individual test, isn't really an improvement.

Right, I don't think the concept of "test this package" maps very well
to this use case. It's useful for complex upgrade tests like we do on
the desktop, where you want to run post-upgrade tests, but not if the
only thing you need to call is system-image-cli, then your test
control will essentially only have that one command.

> How do we automate the setup of the test bed?  If we assume that's always
> external to DEP8, then that just means I need some other interface to write
> to that isn't DEP8.

In theory you could put those into a test run by adt-run, and then
pipe this through the null runner and require that you have an adb
device connected. I'm not sure whether this is any more elegant or
easier to support (my gut feeling is that it isn't).

Martin

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

Attachment: signature.asc
Description: Digital signature