canonical-ci-engineering team mailing list archive

[Steve Langasek <steve.langasek@xxxxxxxxxxxxx>]

On Wed, Dec 10, 2014 at 12:04:53AM -0800, Steve Langasek wrote:
> On Wed, Dec 10, 2014 at 08:08:25AM +1300, Thomi Richards wrote:
> > Hi everyone,

> > Thanks for the replies. I'd like to summarise my understanding:

> >    - Test authors *should* add test classes that describe the test
> >    environment they require. However, adt-run ignores these, and there's no
> >    infrastructure piece that reads and understands these classes, so this is
> >    purely for future-proofing. I don't think we've defined these - I suggest
> >    just using whatever makes sense, and we can clean these up later.
> >    - Test suites that fail will not block package promotion, unless those
> >    failures are regressions (i.e.- they have passed in the past).
> >    - Test authors *must* make tests skip if they're run in an incompatible
> >    environment (hint: if you're writing autopilot tests, the
> >    autopilot.platform module might help here).

> > Did I miss anything?

> > Steve - does this answer your question?

> Ok, so just to be clear, the tests should:

>  - be written assuming that the adt harness will run them on the appropriate
>    device;
>  - set a 'Class' to hint this;
>  - use the interfaces described in
>    https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html to
>    trigger and resume testing across reboots;
>  - and make these tests fail gracefully when run on jenkins today.

> Is that correct?

> I think that provides most of the guidance we need.  The only thing I'm
> still not sure about is the "fail gracefully" part.  All of the interfaces
> invoked by our tests are things that, under DEP8, would be declared as
> dependencies of the tests (AIUI), and there's no interface for querying the
> key bit that will make the test work (i.e., "is the recovery partition set
> up to upgrade us, and does this device support rebooting into recovery
> mode?").  So for the time being I don't think we can be very graceful.

Well, except that still doesn't help us at all, because the specific tests
we're trying to implement here are system-image upgrade tests - which means
we need to have control over the base version of OS that's installed on the
device.  And this has been specifically called out as something that's not
supported.

So I think that still leaves us at square one regarding automation of the
tests.  Provisioning the devices is the time-consuming part anyway; wrapping
a call to 'system-image-cli' on the phone inside a call to 'adt-run', when a
QA person is still going to have to manually flash the device between each
individual test, isn't really an improvement.

How do we automate the setup of the test bed?  If we assume that's always
external to DEP8, then that just means I need some other interface to write
to that isn't DEP8.

Thanks,
-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@xxxxxxxxxx                                     vorlon@xxxxxxxxxx

Attachment: signature.asc
Description: Digital signature