canonical-ci-engineering team mailing list archive

[Steve Langasek <steve.langasek@xxxxxxxxxxxxx>]

Hi all,

On Wed, Dec 10, 2014 at 08:08:25AM +1300, Thomi Richards wrote:
> Hi everyone,

> Thanks for the replies. I'd like to summarise my understanding:

>    - Test authors *should* add test classes that describe the test
>    environment they require. However, adt-run ignores these, and there's no
>    infrastructure piece that reads and understands these classes, so this is
>    purely for future-proofing. I don't think we've defined these - I suggest
>    just using whatever makes sense, and we can clean these up later.
>    - Test suites that fail will not block package promotion, unless those
>    failures are regressions (i.e.- they have passed in the past).
>    - Test authors *must* make tests skip if they're run in an incompatible
>    environment (hint: if you're writing autopilot tests, the
>    autopilot.platform module might help here).
> 
> Did I miss anything?

> Steve - does this answer your question?

Ok, so just to be clear, the tests should:

 - be written assuming that the adt harness will run them on the appropriate
   device;
 - set a 'Class' to hint this;
 - use the interfaces described in
   https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html to
   trigger and resume testing across reboots;
 - and make these tests fail gracefully when run on jenkins today.

Is that correct?

I think that provides most of the guidance we need.  The only thing I'm
still not sure about is the "fail gracefully" part.  All of the interfaces
invoked by our tests are things that, under DEP8, would be declared as
dependencies of the tests (AIUI), and there's no interface for querying the
key bit that will make the test work (i.e., "is the recovery partition set
up to upgrade us, and does this device support rebooting into recovery
mode?").  So for the time being I don't think we can be very graceful.

Also, I saw "touch" suggested in this thread as a class name.  What
semantics do we expect that to express?  What I need to express is
"this test needs to be run on a real device, because that's the only place
we can reboot into the recovery partition".  I'm not sure that's the obvious
semantic of "touch"; I think we should agree on what we're using these
classes for before we start declaring them, otherwise I might as well not
declare the class at all in the near term since there's nothing that can
consume it meaningfully.

Thanks,
-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@xxxxxxxxxx                                     vorlon@xxxxxxxxxx

Attachment: signature.asc
Description: Digital signature