canonical-ci-engineering team mailing list archive

[Martin Pitt <martin.pitt@xxxxxxxxxx>]

Hey all,

Thomi Richards [2014-12-08  8:19 +1300]:
> We're starting to hit cases where a particular dep8 test suite can really
> only be run on an actual phone. We're encouraging development teams to
> express their tests in dep8, but we need some guidance from CI on how to
> declare hardware-type restrictions.
> 
> How should a test author declare that a particular test suite should only
> be run on a real phone?

The only firm restriction that's built into DEP-8 right now is "needs
at least a container" ("Restrictions: isolation-container", i. e. a
schroot isn't good enough) or "needs a full machine"
(isolation-machine, i. e. container doesn't provide enough isolation).
We don't have other enforced restrictions.

Tests can certainly declare the "Classes:" (see
https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html)
which hints to the environment that executes the tests which kind of
machines/targets we want to run a test on. But adt-itself can't of
course not enforce that (it doesn't even know what a "phone" is) -- if
you have a phone app test and try to run it in a container, it will
happily do that. In fact, we have setup scripts which prepare a
container to look like a phone, so you can do exactly that. :-)

> My understanding is that all dep8 tests are run for packages uploaded to
> debian & ubuntu. I think it's reasonable to provide test authors with a way
> to not introduce broken tests..

A test can always skip itself if it detects that it runs in an
inadequate environment. Other than that, using Classes: is how we
intend to generically solve "where to run which tests at" for our CI
infrastructure. Then, if someone manually tries to run a test
someplace else, (s)he can see how far it gets, and if the test isn't
appropriate for the target at all (like trying an autopilot test in a
plain schroot), it'll just fail and be a "don't do that then"
situation.

Thanks,

Martin

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)