canonical-ci-engineering team mailing list archive

[Vincent Ladeuil <vila+ci@xxxxxxxxxxxxx>]

Hi all,

I'm currently blocked on many issues because our low-level test
infrastructure doesn't give me some features that are IMNSHO required.

It's an issue I already ran into when I was in the U1 QA team: I come
from the Bazaar team which was considered by many contributors like some
sort of TDD paradise and it took me a while to realize that many
features I took for granted were just unknown. So showing the code was
the best way to publicize them.

For various reasons, just re-using the bzr test infrastructure is not
realistic. So for lp:selelnium-simple-test I started extracting and
improving some key features that are not easy to reuse (or not
implemented at all) in lp:testtools, lp:subunit and python's unittest.

Running tests involves several key classes:

- a test loader that can walk a file system hierarchy to gather and
  parametrize tests,

- test filters that can include or exclude from the whole test suite to
  select only the test relevant to the task at hand (as a dev I may want
  to run a single test, all the tests in a class, in a module, in a
  subtree or some combinations),

- test results in various formats, most importantly subunit but also
  various display options (test ids, feedback during execution) and the
  ability to properly handle specific tests (expected failures,
  unexpected successes),

- a test runner that generally involves one of the above and as such
  needs to be under our control. Parallelisation is also a feature that
  requires controlling the test runner.

I chose selenium-simple-test as a home for this code at the time because
that was the primary client. I think it's time to give it a proper
project on launchpad so we can more easily use it in the CI engine.

Feedback, thoughts, should I JFDI or do you have objections ?

          Vincent