canonical-ci-engineering team mailing list archive

[Francis Ginther <francis.ginther@xxxxxxxxxxxxx>]

Thomi,

If you're willing to iterate on this a couple times, I think we can
converge on a set of guidelines rather then try to define the perfect set
up front. I agree with your first two major starting points:

   - All test suites in dep8/autopkgtest format.
      - Results to be stored in a subunit binary stream.
      - ...which is an archived artifact.
   - Tests to be in a launchpad branch (of a specific project? any project?)

I think we start with a launchpad branch as that gives access to the dep8
tests in a known location. I don't think the project itself matters as long
as it contains the right set of tests. We want to point the test runner at
a dep8 test and just say 'go' without having to skip certain tests or
filter resutls [1]. If the tests can be executed by 'bzr branch ...' and
'adt-run ...', then I think this first set of criteria is met.

The third point:

   - Evan mentioned that he didn't want to keep running tests in jenkins.
   The tests we'd like to hand over currently run in jenkins. What's the
   proposed alternative? How can we set up those test jobs to prove that these
   suites are stable?

Evan may have additional insight, but at a minimum, we want to get to a
point where it doesn't matter where the tests are run and therefore tests
shouldn't rely on a specific jenkins plugin or feature. The touch devices
are currently available only through jenkins, but the expectation is that
these will eventually be just adt-run slaves attached to a queue of test
requests.

Based on our session in DC [2], there a couple of additional bits of
criteria:

1) Pushing results to an external service (nfss) and keeping the
credentials secret.
    - Can we specify that the data to push to nfss be in a consistent
format within the subunit stream? The idea being that a generic worker
could extract the data from the stream and push it regardless of what test
generated the results. Only that worker then needs to have the credentials.

2) Dealing with potentially destructive tests (e.g. the lrt tests are known
to put the device in a non-usable state)
    - For the handoff criteria, we would need to know that this is an
expected outcome of the test and what the completion criteria is. If we
treat this as an infrastructure failure and just retry the test, it is
likely to never complete. I don't know a good way to handle this
generically and perhaps there ultimately isn't one.

3) Scheduling/Triggering/Monitoring criteria
    - What is the trigger to start a test (i.e. per-image, daily, per-MP,
etc.)
    - What, if anything, needs to be monitored. I think you already have a
start here, but we may need to revisit this to find the best location to
store the alerts.

[1] - We know that the dep8/autopkgtest format does not yet support
hardware/platform constraints (i.e. this test needs a touch device), but we
can hopefully keep the need for this minimized for now.
[2] -
https://docs.google.com/a/canonical.com/document/d/1LlyWTf7SiPU305uNhJ5H4YH8wOGtTVbbgq6QJ0L6HkE/edit

Francis

On Wed, Oct 29, 2014 at 9:48 PM, Thomi Richards <
thomi.richards@xxxxxxxxxxxxx> wrote:

> Hello,
>
>
> It's highly likely that goal for the inaugural sprint for the QA projects
> team will revolve around getting a number of test cases 'up to scratch' to
> be handed over to the CI team.
>
> The problem is that the CI team haven't had time to complete the 'test
> handover checklist' we discussed in D.C. (as an aside, I'd love to know
> from the scrum-experts on this list whether there's a standard way to
> manage dependencies between multiple scrum teams). Additionally, the
> 'handover' doesn't seem like something that fits within the scrum
> framework. I'd like some direction from the CI team as to how you see this
> happening.
>
>
> Regarding the tests themselves, I'd love to get a rough idea of what your
> requirements are. I have jotted down some thoughts that might help get us
> started:
>
>
>    - All test suites in dep8/autopkgtest format.
>       - Results to be stored in a subunit binary stream.
>       - ...which is an archived artifact.
>    - Tests to be in a launchpad branch (of a specific project? any
>    project?)
>    - Evan mentioned that he didn't want to keep running tests in jenkins.
>    The tests we'd like to hand over currently run in jenkins. What's the
>    proposed alternative? How can we set up those test jobs to prove that these
>    suites are stable?
>
>
> I'd love some guidance on the points above, so the work we produce is
> aligned with what CI will accept.
>
> Please accept my apologies if this seems like me trying to interrupt your
> sprint. I guess that 's exactly what I'm doing, but you should let me,
> because... reasons :D
>
> Cheers,
> --
> Thomi Richards
> thomi.richards@xxxxxxxxxxxxx
>
> --
> Mailing list: https://launchpad.net/~canonical-ci-engineering
> Post to     : canonical-ci-engineering@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~canonical-ci-engineering
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Francis Ginther
Canonical - Ubuntu Engineering - Continuous Integration Team