canonical-ci-engineering team mailing list archive

[Francis Ginther <francis.ginther@xxxxxxxxxxxxx>]

To your scrum questions. I don't have any experience here either. The
handoff checklist looks a lot like acceptance criteria for the QA story to
add test X, but there's also an expectation that once that acceptance
criteria is met, the CI team will prioritize the next step work to complete
the process. I need to do a little research.

Also for defining this criteria, I normally would rely on the product owner
and/or scrummaster to be the contact person for this, but since both are on
holiday all week...

Francis

On Thu, Oct 30, 2014 at 11:40 AM, Francis Ginther <
francis.ginther@xxxxxxxxxxxxx> wrote:

> Thomi,
>
> If you're willing to iterate on this a couple times, I think we can
> converge on a set of guidelines rather then try to define the perfect set
> up front. I agree with your first two major starting points:
>
>    - All test suites in dep8/autopkgtest format.
>       - Results to be stored in a subunit binary stream.
>       - ...which is an archived artifact.
>    - Tests to be in a launchpad branch (of a specific project? any
>    project?)
>
> I think we start with a launchpad branch as that gives access to the dep8
> tests in a known location. I don't think the project itself matters as long
> as it contains the right set of tests. We want to point the test runner at
> a dep8 test and just say 'go' without having to skip certain tests or
> filter resutls [1]. If the tests can be executed by 'bzr branch ...' and
> 'adt-run ...', then I think this first set of criteria is met.
>
> The third point:
>
>    - Evan mentioned that he didn't want to keep running tests in jenkins.
>    The tests we'd like to hand over currently run in jenkins. What's the
>    proposed alternative? How can we set up those test jobs to prove that these
>    suites are stable?
>
> Evan may have additional insight, but at a minimum, we want to get to a
> point where it doesn't matter where the tests are run and therefore tests
> shouldn't rely on a specific jenkins plugin or feature. The touch devices
> are currently available only through jenkins, but the expectation is that
> these will eventually be just adt-run slaves attached to a queue of test
> requests.
>
> Based on our session in DC [2], there a couple of additional bits of
> criteria:
>
> 1) Pushing results to an external service (nfss) and keeping the
> credentials secret.
>     - Can we specify that the data to push to nfss be in a consistent
> format within the subunit stream? The idea being that a generic worker
> could extract the data from the stream and push it regardless of what test
> generated the results. Only that worker then needs to have the credentials.
>
> 2) Dealing with potentially destructive tests (e.g. the lrt tests are
> known to put the device in a non-usable state)
>     - For the handoff criteria, we would need to know that this is an
> expected outcome of the test and what the completion criteria is. If we
> treat this as an infrastructure failure and just retry the test, it is
> likely to never complete. I don't know a good way to handle this
> generically and perhaps there ultimately isn't one.
>
> 3) Scheduling/Triggering/Monitoring criteria
>     - What is the trigger to start a test (i.e. per-image, daily, per-MP,
> etc.)
>     - What, if anything, needs to be monitored. I think you already have a
> start here, but we may need to revisit this to find the best location to
> store the alerts.
>
> [1] - We know that the dep8/autopkgtest format does not yet support
> hardware/platform constraints (i.e. this test needs a touch device), but we
> can hopefully keep the need for this minimized for now.
> [2] -
> https://docs.google.com/a/canonical.com/document/d/1LlyWTf7SiPU305uNhJ5H4YH8wOGtTVbbgq6QJ0L6HkE/edit
>
> Francis
>
> On Wed, Oct 29, 2014 at 9:48 PM, Thomi Richards <
> thomi.richards@xxxxxxxxxxxxx> wrote:
>
>> Hello,
>>
>>
>> It's highly likely that goal for the inaugural sprint for the QA projects
>> team will revolve around getting a number of test cases 'up to scratch' to
>> be handed over to the CI team.
>>
>> The problem is that the CI team haven't had time to complete the 'test
>> handover checklist' we discussed in D.C. (as an aside, I'd love to know
>> from the scrum-experts on this list whether there's a standard way to
>> manage dependencies between multiple scrum teams). Additionally, the
>> 'handover' doesn't seem like something that fits within the scrum
>> framework. I'd like some direction from the CI team as to how you see this
>> happening.
>>
>>
>> Regarding the tests themselves, I'd love to get a rough idea of what your
>> requirements are. I have jotted down some thoughts that might help get us
>> started:
>>
>>
>>    - All test suites in dep8/autopkgtest format.
>>       - Results to be stored in a subunit binary stream.
>>       - ...which is an archived artifact.
>>    - Tests to be in a launchpad branch (of a specific project? any
>>    project?)
>>    - Evan mentioned that he didn't want to keep running tests in
>>    jenkins. The tests we'd like to hand over currently run in jenkins. What's
>>    the proposed alternative? How can we set up those test jobs to prove that
>>    these suites are stable?
>>
>>
>> I'd love some guidance on the points above, so the work we produce is
>> aligned with what CI will accept.
>>
>> Please accept my apologies if this seems like me trying to interrupt your
>> sprint. I guess that 's exactly what I'm doing, but you should let me,
>> because... reasons :D
>>
>> Cheers,
>> --
>> Thomi Richards
>> thomi.richards@xxxxxxxxxxxxx
>>
>> --
>> Mailing list: https://launchpad.net/~canonical-ci-engineering
>> Post to     : canonical-ci-engineering@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~canonical-ci-engineering
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Francis Ginther
> Canonical - Ubuntu Engineering - Continuous Integration Team
>



-- 
Francis Ginther
Canonical - Ubuntu Engineering - Continuous Integration Team