canonical-ci-engineering team mailing list archive

[Thomi Richards <thomi.richards@xxxxxxxxxxxxx>]

Hi,

After a quick hangout, we hashed out a rough first draft:

https://wiki.canonical.com/UbuntuEngineering/CI/HandoffCriteria

...with emphasis on the *rough* part. I'd love feedback on this - what have
I forgotten?


Cheers,

On Fri, Oct 31, 2014 at 10:13 AM, Francis Ginther <
francis.ginther@xxxxxxxxxxxxx> wrote:

> To your scrum questions. I don't have any experience here either. The
> handoff checklist looks a lot like acceptance criteria for the QA story to
> add test X, but there's also an expectation that once that acceptance
> criteria is met, the CI team will prioritize the next step work to complete
> the process. I need to do a little research.
>
> Also for defining this criteria, I normally would rely on the product
> owner and/or scrummaster to be the contact person for this, but since both
> are on holiday all week...
>
> Francis
>
> On Thu, Oct 30, 2014 at 11:40 AM, Francis Ginther <
> francis.ginther@xxxxxxxxxxxxx> wrote:
>
>> Thomi,
>>
>> If you're willing to iterate on this a couple times, I think we can
>> converge on a set of guidelines rather then try to define the perfect set
>> up front. I agree with your first two major starting points:
>>
>>    - All test suites in dep8/autopkgtest format.
>>       - Results to be stored in a subunit binary stream.
>>       - ...which is an archived artifact.
>>    - Tests to be in a launchpad branch (of a specific project? any
>>    project?)
>>
>> I think we start with a launchpad branch as that gives access to the dep8
>> tests in a known location. I don't think the project itself matters as long
>> as it contains the right set of tests. We want to point the test runner at
>> a dep8 test and just say 'go' without having to skip certain tests or
>> filter resutls [1]. If the tests can be executed by 'bzr branch ...' and
>> 'adt-run ...', then I think this first set of criteria is met.
>>
>> The third point:
>>
>>    - Evan mentioned that he didn't want to keep running tests in
>>    jenkins. The tests we'd like to hand over currently run in jenkins. What's
>>    the proposed alternative? How can we set up those test jobs to prove that
>>    these suites are stable?
>>
>> Evan may have additional insight, but at a minimum, we want to get to a
>> point where it doesn't matter where the tests are run and therefore tests
>> shouldn't rely on a specific jenkins plugin or feature. The touch devices
>> are currently available only through jenkins, but the expectation is that
>> these will eventually be just adt-run slaves attached to a queue of test
>> requests.
>>
>> Based on our session in DC [2], there a couple of additional bits of
>> criteria:
>>
>> 1) Pushing results to an external service (nfss) and keeping the
>> credentials secret.
>>     - Can we specify that the data to push to nfss be in a consistent
>> format within the subunit stream? The idea being that a generic worker
>> could extract the data from the stream and push it regardless of what test
>> generated the results. Only that worker then needs to have the credentials.
>>
>> 2) Dealing with potentially destructive tests (e.g. the lrt tests are
>> known to put the device in a non-usable state)
>>     - For the handoff criteria, we would need to know that this is an
>> expected outcome of the test and what the completion criteria is. If we
>> treat this as an infrastructure failure and just retry the test, it is
>> likely to never complete. I don't know a good way to handle this
>> generically and perhaps there ultimately isn't one.
>>
>> 3) Scheduling/Triggering/Monitoring criteria
>>     - What is the trigger to start a test (i.e. per-image, daily, per-MP,
>> etc.)
>>     - What, if anything, needs to be monitored. I think you already have
>> a start here, but we may need to revisit this to find the best location to
>> store the alerts.
>>
>> [1] - We know that the dep8/autopkgtest format does not yet support
>> hardware/platform constraints (i.e. this test needs a touch device), but we
>> can hopefully keep the need for this minimized for now.
>> [2] -
>> https://docs.google.com/a/canonical.com/document/d/1LlyWTf7SiPU305uNhJ5H4YH8wOGtTVbbgq6QJ0L6HkE/edit
>>
>> Francis
>>
>> On Wed, Oct 29, 2014 at 9:48 PM, Thomi Richards <
>> thomi.richards@xxxxxxxxxxxxx> wrote:
>>
>>> Hello,
>>>
>>>
>>> It's highly likely that goal for the inaugural sprint for the QA
>>> projects team will revolve around getting a number of test cases 'up to
>>> scratch' to be handed over to the CI team.
>>>
>>> The problem is that the CI team haven't had time to complete the 'test
>>> handover checklist' we discussed in D.C. (as an aside, I'd love to know
>>> from the scrum-experts on this list whether there's a standard way to
>>> manage dependencies between multiple scrum teams). Additionally, the
>>> 'handover' doesn't seem like something that fits within the scrum
>>> framework. I'd like some direction from the CI team as to how you see this
>>> happening.
>>>
>>>
>>> Regarding the tests themselves, I'd love to get a rough idea of what
>>> your requirements are. I have jotted down some thoughts that might help get
>>> us started:
>>>
>>>
>>>    - All test suites in dep8/autopkgtest format.
>>>       - Results to be stored in a subunit binary stream.
>>>       - ...which is an archived artifact.
>>>    - Tests to be in a launchpad branch (of a specific project? any
>>>    project?)
>>>    - Evan mentioned that he didn't want to keep running tests in
>>>    jenkins. The tests we'd like to hand over currently run in jenkins. What's
>>>    the proposed alternative? How can we set up those test jobs to prove that
>>>    these suites are stable?
>>>
>>>
>>> I'd love some guidance on the points above, so the work we produce is
>>> aligned with what CI will accept.
>>>
>>> Please accept my apologies if this seems like me trying to interrupt
>>> your sprint. I guess that 's exactly what I'm doing, but you should let me,
>>> because... reasons :D
>>>
>>> Cheers,
>>> --
>>> Thomi Richards
>>> thomi.richards@xxxxxxxxxxxxx
>>>
>>> --
>>> Mailing list: https://launchpad.net/~canonical-ci-engineering
>>> Post to     : canonical-ci-engineering@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~canonical-ci-engineering
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>>
>> --
>> Francis Ginther
>> Canonical - Ubuntu Engineering - Continuous Integration Team
>>
>
>
>
> --
> Francis Ginther
> Canonical - Ubuntu Engineering - Continuous Integration Team
>



-- 
Thomi Richards
thomi.richards@xxxxxxxxxxxxx