canonical-ci-engineering team mailing list archive

[Francis Ginther <francis.ginther@xxxxxxxxxxxxx>]

Thomi,

Just want to give some quick feedback to your request.

On Tue, Nov 19, 2013 at 2:28 PM, Thomi Richards
<thomi.richards@xxxxxxxxxxxxx> wrote:
> Hi CI team,
>
>
> The mir team would like to run the glmark2 benchmarks on a native mir server
> across Desktop, N4 and N10 devices at the MP stage, and block merges on a
> significant drop in FPS performance.
>
>
> We already have a glmark2 version that runs against a native mir server. The
> parts that are missing includes:
>
> Infrastructure to run that test (jenkins and whatnot).
>
> Policy around exactly when to fail the test, when to warn about a drop in
> performance, and when to pass the test.
>
> A way to convert the data from glmark2 into something the CI team can report
> on.
>
> A place to show the graphs of performance over time.
>
>
> So, addressing these in reverse order (why not):
>
> 4: We already have performance graphs for mir, so I suspect this part will
> be easy.

I'll assume this can be accomplished under a glmark ci specific page
like this one: http://reports.qa.ubuntu.com/graphics/openarena/

> 3: This needs a small amount of python code, I suggest we emit a subunit
> result stream here - we could use this as a testbed for subunit. I imagine
> that this script would be done by someone in the QA team? Is the CI team
> ready to read subunit result streams? I'm more than happy to walk someone
> through what needs to happen here (the code is trivially easy)...

I agree on using subunit as mentioned here:
https://wiki.ubuntu.com/CI/AddingTests  (yeah! a new wiki page)

> 2: I suggest we get the tests running for a few weeks, graph the results,
> and decide the policy based on that data. We're going to be blocking merge
> proposals from this, so we need to make sure we get it right. I expect that
> the policy would be decided upon by a mixture of the mir and QA teams. The
> current idea is that < 5% drop would result in a warning, and >5% would
> result in a fail. It would be nice if we could tweak this policy easily
> without having to go through the CI team... but that's a minor detail.

This may not be so easy. Failing an MP because performance drops below
X% is a pass/fail action. Providing a warning when it's not quite so
bad is not. We'll need to iterate on how do make this visible to
developers.

> 1: Chris Gagnon is already working on a jenkins job that runs the mir demos
> and integration tests, but I think this should be a separate job, for a
> couple of reasons:
>
> First, we're likely to want to run these performance tests in several places
> (MP CI runs to begin with, perhaps on a daily basis on top of the distro
> images as well?).
>
> Second, I think it's worth keeping the concerns of "have we broken the mir
> demos or integration tests by altering API without updating them" and the
> concern of "have we regressed performance in this release" separate.

This should work. The current cupstream2distro-config tools support
adding pass/fail child jobs to a project configuration. And this type
of test should migrate into the CI Airline world.

> So, I don't think there's anything too controversial in here - Can we get a
> plan together to figure out who's going to do what, so we can get the mir
> team hooked up?

I opened a bug to track this request:
https://bugs.launchpad.net/ubuntu-ci-services-itself/+bug/1252933

> Cheers,
>
> --
> Thomi Richards
> thomi.richards@xxxxxxxxxxxxx
>
> --
> Mailing list: https://launchpad.net/~canonical-ci-engineering
> Post to     : canonical-ci-engineering@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~canonical-ci-engineering
> More help   : https://help.launchpad.net/ListHelp
>



-- 
Francis Ginther
Canonical - Ubuntu Engineering - Continuous Integration Team