canonical-ci-engineering team mailing list archive

[Ursula Junque <ursula.junque@xxxxxxxxxxxxx>]

Hi,

Adding CI team so they are aware of this, and can chime in if appropriated.

On Thu, Mar 12, 2015 at 4:00 PM, Nicholas Skaggs <
nicholas.skaggs@xxxxxxxxxxxxx> wrote:

> Greetings all! It's come to my attention recently there is a bit of a
> fatal flaw in our testing process for images. I'll be brief and to the
> point. As of today we cannot:
>
> - recreate test results with certainty
> - test old images
>
> The reason for both of these issues is because we grab our test
> dependencies on the fly. In doing so, we are at the mercy of the archive
> being unchanging. As Max found out yesterday, in practice this means
> sometimes you can't even run tests against the latest images because the
> archive changed after it was built! It also means, we most certainly cannot
> run tests on older images. The archive changes, even for things like the
> rtm images which has a released version of ubuntu as it's base.
>
> So what workarounds exist? Martin has added a nicer error message when you
> encounter this [1], and also has supported changing to r/w mode just to
> update the apt-indexes [2]. Sometimes this helps, but usually not as the
> archive wants to update critical packages that not only change the system
> you are attempting to test, but also fail to install.
>
> So what solutions might be possible? What is needed is a snapshot of the
> archive more or less to coincidence with the image. Barring the archive
> suddenly deciding to retain old packages, this really isn't possible. So we
> could perhaps explore creating and building click packages that contain
> tests and their dependencies. As these are dependent on the image
> themselves for our purposes they would need to be created (and hosted?) by
> CI. Then presumably autopkgtest would have to know where they are and how
> to get them. And of course, it's still a broken story for our community who
> doesn't have such a service willing to create copies of needed packages at
> image build time. Presumably these packages might also be possible to
> create at click build time, but they would need to be built at the same
> time and then kept.
>
> Since the aforementioned options are really hacks anyway, there might be
> another option. In practice most test dependencies aren't hugely connected
> with the platform, so the update hack could work for them. This fails for
> autopilot however. If we packaged autopilot and the helpers as a click (or
> debian package ::shudder::) that is tied with unity8 and the image, it's
> likely we could hack / workaround not being able to test at all.
>
> So in summary we really can't test old images. And sometimes we can't even
> test the current image. That is definitely a problem, but we until now we
> have simply retained old results and worked-around / ignored the pain of
> sometimes having an untestable image. Since CI hasn't yet adopted using
> adt, this issue hasn't yet completely reared it's head



> The even greater issue IMHO, is that we can't re-run tests with certainty.
> Ideally the image should be read-only, should remain static and we should
> have a static testing payload to deploy. The payload should be the same
> today as it was 2 weeks ago when we first ran the tests. The results should
> be consistent. We can't say that today and that makes the testing we do
> suspect.
>
>
> Nicholas
>
> 1. https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1425682
> 2. https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1342838
>
> --
> qa-team mailing list
> qa-team@xxxxxxxxxxxxxxxxxxx
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/qa-team
>



-- 
Úrsula Junque
Ubuntu CI Engineer

ursula.junque@xxxxxxxxxxxxx
ursinha@xxxxxxxxxxx
ursinha@xxxxxxxxxx

Ubuntu - "I am what I am because of who we all are."
Linux user #289453 - Ubuntu user #31144