canonical-ci-engineering team mailing list archive

[Vincent Ladeuil <vila+ci@xxxxxxxxxxxxx>]

>>>>> Andy Doan <andy.doan@xxxxxxxxxxxxx> writes:

    > Hey Guys,
    > A heads up. Chris and I just got his tarmac job updated to run unit tests as
    > part of new MPs.

\o/

    > The test script is very lightweight and only does out
    > unit-tests. It takes just at 60 seconds right now, so it shouldn't
    > be too cumbersome.

/me nods

    > If you want to test things locally, you can simply run:
    >  ./tarmac.sh

I did that and it succeeded successfully, well done.

Yet, there are a bunch of alarming error messages there.

Among others:

    Ensure a failed queue connection returns an HTTP 503 error ... ERROR:root:unable to queue up build_source request
    Traceback (most recent call last):
      File "/home/vila/ci/ubuntu-ci-services-itself/trunk/ci-utils/ci_utils/amqp_utils.py", line 57, in send
        con = connection(config)
      File "build/bdist.linux-x86_64/egg/mock.py", line 955, in __call__
        return _mock_self._mock_call(*args, **kwargs)
      File "build/bdist.linux-x86_64/egg/mock.py", line 1010, in _mock_call
        raise effect
    RuntimeError: mocked test exception
     
     
    Ensure this times out after the right amount of time ... ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:lost connection to Rabbit
    ERROR:root:Rabbit did not reappear quickly enough.
    ok
     
    testPPAAssigner404 (lander.tests.test_service_wrapper.TestWrapper)
    Ensure a "no ppas" type error generates a response ... DEBUG: Executing service wrapper: ppa_assigner
    DEBUG:lander_service_wrapper:Executing service wrapper: ppa_assigner
    ERROR:root:Unable to handle request: HTTP Error 404: 404: foo bar
    Traceback (most recent call last):
      File "/home/vila/ci/ubuntu-ci-services-itself/trunk/lander/lander/tests/../../bin/lander_service_wrapper.py", line 159, in main
        results = handler(args)
      File "/home/vila/ci/ubuntu-ci-services-itself/trunk/lander/lander/tests/../../bin/lander_service_wrapper.py", line 82, in _handle_ppa_assigner
        loc = _post(url, {'ticket_id': config['request_id']})
      File "build/bdist.linux-x86_64/egg/mock.py", line 955, in __call__
        return _mock_self._mock_call(*args, **kwargs)
      File "build/bdist.linux-x86_64/egg/mock.py", line 1010, in _mock_call
        raise effect
    HTTPError: HTTP Error 404: 404
    ok
     
    Using /home/vila/ci/ubuntu-ci-services-itself/trunk/ci-utils/six-1.5.2-py2.7.egg
    Finished processing dependencies for ppa-assigner==0.1
    Creating test database for alias 'default'...
    Traceback (most recent call last):
      File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
        msg = self.format(record)
      File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
        return fmt.format(record)
      File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
        record.message = record.getMessage()
      File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
        msg = msg % self.args
    TypeError: not all arguments converted during string formatting
    Logged from file models.py, line 86
    ERROR:root:Unable to populate PPAs from launchpad. Check that LAUNCHPAD_USER is configured correctly in local_settings.py.
    ....Traceback (most recent call last):
      File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
        msg = self.format(record)
      File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
        return fmt.format(record)
      File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
        record.message = record.getMessage()
      File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
        msg = msg % self.args
    TypeError: not all arguments converted during string formatting
    Logged from file models.py, line 86
    .../home/vila/ci/ubuntu-ci-services-itself/trunk/ppa-assigner/ppa_assigner/api.py:45: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
      raise ImmediateHttpResponse(HttpNotFound(e.message))
     
    WARNING:py.warnings:/home/vila/ci/ubuntu-ci-services-itself/trunk/ppa-assigner/ppa_assigner/api.py:45: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
      raise ImmediateHttpResponse(HttpNotFound(e.message))
     
There are also debug messages that pollutes the test output and makes it
harder than needed to read:
     
    testBuildSource (bsbuilder.tests.test_v1.TestAPI) ... DEBUG:bsbuilder.resources.root:[api]: http://localhost/api/v1/build_source [u'build_source']
    /home/vila/ci/ubuntu-ci-services-itself/trunk/ci-utils/WebTest-2.0.10-py2.7.egg/webtest/lint.py:475: WSGIWarning: Content-Type header found in a 204 response, which not return content.
      WSGIWarning)
    ok
    testBuildSourceBadParams (bsbuilder.tests.test_v1.TestAPI)
    Ensure proper error message is returned for incorrect params. ... DEBUG:bsbuilder.resources.root:[api]: http://localhost/api/v1/build_source [u'build_source']
    ok
    testBuildSourceUnconfigured (bsbuilder.tests.test_v1.TestAPI) ... DEBUG:bsbuilder.resources.root:[api]: http://localhost/api/v1/build_source [u'build_source']
    ok
    testStatus (bsbuilder.tests.test_v1.TestAPI) ... DEBUG:bsbuilder.resources.root:[api]: http://localhost/api/v1/status [u'status']
    DEBUG:bsbuilder.resources.v1:[status]: http://localhost/api/v1/status []
    DEBUG:bsbuilder.resources.root:[api]: http://localhost/api/v1/status [u'status']
    DEBUG:bsbuilder.resources.v1:[status]: http://localhost/api/v1/status []
    ok
     
    Ran 4 tests in 0.004s

4 passing tests should output 4 lines ;)

This seems easier said than done but if we don't start sticking to that
rule now, the tech debt will severely reduce the value of our tests.

Test output (on both stdout and stderr) can and should be captured. If
the content is relevant it should also be asserted against.

The explicit goal here is that *any* output (bar the test name and the
test status[1]) to the console during test execution can be considered as a
useful warning that something went wrong and can be fixed.

    > This creates a temp-directory with a python-venv, runs our tests, and
    > deletes the temp-directory. Thus, it *should* be a non-destructive thing you
    > can easily try out.

Well done again, anybody should be able to run tests blindly by default
without requiring *any* setup.

If some tests requires a specific setup they should gracefully be
skipped (but as part of tarmac no tests should be skipped right ?).

    > Its not perfect, but mathematically speaking is now infinitely
    > better than what we had.

Indeed, thanks a lot for starting that ! 

    > I'd gladly accept patches

Brace yourself ;)

    > that clean up the output or deal with the results better (subunit
    > would be ideal) so its easier to track what exactly failed in an
    > MP.

I couldn't have said it better ;)

My remarks above are indeed targeted at a better test run output. 

  Vincent

[1]: And timing but we'll get to that later ;)