canonical-django team mailing list archive

[Jamu Kakar <jamshed.kakar@xxxxxxxxxxxxx>]

Hi James,

On Mon, Jun 14, 2010 at 12:14 PM, James Westby
<james.westby@xxxxxxxxxxxxx> wrote:
> On Mon, 14 Jun 2010 11:51:44 +0200, Jamu Kakar <jamshed.kakar@xxxxxxxxxxxxx> wrote:
>> Are you using SQLite for tests with something else for production?
>>
>> We tried this in Landscape, with SQLite for tests and PostgreSQL for
>> production, and eventually switched to using PostgreSQL for
>> everything.  We ran into issues where SQLite didn't behave like
>> PostgreSQL and we want to use PostgreSQL-specific features.  Our
>> motivation for using SQLite was to keep our test suite fast, but in
>> practise we've been able to keep it fast and use PostgreSQL.
>
> That's good to know. We're not at the stage of deploying anything yet,
> we just had our planning sprint last week, but I made the tests default
> to sqlite partly for speed, and partly for ease of setup.
>
> If you have specific tips about quick tests with Django and Postgres
> that would be great to share.

I don't have specific Django advice, but things we've discovered in
Landscape that might help you:

- Make your application run with completely empty databases.  It'll
  probably be completely useless in this configuration, but that's
  fine.  This will create a situation where the only thing you need
  to do before a particular test runs is make sure you have a valid
  schema with empty tables, which in turn will mean that setup cost
  between tests is minimized.

- Create whatever sample data you need in each test.  Use helper
  methods to create sample objects and avoid boilerplate.  Robert's
  testresources package is also a very nice way to encapsulate and
  share common "this test needs a thing" code.

- With PostgreSQL at least, the cost of dropping and recreating
  tables is higher than deleting all the data in every table, when
  you have a small amount of data.  We've found that DELETE FROM is
  faster than TRUNCATE in these cases, too.

- Use a patching system that can upgrade a database to the patch
  level expected by the code and make your test suite do this
  automatically, when it detects a mismatch.  There's some code in
  canonical.schema in the Landscape code base [1] that we really
  ought to move out to a lazr package, but until then you might find
  it useful.  This auto-updating helps to keep branch switching a
  fairly painless process.  You just run 'make check' in the new
  branch without needing to worry about database state.

>> If that's what you're doing, maybe it'll work out for you, but in
>> general I think it's best to test with the same database as you
>> deploy to.
>
> Agreed.
>
>> Behaviour should go in your model, not in your views.  The behaviour
>> exposed by your model should be covered by unit tests.  For that
>> matter, the behaviour of unit tests will also benefit from being
>> covered by unit tests.
>
> I'm not sure I follow the last sentence.

Oops, I meant to say:

For that matter, the behaviour of views will also benefit from being
covered by unit tests.

>> In Landscape we use Zope's test browser, the analog to Django's test
>> client, and it works well for us for the kinds of tests you
>> describe.  We've found that making assertions about HTML content is
>> prone to error, but using the browser to get named controls works
>> well.  For example, we often have assertions about links, like:
>>
>>     link = browser.getLink("Logout")
>>     self.assertEqual("http://localhost/logout";, link.url)
>
> getLink works on the text contained in the <a> tag?

Yep, that's right.  It also takes some other parameters so you can
match on element ID, URL, depending on what you need.  It's here if
you're curious:

http://bazaar.launchpad.net/~landscape/zope3/trunk/annotate/head:/src/zope/testbrowser/browser.py

> I'm working on a project that might help in writing robust tests for
> html, I hope to post to canonical-tech about it soon. Information on
> what makes such tests brittle would be good though.

Sounds cool!  The most brittle things we've found are assertions
about strings in HTML content.  Things like:

    self.assertIn("Logout", browser.contents)

tend to be flaky.

>> These clients breaks down when Javascript enters the picture.  At
>> that point you need something different, like Windmill or Selenium,
>> to emulate a human opening a page in the browser, clicking around,
>> etc.  Also, if you're writing Javascript code you should definitely
>> have unit tests to cover your code.  Landscape has infrastructure
>> for running YUI3-based unit tests that you might want to look at if
>> you're planning on writing Javascript.
>
> I'd love to look at that. Unit tests will help avoid requiring too much
> of windmill/selenium-based tests.

Take a peek at src/canonical/landscape/javascript to see our code
and related tests.  I suggest you talk to Sidnei about the test
suite integration details, he knows more about it than I do.

Thanks,
J.

[1] lp:~landscape/landscape/trunk