quickly team mailing list archive

[Didier Roche <didrocks@xxxxxxxxxx>]

On Thu, Oct 29, 2009 at 4:09 AM, John Barstow <jbowtie@xxxxxxxxxxxxx> wrote:
> I'm looking at refactoring Quickly in a number of ways to get some
> test coverage, and I wanted to discuss them with the team before
> investing a whole lot more time in them.
Great. For your information, I've already began to create a test suite
too. We should gather our work (which will be hard before UDS for me)

>
> I have a toy implementation in my published testability branch where
> you can see the effects of what I'm talking about. It's not actually
> something that can be merged cleanly, but I can pretty easily create
> and publish a branch for that if my proposal makes sense.
>
> Current Issues
> ------------------
>
> Currently a command is implemented by writing a python (or other)
> script and including it a template. This has several undesirable
> properties.
>
> * Common commands such as edit or glade need to be maintained in each
> template, duplicating code.

Not really, see https://bugs.launchpad.net/quickly/+bug/452306
I have a branch containing some code, I just want to add few features
now to enable/prevent automatic other template command launch.

> * Test modules cannot import individual functions or classes for
> testing (importing the script executes it...)
> * Standards for parsing arguments or reporting errors cannot be
> enforced meaningfully.

That's correct. optparse can't be use as we can have some options and
values concerning template scripts which can be in whatever language.

> * A number of support functions call sys.exit, limiting their
> suitablity for reuse.

See https://bugs.launchpad.net/quickly/+bug/449534
I'm already aware of that and 0.2.5 contained a lot of fixes related
to that. 0.4 should be clean.

> * Commands are mixed with template files, making the source a bit
> harder to navigate for no good reason.

Hum? No, the spec is normally "all commands should be at the root,
having execution right". Other things are in subfolder, see
http://bazaar.launchpad.net/~quickly/quickly/trunk/files/head%3A/data/templates/ubuntu-project/
for ubuntu-project template.


>
> Proposed Solution
> -----------------------
>
> I'd like to rework the whole command infrastructure, following the
> example of Django (which makes it extremely easy to extend the
> manage.py command-line app).
>
> The new functionality will go into a new quickly.management namespace
> to avoid compatibility issues.
>
> Commands should be moved into a new commands directory within the
> template. Quickly will just look in this directory for a given
> template.
>
> All commands will inherit from BaseCommand. The ManagementUtility
> class will be responsible for locating, instantiating, and executing
> commands.  The existing Command class (which executes scripts) will be
> renamed to ShellCommand and inherit from BaseCommand.

This was my first draft on Quickly, but I changed it for dynamic
introspection and seeking for execution rights as we really want
template developers to be able to develop their templates in whatever
languages they want.

I've already met some people developing their own template in
~/quickly-templates in shell and I think that to have the largest
userbase, we don't want to change this now.
See http://blog.didrocks.fr/index.php/post/Build-your-application-quickly-with-Quickly%3A-Inside-Quickly-part-6

>
> The basic flow is like so:
> * ManagementUtility.execute() will parse the argv using OptionParser
> and handle default options (like --version, --help and --template).
> * ManagementUtility.execute() will also handle the generic help (i.e.,
> listing available commands).
> * ManagementUtility.fetch_command() will load and instantiate the
> appropriate command (consulting the template first, then the core and
> finally the built-ins). Note that we consult the template first, so
> templates can easily override the built-ins if required.
> * BaseCommand.print_help() will consult the help, args, and
> option_list attributes to construct the help for individual commands.
> * BaseCommand.execute() will parse the argv with an OptionParser, call
> handle(*args, **kwargs) to do the actual work for a command, catch any
> exceptions and write a nicely formatted message to stderr.
>
> So to implement a command, we inherit from BaseCommand and override:
> *  the handle() function to do actual work. If there's an error, throw
> a CommandError exception instead of calling sys.exit().
> *  the help attribute to provide a short description of the command.
> *  the option_list attribute to customize the list of optparse options
> which will fed into the command's OptionParser.
> *  the args attribute to list the arguments accepted by the command
> (this is used to create the usage line in help messages, so something
> like "SRC DEST" would be appropriate).

That's a great idea, but we can't implement that if we still want
using other template languages than python. That's why we just call
the template with help or shell-completion argument to enable our own
advanced shell completion method.

Note that we have advanced introspective and dynamic completion than
we can't have with OptionParser, taking countext into account and so
on (cf http://blog.didrocks.fr/index.php/post/Build-your-application-quickly-with-Quickly%3A-Inside-Quickly-part-4)

>
> Finally, we create a collection of core commands, like "edit" and
> "glade" which can be reused across many templates, and move them to a
> quickly.management.core namespace. A template can replace these
> commands easily enough and in most cases probably won't.

I'm not sure that every templates wants those commands.
We already have "builtin commands" (again, sorry for quoting my blog
post but all those notions are described there:
http://blog.didrocks.fr/index.php/post/Build-your-application-quickly-with-Quickly%3A-Inside-Quickly-part2)
So, builtin command are commons to every templates and we will have
very soon inheritance model to state command compatibility and
incompatibility.

>
> Benefits
> ----------
>
> Commands no longer have to worry about parsing arguments or handling
> the default options.
> Commands can be easily tested by calling handle() directly with
> appropriate test arguments. This means that test suites can also set
> up and tear down any resources in /tmp needed to test a command.
> Templates that want to reuse existing commands can do so without
> maintaining duplicate code.
> We can colorize terminal output - my testability branch prints errors
> in bold red, for example - by implementing functionality in
> BaseCommand.

I'm really aware of the benefits and again, thanks to proposing that, but :
1- that prevents having multiple languages in templates, which was the
main reason of Quickly design
2- we won't have access anymore to "advanced completion"

>
> Testing
> ---------
>
> Once we have reworked this infrastructure, I'd like to propose a new
> core command - "quickly test". This should instantiate a TestRunner
> that locates and executes all unit tests and doctests in a quickly
> project. We should also generate a "tests" directory containing a
> basic unit test when a project is created.
>
> For quickly itself, we can reuse the test runner to run any doc tests
> or unit tests (which I will happily write!).

Yeah, I must have already done the test suite earlier, I know, but
it's in progress. I'll happily share with you that once UDS has passed
(do you attend to it, we will have some Quickly session?).

On the design tests, we agree and that's what I've already begun. I'll
also write some overrides on modules depending of the network
(launchpadlib for instance) so that we can launch our testsuite in an
isolated area.

Once my first draft available, I'm really happy to share that with
your (more advanced than I, I guess) knowledge on that subject so that
Quickly 0.4 is really "strong, but quickly" :)

>
>
> OK, I think that's everything. Thoughts, anyone?
>

I just think you miss the big starting point despite your excellent
suggestions. :)
If you didn't read my blog post, I urge to do it so that we can have
the same bases. There are a lot of areas where quickly is weak (test
suite for instance, as already stated ;)).

Welcome on board, I hope that this won't prevent your for starting
contributing to Quickly. Do not hesitate to come on #quickly in
freenode so that we can discuss together :)


Didier