canonical-ci-engineering team mailing list archive

[Celso Providelo <celso.providelo@xxxxxxxxxxxxx>]

Vincent,

You have raised this issue while reviewing on of my cleanup branches
and I decided to move the discussion to the public ML, so the rest of
the team can easily participate.

On Fri, May 9, 2014 at 6:04 AM, Vincent Ladeuil <vila@xxxxxxxxxxxxx> wrote:
> Review: Approve
>
> Lovely cleanup of code and namespace \o/
>
> 337     with LogCapture() as lc:
> 338     logger = logging.getLogger()
> 339     - self.cli.main(args, log=logger)
> 340     + with mock.patch('sys.stdout'):
> 341     + self.cli.main(args, log=logger)
>
> Care to have an out of review discussion about that ? Some ideas: I'd live to see a better story around noisy tests. If I get your intent right here, you're silenting one by ignoring sys.stdout, may be there a re valuable data there to check in the test, also, couldn't we unify all output in loggers ? (Even if that means having one still going to stdout with the same content we would then have a single point to capture/redirect for tests).

Right, as part of this and other previous cleanups I have *quieten*
lots of tests that leaked data through STDOUT, solely because it
bothered me while reading the test suite results (I have broken the
test suite a lot lately).

So, first, we have to agree, as a team, that stdout noise in the test
suite output is a real issue for us, because in practice it does not
really affect the tests results.

I say that because, from what I saw, the relevant output information
that is important for tests is already using logger and is captured
and checked accordingly.

If that is the case, we should probably instrument our default test
runner to do the STDOUT suppressing and register/list tests that are
offending this rule, turning it into a failure once the existing tests
are all fixed (so it will never happen again).

If we all agree, It would be a nice feature to be implemented right
after Phase-0 release, as part of the project "Technical Dept" list
that we could build together.

Im sure there are other testing-infrastructure issues that should be
tackled across the entire project as soon as possible, ideally in the
beginning of a new development cycle to avoid major
disruption/distractions.

That's my $0.02 on this.

-- 
Celso Providelo
celso.providelo@xxxxxxxxxxxxx