canonical-ci-engineering team mailing list archive

[Vincent Ladeuil <vila+ci@xxxxxxxxxxxxx>]

>>>>> Paul Larson <paul.larson@xxxxxxxxxxxxx> writes:

    > Quick share of something I'm looking at based on discussions we had last
    > week.  I did some experimenting with a logstash+redis+elasticsearch setup
    > last night, and as of this morning, I also have the latest jenkins scripts
    > running on a local machine, running touch_ro jobs, and logging there.

\o/

    > If you're interested to see what it looks like (very barebones
    > setup right now) go to: http://paullarson.net:9292

    > All that's getting logged here is the console messages from the
    > jenkins jobs.

the devscripts package provides the 'annotate-output' utility which
produces stuff like (borrowed from some the u1 jenkins which requires
team membership in ubuntuone-hackers or canonical-isd-hackers :-/):

+ annotate-output sudo -H -u tarmac /mnt/tarmac/run-juju-build
22:14:28 I: Started sudo -H -u tarmac /mnt/tarmac/run-juju-build
22:14:28 O: JUJU_DEPLOY_ENV=dev
22:14:28 O: JUJU_STACK=u1-app-backend
22:14:28 O: JUJU_DEPLOY_BRANCH=lp:~ubuntuone-pqm-team/canonical-is-charms/ubuntuone-servers-deploy
22:14:28 O: + cd /mnt/tarmac
22:14:28 O: + rm -rf deploy

I.e. each line is time-stamped and origin-stamped (I: stdin, O: stdout,
E: stderr).

It's darn handy on jenkins as you get timings for all actions for free
(live saver against timeouts or their fallouts). Not to mention the
distinction between stdout/sdterr with the added benefit that they are
not intermixed (as happens so often in jenkins consoles).

    > I could see this being pretty useful in cases where we find an odd
    > error message and want to know how common it is.  Right now, we
    > try to spot stuff like that and log it by hand in a spreadsheet.
    > This is very time consuming, error prone, and highly likely to
    > miss things that we just don't know we care about *yet*.

Speaking of that, have we investigate using statsd manually for the
metrics we cannot automate ?

    > If you want a quick example, have it search for "Failed to fetch"
    > (yes, it's case sensitive) and you can also turn on things at the
    > bottom like job, message, etc. to see details.

I couldn't get it work :-(

Can we hangout tomorrow so you show me that in a shared screen ? ;)

    > This will show you some previous failed builds today where I was
    > able to reproduce some networking problems after provisioning,
    > which I'm talking about on irc now.

Unfortunately *now* was long gone when I replied and I wasn't able to
follow the pointer ;( 

Do you have an url for that ?

   Vincent