canonical-ci-engineering team mailing list archive

[Evan Dandrea <evan.dandrea@xxxxxxxxxxxxx>]

While busying myself in an airport, I took some terrible notes on a
video the Yelp team put together on building their service on top of
Python:

https://www.youtube.com/watch?v=MLM2acV_1yo

Moving from monolithic to SOA
- lets pieces move independently
Global deps bad (you have to upgrade the world). Use virtualenv.
They're looking into docker. How do we get this with juju? kapil's stuff?
Using pip.
Make sure you always give versions deps in pip. Great post on
kermed(?).io on this.
We're using fit submodules. Now release libraries on a regular basis.
Their Jenkins looks for git tags for releases, builds and push to
their internal pypi server.
They've moved from tornado to pyramid. Very successful migration.
Highly recommended.
Using uwsgi after evaluating gunicorn, modwsi, circus, etc.
Using scribd for logging. Uwsgi to python logging to scribd.
Metrics!
What's the 99th percentile time for this endpoint
Are all services slow or is it just this one
How many queries per second is this endpoint handling?
Which downstream service is killing our performance?
Are any clients using the old API?
Did this new service introduce a performance regression?
Codeahale(?), makers of Metrics mind the gap for Java, gives an
excellent talk on metrics
Wrote their own metrics lib on top of uwsgi
Using uwsgi mules as way of sharing metrics data across many workers
If you bring up a service instance you want it to immediately recv
traffic. If you take it down, you want it to stop recv traffic. Enter
airbnb's nerve and mule
Sounds like juju gives us this
Smartstack from airbnb
One nice thing is that it does a health check before accepting traffic.
Could we automatically take a node out of the pool if it's failing the
health check?
Write an ops dashboard with all of this
Service, commit hash version(links to codebrowse), uptime, qps, p50, p99
Could extend our health check page for these
They encourage Devs to write client libraries that simplify
interacting with each rest service
Using memcache, thinking about nginx for caching
Looking into ssl mutual auth for security between services