canonical-ci-engineering team mailing list archive

[Celso Providelo <celso.providelo@xxxxxxxxxxxxx>]

Hi folks,

Some of us (Francis, Chris, Ursula and me) were investigating solutions to
avoid sharing Swift credentials across UCI-E components, specially the CLI.

Disclosing the system swift credentials to all end-users is unacceptable,
but CLI should keep the ability to upload files (source packages) to the
UCI-E (read directly to Swift) as input of a new Ticket.

The initial proposal, naturally suggested proxying HTTP uploads. But
dealing with all the load involved with linux tarballs didn't sound
appealing, to say the least.  Then William pointed us to a very cool Swift
feature called TempURL [1].

Basically, Swift provides an API for accessing stored objects from an
end-point HTTP(S) URL. These URLs can be created for GET+HEAD (download) or
PUT+HEAD (upload) with a limited lifetime and a secure one-time signature.

The idea is to leverage tempurls and create a new component for providing
'supervised storage sandboxes' in Swift upon request. This approach suits
the package upload workflow very well because the upload contents are
authenticated (gpg signatures) not the uploader. We will end up proxying
the ticket creation (from an uploaded sandbox) instead of proxying each
individual upload.

More details on the former Package Upload Proxy (PUP, what bad name!)
proposal on [2].

I have started a *small* proof-of-concept project that implements a
thin-wrapper for tempurls in [3]. At some point it will evolve to a
webservice (possibly restish, because it's already in our toolbox) that
will allow creating, populating, validating and authorising a set of files
server-side without using swift credentials.

As we know, the designing of a new self-contained component to an existing
system is always a good opportunity to revisit past-decisions and innovate.
Evan mentioned the 'poor integration tests' issue during our 1:1 call today
and it looks like this new development might be a great scenario for
experimenting new solutions.

For testing swift tempurls easily, since canonistack does not allow it yet
and HP cloud support diverges from OpenStack [4], I had to setup a local
Swift instance, and it felt so 2001 ...

I ended up adjusting a shell script stollen from Martin Pitti [5] for
creating and configuring a bare swift instance within a LXC container.

Creation takes a couple of minutes, but once the container is created it's
runs almost instantly, in a way I would not be bothered by firing it up for
a set of automated tests.

I wonder if we can have a hangout to discuss ideas for hooking Juju and LXC
into UCI-E test suite and provide real infrastructure bits like
Swift/Keystone and RabbitMQ so we can write proper system integration tests
that do not take ages to run.


[1]
http://docs.openstack.org/trunk/config-reference/content/object-storage-tempurl.html

[2]
https://docs.google.com/a/canonical.com/document/d/1ggIP5iIibABWKYBNr9n9HSnrBzi6wq4__g9uzV5QIHw
[3] https://code.launchpad.net/~cprov/+junk/swift-tempurl
[4] http://docs.hpcloud.com/api/object-storage#temp_url-jumplink-span
[5]
http://www.piware.de/2014/03/creating-a-local-swift-server-on-ubuntu-for-testing/

-- 
Celso Providelo
celso.providelo@xxxxxxxxxxxxx