canonical-ci-engineering team mailing list archive

[Vincent Ladeuil <vila+ci@xxxxxxxxxxxxx>]

Hi all,

Soon after the Friday meeting I had a weird idea: what if we were using
lxcs for all of our actual physical hosts ?

Basically the idea would be:

- move all hosts to lxcs keeping their actual IP addresses,
- ask IS to host the physical ones on their new IP addresses,
- ask IS to guarantee the firewalls around the machines and the network.

I.e. we keep the actual IP addresses inside the lxcs and let IS use
whatever they see fit.

The primary benefits would be that:

- we separate the move from the IP redeployment,

- we prepare the move to prodstack by taking a snapshot of our existing
  infrastructure (I'm talking about using juju yet ! But knowing that
  everything runs inside lxcs is already a significant step in the right
  direction).

That gives us:

- more time to switch to name-only services (we can start before and
  finish after the move instead on requiring a big switch at move time),

- we can swap an lxc from one physical host to another if one is broken
  or lost or doesn't came up after the move,

- we ensure we've collected all configurations before the move (a kind
  of snapshot) and have even (optimistic target) tested them.

Feedback welcome if I'm missing some obvious part but it seems to me we
need:

- /etc (etckeeper is nearly there already and we talked about deploying
  it everywhere if only as a rough backup),

- kernel (not sure this is a hard constraint apart from the architecture
  itself i386/arm),

- dpkg --list

- some stuff under /var or /usr/lib ? (I can never get that part right
  ;) May etckeeper already takes care of that)

- astute use of existing munin metrics to make sure we know the actual
  constraints on the physical interfaces (network, disk, usb hubs).


If we manage to pull that off before the move, we can be hosted on
different hardware as long as we match minimal constraints (arch,
memory, but what else is truly needed that can't be provided to the
guest ? (even if it's not by the host anymore ?). 

And munin can already tell us what is the peak charge for every? server
to ensure our lxcs are on hosts that can provide what they need.

All feedback highly welcome about whether this apply to all hosts or
part of them, whether this helps us collect all the existing
configurations, whether this provides fallbacks if part of the hosts are
lost, ... anything I may forget ;)

      Vincent