canonical-ci-engineering team mailing list archive

[Celso Providelo <celso.providelo@xxxxxxxxxxxxx>]

On Tue, Mar 31, 2015 at 5:49 PM, Thomi Richards
<thomi.richards@xxxxxxxxxxxxx> wrote:
> Hi,
>
> On Wed, Apr 1, 2015 at 9:16 AM, Celso Providelo
> <celso.providelo@xxxxxxxxxxxxx> wrote:

[snip]

>> Instead of discussing things that represent direct value to our users, as:
>>
>> - How to make ADT wrapper/setups more robust ?
>> - How to deal with deadletter-ed messages ?
>> - How to survive and isolate broken workers ?
>> - Can we show results properly in the Dashboard ?
>>
>
> You're correct, but that's not the full picture. Right now, when we need to
> create a new service, there's a lot of infrastructure that has to be copied
> around. What's worse, is that when we need to change that infrastructure
> code it's a real pain in the ass. Remember when we wanted to change all the
> logging handlers to rotate their log files? That took me over an hour, and
> multiple MPs, when it should have been a one-MP change. Right now I'm slowly
> refining our service code, one service at a time. It's slow, painful work,
> and I'd really like to be able to apply that work across all the services we
> create.

A fact, by the end of this week we will have done 8 microservices
already (quite a lot, TBH) but we haven't invested the time to have
proper in-production testing/monitoring for any of them. My hypothesis
is that if we had done one and stressed it enough in production
(chaos-monkeys) we would have copied over much better code.

Bringing a library just for creating this single place where the
worlds collide is not going to fix this per-se. We need other things
(code-related, not talking about the so-much wanted *unicorn* ...)

Also, it looks like we are *steering* our focus too early from what we
decided to do, why don't we get microservices done right, with proper
deployment automation and in-product checks ? Why going back to
sharing code between services, which is known to cause more pain than
benefit in µservices, before we experiment the full microservices
experience ?

> Another way to think about it is: This code needs to be written; why do you
> think putting it in a library costs more than putting it in the service?
> Where's the additional overhead coming from?

You proposed it ;-) library MPs will have a extra careful
review/approval criteria, double approval (who else than you will be
able to approve my overnight MPs ?), release cycles, etc ...

> A good library can, I think, massively reduce the overhead in creating a new
> service. My goal is to have to write fewer than 10 lines of code to create a
> best-practise no-op rabbitmq-listening uService. Right now it takes ~ 100
> lines.

It probably will, but at the same time it will also block any
development initiative that does not follow what the framework
*dictates*, which for me is way more expensive than the 5 minutes
spent copying files around.

Again, maybe copying files is not our biggest problem, have we consider it ?

And we should, at least, think about the fact that the services (and
the library) code is disposable. It's not meant to live forever, we
still want to be able to drop one or more services off and re-write
them entirely with some other technology if there is a need.

>> We are debating new and old ways to create rigid and artificial
>> development workflows to lock ourselves in a very dark and lonely
>> place where changing code is a burden that should be avoided ...
>>
>
> That's a total straw-man argument.
>
> I understand that this team had a bad experience with ci-utils, but it feels
> to me like your argument boils down to "we had a bad experience with
> ciutils, so let's never consider using libraries ever again". Rather, I'd
> suggest a more nuanced approach: What made ciutils a bad experience? It
> sounds like perhaps backwards compatibility wasn't enforced, and the library
> became a dumping ground for anything and everything. We can avoid those
> traps, and reap the benefits of not duplicating code everywhere.
>

As any library, ci_utils had the same honourable intent of
systematically sharing common code across services. We will try harder
this time ...

Also, my point remains, the sum of time invested in:

 * creating a library,
 * maintaining compatibility,
 * releasing it properly,
 * deciding if a change should be local or common,
 * coordinate service and library changes/deployment,

May be higher than the time we spend copying code around ...

>>
>> My personal opinion is that I'd rather copy the whole worker module
>> and run a full sed renaming for creating a service, and become famous
>> in the process as the "CI copy & paste clown", than invest our energy
>> and bright brains into something that we already know will end up
>> badly.
>>
>
> Again, that's a straw-man argument. We _don't_ "know [it] will end up
> badly".

Point taken, but it's more like "It cannot end up well if it's called
*utils*"  and that totally (dis-)qualifies as an straw-man argument.

Summing up, let's create that code backpack and try to use it wisely
to reduce the copy & paste burden (eats time and is error prone) and
hope it will free our time to work on other bigger & heavier problems
we have been dodging for 5 weeks already (in-production checks and
deployment-automation).

[]
-- 
Celso Providelo
celso.providelo@xxxxxxxxxxxxx