Hi,
Thanks for the trip report, there looks like some really interesting things
there.
I've been thinking about event-sourced services and the CQRS pattern for a
while now. It seems like a reasonably good strategy for a service where
data consistency is important. I don't think we have built such a service
yet. However, imagine a hypothetical example.....
We want to build a UI showing the current state of the proposed-migration
system for developers. It should show what packages are being processed
("in-flight"), where in the system they are, as well as the results of any
that have completed, in chronological order.
The issue we hit straight away is that you can't easily inspect the
contents of a rabbit queue without de-queueing and re-queueing all the
messages. Let's not do that.
We could solve this with a service (package-tracker) that listened on HTTP.
All the other services would send it messages ("I got a new request for
package 'linux'", or "I'm starting to test package 'autopilot'", or "I'm
publishing results for package 'unity'"). The package-tracker could then
store those in a database, and provide a nice API & UI on top of the
database.
That would work, but every time we need to re-deploy the service we'd need
to either:
- Copy the old database file to the new host (if it's using something
on-disk like sqlite, json flat file or whatever). This makes our
deployments more icky than we'd want, I think
- Point the new instance at the separate postgres server. This means we
now have to keep a sql database schema compatible, or deal with schema
upgrades. noooooo!
- Use event-sourced service (see below)
With an event sourced model, we would:
1. Use a tool to store those events (new package, start testing, publish
results etc) in a queue. We could use rabbit, but there are better systems
out there.
2. Build a service that connects to that tool, and when it starts, builds
an in-memory model of everything that's happened by replaying *all* events,
since the dawn of time.
This is surprisingly efficient:
- You can scan millions of events in seconds.
- If you ever get too many events, you can safely discard the oldest
ones.
- Since we're only showing a UI on the last N days (a week? a month?)
you can discard packages from your in-memory model as soon as you get a
"published results" message with a timestamp outside your time-window.
There are some disadvantages as well though:
- You need to maintain compatibility in the service for old event
formats. Then again, if you do break compatibility, you can just blow away
the event queue and your UI will be empty until it gets repopulated.
- It requires a separate event queue system, maintenance etc. etc.
Fowler does a better job of explaining CQRS than I can:
http://martinfowler.com/bliki/CQRS.html
If we ever build such a system, I think we should consider something like
this....
Cheers,
On Sun, May 10, 2015 at 2:27 AM, psivaa <para.siva@xxxxxxxxxxxxx> wrote:
Hi,
Here comes the trip report as promised,
There were 3 talks in this meet up, containing a lot very
interesting content. They also had some debatable content. Because it
was about the subject of uServices, the authors had their own take on
the key part of what micro services are for and advocated the communication
methods based on that. **This** should not be new to us :P.
The link to the whole meet-up is:
https://skillsmatter.com/meetups/7005-inter-service-communication
The report on it is in:
https://docs.google.com/document/d/1ejlPMGwWrpjtmdlfqUGFDSsJ8YKRqSXTltkgZowT84M/edit#
Thanks
--
Mailing list: https://launchpad.net/~canonical-ci-engineering
Post to : canonical-ci-engineering@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~canonical-ci-engineering
More help : https://help.launchpad.net/ListHelp
