Damien Gasparina on Data dichotomy

08-10-2018 Antonio Goncalves 0 all

Hi Damien, tell us who you are and what lead you into microservices?

Hi there! I am an engineer and an open-source enthusiast working for Confluent, the company founded by the creators of Apache Kafka. In the past couple of years, I have worked closely with clients of all shapes and sizes – from small startups to Tier 1 bank – on various topics such as designing event sourcing systems and data integration.

I believe that event sourcing and microservices are a good that helps to deal with the growing complexity of software development. I expect that this approach will becomes a standard in the future. Furthermore, playing lego with containers is really funny!

What will you be talking about at Voxxed Days Microservices?

I will be talking about one of my favorite topic: how to handle data in a loosely coupled service-based system. While designing microservices, we are taught to encapsulate and decouple one service from another so they can continue to change and grow independently. But on the other side, we still need manipulate a shared dataset. So we find ourselves faced with a a contradiction. A dichotomy: Data systems are all about making data available. Services are about hiding it.

In this talk, I will discuss about the benefits of an immutable changelog of state and why this approach provides most of the benefits of a data warehouse without the inconvenience of centralization.

So we have data on one side, and services on the other side. And now we have streams in the middle. How does all this work together ?

A stream of data is an immutable sequence of state, a bus of data than can be accessed by everyone. This is a convenient structure allowing share state between services: each services is in charge of manipulating its local view of the shared data, making it easy to iterate quickly; yet the immutable stream of state is available as a common source of facts. This approach balance the data dichotomy mentioned previously.