Gunnar Morling on Data Streaming with Debezium

14-06-2018 Antonio Goncalves 0 all

Hi Gunnar, tell us who you are and what lead you into microservices?

I’m an open-source enthusiast and software engineer working at Red Hat, where I’m leading the Debezium project. Debezium is a platform for change data capture (CDC) based on Apache Kafka, allowing you to react to all the inserts, updates and deletes in your database in near real-time. That’s a very powerful tool in the box when working with microservice architectures, e.g. for propagating data between different services or when extracting new microservices from existing monolithic applications.

Before working on Debezium, I was the spec lead for the Bean Validation 2.0 spec (JSR 380) and contributed towards the MicroProfile Config API. Both are very useful APIs for implementing microservices; after all, you should make sure the data you receive from other services is valid and also there’s plenty of need for configuring things like service endpoints etc. Overall, microservices relate to many parts of my work.

What will you be talking about at Voxxed Days Microservices?

I’ll talk about streaming data changes out of your database (e.g. MySQL, Postgres or MongoDB) with Debezium. We’ll discuss different use cases for CDC – e.g. synchronizing data between microservices, but also updating caches, full-text search indexes and others – and how it can be implemented using Debezium and Kafka. In my demo, I’ll show how to set up Debezium, stream change events out of MySQL and push life updates to a web application via WebSockets. That is, if the demo gods are in my favor 🙂

Debezium implements the change data capture pattern, right? Is it the only pattern we can use in a microservice architecture to synchronize data?

It’s not the only one, but certainly one I recommend very much. E.g. you could consider doing (synchronous) REST calls from one service to another, but this easily leads to a high coupling between services. What to do for instance if the called service isn’t available? Should you buffer and retry the requests? Fail the original request of the invoking system? These kinds of issues can be prevented by setting up change data streams to which interested services can subscribe, allowing them to keep their own local (and optimized) view of other services data.