Batch and stream processing finally together with Apache Beam
Mete Atamel is a Developer Advocate at Google, currently focused on helping developers with Google Cloud Platform. As a long-time Java and a recent C# developer, he like to compare the two ecosystems. Prior to Google, he worked at Microsoft, Skype, Adobe, EMC, and Nokia building apps and services on various web, mobile and cloud platforms. Originally from Cyprus, he currently live in Greenwich, not too far away from the prime meridian.
In this session, we will first take a look at the history of massive-scale data processing at Google with MapReduce, FlumeJava, MillWheel and how they contributed to the creation of Dataflow, a new paradigm integrating batch and stream processing. Next, we will learn about the details of Dataflow and some of its differentiating elements. In 2016, Dataflow SDK further transformed into an open source project called Apache Beam. We will take a look at Apache Beam and how it enables pipelines to run against multiple backends such as Dataflow, Flink, Spark and more.