Neil is a Technologist in the Office of the CTO at Confluent, the company founded by the creators of Apache Kafka. His role is as an industry expert in the world of streaming, distributed systems and the next generation of technology as the world becomes real-time. This includes working on fun things like event streaming apps, stream processing, serverless, developing crazy ideas and bending Kafka in all kinds of directions. He has over 25 years of expertise in distributed computing, messaging and stream processing. He is an ex-fintech CTO, member of the CNCF serverless working group and startup founder
Today's world operates under a new set of constraints and tries to solve problems that were never thought possible. Everything has changed.
In this talk, I discuss the origins of how Apache Kafka came to be, but also how it now underpins some of the worlds largest microservice eco systems. Kafka and the microservice moment gathered momentum at the same time, Kafka for streams or data-in-motion and microservices as a new, fine grained architectural principle. The thorn in the side for microservices has always been about the ability to handle, state, and scale. Kafka, meanwhile has always been good at distributing state, which means, that now as we build microservices on top of Kafka, there is a natural affinity between them. I will cover the evolution of microservices until the point of today and then work through how we can build stateful scalable microservices using Kafka. I will break down how scaling state works, and also how it can be queried with ‘interactive queries’.
Lastly, I cover key architectural principles of:
o Scalable business processing o Run it on rails: Instrumentation and monitoring o Control flow patterns (start, stop, pause) o Error handling strategies