Jeremy Irons
Kafka
Search results
Franz Kafka (3 July 1883 – 3 June 1924) was a German-language novelist and writer from Prague. He is widely regarded as one of the major figures of 20th-century literature. His work fuses elements of realism and the fantastic.
Apache Kafka is a scalable, high-performance, and fault-tolerant platform for data pipelines, streaming analytics, and mission-critical applications. Learn how Kafka is used by thousands of companies across various industries and how to connect, process, and store streams of events.
- Introduction
- Why Kafka? Benefits and Use Cases
- Kafka Architecture – Fundamental Concepts
- Kafka Components and Ecosystem
Apache Kafka is an event streaming platform used to collect, process, store, and integrate data at scale. It has numerous use cases including distributed streaming, stream processing, data integration, and pub/sub messaging. In order to make complete sense of what Kafka does, we'll delve into what an event streaming platform is and how it works. So...
Kafka is used by over 100,000 organizations across the world and is backed by a thriving community of professional developers, who are constantly advancing the state of the art in stream processing together. Due to Kafka's high throughput, fault tolerance, resilience, and scalability, there are numerous use cases across almost every industry - from...
Kafka Topics
Events have a tendency to proliferate—just think of the events that happened to you this morning—so we’ll need a system for organizing them. Kafka’s most fundamental unit of organization is the topic, which is something like a table in a relational database. As a developer using Kafka, the topic is the abstraction you probably think the most about. You create different topics to hold different kinds of events and different topics to hold filtered and transformed versions of the same kind of e...
Kafka Partitioning
If a topic were constrained to live entirely on one machine, that would place a pretty radical limit on the ability of Kafka to scale. It could manage many topics across many machines—Kafka is a distributed system, after all—but no one topic could ever get too big or aspire to accommodate too many reads and writes. Fortunately, Kafka does not leave us without options here: It gives us the ability to partitiontopics. Partitioning takes the single topic log and breaks it into multiple logs, eac...
How Kafka Partitioning Works
Having broken a topic up into partitions, we need a way of deciding which messages to write to which partitions. Typically, if a message has no key, subsequent messages will be distributed round-robin among all the topic’s partitions. In this case, all partitions get an even share of the data, but we don’t preserve any kind of ordering of the input messages. If the message does have a key, then the destination partition will be computed from a hash of the key. This allows Kafka to guarantee t...
If all you had were brokers managing partitioned, replicated topics with an ever-growing collection of producers and consumers writing and reading events, you would actually have a pretty useful system. However, the experience of the Kafka community is that certain patterns will emerge that will encourage you and your fellow developers to build the...
Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala . The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
- Get Kafka. Download the latest Kafka release and extract it: $ tar -xzf kafka_2.13-2.8.0.tgz $ cd kafka_2.13-2.8.0.
- Start the Kafka environment. NOTE: Your local environment must have Java 8+ installed. Run the following commands in order to start all services in the correct order
- Create a topic to store your events. Kafka is a distributed event streaming platform that lets you read, write, store, and process events (also called records or messages in the documentation) across many machines.
- Write some events into the topic. A Kafka client communicates with the Kafka brokers via the network for writing (or reading) events. Once received, the brokers will store the events in a durable and fault-tolerant manner for as long as you need—even forever.
Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data.
Learn what Apache Kafka is, how it works, and why it is useful for real-time data processing. Find out how to use Kafka as a managed service on Google Cloud with Confluent Cloud.
