{"componentChunkName":"component---src-templates-tag-js","path":"/tags/kafka/","result":{"data":{"site":{"siteMetadata":{"title":"LoginRadius Blog"}},"allMarkdownRemark":{"totalCount":2,"edges":[{"node":{"fields":{"slug":"/engineering/quick-kafka-installation/"},"html":"

In this post, we will look at the step-by-step process for Kafka Installation on Windows. Kafka is an open-source stream-processing software platform and comes under the Apache software foundation.

\n

What is Kafka?

\n

Kafka is used for real-time streams of data, to collect big data, or to do real-time analysis (or both). Kafka is used with in-memory microservices to provide durability and it can be used to feed events to complex event streaming systems and IoT/IFTTT-style automation systems.

\n

Installation :

\n

1. Java Setup:

\n

Kafka requires Java 8 for running. And hence, this is the first step that we should do to install Kafka. To install Java, there are a couple of options. We can go for the Oracle JDK version 8 from the Official Oracle Website.

\n

2. Kafka & Zookeeper Configuration:

\n

Step 1: Download Apache Kafka from its Official Site.

\n

Step 2: Extract tgz via cmd or from the available tool to a location of your choice:

\n
tar -xvzf kafka_2.12-2.4.1.tgz
\n

Step 3: Copy the path of the Kafka folder. Now go to config inside Kafka folder and open zookeeper.properties file. Copy the path against the field dataDir and add /zookeeper-data to the path.

\n

\n \n \n \nStep 4: we have to modify the config/server.properties file. Below is the change:

\n
fileslog.dirs=C:\\kafka\\kafka-logs
\n

Basically, we are pointing the log.dirs to the new folder /data/kafka.

\n

Run Kafka Server:

\n

Step 1: Kafka requires Zookeeper to run. Basically, Kafka uses Zookeeper to manage the entire cluster and various brokers. Therefore, a running instance of Zookeeper is a prerequisite to Kafka.

\n

To start Zookeeper, we can open a PowerShell prompt and execute the below command:

\n
.\\bin\\windows\\zookeeper-server-start.bat .\\config\\zookeeper.properties
\n

If the command is successful, Zookeeper will start on port 2181.

\n

Step 2: Now open another command prompt and change the directory to the kafka folder. Run kafka server using the command:

\n
.\\bin\\windows\\kafka-server-start.bat .\\config\\server.properties
\n

Now your Kafka Server is up and running, you can create topics to store messages. Also, we can produce or consume data directly from the command prompt.

\n

Create a Kafka Topic:

\n
    \n
  1. Open a new command prompt in the location C:\\kafka\\bin\\windows.
    2. \n
  2. Run the following command:
    3. \n
\n
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
\n

Creating Kafka Producer:

\n
    \n
  1. Open a new command prompt in the location C:\\kafka\\bin\\windows
    2. \n
  2. Run the following command:
    3. \n
\n
kafka-console-producer.bat --broker-list localhost:9092 --topic test
\n

Creating Kafka Consumer:

\n
    \n
  1. Open a new command prompt in the location C:\\kafka\\bin\\windows.
    2. \n
  2. Run the following command:
    3. \n
\n
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
\n

If you see these messages on consumer console,Congratulations!!! you all done. Then you can play with producer and consumer terminal bypassing some Kafka messages.

\n","frontmatter":{"date":"August 25, 2020","updated_date":null,"title":"Setting Up and Running Apache Kafka on Windows OS","tags":["Kafka","Windows"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/cf0f7f91117e9c259a3ad57b67a89c15/ee604/messagelog.png","srcSet":"/static/cf0f7f91117e9c259a3ad57b67a89c15/69585/messagelog.png 200w,\n/static/cf0f7f91117e9c259a3ad57b67a89c15/497c6/messagelog.png 400w,\n/static/cf0f7f91117e9c259a3ad57b67a89c15/ee604/messagelog.png 800w,\n/static/cf0f7f91117e9c259a3ad57b67a89c15/f3583/messagelog.png 1200w,\n/static/cf0f7f91117e9c259a3ad57b67a89c15/5707d/messagelog.png 1600w,\n/static/cf0f7f91117e9c259a3ad57b67a89c15/7ddcb/messagelog.png 2700w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Ashish Sharma","github":"ashish8947","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/stream-processing-using-kafka/"},"html":"

Kafka Streams is a Java library developed to help applications that do stream processing built on Kafka. To learn about Kafka Streams, you need to have a basic idea about Kafka to understand better. If you’ve worked with Kafka before, Kafka Streams is going to be easy to understand.

\n

What are Kafka Streams?

\n

Kafka Streams is a streaming application building library, specifically applications that turn Kafka input topics into Kafka output topics. Kafka Streams enables you to do this in a way that is distributed and fault-tolerant, with succinct code.

\n

What is Stream processing?

\n

Stream processing is the ongoing, concurrent, and record-by-record real-time processing of data.

\n

Let us get started with some highlights of Kafka Streams:

\n\n

Kafka Streams Concepts:

\n\n

There are two individual processors in the topology:

\n\n

\n \n \n

\n\n

There is actually a close relationship between streams and tables, the so-called stream-table duality

\n

Implementing Kafka Streams

\n

Let's Start with the Setup using Scala instead of Java. The Kafka Streams DSL for Scala library is a wrapper over the existing Java APIs for Kafka Streams DSL.

\n

To Setup things, we need to create a KafkaStreams Instance. It needs a topology and configuration (java.util.Properties). We also need a input topic and output topic. Let's look through a simple example of sending data from an input topic to an output topic using the Streams API

\n

You can create a topic using the below commands (need to have Kafka pre installed)

\n
kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic inputTopic\nkafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic outputTopic
\n
val config: Properties = {\n    val properties = new Properties()\n    properties.put(StreamsConfig.APPLICATION_ID_CONFIG, "your-application")\n    properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")\n    properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest")\n    properties.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE)\n    properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String())\n    properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String())\n    properties\n    }
\n

StreamsBuilder provide the high-level Kafka Streams DSL to specify a Kafka Streams topology.

\n
val builder: StreamsBuilder = new StreamsBuilder
\n

Creates a KStream from the specified topics.

\n
val inputStream: KStream[String,String] = builder.stream(inputTopic, Consumed.`with`(Serdes.String(), Serdes.String()))
\n

Store the input stream to the output topic.

\n
inputStream.to(outputTopic)(producedFromSerde(Serdes.String(),Serdes.String())
\n

Starts the Streams Application

\n
val kEventStream = new KafkaStreams(builder.build(), config)\nkEventStream.start()\nsys.ShutdownHookThread {\n      kEventStream.close(10, TimeUnit.SECONDS)\n    }
\n

You can send data to the input topic using

\n
kafka-console-producer --broker-list localhost:9092 --topic inputTopic
\n

And can fetch the data from the output topic using

\n
kafka-console-consumer --bootstrap-server localhost:9092 --topic outputTopic --from-beginning
\n

You can add the necessary dependencies in your build file for sbt or pom file for maven. Below is an example for build.sbt.

\n
// Kafka\nlibraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % "2.0.0"\nlibraryDependencies += "javax.ws.rs" % "javax.ws.rs-api" % "2.1" artifacts( Artifact("javax.ws.rs-api", "jar", "jar")) // this is a workaround. There is an upstream dependency that causes trouble in SBT builds.
\n

Let us modify the code a little bit to try out a WordCount example:\nThe code splits the sentences into words and groups by word as a key and the number of occurences or count as value and is being sent to the output topic by converting the KTable to KStream.

\n
val textLines: KStream[String, String] = builder.stream[String, String](inputTopic)\nval wordCounts: KTable[String, Long] = textLines\n\t\t.flatMapValues(textLine => textLine.toLowerCase.split("\\\\W+"))\n\t\t.groupBy((_, word) => word)\n\t\t.count()(materializedFromSerde(Serdes.String(),Serdes.Long()))\n\twordCounts.toStream.to(outputTopic)(producedFromSerde(Serdes.String(),Serdes.Long())
\n

With the above process, we can now implement a simple streaming application or a word count application using Kafka Streams in Scala. If you want to set up kafka on Windows, here is a quick guide to implement apache kafka on windows OS.

\n","frontmatter":{"date":"July 01, 2020","updated_date":null,"title":"Kafka Streams: A stream processing guide","tags":["Scala","Kafka","Kafka Streams"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/b42ed1fb037f4d27ff3e60a004298488/ee604/header.png","srcSet":"/static/b42ed1fb037f4d27ff3e60a004298488/69585/header.png 200w,\n/static/b42ed1fb037f4d27ff3e60a004298488/497c6/header.png 400w,\n/static/b42ed1fb037f4d27ff3e60a004298488/ee604/header.png 800w,\n/static/b42ed1fb037f4d27ff3e60a004298488/f3583/header.png 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Priyadarshan Mohanty","github":"priyadarshan1995","avatar":null}}}}]}},"pageContext":{"tag":"Kafka"}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}