{"componentChunkName":"component---src-pages-author-author-yaml-id-js","path":"/author/abhilash-k-r/","result":{"data":{"allMarkdownRemark":{"edges":[{"node":{"id":"14efec93-f52d-5d6c-9d1f-61f22d4eb90d","html":"

We'll talk about Apache Beam in this guide and discuss its fundamental concepts. We will begin by showing the features and advantages of using Apache Beam, and then we will cover basic concepts and terminologies.

\n

Ever since the concept of big data got introduced to the programming world, a lot of different technologies, frameworks have emerged. The processing of data can be categorized into two different paradigms. One is Batch Processing, and the other is Stream Processing.

\n

Different technologies came into existence for different paradigms, solving various big data world problems, for, e.g., Apache Spark, Apache Flink, Apache Storm, etc.

\n

As a developer or a business, it's always challenging to maintain different tech stacks and technologies. Hence, Apache Beam to the rescue!

\n

\n \n

Let's take a sample input. Name the file as input.json

\n
{"name":"abhi", "score":12}\n{"name":"virat", "score":23}\n{"name":"dhoni", "score":45}\n{"name":"rahul", "score": 156}\n{"name": "Edmund"}\n{"name": "Ojha"}
\n

The input should be a newline delimited JSON.

\n

Include the following dependencies in your pom.xml

\n
<dependency>\n    <groupId>org.apache.beam</groupId>\n    <artifactId>beam-sdks-java-core</artifactId>\n    <version>2.24.0</version>\n</dependency>\n\n<dependency>\n    <groupId>org.apache.beam</groupId>\n    <artifactId>beam-runners-direct-java</artifactId>\n    <version>2.24.0</version>\n</dependency>
\n

Let's code the beam pipeline. Follow the steps

\n
    \n
  1. \n

    Create a pipeline.

    \n
    Pipeline pipeLine = Pipeline.create();\n// OR \n// Pipeline pipeLine = Pipeline.create(options);
    \n

    Create a pipeline which binds all the pcollections and transforms. Optionally you can pass the PipelineOptions options if needed.

    \n
    2. \n
  2. \n

    Read the input file

    \n
    PCollection<String> inputCollection = pipeLine.apply("Read My File", TextIO.read().from("input.json"));
    \n

    Use the TextIO transform to read the input files. Every line is a different json record.

    \n
    3. \n
  3. \n

    Apply a transform to filter out the names starting from a vowel

    \n
    PCollection filteredCollection = inputCollection.apply("Filter names starting with vowels", Filter.by(new SerializableFunction<String, Boolean>() {\n\n        public Boolean apply(String input) {\n            ObjectMapper jacksonObjMapper = new ObjectMapper();\n            try {\n                JsonNode jsonNode = jacksonObjMapper.readTree(input);\n                String name = jsonNode.get("name").textValue();\n                return vowels.contains(name.substring(0,1).toLowerCase());\n            } catch (JsonProcessingException e) {\n                e.printStackTrace();\n            }\n            return false;\n        }\n    }))
    \n

    The filter transform takes a SerializableFunction Object where the apply method is overridden. Every json-string record is converted to a JSON. The first character of the name is checked if it's a vowel. The transform is applied to each input JSON record. Based on the boolean value returned, the record is retained or discarded.

    \n
    4. \n
  4. \n

    Write the results to a file

    \n
    inputCollection.apply("write to file", TextIO.write().to("result").withSuffix(".txt").withoutSharding());
    \n

    The results of the Filter transform are stored in a text file using the write method of the TextIO transform. As PCollections are distributed across machines, the results are written to multiple files/shards. To avoid this, we use withoutSharding where all the output is written to a single file.

    \n
    5. \n
\n

Output:

\n
{"name": "Edmund"}\n{"name": "Ojha"}\n{"name":"abhi", "score":12}
\n
\n

Complete Code:

\n
Pipeline pipeLine = Pipeline.create();\n\nfinal Set<String> vowels = new HashSet<String>(Arrays.asList("a","e","i","o","u"));\n\npipeLine.apply("Read My File",\n                TextIO.read().from("input.json"))\n        .apply("Filter names starting with vowels", Filter.by(new SerializableFunction<String, Boolean>() {\n\n            public Boolean apply(String input) {\n                ObjectMapper jacksonObjMapper = new ObjectMapper();\n                try {\n                    JsonNode jsonNode = jacksonObjMapper.readTree(input);\n                    String name = jsonNode.get("name").textValue();\n                    return vowels.contains(name.substring(0,1).toLowerCase());\n                } catch (JsonProcessingException e) {\n                    e.printStackTrace();\n                }\n                return false;\n            }\n        }))\n        .apply("write to file", TextIO.write().to("result").withSuffix(".txt").withoutSharding());\n\npipeLine.run().waitUntilFinish();
\n

For more advanced concepts, refer to the official site - beam.apache.org

\n","frontmatter":{"title":"Apache Beam: A Basic Guide","author":{"id":"Abhilash K R","github":"Better-Boy","avatar":null},"date":"October 16, 2020","updated_date":null,"tags":["Engineering","Big Data","Streaming","Apache Beam","Java"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/17c96c6367455f79bdb854c4608ebaba/ee604/main.png","srcSet":"/static/17c96c6367455f79bdb854c4608ebaba/69585/main.png 200w,\n/static/17c96c6367455f79bdb854c4608ebaba/497c6/main.png 400w,\n/static/17c96c6367455f79bdb854c4608ebaba/ee604/main.png 800w,\n/static/17c96c6367455f79bdb854c4608ebaba/f3583/main.png 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}}},"fields":{"authorId":"Abhilash K R","slug":"/engineering/apache-beam/"}}}]},"authorYaml":{"id":"Abhilash K R","bio":null,"github":"Better-Boy","stackoverflow":"6849682","linkedin":null,"medium":null,"twitter":null,"avatar":null}},"pageContext":{"id":"Abhilash K R","__params":{"id":"abhilash-k-r"}}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}