{"componentChunkName":"component---src-templates-tag-js","path":"/tags/streaming/","result":{"data":{"site":{"siteMetadata":{"title":"LoginRadius Blog"}},"allMarkdownRemark":{"totalCount":2,"edges":[{"node":{"fields":{"slug":"/engineering/guest-post/http-streaming-with-nodejs-and-fetch-api/"},"html":"

When your webapp has a large amount of data to visualize, you don't want your users to wait 10 seconds before seeing something.

\n

One technique that is often overlooked is HTTP streaming. It's broadly supported, works well, and doesn't require fancy libraries.

\n

We're going to go through how we can use HTTP streaming in our applications and what to consider when we do so.

\n

Introduction

\n

When building web applications, we typically have a REST API with GET endpoints that do something like this:

\n
    \n
  1. Parse the request (URL, query params, etc..)
    2. \n
  2. Query data from a database
    3. \n
  3. Convert the database results into JSON
    4. \n
  4. Send the JSON response back
    5. \n
\n

The API will typically wait for each step to complete before going on to the next one, and by step 4, the database result and the JSON objects are all in memory before the request is handled, and everything can be cleaned up.

\n

This works, and there is nothing wrong with it (KISS, right?) as long as your database query results are small and quickly available.

\n

But let's say you want to render a chart with 10k data points. Querying will not be as smooth anymore.\nThe simple way will work, but ideally, you don't want to accumulate all the data in memory before sending the response.

\n

With HTTP streaming, you can start rendering the chart even before your query is complete.

\n

To make it happen:

\n
    \n
  1. Your API should use HTTP streaming to send its response.
    2. \n
  2. Your webapp should use the Fetch API to make the request so that it can process the streaming response.
    3. \n
\n

Create a Streaming API

\n

In this example, we use Koa for the API, but you can use other libraries like Express or plain Node.js. Most will have support for streaming.

\n

Let's create an API:

\n
const Koa = require('koa');\nconst app = new Koa();\n\napp.use(async (ctx) => {\n    if (ctx.request.url === '/measurements.json') {\n        ctx.response.set('content-type', 'application/json');\n        // This is where the magic happens: set a stream as the response body\n        ctx.body = fs.createReadStream('./measurements.json');\n    }\n});\n\nhttp.createServer(app.callback()).listen(3000);
\n

This code creates an API with 1 endpoint GET /measurements that will respond with the contents of a JSON file with 10k measurements.

\n

The file looks like this:

\n
[\n  { "id": "1", "timestamp":  "2022-01-19T10:39:00.000Z", "value":  239.34 },\n  { "id": "2", "timestamp":  "2022-01-19T10:40:00.000Z", "value":  820.14 },\n  { "id": "3", "timestamp":  "2022-01-19T10:41:00.000Z", "value":  926.03 },\n  { "id": "4", "timestamp":  "2022-01-19T10:42:00.000Z", "value":  513.01 },\n  // ...\n  { "id": "99998", "timestamp":  "2022-03-29T21:16:00.000Z", "value":  13.81 },\n  { "id": "99999", "timestamp": "2022-03-29T21:17:00.000Z", "value": 465.28 },\n  { "id": "100000", "timestamp":  "2022-03-29T21:18:00.000Z", "value":  71.95 }\n]
\n

We're creating an HTTP/1.1 server with Node's http module. By setting ctx.body to a stream, data will be sent to the requester as soon as it is loaded using a mechanism called chunked transfer encoding.

\n

This saves time and memory of your API because it doesn't have to accumulate the whole result in memory before sending the response.

\n

In a real application, you're probably using a database instead of a pre-created JSON file.\nIf you're using MongoDB, you can create a stream with the cursor's stream() method.

\n

Consume a Streaming API From a Webapp

\n

The GET /measurements endpoint we created can be consumed with any HTTP client, but you have to use the Fetch API to take advantage of streaming.

\n

fetch() will set response.body to a ReadableStream for streaming responses.

\n

Here's how you can read a streaming response:

\n
fetch('http://localhost:3000/measurements.json')\n    .then(async (response) => {\n        // response.body is a ReadableStream\n        const reader = response.body.getReader();\n        for await (const chunk of readChunks(reader)) {\n            console.log(`received chunk of size ${chunk.length}`);\n        }\n    });\n\n// readChunks() reads from the provided reader and yields the results into an async iterable\nfunction readChunks(reader) {\n    return {\n        async* [Symbol.asyncIterator]() {\n            let readResult = await reader.read();\n            while (!readResult.done) {\n                yield readResult.value;\n                readResult = await reader.read();\n            }\n        },\n    };\n}
\n

If you run the API that we created and then run the code above in a web browser, you'll see received chunk of size x several times in the console with varying x:

\n

\"The

\n

To see it more clearly, open up Developer Tools (F12) and set network throttling to 3G. It will take longer for the file to download, so you can see that the chunks are being processed gradually.

\n

That's all very nice, but the chunks themselves are not very useful. You want to process the measurements that are in these chunks. Ideally, you would have an async iterable that gradually yields the JSON objects as they come in so that you can use for await to iterate over the measurements instead of the chunks.

\n

JavaScript doesn't have a JSON parser that can deal with streams. Let's assume we have a function that can do this called parseJsonStream(readableStream). More details and a simple implementation can be found in \"Streaming Considerations\" below.

\n

We would then be able to do this:

\n
fetch('http://localhost:3000/measurements.json')\n    .then(async (response) => {\n        let measurementsReceived = 0;\n        for await (const measurement of parseJsonStream(response.body)) {\n            measurementsReceived++;\n            // To prevent the console from flooding we only show 1 in every 100 measurements\n            if (measurementsReceived % 100 === 0) {\n                console.log(`measurement with id {${measurement.id}} at time ${measurement.timestamp} has value [${measurement.value}]`);\n            }\n        }\n    });
\n

This is the result:

\n

\"The

\n

This is a very powerful mechanism. Instead of console.log() statements, imagine a line chart where the measurements gradually become visible as more data comes in.

\n

This is an example created with Apache ECharts:

\n

\"A

\n

Before the Fetch API, it was impossible to do this because the alternative, XMLHttpRequest, will load the whole response into memory before providing it to your code.

\n

Modern applications don't use XMLHttpRequest directly, but a lot of libraries like Axios or Angular's HttpClient rely on it to make requests.

\n

People would rely on more advanced technologies like WebSockets to stream data.

\n

HTTP/2 for Streaming

\n

You might have heard that HTTP/2 has a more efficient mechanism for streaming, and you would be right. So let's see if we can use HTTP/2.

\n

From the side of your webapp, the browser will automatically determine if it can communicate with the server over HTTP/2 and use it if it's available. The Fetch API will do this transparently, so you do not need to make any changes to your webapp.

\n

What you need to do is make your API available over HTTP/2:

\n
const Koa = require('koa');\nconst app = new Koa();\n\napp.use(async (ctx) => {\n    if (ctx.request.url === '/measurements.json') {\n        ctx.response.set('content-type', 'application/json');\n        ctx.body = fs.createReadStream('./measurements.json');\n    }\n});\n\nhttp2.createSecureServer(\n    {\n        key: fs.readFileSync(path.resolve('path/to/localhost-key.pem')),\n        cert: fs.readFileSync(path.resolve('path/to/localhost.pem')),\n    },\n    app.callback(),\n).listen(3000);
\n

This code is mostly the same as for HTTP/1.1 with two notable differences:

\n\n

That's it!

\n

If you restart the API and check the network tab in Developer Tools, you'll see that your application will now stream over HTTP/2 (don't forget to update the URL in your webapp, start with https:// instead of http://).

\n

HTTP Streaming Considerations

\n

In the code above, we assumed that there was a function parseJsonStream(readableStream) that would parse a ReadableStream containing JSON into an async iterable of objects.

\n

The difficulty is that reading from a ReadableStream will give you chunks of data that don't necessarily correspond to anything meaningful. To illustrate, let's take a look at this example:

\n
[\n  { "id": "1", "timestamp":  "2022-01-19T10:39:00.000Z", "value":  239.34 },\n  { "id": "2", "timestamp":  "2022-01-19T10:40:00.000Z", "value":  820.14 },\n  { "id": "3", "timestamp":  "2022-01-19T10:41:00.000Z", "value":  926.03 },\n  { "id": "4", "timestamp":  "2022-01-19T10:42:00.000Z", "value":  513.01 }\n]
\n

We could receive this JSON in chunks like this:

\n
[\n  { "id": "1", "timestamp":  "2022-01-19T10:39:00.000Z", "value":  239.34 },\n  { "id": "2", "tim\n\n\n\n estamp":  "2022-01-19T10:40:00.000Z", "value":  820.14 },\n  { "id": "3", "timestamp":  "2022-01-19T10:41:00.000Z", "value":  926.03 },\n  { "id": "4", "timestamp":  "2022-01-19T10:42\n\n\n\n:00.000Z", "value":  513.01 }\n]
\n

You need some way to determine where one measurement object starts and ends. To simplify this, we created a JSON file where each line contains precisely one object. Parsing the stream becomes manageable when we can make this assumption.

\n

Here is the implementation of parseJsonStream(readableStream)

\n
async function *parseJsonStream(readableStream) {\n    for await (const line of readLines(readableStream.getReader())) {\n        const trimmedLine = line.trim().replace(/,$/, '');\n\n        if (trimmedLine !== '[' && trimmedLine !== ']') {\n            yield JSON.parse(trimmedLine);\n        }\n    }\n}\n\nasync function *readLines(reader) {\n    const textDecoder = new TextDecoder();\n    let partOfLine = '';\n    for await (const chunk of readChunks(reader)) {\n        const chunkText = textDecoder.decode(chunk);\n        const chunkLines = chunkText.split('\\n');\n        if (chunkLines.length === 1) {\n            partOfLine += chunkLines[0];\n        } else if (chunkLines.length > 1) {\n            yield partOfLine + chunkLines[0];\n            for (let i=1; i < chunkLines.length - 1; i++) {\n                yield chunkLines[i];\n            }\n            partOfLine = chunkLines[chunkLines.length - 1];\n        }\n    }\n}\n\nfunction readChunks(reader) {\n    return {\n        async* [Symbol.asyncIterator]() {\n            let readResult = await reader.read();\n            while (!readResult.done) {\n                yield readResult.value;\n                readResult = await reader.read();\n            }\n        },\n    };\n}
\n

If you have control over the API you're calling, you can use the \"1 object per line\" formatting as part of the contract, but know that it could be prone to breaking. For robust JSON support, we need a real streaming parser.

\n

Other options include using a format with one object per line by default, like CSV, or a more advanced format with built-in support for streaming like Apache Arrow.

\n

Advantages

\n
    \n
  1. Snappy User Experience: You can start showing data as soon as it's available.
    2. \n
  2. Scalable API: No memory usage spikes from accumulating results in memory.
    3. \n
  3. Uses plain HTTP and a standard JavaScript API. There are no connections to manage or complicated frameworks that might become obsolete in a few years.
    4. \n
\n

Disadvantages

\n
    \n
  1. Implementation is slightly more involved than using regular API calls.
    2. \n
  2. \n

    Error handling becomes more difficult because HTTP status code 200 will be sent as soon as streaming starts. What do we do when something goes wrong in the middle of the stream?

    \n

    When something goes wrong, your API should close the stream. Your webapp can then determine if the stream was complete and show a fitting message to the user if it's not. For example: when using a JSON response, as we discussed, you can check that the last line contains only \"]\".

    \n
    3. \n
  3. No streaming JSON parser is currently available. Needs formatting assumptions as part of the contract or a more unconventional format.
    4. \n
\n

Conclusion

\n

You have learned how to stream HTTP data efficiently using Node.js and HTTP without congesting or burdening the memory. You have also understood the advantages and disadvantages of HTTP streaming.

\n","frontmatter":{"date":"April 27, 2022","updated_date":null,"title":"Implement HTTP Streaming with Node.js and Fetch API","tags":["Node.js","Fetch API","Streaming"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/15f56fac896fb3f3370431c08e4ffbac/ee604/http-streaming-with-nodejs-and-fetch-api.png","srcSet":"/static/15f56fac896fb3f3370431c08e4ffbac/69585/http-streaming-with-nodejs-and-fetch-api.png 200w,\n/static/15f56fac896fb3f3370431c08e4ffbac/497c6/http-streaming-with-nodejs-and-fetch-api.png 400w,\n/static/15f56fac896fb3f3370431c08e4ffbac/ee604/http-streaming-with-nodejs-and-fetch-api.png 800w,\n/static/15f56fac896fb3f3370431c08e4ffbac/f3583/http-streaming-with-nodejs-and-fetch-api.png 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Nick Van Nieuwenhuyse","github":"Nivani","avatar":null}}}},{"node":{"fields":{"slug":"/engineering/apache-beam/"},"html":"

We'll talk about Apache Beam in this guide and discuss its fundamental concepts. We will begin by showing the features and advantages of using Apache Beam, and then we will cover basic concepts and terminologies.

\n

Ever since the concept of big data got introduced to the programming world, a lot of different technologies, frameworks have emerged. The processing of data can be categorized into two different paradigms. One is Batch Processing, and the other is Stream Processing.

\n

Different technologies came into existence for different paradigms, solving various big data world problems, for, e.g., Apache Spark, Apache Flink, Apache Storm, etc.

\n

As a developer or a business, it's always challenging to maintain different tech stacks and technologies. Hence, Apache Beam to the rescue!

\n

\n \n

Let's take a sample input. Name the file as input.json

\n
{"name":"abhi", "score":12}\n{"name":"virat", "score":23}\n{"name":"dhoni", "score":45}\n{"name":"rahul", "score": 156}\n{"name": "Edmund"}\n{"name": "Ojha"}
\n

The input should be a newline delimited JSON.

\n

Include the following dependencies in your pom.xml

\n
<dependency>\n    <groupId>org.apache.beam</groupId>\n    <artifactId>beam-sdks-java-core</artifactId>\n    <version>2.24.0</version>\n</dependency>\n\n<dependency>\n    <groupId>org.apache.beam</groupId>\n    <artifactId>beam-runners-direct-java</artifactId>\n    <version>2.24.0</version>\n</dependency>
\n

Let's code the beam pipeline. Follow the steps

\n
    \n
  1. \n

    Create a pipeline.

    \n
    Pipeline pipeLine = Pipeline.create();\n// OR \n// Pipeline pipeLine = Pipeline.create(options);
    \n

    Create a pipeline which binds all the pcollections and transforms. Optionally you can pass the PipelineOptions options if needed.

    \n
    2. \n
  2. \n

    Read the input file

    \n
    PCollection<String> inputCollection = pipeLine.apply("Read My File", TextIO.read().from("input.json"));
    \n

    Use the TextIO transform to read the input files. Every line is a different json record.

    \n
    3. \n
  3. \n

    Apply a transform to filter out the names starting from a vowel

    \n
    PCollection filteredCollection = inputCollection.apply("Filter names starting with vowels", Filter.by(new SerializableFunction<String, Boolean>() {\n\n        public Boolean apply(String input) {\n            ObjectMapper jacksonObjMapper = new ObjectMapper();\n            try {\n                JsonNode jsonNode = jacksonObjMapper.readTree(input);\n                String name = jsonNode.get("name").textValue();\n                return vowels.contains(name.substring(0,1).toLowerCase());\n            } catch (JsonProcessingException e) {\n                e.printStackTrace();\n            }\n            return false;\n        }\n    }))
    \n

    The filter transform takes a SerializableFunction Object where the apply method is overridden. Every json-string record is converted to a JSON. The first character of the name is checked if it's a vowel. The transform is applied to each input JSON record. Based on the boolean value returned, the record is retained or discarded.

    \n
    4. \n
  4. \n

    Write the results to a file

    \n
    inputCollection.apply("write to file", TextIO.write().to("result").withSuffix(".txt").withoutSharding());
    \n

    The results of the Filter transform are stored in a text file using the write method of the TextIO transform. As PCollections are distributed across machines, the results are written to multiple files/shards. To avoid this, we use withoutSharding where all the output is written to a single file.

    \n
    5. \n
\n

Output:

\n
{"name": "Edmund"}\n{"name": "Ojha"}\n{"name":"abhi", "score":12}
\n
\n

Complete Code:

\n
Pipeline pipeLine = Pipeline.create();\n\nfinal Set<String> vowels = new HashSet<String>(Arrays.asList("a","e","i","o","u"));\n\npipeLine.apply("Read My File",\n                TextIO.read().from("input.json"))\n        .apply("Filter names starting with vowels", Filter.by(new SerializableFunction<String, Boolean>() {\n\n            public Boolean apply(String input) {\n                ObjectMapper jacksonObjMapper = new ObjectMapper();\n                try {\n                    JsonNode jsonNode = jacksonObjMapper.readTree(input);\n                    String name = jsonNode.get("name").textValue();\n                    return vowels.contains(name.substring(0,1).toLowerCase());\n                } catch (JsonProcessingException e) {\n                    e.printStackTrace();\n                }\n                return false;\n            }\n        }))\n        .apply("write to file", TextIO.write().to("result").withSuffix(".txt").withoutSharding());\n\npipeLine.run().waitUntilFinish();
\n

For more advanced concepts, refer to the official site - beam.apache.org

\n","frontmatter":{"date":"October 16, 2020","updated_date":null,"title":"Apache Beam: A Basic Guide","tags":["Engineering","Big Data","Streaming","Apache Beam","Java"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/17c96c6367455f79bdb854c4608ebaba/ee604/main.png","srcSet":"/static/17c96c6367455f79bdb854c4608ebaba/69585/main.png 200w,\n/static/17c96c6367455f79bdb854c4608ebaba/497c6/main.png 400w,\n/static/17c96c6367455f79bdb854c4608ebaba/ee604/main.png 800w,\n/static/17c96c6367455f79bdb854c4608ebaba/f3583/main.png 1200w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Abhilash K R","github":"Better-Boy","avatar":null}}}}]}},"pageContext":{"tag":"Streaming"}},"staticQueryHashes":["1171199041","1384082988","2100481360","23180105","528864852"]}