I ❤️ .NET

In this article, let's learn about how to work with API's that support Remote Streaming using HTTP Client in .NET.

Note: If you have not done so already, I recommend you read the article on Save bandwidth with Compression when sending and reading data using HTTPClient in dotnet.

Table of Contents

1. Introduction
2. Integrating with Streaming API
3. Summary

Introduction

Streaming, in essence, is a method of transmitting or receiving data over a network as a continuous and steady flow, as opposed to receiving it all at once. While commonly associated with video and audio content, streaming has also become valuable for web APIs that return JSON data, especially when dealing with large datasets.

In our case, we're focusing on a streaming endpoint from our API. When we send a request to this endpoint, the results start arriving one by one, effectively streaming to the consumer in this case, a web browser. This capability has been made possible thanks to the introduction of IAsyncEnumerable in ASP.NET Core 3 and changes in System.Text.Json to accommodate streaming in ASP.NET Core 6.

Integrating with Streaming API

To better grasp this concept, let's briefly look at the API code.

At its core, data access begins with the return of an IAsyncEnumerable. Within the related controller action, results are yielded individually to the API consumer, ensuring an asynchronous streaming process. For now, this illustrates that data flows asynchronously from the data store to the client via API.

When it comes to working with the streaming API on the client side, a straightforward HttpClient approach won't suffice. Deserializing the entire stream into an IEnumerable of integer would cause the client to wait until all the data has been received before proceeding. In order to support streaming at the client level, we need a different approach.

HttpClient provides this support through two components:

1. the IAsyncEnumerable interface, which has been available since .NET Core 3 and allows asynchronous iteration over a set of values, aligning perfectly with the streaming paradigm.
2. When we call System.Text.Json's streaming capabilities like DeserializeAsyncEnumerable, the result is an asynchronous enumerable that lets us work on incoming data incrementally.

When we combine these components, we can asynchronously iterate over the incoming objects from the integer stream, processing them one by one.

Summary

In this article, we've delved into the world of streaming a method of transmitting or receiving data as a continuous flow over a network. This approach allows us to begin processing parts of the data while the rest is still being received, offering improved efficiency and responsiveness. On the client side, HttpClient's support for streaming is made possible through two key components IAsyncEnumerable Interface and System.Text.Json.