
Working with API that supports remote streaming using HTTPClient in dotnet
http client
6 Articles
In this article, let's learn about how to work with API's that support Remote Streaming
using HTTP Client
in .NET.
Note: If you have not done so already, I recommend you read the article on Save bandwidth with Compression when sending and reading data using HTTPClient in dotnet.
Table of Contents
Introduction
Streaming
, in essence, is a method of transmitting or receiving data over a network as a
continuous and steady flow, as opposed to receiving it all at once
. While commonly associated with video and audio content,
streaming has also become valuable for web APIs that return JSON data, especially when dealing with large datasets.
In our case, we're focusing on a streaming endpoint
from our API. When we send a request to this endpoint,
the results start arriving one by one, effectively streaming to the consumer in this case, a web browser. This capability has been made possible
thanks to the introduction of IAsyncEnumerable
in ASP.NET Core 3 and changes in
System.Text.Json
to accommodate streaming in ASP.NET Core 6.
Integrating with Streaming API
To better grasp this concept, let's briefly look at the API code.
Code Sample - API Supporting Streaming
At its core, data access begins with the return of an IAsyncEnumerable
. Within the related controller action,
results are yielded
individually to the API consumer, ensuring an asynchronous streaming
process
. For now, this illustrates that data flows asynchronously from the data store to the client via API.
Code Sample - Consuming Data from API that supports Streaming
When it comes to working with the streaming API on the client side, a straightforward HttpClient
approach won't
suffice. Deserializing
the entire stream
into an
IEnumerable
of integer
would cause the client to wait until all the data has
been received before proceeding. In order to support streaming at the client level, we need a different approach.
HttpClient
provides this support through two components:
-
the
IAsyncEnumerable
interface
, which has been available since .NET Core 3 and allowsasynchronous
iteration over a set of values, aligning perfectly with the streaming paradigm. -
When we call
System.Text.Json
's streaming capabilities likeDeserializeAsyncEnumerable
, the result is anasynchronous enumerable
that lets us work on incoming data incrementally.
When we combine these components, we can asynchronously
iterate over the incoming objects from the integer
stream, processing them one by one.
Summary
In this article, we've delved into the world of streaming
a method of transmitting or
receiving data as a continuous flow over a network
. This approach allows us to begin processing parts of the data while the
rest is still being received, offering improved efficiency and responsiveness. On the client side, HttpClient's support for streaming is made
possible through two key components IAsyncEnumerable
Interface and
System.Text.Json
.