I ❤️ .NET

In this article, let's learn about how to improve performance and memory use while accessing APIs with Stream using HTTP Client in .NET.

Table of Contents

1. Introduction
2. Advantages of working with Stream
3. Clarifying Streams
4. Using Stream while reading data from API
5. Using Stream while sending data to API
6. Summary

Introduction

Streams help us to keep the memory footprint of our application as low as possible and the performance high. In this article let's get clarified with Stream and learn the advantages of working with Streams and use that knowledge with HTTP Client. We will read the data using Streams and send the data using Streams and get the performance improvement.

A Stream is an abstract representation of a sequence of bytes, which can be data like files, input/output devices, or network traffic. Streams offer a unified way to interact with various types of input and output without getting into specific technical details of the underlying system or devices.

Advantages of working with Stream

Stream enable both reading from and writing to above mentioned data sources, even in cases of network traffic and in-memory objects. This eliminates the need for large in-between temporary variables thus helps in,

• Reducing memory usage
• Enhancing performance

Clarifying Streams

Before we proceed, let's address a potential confusion about the term Stream. The stream I'm referring to now is the abstraction we establish using the Stream class. This abstraction represents a coding concept that handles sequences of bytes. It applies to code written at various levels, including,

• Clients
• APIs
• Intermediatery Stages

Importantly, these levels are distinct. APIs don't necessarily require streams for clients to benefit, and vice versa. The knowledge gained in this article is universally applicable, regardless of the API's coding approach.

Shifting to the context of data transfer, Stream also refers to the movement of data. This data transfer concept can lead to confusion. Since we're working with HTTP, it involves the transmission of bytes from or to the API. As HTTP operates over TCP, packets have a maximum size of 64 KB, often smaller. Whether it's streaming movies, browsing a site, or interacting with an HTTP-based API, data arrives sequentially in packages.

However, the confusion doesn't end there. The API can manage data streaming in different ways. It can push all the data at once or stream it asynchronously, allowing gradual transmission. With asynchronous streaming support, the API can send response portions individually over the network as a continuous flow. For example, data could come directly from an underlying database. This distinction doesn't directly involve client-side stream support.

Using Stream while reading data from API

Let's examine what occurs when we call our API. Our focus will be on the response, as we're beginning with data retrieval. So, when a request is made for instance, to obtain a any data the response is received. Because of HTTP's nature, the content arrives in a continuous flow over the network, regardless of whether the client or API is designed for streaming or supports it. Once all the content is received, we can extract it from response.Content.

Mostly I notice developers achieving this using ReadAsStringAsync, which generates an in-memory string equivalent to the entire data response body. Following this, the string is transformed through deserialization into a data object using System.Text.Json. This is the current process. Several aspects could be enhanced here. Let's begin with the intermediary string creation. Rather than deserializing the object from a string, we can utilize a stream. This approach eliminates the need for a potentially large intermediary string. The string doesn't have to be generated in memory beforehand. Consequently, this reduces memory usage and enhances performance. Let's explore how we can implement this improvement.

Using Stream while sending data to API

When we transmit data to an API, let's say a any data, a similar process occurs in reverse as when we receive a response. We create a data object, convert it into a JSON string through serialization resulting in an in-memory string and then transmit the entire content all at once. However, it's conceivable to enhance this procedure.

Instead of serializing to a JSON string, we can utilize a stream to bypass this step, thereby avoiding the need for an in-memory string. Additionally, we can apply a stream to send content over the network, doing away with the need to send it all at once. This can be achieved using StreamContent in place of StringContent. Let's explore this through a demonstration.

Summary

In this article, we learned about how to improve performance and memory use while accessing APIs with Stream using HTTP Client in .NET. We learned about the advantages of working with Streams and use that knowledge with HTTP Client. We read the data using Streams and send the data using Streams and get the performance improvement and low memory usage benefit.