Characterization Tests: The Safety Net Every NuGet Package Swap Needs in .NET

Characterization Tests: The Safety Net Every NuGet Package Swap Needs in .NET

Author - Abdul Rahman (Bhai)

NuGet

6 Articles

 Improve

Table of Contents

  1. What we gonna do?
  2. Why we gonna do?
  3. How we gonna do?
  4. Summary

What we gonna do?

You have identified a vulnerable NuGet package. You have found a suitable replacement. You have read the migration guide, updated the reference, and run the build — everything compiles. Then you deploy to staging and a downstream integration test starts returning subtly wrong data. The replacement package serializes DateTime fields in a slightly different format than the old one, and three consumers of your API silently break.

The compiler could not catch this. Your unit tests could not catch it either, because unit tests assert what code should do — they have no way to compare what the old code actually did against what the new code does. That gap is precisely where characterization tests live.

A characterization test records the real, verified output of a piece of code at a known-good point in time — before any replacement happens — and commits that record as a golden master. When you swap the dependency, the same test runs against the new code and compares its output byte-for-byte against the golden master. Any divergence is a test failure, and you know exactly what changed before it reaches production. In this article you will learn what characterization tests are, how they differ from unit tests, and how to implement them in .NET using the Verify library to protect every NuGet package replacement you ever make.

Why we gonna do?

The Silent Breakage Problem With NuGet Replacements

Imagine your application uses a fictional JSON library, LegacyJson, to serialize invoice records. The library has a known vulnerability, and you replace it with System.Text.Json. Both libraries serialize C# objects to JSON strings, so the replacement feels mechanical. But serialization is not a single universal behavior — it is a collection of decisions: How are null properties handled? Are property names camel-cased or Pascal-cased? Is DateTime emitted as ISO 8601 or a custom format? Are empty collections serialized as [] or omitted entirely?

These differences are invisible until something downstream depends on the exact shape of the output. An external partner consuming your invoice feed may rely on the specific date format. An event bus may reject messages with unexpected property casing. A legacy mobile client may fail to deserialize a payload where null fields are now omitted. None of these failures are loud — they manifest as data anomalies hours or days after a seemingly successful deployment.

Unit Tests Cannot Protect You Here — and That Is Not Their Job

A unit test for an invoice serializer typically looks like this: 


[Fact]
public void Serialize_BasicInvoice_ReturnsValidJson()
{
    var serializer = new InvoiceSerializer();
    var invoice = new Invoice { Id = 1, CustomerName = "Acme Corp", Amount = 250.00m };

    var json = serializer.Serialize(invoice);

    Assert.False(string.IsNullOrWhiteSpace(json));
    Assert.Contains("Acme Corp", json);
}

This test passes whether the serializer produces {"customerName":"Acme Corp"} or {"CustomerName":"Acme Corp"}. It tells you the serializer ran without throwing. It says nothing about the exact shape of the output. When you swap the serializer library, this test continues to pass — and gives you a false sense of safety while the casing difference silently breaks a downstream consumer.

Unit tests assert correctness against a specification. Characterization tests assert stability of output across a replacement. The two are complementary — you need both.

The Golden Master Concept

A golden master is a committed, version-controlled snapshot of what your code actually produced at a known-good point in time. It is the authoritative reference every future run is compared against. When the replacement library produces output that is byte-for-byte identical to the golden master, you have strong evidence that the swap is safe. When it diverges, the test fails and tells you exactly what changed — before you deploy.

The process is intentionally a two-phase loop: 


  Phase 1 — Create the golden master (run against the old package)
  ──────────────────────────────────────────────────────────────────
  Run test → Verify creates *.received.txt (actual output)
           → Test FAILS (no *.verified.txt exists yet)
  Inspect received.txt → if the output looks correct, accept it
  Copy contents into *.verified.txt → commit to source control
  Delete received.txt

  Phase 2 — Guard the replacement (run against the new package)
  ──────────────────────────────────────────────────────────────────
  Swap the NuGet reference → rerun the test
  Verify creates *.received.txt for the new output
  Verify diffs received.txt against *.verified.txt
  → Match: test PASSES, received.txt is deleted
  → Mismatch: test FAILS, diff opens showing exactly what changed

How we gonna do?

The Scenario: Replacing a Serialization Library Without Breaking Invoice Output

Your application serializes Invoice records to JSON before publishing them to a message queue. Today the serializer is backed by a fictional package called LegacyJson. You need to replace it with System.Text.Json from the .NET base class library. Before you touch the reference, you will capture a golden master so the test suite can guarantee the replacement produces identical output.

Step 1 — Define the Domain Model and Serializer Interface

Start with the domain types. These belong to your application and do not change when you swap the library: 


// Domain/Invoice.cs
namespace InvoiceQueue.Domain;

public sealed record Invoice(
    int Id,
    string CustomerName,
    decimal Amount,
    DateTimeOffset DueDate,
    IReadOnlyList<InvoiceLineItem> LineItems);

public sealed record InvoiceLineItem(
    string Description,
    int Quantity,
    decimal UnitPrice);

Define a serializer interface in a project with no NuGet package references. Both the legacy implementation and the replacement will satisfy this contract: 


// Application/IInvoiceSerializer.cs
namespace InvoiceQueue.Application;

public interface IInvoiceSerializer
{
    string Serialize(Invoice invoice);
    Invoice Deserialize(string json);
}

Step 2 — Write the Legacy Serializer (the One You Are Replacing)

Implement the interface using the fictional LegacyJson library. This is the code you want to characterize before discarding it: 


// Infrastructure/LegacyJsonInvoiceSerializer.cs
// <PackageReference Include="LegacyJson" Version="5.2.1" />

using LegacyJson;
using InvoiceQueue.Application;
using InvoiceQueue.Domain;

namespace InvoiceQueue.Infrastructure;

public sealed class LegacyJsonInvoiceSerializer : IInvoiceSerializer
{
    // LegacyJson uses Pascal-case by default, includes null fields,
    // and formats DateTimeOffset as "yyyy-MM-ddTHH:mm:sszzz".
    private static readonly JsonSerializerSettings Settings = new()
    {
        NullValueHandling = NullValueHandling.Include,
        Formatting = Formatting.Indented,
        DateFormatString = "yyyy-MM-ddTHH:mm:sszzz"
    };

    public string Serialize(Invoice invoice) =>
        JsonConvert.SerializeObject(invoice, Settings);

    public Invoice Deserialize(string json) =>
        JsonConvert.DeserializeObject<Invoice>(json, Settings)!;
}

Step 3 — Add the Verify NuGet Package to the Test Project

Verify is a snapshot-testing library that automates the golden master workflow. Add it to your xUnit test project alongside the standard test dependencies: 


<!-- InvoiceQueue.Tests.csproj -->
<ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.11.1" />
    <PackageReference Include="xunit" Version="2.9.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.8.2">
        <PrivateAssets>all</PrivateAssets>
    </PackageReference>
    <PackageReference Include="Verify.Xunit" Version="26.5.0" />
</ItemGroup>

Step 4 — Write the Characterization Tests Against the Legacy Serializer

Create the test class. Inherit from VerifyBase — the recommended approach in Verify v22 and later — to wire Verify into the xUnit pipeline without any per-class attribute. Each test method returns Task rather than void because Verify() is asynchronous: 


// InvoiceSerializerCharacterizationTests.cs
using InvoiceQueue.Domain;
using InvoiceQueue.Infrastructure;
using VerifyXunit;

namespace InvoiceQueue.Tests;

public class InvoiceSerializerCharacterizationTests : VerifyBase
{
    // ── Shared factory ───────────────────────────────────────────────────────
    private static Invoice BuildFullInvoice() => new Invoice(
        Id: 42,
        CustomerName: "Contoso Retail",
        Amount: 1_850.75m,
        DueDate: new DateTimeOffset(2026, 3, 31, 0, 0, 0, TimeSpan.Zero),
        LineItems:
        [
            new InvoiceLineItem("Standing desk", 2, 599.99m),
            new InvoiceLineItem("Ergonomic chair", 1, 649.77m),
        ]);

    private static Invoice BuildInvoiceWithNullCustomer() => new Invoice(
        Id: 7,
        CustomerName: null!,
        Amount: 0m,
        DueDate: DateTimeOffset.MinValue,
        LineItems: []);

    // ── Characterization tests ────────────────────────────────────────────────

    [Fact]
    public Task Serialize_FullInvoice_MatchesGoldenMaster()
    {
        var serializer = new LegacyJsonInvoiceSerializer();
        var result = serializer.Serialize(BuildFullInvoice());
        return Verify(result);
    }

    [Fact]
    public Task Serialize_EmptyLineItems_MatchesGoldenMaster()
    {
        var serializer = new LegacyJsonInvoiceSerializer();
        var invoice = BuildFullInvoice() with { LineItems = [] };
        var result = serializer.Serialize(invoice);
        return Verify(result);
    }

    [Fact]
    public Task Serialize_NullCustomerName_MatchesGoldenMaster()
    {
        var serializer = new LegacyJsonInvoiceSerializer();
        var result = serializer.Serialize(BuildInvoiceWithNullCustomer());
        return Verify(result);
    }

    [Fact]
    public Task Roundtrip_FullInvoice_SurvivesSerializeDeserialize()
    {
        var serializer = new LegacyJsonInvoiceSerializer();
        var original = BuildFullInvoice();
        var json = serializer.Serialize(original);
        var restored = serializer.Deserialize(json);
        // Re-serialize the restored object through the same serializer so the
        // golden master captures the serializer's output, not Verify's own renderer.
        var roundtrippedJson = serializer.Serialize(restored);
        return Verify(roundtrippedJson);
    }
}

Notice there are no Assert calls. Verify handles the assertion by diffing the actual output against the golden master file. If no golden master exists, Verify creates a .received.txt file and fails the test — prompting you to inspect and accept the output.

Step 5 — Accept the First Run to Create the Golden Master

Run the tests for the first time. Every test fails because no .verified.txt files exist yet. Verify places the actual output in *.received.txt files alongside the test class. For the Serialize_FullInvoice_MatchesGoldenMaster test, the received file looks like this: 


// InvoiceSerializerCharacterizationTests.Serialize_FullInvoice_MatchesGoldenMaster.received.txt

{
  "Id": 42,
  "CustomerName": "Contoso Retail",
  "Amount": 1850.75,
  "DueDate": "2026-03-31T00:00:00+00:00",
  "LineItems": [
    {
      "Description": "Standing desk",
      "Quantity": 2,
      "UnitPrice": 599.99
    },
    {
      "Description": "Ergonomic chair",
      "Quantity": 1,
      "UnitPrice": 649.77
    }
  ]
}

Inspect this output carefully. Check the date format, the casing of property names, the presence of null fields, and the shape of nested objects. If everything looks correct, accept it: rename the file from .received.txt to .verified.txt and commit it to source control. This is your golden master — the authoritative record of what the legacy serializer produces.

Verify provides a CLI tool to accept all received files in one step: 


dotnet tool install -g verify.tool
dotnet verify accept --directory Tests/

Re-run the tests. All four tests now pass because the received output matches the verified files. The .received.txt files are deleted automatically.

Step 6 — Implement the Replacement Serializer

Now write the SystemTextJsonInvoiceSerializer that you want to promote to production. Implement the same interface: 


// Infrastructure/SystemTextJsonInvoiceSerializer.cs
// No additional PackageReference needed — System.Text.Json ships with .NET

using System.Text.Json;
using System.Text.Json.Serialization;
using InvoiceQueue.Application;
using InvoiceQueue.Domain;

namespace InvoiceQueue.Infrastructure;

public sealed class SystemTextJsonInvoiceSerializer : IInvoiceSerializer
{
    // System.Text.Json defaults to camel-case property names and
    // omits null properties unless configured otherwise.
    private static readonly JsonSerializerOptions Options = new()
    {
        WriteIndented = true,
        PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
        DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
        Converters = { new JsonStringEnumConverter() }
    };

    public string Serialize(Invoice invoice) =>
        JsonSerializer.Serialize(invoice, Options);

    public Invoice Deserialize(string json) =>
        JsonSerializer.Deserialize<Invoice>(json, Options)!;
}

Step 7 — Swap the Serializer in the Characterization Tests and Run

Change the one line in your test class that instantiates the serializer: 


// Before
var serializer = new LegacyJsonInvoiceSerializer();

// After
var serializer = new SystemTextJsonInvoiceSerializer();

Run the tests. The output reveals exactly what the Serialize_FullInvoice_MatchesGoldenMaster test actually found: 


// Diff: received vs verified

  {
-   "Id": 42,
-   "CustomerName": "Contoso Retail",
-   "Amount": 1850.75,
-   "DueDate": "2026-03-31T00:00:00+00:00",
-   "LineItems": [ ... ]
+   "id": 42,
+   "customerName": "Contoso Retail",
+   "amount": 1850.75,
+   "dueDate": "2026-03-31T00:00:00+00:00",
+   "lineItems": [ ... ]
  }

The characterization test caught two categories of divergence before any deployment:

  • Property name casing — LegacyJson used Pascal-case; System.Text.Json with CamelCase policy emits camel-case. Any consumer expecting CustomerName will now receive customerName.
  • Null handling — Serialize_NullCustomerName_MatchesGoldenMaster fails because the old library serialized null fields as "CustomerName": null, while the new serializer omits them entirely under WhenWritingNull.

These are not hypothetical edge cases — they are real behavioral differences between two production-grade serializers. The characterization test made them visible in seconds.

Step 8 — Resolve the Divergence Before Accepting

You now have two choices. Either configure the replacement serializer to produce identical output (matching the golden master), or consciously decide the output change is acceptable and update the golden master to reflect the new behaviour.

To match the existing golden master, configure System.Text.Json to use Pascal-case names and include null fields: 


private static readonly JsonSerializerOptions Options = new()
{
    WriteIndented = true,
    // Remove CamelCase policy — Pascal-case is the default in System.Text.Json
    DefaultIgnoreCondition = JsonIgnoreCondition.Never, // include null fields
    Converters = { new JsonStringEnumConverter() }
};

Rerun the tests after the configuration change. When all four pass, the replacement produces output that is byte-for-byte identical to the golden master and you can proceed to swap the production reference with confidence.

Characterization Tests vs Unit Tests: A Quick Reference


┌─────────────────────────────────┬──────────────────────────────────────┬──────────────────────────────────────┐
│ Dimension                       │ Unit Test                            │ Characterization Test                │
├─────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────┤
│ Purpose                         │ Verify new code meets specification  │ Record what code actually does       │
│ Written when                    │ While writing code                   │ Before changing a dependency         │
│ Passes when                     │ Output matches expected assertion    │ Output matches golden master         │
│ First run result                │ Pass or Fail                         │ Always Fails (creates golden master) │
│ Assertion style                 │ Assert.Equal / FluentAssertions      │ Verify(result) — snapshot diff       │
│ Granularity                     │ Single method or class               │ Observable output across methods     │
│ Guards against                  │ Logic errors in new code             │ Behavioural changes in replacements  │
└─────────────────────────────────┴──────────────────────────────────────┴──────────────────────────────────────┘

Practical Guidelines for Effective Characterization Tests

  • Cover edge cases, not just the happy path. Serialization differences most often surface at the boundaries — null values, empty collections, large decimals, extreme date ranges, and Unicode characters. Include a test for each.
  • Commit the verified files to source control. The .verified.txt files are your test fixtures. They must travel with the code or the tests are worthless on another developer's machine.
  • Do not accept a golden master without inspecting it. The entire value of the approach collapses if the golden master is incorrect. Verify deliberately fails the first run to force this inspection.
  • Delete received files after each run. Verify handles this automatically on a passing run. If a received file lingers, a previous run failed — investigate before accepting.
  • Add *.received.txt to your .gitignore. Received files are ephemeral diff artifacts — committing them would pollute history and cause false failures on other machines. Only *.verified.txt files belong in source control.
  • Keep characterization tests separate from unit tests. Place them in a distinct folder or assembly so the team can identify their purpose at a glance.

Summary

  • A characterization test records what code actually does rather than asserting what it should do — making it the right tool for guarding NuGet package replacements.
  • A golden master is a committed snapshot of real output created by running the old implementation. Every future run is diffed against it.
  • The Verify NuGet library automates the golden master workflow in .NET: it creates .received.txt files on first run, lets you accept them as .verified.txt golden masters, and diffs on every subsequent run.
  • A first run always fails by design — this is Verify prompting you to inspect the output before committing it as authoritative.
  • Characterization tests complement unit tests. They guard behavioural stability across replacements; unit tests guard correctness against specifications. You need both.
  • When a characterization test fails after a swap, you have two options: configure the replacement to produce identical output, or consciously accept the change and update the golden master.

If you are working through building a safe NuGet dependency management strategy, the previous articles in this series cover how to audit your full dependency tree for vulnerabilities, how to swap out a vulnerable package without touching domain code, and how to enforce an Anti-Corruption Layer so package types never leak into your domain. Characterization tests are the final safety net that ties those strategies together.

  • Nuget
  • Characterization Tests
  • Golden Master
  • Verify
  • Snapshot Testing
  • Package Replacement
  • Dependency Safety
  • xUnit
  • Testing
  • .NET