Every Comma Was Costing Me an Array

A nightly job that chews through a CSV export got moved onto a smaller box, and it started spending more time in gen0 collections than I was comfortable with. The work is boring: read a few hundred thousand lines, pull one numeric column, add it up. Nothing in that loop should be stressing the garbage collector. The allocation counter disagreed, and that counter has never once lied to make me feel better.

The culprit turned out to be the least suspicious line in the whole thing: line.Split(',').

Here's the part I keep under-weighting. string.Split doesn't just find the commas. It allocates a string[] for the pieces and then a fresh string for every field it carves out, including the ones I never look at. I wanted field number three. I was paying for all of them, a million times over, then handing each little string straight to the GC a microsecond later.

Measuring it instead of trusting the vibe

I didn't want to argue from a profiler screenshot, so I wrote the smallest honest version of the job. A million CSV-shaped lines of id,sku,amount, built once up front so the data itself doesn't land on either scorecard. Then two functions that sum the amount column and differ only in how they split a line.

static long SumWithStringSplit(string[] lines)
{
    long total = 0;
    foreach (string line in lines)
    {
        string[] fields = line.Split(',');
        total += int.Parse(fields[2], CultureInfo.InvariantCulture);
    }
    return total;
}

I wrapped a best-of-five loop with GC.GetAllocatedBytesForCurrentThread() and kept a checksum so I'd notice if either version quietly computed the wrong total.

Summing the amount column over 1,000,000 lines, best-of-5:

string.Split(',')     157,759,960 bytes/run      57 ms   (checksum 25500000)

Just over 150 MB for one pass, to read one column. That's an array plus three substrings per line, a million times, all of it garbage before the next iteration even starts.

Keeping the commas on the stack

ReadOnlySpan<char> has had a Split overload for a few versions now that I'd somehow never reached for on a real problem. Instead of handing back new strings, it writes the slice boundaries as Range values into a buffer you pass in. Stack-allocate that buffer and the whole operation touches the heap zero times.

static long SumWithSpanSplit(string[] lines)
{
    long total = 0;
    Span<Range> fields = stackalloc Range[4];
    foreach (string line in lines)
    {
        ReadOnlySpan<char> span = line;
        span.Split(fields, ',');
        total += int.Parse(span[fields[2]], CultureInfo.InvariantCulture);
    }
    return total;
}

span[fields[2]] slices the third range back out of the original line without copying it, and int.Parse has a ReadOnlySpan<char> overload, so the number comes straight off that slice.

Span<Range> split              40 bytes/run      21 ms   (checksum 25500000)

Same checksum, so same answer. The 40 bytes is measurement noise rather than per-line cost; the per-line allocation is flat zero. About 150 MB of garbage gone, and it finished in roughly a third of the time as a bonus. I came for the allocations and the speed tagged along.

Where I'd stop short

This isn't a "go replace every Split you own" post. If you split a config value once at startup, the array version is clearer and its allocation is a rounding error, so reach for the readable thing. The span version earns its keep on the hot paths where the same shape of line streams past millions of times.

Two sharp edges are worth knowing. The Span<Range> overload won't grow your buffer for you. If a line has more fields than the destination holds, the last Range gets the entire unsplit remainder instead of throwing. For fixed-shape data that's fine, since you size the buffer to the columns you expect, but ragged input will surprise you quietly. The other edge: the moment you actually need those fields as string values to keep past the loop, you're allocating them anyway and the span version buys you nothing. Its whole advantage is reading through data you're about to throw away.

My honest take is that the Span/Range APIs still read a little awkwardly. Slicing with span[fields[2]] isn't as obvious as fields[2], and that friction is probably why I skipped over them for so long. On a cold path that awkwardness isn't worth it. On the one loop lighting up your gen0 graph, 150 MB of avoidable garbage buys a slightly uglier line of code every time.

Here's the whole thing, runnable as-is on the .NET 10 SDK with dotnet run -c Release:

using System.Diagnostics;
using System.Globalization;

const int LineCount = 1_000_000;
const int Runs = 5;

string[] lines = new string[LineCount];
for (int i = 0; i < LineCount; i++)
    lines[i] = $"{i},widget-{i % 97},{(i % 50) + 1}";

// Warm up the JIT for both paths before measuring.
_ = SumWithStringSplit(lines);
_ = SumWithSpanSplit(lines);

Console.WriteLine($"Summing the amount column over {LineCount:n0} lines, best-of-{Runs}:");
Console.WriteLine();
Measure("string.Split(',')  ", () => SumWithStringSplit(lines));
Measure("Span<Range> split  ", () => SumWithSpanSplit(lines));

static long SumWithStringSplit(string[] lines)
{
    long total = 0;
    foreach (string line in lines)
    {
        string[] fields = line.Split(',');
        total += int.Parse(fields[2], CultureInfo.InvariantCulture);
    }
    return total;
}

static long SumWithSpanSplit(string[] lines)
{
    long total = 0;
    Span<Range> fields = stackalloc Range[4];
    foreach (string line in lines)
    {
        ReadOnlySpan<char> span = line;
        span.Split(fields, ',');
        total += int.Parse(span[fields[2]], CultureInfo.InvariantCulture);
    }
    return total;
}

static void Measure(string label, Func<long> work)
{
    GC.Collect();
    GC.WaitForPendingFinalizers();
    GC.Collect();

    long before = GC.GetAllocatedBytesForCurrentThread();
    long best = long.MaxValue;
    long checksum = 0;
    for (int r = 0; r < Runs; r++)
    {
        var sw = Stopwatch.StartNew();
        checksum = work();
        sw.Stop();
        best = Math.Min(best, sw.ElapsedMilliseconds);
    }
    long after = GC.GetAllocatedBytesForCurrentThread();

    long perRun = (after - before) / Runs;
    Console.WriteLine($"{label}{perRun,14:n0} bytes/run   {best,5} ms   (checksum {checksum})");
}

Measured in a container, best-of-five, nothing fancy. I care about the ratio, not the exact milliseconds your hardware would print.

Ever found a hot Split hiding inside a loop you'd written off as simple? Tell me where yours turned up.

— still benchmarking the boring loops nobody asked me to