[Blackclaws]

Builds on #193 (checksum structures from their encoded bytes). Until that is merged, this PR's diff includes its two commits.

As discussed in #190, the writing changes are split into smaller PRs. This one carries metadata placement and single-pass front loading; the performance changes and the streaming API are stacked on it as separate PRs, each listing the one it builds on.

Structure is allocated in encode order, so it ends up spread evenly through the file. A reader that fetches ranges rather than seeking freely pays for the whole file to read the structure: on a 620 MB file of measurement series, walking the 8.2% of bytes that are structure touched 591 of 591 one-megabyte ranges.

H5WriteOptions.MetadataPlacement adds two alternatives to the layout the writer produces by default:

Placement	Allocation	Effect
Interleaved	in encode order	the default; files are byte-for-byte what they were
Aggregated	from blocks that double in size, forming a few clusters	needs no estimate of the total; costs a proportion of the metadata, not a fixed block
FrontLoaded	from one region at the front, exactly as large as the structure	a reader fetches all structure in one range

Measured in RangeReadCostTests, over a stream counting what an HTTP range client would transfer (600 groups of deflated measurement series, 256 kB ranges):

interleaved    fetched 4,058,242 B of 4,058,242 B   100.0% of file   16 requests
aggregated     fetched   786,432 B of 4,060,404 B    19.4% of file    3 requests
front-loaded   fetched   262,144 B of 4,058,242 B     6.5% of file    1 request

Front loading costs no file size at all here: the front-loaded file is byte-for-byte as long as the interleaved one.

How it works

The writer allocates every address before writing and then seeks to it, so segregating regions needs no change to any encoding logic - only the allocator, plus classifying each allocation site as structure or payload. Three of those are not what they look like:

• An implicit chunk index is not an index. The chunks sit contiguously from the address in the layout message, so what is allocated there is the whole payload of an unfiltered chunked dataset.
• The global heap serves both kinds. An attribute's variable-length value is structure - it is what a viewer reads while browsing - while a dataset's own variable-length elements are payload. The heap manager keeps a collection open per kind. A compact dataset's variable-length values are structure, since its data is part of its header.
• A compact dataset's payload lives inside its object header, so for compact datasets payload is structure and there is nothing to separate.

A filtered dataset's fixed array index has its data block allocated with its header: the block's size follows from the chunk count, which is known when the layout is created, so all of the index is structure in place when the structure is placed.

Front loading without a measuring pass

Payload is placed after the structure, so every payload address depends on how large the structure turns out to be. The writer does not encode the tree twice to find out. It reserves a provisional region, places payload provisionally beyond any real file (FreeSpaceManager.PROVISIONAL_PAYLOAD_BASE) and holds the encoded structure in memory (StructureStagingStream). When the structure is complete it closes the region where the structure ends, moves the payload down behind it, corrects and re-checksums the headers that name a provisional address, and writes the structure front to back in one run. Payload given with the tree is written after that.

The region is exactly what the structure uses, so nothing is abandoned and no estimate is involved. The price is memory the size of the structure while it is held. Against the earlier measuring pass, on 300,000 datasets declared through BeginWrite, median of 5 runs:

                       BeginWrite      allocated by BeginWrite   peak RSS
front-loaded           3.03 s -> 1.75 s      1406 -> 864 MB      343 -> 346 MB
front-loaded, chunked  4.21 s -> 2.34 s      1970 -> 1111 MB     701 -> 618 MB
interleaved            unchanged

SinglePassFrontLoadingTests asserts per shape - contiguous, compact, chunked filtered and unfiltered, single chunk, variable-length strings, Nullable<T>, attributes, object references, many groups - that front loading opens exactly one region, abandons no structure, and produces the same content an interleaved write does; and that BeginWrite writes the structure front to back, never behind an earlier write.

MetadataReservation still exists for a caller who would rather not hold the structure: the structure is then written as it is encoded into a region of that size, and a reservation that falls short spills into blocks, losing locality for the remainder rather than failing.

Worth flagging

• There is no free list, so a region's unused remainder is abandoned rather than reclaimed. Front loading without a reservation abandons nothing; the cost appears with an explicit MetadataReservation or with Aggregated blocks.
• Variable-length payload is clustered too. A dataset's heap collections are allocated now and written later, so they are gathered into blocks of their own rather than scattered between headers. The unused tail of the last such block makes a clustered file slightly larger than an interleaved one; FreeSpaceManager.PayloadAbandoned reports it.
• Metadata blocks grow rather than being fixed. A block is claimed in full, so a fixed size would be a floor on file size: 33 kB of metadata in an 8.4 MB file at 8 MB blocks. Blocks start at 64 kB and double up to MetadataBlockSize, which is therefore a ceiling, and must be positive for any placement that uses blocks.
• The end-of-file address covers allocated rather than written space, because an address can be referenced without ever being written to - a dataset declared through BeginWrite and never written. The stream is extended to the allocator's mark, translated by the base address since the allocator counts from the superblock and the stream from the start of the file.
• No CHANGELOG.md entry, assuming you would rather write it yourself - happy to add one.

Changes since the last review

• The measuring pass is gone. Front loading encodes the structure once and stages it, as described above. This replaces the sizing pass, SizingStream and the SizeOnly paths you reviewed, including the shortcut for fixed-size data from your suggestion: with no second encode there is nothing left to skip. NothingDeferredOutrunsTheSizingPass and TheSizingPassMeasuresExactlyWhatAWriteAllocates give way to SinglePassFrontLoadingTests.
• Rebased on current dev and on perf(writing): checksum structures from their encoded bytes #193, which the staging relies on: a staged header is corrected and re-checksummed in memory, and a structure that is written once can no longer be read back for its checksum.
• A fixed array index is allocated with its data block, so a filtered dataset's index is part of the structure when it is placed instead of arriving at Dispose.
• MetadataLayoutTests counts an object header's whole extent when mapping structure onto ranges, not only the slot its signature is in: a group with many links has a header large enough to cross a range boundary.

🤖 Generated with Claude Code

[Apollo3zehn]

Thanks for this PR as well! This is a nice feature for high latency environments :-)

I started the review and put some time into figuring out why it is necessary to still write the data into the Stream that measures the metadata size because preparing the data for write does not come for free and it would be good to skip unnecessary heavy work. The answer is two fold: There are cases where we can skip that and some where it is not possible, i.e. when writing the data itself causes metadata modifications which is the case for variable length data. So I asked GPT for a plan. Please tell me if you agree to its content or if you have any doubts. If the plan is fine I will add a commit with the proposed changes. The plan:

Suggested improvement: skip raw data writes during metadata sizing for fixed-size datasets

The metadata sizing pass currently avoids compression for filtered chunks, but it still walks dataset data, allocates chunk buffers, copies bytes, and registers raw chunk data. For fixed-size unmanaged datasets, that work appears unnecessary because it does not affect metadata size.

Why this should be safe

MeasureMetadataSize already performs the metadata-relevant work:

• It encodes the file structure with MetadataPlacement = Interleaved and SizeOnly = true.
• It creates each dataset layout and H5D_Base.
• It stores each dataset in Context.DatasetToInfoMap.
• It disposes the writer, which disposes each H5D and finalizes chunk index metadata.
• It encodes global heap collections before writing the final superblock.

For fixed-size unmanaged dataset payloads, WriteData only copies raw data into the target layout. That raw data copy does not change the amount of metadata that must be reserved.

Chunked layout details

For chunked datasets, skipping raw data writes during the sizing pass should still preserve metadata sizing:

• Filtered multi-chunk datasets use fixed-array indexing.
• The fixed-array metadata size depends on WriteChunkInfos.Length, ChunkSizeLength, and page information derived from the entry count.
• It does not depend on the actual chunk address, compressed chunk size, or filter mask values.
• If data writing is skipped, chunk entries may remain at default values, but their encoded width and count remain unchanged.
• Unfiltered multi-chunk datasets use implicit indexing; their data region is allocated during layout creation, so WriteData does not add metadata.
• Single-chunk datasets store the relevant indexing information in the layout message.
• Contiguous datasets allocate payload as AllocationKind.RawData, not metadata.

This matches the existing observation that compressed output size is not metadata-size-relevant: fixed-array entries are fixed-width, and ChunkSizeLength is derived from the uncompressed chunk size.

Keep data encoding where it has metadata side effects

The sizing pass must still run WriteData for datasets whose element encoding can allocate or alter metadata:

• Compact datasets, because the encoded data is stored in the object header.
• Variable-length strings and variable-length sequences, because encoding allocates global heap metadata.
• Object references, because encoding may recursively encode referenced groups or datasets.
• Nullable/reference-containing element types, because they are routed through the per-element encoding path.

Proposed code shape

Add a small guard in the write path, after dataset encoding/layout creation but before WriteData:

if (context.SizeOnly && !RequiresSizeOnlyDataWrite<T>(dataset))
    return;

A possible predicate:

private static bool RequiresSizeOnlyDataWrite<T>(H5Dataset dataset)
{
    return
        dataset.Layout is DataLayoutMessage4
        {
            Properties: CompactStoragePropertyDescription
        }
        || DataUtils.IsReferenceOrContainsReferences(typeof(T))
        || Nullable.GetUnderlyingType(typeof(T)) is not null;
}

The exact location would likely be InternalWriteDataset<T, TElement>, since InternalEncodeDataset should still run in the sizing pass to create layouts, datatype messages, dataspace messages, H5D instances, and deferred metadata structures.

Expected benefit

This should avoid unnecessary work during metadata measurement for fixed-size datasets:

• no source-data traversal
• no chunk-buffer allocation
• no raw byte copying
• no raw chunk registration/allocation for filtered chunks
• no selection walking beyond what is needed for metadata

Compression is already skipped, so this targets the remaining fixed-size raw-data overhead in the measurement pass.

[Apollo3zehn]

Apart from that, due to the complexity involved and the limitations being imposed ("BeginWrite needs an explicit MetadataReservation") I would like to keep Interleaved the default.

BeginWrite needs an explicit MetadataReservation. The measuring pass cannot know what a caller will write later, so it under-counts those chunk indexes and the shortfall is placed as Aggregated would place it.

I am not sure I fully understand this statement. Why is it under-counting chunk indexes? The number of chunks are always known beforehand and will not change during BeginWrite.

[Blackclaws]

I'm on it. I've taken your proposal and Claude implemented it after fixing a couple of things it found in there that didn't work. Also found a number of other edge cases, an update to the PR is coming either today or tomorrow.

[Blackclaws]

Thanks — and yes, I agree with the premise, and with where the plan draws the line. I have implemented it (perf(writing): skip fixed-size data in the metadata sizing pass) rather than leaving you to write the commit, since checking it turned up two things worth flagging. Drop that commit and take your own if you would rather.

Where the plan is right: nothing about a fixed-size dataset's metadata footprint depends on its data. I checked every encode size involved — SingleChunkIndexingInformation.GetEncodeSize and ImplicitIndexingInformation.GetEncodeSize ignore their values entirely, and the chunk-index case is the one you raise on the other thread. The list of things that must still be encoded is right too: compact, variable-length, object references, Nullable<T>.

Two corrections it needed:

The guard has to sit at the WriteData call inside InternalEncodeDataset, not in InternalWriteDataset. InternalWriteDataset is only the deferred path — what writer.Write(dataset, data) reaches. The sizing pass never calls it: it calls writer.Write() and then Dispose, so all of its data encoding goes through InternalEncodeDataset's own WriteData call. Guarding InternalWriteDataset would have skipped nothing at all, and nothing would have failed to say so.

The predicate has to test TElement, not T. T is the dataset type, and an array is itself a reference, so IsReferenceOrContainsReferences(typeof(int[])) is true — asked of T the predicate answers "needs data" for every array-backed dataset, which is very nearly all of them. Same silent no-op. On TElement it answers as intended, and it also covers the object-reference case for free, since H5ObjectReference is a class.

One addition: Nullable<T> is worth keeping in the predicate for a reason the plan does not give. It is not that nullables go through the per-element path — it is that GetTypeInfoForNullableValueType encodes them as a variable-length sequence of one and calls GlobalHeapManager.AddObject, so they take global heap space. Nullable<int> contains no reference, so the reference test does not see it. MetadataLayoutTests has a nullable ints shape that pins this: drop the clause and it measures 200 bytes against the 12,488 a write allocates.

What I did not do is make the compact case conditional. It is safe to skip too — CompactStoragePropertyDescription.Data is allocated by size in DataLayoutMessage4.Create, so the object header's size is already settled — but a compact dataset is capped at 64 kB, so there is nothing to win, and keeping it documents that payload and structure are the same bytes there.

Measured on a 64 MB file, sizing pass against a full write of the same file, best of seven runs each — the fastest run being the one least perturbed by whatever else the machine is doing:

unfiltered    34.9 ms -> 1.3 ms    43% of a write -> 2%
filtered      29.4 ms -> 0.9 ms   2.6% of a write -> 0.1%

The safety net is TheSizingPassMeasuresExactlyWhatAWriteAllocates, which asserts across ten shapes that the pass reports exactly what a write allocates. Equality, not a bound: over-counting wastes the reservation's tail and under-counting spills, so the number has to be the number. That is what makes both shortcuts — no compression, and now no fixed-size data — checkable rather than argued, and it went on to catch two allocation bugs of mine that had nothing to do with this commit.

[Blackclaws]

Interleaved is the default now. Agreed on the reasoning, and it makes the change purely additive — default-option files are byte-for-byte what they were.

On the under-counting: you are right, and my statement was wrong. I checked it rather than argued it, and the shortfall for a deferred chunk index is exactly zero:

filtered chunked int (fixed array)        measured 495   allocated 495   shortfall 0
unfiltered chunked int (implicit index)   measured 192   allocated 192   shortfall 0

Exactly as you say: the chunk count is known before anything is written. WriteChunkInfos is sized to TotalChunkCount in H5D_Chunk4.Initialize, the fixed-array data block is sized from WriteChunkInfos.Length, and Initialize runs during the structure write, which the sizing pass performs in full. Writing the data later fills entries in an array whose length was already fixed.

I then replaced that wrong claim with a second wrong claim, so it is worth being explicit about where it ended up. I said the real shortfall was variable-length data, which was true at the time and is no longer: variable-length values were being allocated on the global heap as structure, and reviewing this again showed that is wrong for a dataset's own elements. An attribute's value is structure — it is what a viewer reads while browsing a file — but a dataset's variable-length elements are its payload, and both went through the same manager. A dataset of variable-length strings with no attributes at all measured 97.3% structure, which meant a front-loaded region swallowed the payload and there was nothing left to separate. Silently, too: the file stays valid and the same size, and only the locality quietly fails to materialise.

With that fixed, a variable-length value's heap space is payload, so writing more of it later does not move the metadata total either:

contiguous strings (global heap)          measured 200   allocated 200   shortfall 0

So the honest answer to your question is better than either of my attempts at it: nothing a caller writes after BeginWrite changes how much structure the file has, and MetadataReservation is not needed for a deferred write at all. A deferred variable-length write under FrontLoaded now opens one region, abandons nothing, and produces a file the same size as the interleaved one. NothingDeferredOutrunsTheSizingPass asserts all three shortfalls are zero — this guidance has been wrong twice, so it seemed worth pinning rather than restating.

The same review turned up a second misclassification of the same kind — an implicit chunk index is not an index, so what is allocated there is an unfiltered chunked dataset's whole payload — plus a corruption bug and a few other things. Those are in the rewritten description rather than here. The pattern they share is worth one sentence though: all of them were invisible from outside, because a misclassified allocation still produces a valid file of the same size and only the locality quietly fails to appear. It follows that the measurements in the first version of this description did not cover unfiltered chunked data or variable-length datasets, and the ones now in it do.

[Blackclaws]

Force-pushed, and the description is rewritten rather than amended — worth a fresh read, since editing it does not notify.

The two replies above answer your comments. Everything else came out of reviewing the result again afterwards, and is listed under Changes since the first review rather than repeated here. Four of those are defects in code you had already read, so they are the ones I would not want buried:

• The extension that covers allocated-but-unwritten space compared an allocator address against a stream length without translating for UserBlockSize, so a user block plus a never-written deferred dataset produced a file h5dump rejects.
• The reservation added 12 kB of slack that was never needed. Front loading now costs zero bytes rather than nearly zero.
• A metadata block was claimed in full, so the 8 MB default was a floor on file size: 33 kB of metadata produced an 8.4 MB file. Blocks now grow.
• Three h5dump assertions could not fail, because TestUtils.DumpH5File strips the error stack the assertions were looking for.

Two unrelated pre-existing bugs turned up on the way and are filed separately: #183 and #184. Neither is touched by this PR.

Two things still open for you rather than for me: whether you want a CHANGELOG.md entry from me or would rather write it, and whether Aggregated is worth keeping at all — front loading now dominates it on every axis I can measure except needing no sizing pass, so if you would rather carry one option instead of two, dropping it is a smaller surface to maintain.