Replace Blosc2 with zstd-1 and decode chunks in parallel on read

[aarmey]

Summary

• zstd level 1 replaces Blosc2+LZ4 as the default compression for every numeric array written by write_h5ad (via hdf5plugin.Zstd) and write_zarr (zarr's/numcodecs' zstd codec). There is no option to switch back; dataset_kwargs still overrides it as before.

• .h5ad reads now decode in parallel. HDF5 runs a dataset's filter pipeline one chunk at a time on a single thread. The new _compression.read_array():

  • finds every chunk's file offset in one chunk_iter pass;
  • reads the compressed chunks directly with os.pread;
  • decodes them with numcodecs.Zstd, which releases the GIL, on a thread pool sized by numba.get_num_threads(), giving each thread one contiguous run of chunks.

  Every VCS/IVCS array read through the anndata registry (read_h5ad, load_packed, anndata.io.read_elem) uses it.

• Large arrays are written in 1 MiB chunks. numeric_only_compression now gives compressed 1-D HDF5 arrays larger than one chunk ~1 MiB chunks, unless the caller picked chunks. h5py's automatic 64–330 KiB chunks made the reader's per-chunk Python overhead dominate. The zstd ratio is unchanged from 64 KiB to 4 MiB chunks.

• Blosc2 files still read. Anything the parallel path doesn't handle falls back to an ordinary h5py read: Blosc2 files from earlier versions, uncompressed or contiguous datasets, zstd combined with another filter, chunks that were never written, files open for writing, and non-sec2 drivers.

• Fixes read_h5ad in a fresh process. It never imported hdf5plugin, so reading compressed obs/var columns (which happens before X) failed unless something else had already loaded the plugin.

• numcodecs is now a direct dependency. It was already installed via zarr.

Results

Recompressed with this branch, all 15 files in /opt/IVCSC went from 18.96 GB to 14.99 GB (1.26×). On packed_indices, Blosc2/LZ4 got ~1.00× on every file, while zstd-1 gets 1.1–1.5× through entropy coding of the mostly single-byte varints. Higher zstd levels gain nothing.

Warm-cache reads, Blosc2 file → zstd-1 file:

File	full packed_indices read	load_packed
lupus (1.26M cells)	1.31 s → 0.11 s	4.00 s → 2.89 s
K562-genome-wide (2.0M cells)	7.99 s → 0.71 s	14.06 s → 6.48 s
full_IBD_cohort (1.19M cells)	1.22 s → 0.54 s	20.46 s → 15.37 s

What's left of load_packed is mostly anndata parsing obs/obsm plus index decoding, which this PR doesn't touch. Subsetting cells during decode is planned in #67, which also closes the design question in #24.

Merge fix

Merging main (#66) into this branch kept this branch's _rapid_load.py, re-adding load_and_normalize and its helpers without the kernels and imports they call: 19 undefined names under ruff, and a NameError if called. cca3f93 restores main's file. The only change this PR needs there is the plugin-import comment, since load_packed already reads X through the registry.

Test plan

• tests/test_parallel_read.py:
  • parallel path across dtypes and lengths (empty, exact multiples of the chunk size, a partial final chunk);
  • files with a userblock;
  • a chunk HDF5 stored without the filter (filter-mask bit set);
  • fallback for unwritten chunks, contiguous, chunked-uncompressed, Blosc2, zstd+shuffle, writable and in-memory (core) files, and zarr arrays;
  • write_h5ad → read_h5ad taking the parallel path;
  • a Blosc2-written file still reading;
  • read_h5ad in a fresh subprocess (fails without the hdf5plugin import fix);
  • numcodecs decoding raw HDF5 zstd chunks.
• tests/test_compression.py: 1 MiB chunking for large 1-D arrays; small arrays, explicit chunks and 2-D arrays are left alone; zstd filter/codec defaults.
• Full suite: 1578 passed, 71 skipped. ruff check, ruff format --check, codespell and ty check are clean.
• vulture, jscpd, radon and xenon report nothing new in the changed files.
• Each /opt/IVCSC file was verified against its Blosc2 original, before and after being overwritten in place. The check covers every group, attribute, dataset, shape, dtype and value, and the new files are read through read_array.

https://claude.ai/code/session_01QBeTZy6BKKGKGJ7fwvkWgM