fix(file): keep image holes and map ext4 journals in bmaps - #158
Draft
JanZachmann wants to merge 11 commits into
Draft
JanZachmann wants to merge 11 commits into
JanZachmann wants to merge 11 commits into
Conversation
write_partition ran `fallocate -d` on the whole image, which turned every zero-filled block into a hole. A bmap created afterwards with `-b` then skipped those blocks, e.g. ext4 journals that the image deliberately writes as zeros, so a bmap-based flash left the old journal data on the device. Write only the blocks that differ from the image instead. Unchanged blocks keep their allocation, and a block that changed to all zeros is now written too (`dd conv=sparse` skipped it and kept the old content). Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
- all direct dependencies to their latest version, omnect-crypto to 0.5.0 (create_cert_and_key no longer takes the unused extensions argument; the extensions of a certificate without CSR are the same) - actix-web without default features: the OAuth callback server only serves plain HTTP on localhost, and the http2 feature pulled in h2 0.3 (RUSTSEC-2026-0258, no fixed 0.3 release) - flate2 selects its rust_backend itself instead of relying on actix-web's compression features to enable one Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
JanZachmann
requested review from
HarryWaschkeit
and removed request for
HarryWaschkeit
September 28, 2026 12:03
JanZachmann
marked this pull request as draft
September 28, 2026 13:50
libfs was built with `default-features = false`, which also drops its `use_linux` feature. Without it libfs copies byte by byte, so every image omnect-cli works on came back fully allocated, and a bmap created with `-b` mapped the whole image. The `fallocate -d` that write_partition used to run hid this. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
A fresh ext4 journal is all zeros, and a sparse image, e.g. one that xz decompressed into a file, holds it as holes. A bmap then skips the journal, so a bmap-based flash leaves the journal of the previous installation on the device, and after a power cut its replay can apply an old transaction to the new filesystem. Before creating the bmap, rewrite the journal blocks of every ext4 partition with their own content. Holes become allocated zero blocks and all other blocks stay unchanged. debugfs lists the journal blocks, so the package now depends on e2fsprogs. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
libfs::copy_file copies each data segment with one copy_file_range call and ignores a short copy, so a segment over 2 GiB was cut off without error. copy_image keeps the holes and loops until each segment is copied. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
- fall back to /usr/sbin/debugfs when debugfs is not in PATH - reject an invalid s_log_block_size before it is used as a shift - skip partitions whose superblock is past the end of the image - skip the MBR extended container entry - share SECTOR_SIZE between partition and functions Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
The deb package depends on e2fsprogs for debugfs. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
Decompress wrote every zero block as data, so a bmap of a compressed input mapped the whole image. The decompressed file now seeks over zero blocks and keeps them as holes. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
The ext4 journal lookup is only used for bmap generation, so it moves with generate_bmap_file from functions.rs to bmap.rs. Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
- a bmap flash onto a device with old content now overwrites the journal - an invalid s_log_block_size is rejected - a partition that starts past the end of the image is skipped - the MBR extended container entry is not returned as a partition Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
write_partitioncompares the partition file with the image in 4 KiB blocks and writes only the blocks that differ, instead of runningfallocate -d.fallocateis removed from the Docker image.copy_image, which keeps the holes (found by libfs withuse_linux) and copies each data segment in a loop. Decompressing an image keeps zero blocks as holes.bmaptool create, the journal blocks of every ext4 partition are rewritten with their own content, so the bmap maps the journals. This code lives in the newfile::bmapmodule.debugfslists the journal blocks, so the package and the Docker builder neede2fsprogs.extensionsargument ofcreate_cert_and_key. actix-web runs without default features, and flate2 setsrust_backenditself.Reason
A fresh ext4 journal is all zeros. In a sparse image, e.g. one that xz decompressed into a file, or one after
fallocate -d, these zeros are holes, and a bmap skips them. A bmap-based flash then leaves the journal of the previous installation on the device, and after a power cut its replay can apply an old transaction to the new filesystem. Rewriting the journal blocks allocates the holes and leaves every other block as it is; for an omnect image this adds the cert, etc and data journals to the flash. This replaces the image-side approach discussed in omnect/meta-omnect#701: the Yocto build already maps the journals, and they are lost only when the image is decompressed sparse.With
default-features = false, libfs made byte copies, so every image came back fully allocated, and the decompress wrote zero blocks as data.fallocate -dhid both; without it, every bmap would map the whole image.libfs::copy_fileis not used because it copies each data segment with a singlecopy_file_rangecall, which moves at most 2 GiB, and ignores a short copy.Writing only changed blocks also fixes a gap of the old
dd conv=notrunc,sparse: a block that changed to all zeros was skipped and kept its old content; now it is written because it differs.actix-web's default
http2feature pulled in h2 0.3.27 (RUSTSEC-2026-0258), which has no fixed 0.3 release. The OAuth callback server serves only plain HTTP on localhost, so it never uses HTTP/2. flate2 got its zlib backend only through actix-web's compression features. For a certificate without CSR, omnect-crypto 0.5.0 adds the same extensions as 0.4.0.