Skip to content

fix(file): keep image holes and map ext4 journals in bmaps - #158

Draft
JanZachmann wants to merge 11 commits into
omnect:mainfrom
JanZachmann:jz-2026-09-28-keep-image-layout
Draft

JanZachmann wants to merge 11 commits into
omnect:mainfrom
JanZachmann:jz-2026-09-28-keep-image-layout

Conversation

@JanZachmann

@JanZachmann JanZachmann commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • write_partition compares the partition file with the image in 4 KiB blocks and writes only the blocks that differ, instead of running fallocate -d. fallocate is removed from the Docker image.
  • The image is copied with copy_image, which keeps the holes (found by libfs with use_linux) and copies each data segment in a loop. Decompressing an image keeps zero blocks as holes.
  • Before bmaptool create, the journal blocks of every ext4 partition are rewritten with their own content, so the bmap maps the journals. This code lives in the new file::bmap module. debugfs lists the journal blocks, so the package and the Docker builder need e2fsprogs.
  • All direct dependencies are bumped to their latest version, including major versions. omnect-crypto 0.5.0 drops the unused extensions argument of create_cert_and_key. actix-web runs without default features, and flate2 sets rust_backend itself.

Reason

A fresh ext4 journal is all zeros. In a sparse image, e.g. one that xz decompressed into a file, or one after fallocate -d, these zeros are holes, and a bmap skips them. A bmap-based flash then leaves the journal of the previous installation on the device, and after a power cut its replay can apply an old transaction to the new filesystem. Rewriting the journal blocks allocates the holes and leaves every other block as it is; for an omnect image this adds the cert, etc and data journals to the flash. This replaces the image-side approach discussed in omnect/meta-omnect#701: the Yocto build already maps the journals, and they are lost only when the image is decompressed sparse.

With default-features = false, libfs made byte copies, so every image came back fully allocated, and the decompress wrote zero blocks as data. fallocate -d hid both; without it, every bmap would map the whole image. libfs::copy_file is not used because it copies each data segment with a single copy_file_range call, which moves at most 2 GiB, and ignores a short copy.

Writing only changed blocks also fixes a gap of the old dd conv=notrunc,sparse: a block that changed to all zeros was skipped and kept its old content; now it is written because it differs.

actix-web's default http2 feature pulled in h2 0.3.27 (RUSTSEC-2026-0258), which has no fixed 0.3 release. The OAuth callback server serves only plain HTTP on localhost, so it never uses HTTP/2. flate2 got its zlib backend only through actix-web's compression features. For a certificate without CSR, omnect-crypto 0.5.0 adds the same extensions as 0.4.0.

write_partition ran `fallocate -d` on the whole image, which turned every
zero-filled block into a hole. A bmap created afterwards with `-b` then
skipped those blocks, e.g. ext4 journals that the image deliberately
writes as zeros, so a bmap-based flash left the old journal data on the
device.

Write only the blocks that differ from the image instead. Unchanged
blocks keep their allocation, and a block that changed to all zeros is
now written too (`dd conv=sparse` skipped it and kept the old content).

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
- all direct dependencies to their latest version, omnect-crypto to
  0.5.0 (create_cert_and_key no longer takes the unused extensions
  argument; the extensions of a certificate without CSR are the same)
- actix-web without default features: the OAuth callback server only
  serves plain HTTP on localhost, and the http2 feature pulled in h2 0.3
  (RUSTSEC-2026-0258, no fixed 0.3 release)
- flate2 selects its rust_backend itself instead of relying on
  actix-web's compression features to enable one

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
libfs was built with `default-features = false`, which also drops its
`use_linux` feature. Without it libfs copies byte by byte, so every image
omnect-cli works on came back fully allocated, and a bmap created with `-b`
mapped the whole image. The `fallocate -d` that write_partition used to run
hid this.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
A fresh ext4 journal is all zeros, and a sparse image, e.g. one that
xz decompressed into a file, holds it as holes. A bmap then skips the
journal, so a bmap-based flash leaves the journal of the previous
installation on the device, and after a power cut its replay can apply
an old transaction to the new filesystem.

Before creating the bmap, rewrite the journal blocks of every ext4
partition with their own content. Holes become allocated zero blocks and
all other blocks stay unchanged. debugfs lists the journal blocks, so
the package now depends on e2fsprogs.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
@JanZachmann JanZachmann changed the title fix(file): keep the image layout when writing a partition back fix(file): keep holes and map ext4 journals in generated bmaps Sep 28, 2026
libfs::copy_file copies each data segment with one copy_file_range call and ignores a short copy, so a segment over 2 GiB was cut off without error. copy_image keeps the holes and loops until each segment is copied.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
- fall back to /usr/sbin/debugfs when debugfs is not in PATH
- reject an invalid s_log_block_size before it is used as a shift
- skip partitions whose superblock is past the end of the image
- skip the MBR extended container entry
- share SECTOR_SIZE between partition and functions

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
The deb package depends on e2fsprogs for debugfs.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
Decompress wrote every zero block as data, so a bmap of a compressed input mapped the whole image. The decompressed file now seeks over zero blocks and keeps them as holes.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
The ext4 journal lookup is only used for bmap generation, so it moves with generate_bmap_file from functions.rs to bmap.rs.

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
@JanZachmann JanZachmann changed the title fix(file): keep holes and map ext4 journals in generated bmaps fix(file): keep image holes and map ext4 journals in bmaps Sep 28, 2026
- a bmap flash onto a device with old content now overwrites the journal
- an invalid s_log_block_size is rejected
- a partition that starts past the end of the image is skipped
- the MBR extended container entry is not returned as a partition

Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant