Skip to content

Keep encoded # in ZIM paths when building document URIs - #342

Merged
benoit74 merged 1 commit into
openzim:mainfrom
rmz-oz:fix-encoded-hash-in-document-uri
Sep 28, 2026
Merged

benoit74 merged 1 commit into
openzim:mainfrom
rmz-oz:fix-encoded-hash-in-document-uri

Conversation

@rmz-oz

@rmz-oz rmz-oz commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Fix #341

normalize() correctly decodes %23 to a literal # in the ZIM path, but get_document_uri() then runs that ZIM path through urlsplit(), which treats the # as a fragment delimiter and drops everything after it. So the rewritten link points to another entry, usually one that does not exist in the ZIM.

The change is one line: urlsplit(item_path.value, allow_fragments=False). A ZIM path is never encoded, so a # in it is always part of the path or query. The existing quote(relative_path, safe="/=,") then turns it back into %23. A real fragment passed as item_fragment is still appended as before.

Using @veloman-yunkan's script from openzim/overview#92 (root https://example.com/):

input before after
a/b%23dont_drop_me_please a/b a/b%23dont_drop_me_please
a/b%23x#frag (full rewrite) a/b#frag a/b%23x#frag
a/b?q=1%232 a/b%3Fq=1 a/b%3Fq=1%232

For comparison, urlRewriteFunction in javascript/src/wombatSetup.js already gives .../a/b%23dont_drop_me_please and .../a/b%3Fq=1%232 for the same inputs, because it goes through encodeURIComponent(). With this change the static (Python) rewriting and the dynamic (wombat) rewriting agree on these URLs.

Tests: three new cases in test_relative_url (encoded # in the path, encoded # followed by a real fragment, encoded # in the query). All three fail on main and pass with the fix. Locally: tests/rewriting/test_url_rewriting.py 139 passed; ruff check, ruff format --check and pyright on the touched files are clean. (I could not run the libmagic-dependent test modules on my machine; the rest of the suite is 627 passed before, 630 after.)

This does not touch the other points from openzim/overview#92 (#340 on //, #321 on dot segments).

CHANGELOG entry added under Unreleased / Fixed.

A `%23` in the path or query of an original URL is decoded to a
literal `#` in the ZIM path, as expected. But get_document_uri() then
passed that ZIM path to urlsplit(), which treated the `#` as a fragment
delimiter and silently dropped everything after it. The rewritten link
therefore pointed to another (usually missing) ZIM entry, e.g.
`a/b%23c` was rewritten to `a/b`.

Tell urlsplit() not to look for a fragment: a ZIM path is never
encoded, so the `#` is part of the path. It is then re-encoded to `%23`
by the existing quote() call, which matches what the JS side
(wombatSetup.js) already produces via encodeURIComponent().

Fix openzim#341

@benoit74 benoit74 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much, a very good PR, nothing to add besides "let's merge". Looking forward for more contribution on "sibling" issues (or any other one actually) if you see fit.

@codecov

codecov Bot commented Sep 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (7908b7b) to head (5ecd60b).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #342   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           42        42           
  Lines         2694      2694           
  Branches       398       398           
=========================================
  Hits          2694      2694           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@benoit74
benoit74 merged commit 164a0c3 into openzim:main Sep 28, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

URL normalisation: URL-encoded # makes its way into the path, but is lost when obtaining the URI of that entry

2 participants