Conversation
`classifyDirectory` gains `DirShape::PartFile` for `<table>/<part>/<file>` (live, detached, moving, non-Atomic); the two `TableSubdir` case bodies become `tableSubdirExists`/`tableSubdirChildren` and the new shape answers through them for now, so no answer changes in this commit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V8mZSGiD8iJumpJMiQnrmC Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…der view `existsDirectory`/`listDirectory` on `<table>/<part>/<file>` used to fall through to the table-subdirectory branch and LIST the life's `_files/` prefix per probe; `MergeTreeDataPartChecksum::checkSize` asks it for every checksum entry of every part at load (77k LISTs on a 1,672-part restart, issue #2439). A resolved ref now answers from its retained view; an unresolved one keeps the old branch. A non-projection nested directory inside a part now reports present, its children and non-empty. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V8mZSGiD8iJumpJMiQnrmC Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…rt-file probes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V8mZSGiD8iJumpJMiQnrmC Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V8mZSGiD8iJumpJMiQnrmC Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
SYSTEM CAS FORGET was missing from 05053_cas_part_file_probes_no_list.sh, unlike 04278_cas_disk.sh's model; without it the custom disk and its GC thread stay registered for the server process's lifetime. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V8mZSGiD8iJumpJMiQnrmC Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…ifest Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V8mZSGiD8iJumpJMiQnrmC Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…n three test oracles, fix the gtest gate filter
Fixes from the final whole-branch review and the codex review of the
part-file directory probes:
- Drop `dirPrefixOf`. Its trailing-slash arm never runs — `PartPathParser`'s
`splitNonEmpty` drops empty components, so `Route::file` never ends in
`/` — and its comment claimed work it never did. Both `PartFile` call
sites now use `dr.r->file + "/"` directly, as `existsFileOrDirectory`
already did.
- `PartFileAnswersFromTheViewWithoutAList`'s cold-cache case now asserts
the manifest `GET` count exactly (nine probes reach `getView`; the tenth,
`isDirectoryEmpty` on a projection directory, short-circuits) instead of
merely `> 0`.
- `FailedManifestReadPropagatesAndDoesNotList` now injects `CORRUPTED_DATA`
instead of `CANNOT_READ_ALL_DATA`. `CORRUPTED_DATA` is a deterministic
local failure, so the read engine propagates it on the first attempt
instead of retrying it to the lease budget (~20s) like a transport
fault; the test now asserts the propagated code, that the message names
the manifest key, and that exactly one manifest `GET` and zero `LIST`s
were issued.
- `INSTANTIATE_TEST_SUITE_P`'s instance name is now `CASCaches` (was
`Caches`), so the two parameterized `PartFileAnswersFromTheViewWithoutAList`
cases match the `CAS*` gate filter used elsewhere; they were silently
excluded from it before.
- `openCountingStorage`'s directory-owning guard is now constructed before
`storage->startup()`, so a throwing `startup()` still cleans up the two
temp directories; `create_directories`'s error is now propagated instead
of ignored.
- The stateless test's oracle now also asserts `l20 > 0`, so it cannot pass
vacuously when the `CASRootList` counter reads zero for both `ATTACH`es.
- Two comments that framed the unresolved-ref invariant as history
("answers exactly as before this shape existed") now state the
invariant directly; the `PartFile` classification comment's parser
description is now qualified to Atomic paths, since a non-Atomic path
anchors on the rightmost part-shaped component instead.
Verified: `ninja -C build unit_tests_dbms clickhouse` clean;
`unit_tests_dbms --gtest_filter='CAS*'` — 2536 tests, all passed
(`PartFileAnswersFromTheViewWithoutAList/Default` and `/Disabled` now
included; `FailedManifestReadPropagatesAndDoesNotList` down from ~20s to
14ms); `05053_cas_part_file_probes_no_list` with `--test-runs 5` against a
standalone server — 5/5 passed.
Related: #2439
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8mZSGiD8iJumpJMiQnrmC
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…nline code The unresolved-ref fallback sentence named only the Atomic-table branch (the table-level file listing). A non-Atomic table's unresolved probe falls back to the mirrored live-tree listing instead; say both. Also wrap `LIST` in inline code, matching every other S3 verb on this page. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V8mZSGiD8iJumpJMiQnrmC Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
to filimonov/ClickHouse
that referenced
this pull request
Sep 27, 2026
Related: Altinity#2440 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V8mZSGiD8iJumpJMiQnrmC
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On a
casdisk, a directory probe on a path inside a part (<table>/<part>/<file>) was routed to the table-subdirectory branch, which runs an S3 LIST of the table's_files/prefix.MergeTreemakes this probe for every checksum entry of every part while it loads (MergeTreeDataPartChecksum::checkSize), so one restart of a 1,672-part server issued 77k LISTs in three minutes, got 537503 Slow Downand failed 139 uploads.This PR gives such a path its own shape,
PartFile, and answers it from the part's folder manifest: a plain file is not a directory, a nested directory is. No LIST is issued when the part resolves. When it does not resolve (a table-level subdirectory that happens to look like a part), the old branch runs unchanged.Measured on
ATTACH TABLE:CASRootListwas 105 for 20 parts and 1005 for 200 parts; it is now 5 for both.One visible change: a nested non-projection directory inside a part now reports present, its children and non-empty. Before, it reported absent and empty. Projection directories keep their answers.
Tests: routing cases in
gtest_ca_wiring, a newgtest_cas_directory_probes(zero LIST for resolved parts, exact one-LIST oracle for unresolved paths, a failed manifest read throws and never lists, a 50-part load profile) and the stateless test05053(DETACH/ATTACH of 20 and 200 parts, equal LIST counts read from the ATTACH query's ownProfileEvents).Closes: #2439
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
casdisk: loading a table no longer issues one object-store LIST per part file. Directory probes inside a part are answered from the part manifest, so a restart costs a fixed number of LISTs per table instead of one per file.Documentation entry for user-facing changes