Skip to content

Skip remote database engines when scanning materialized views - #804

Open
alexsubota wants to merge 2 commits into
ClickHouse:mainfrom
alexsubota:fix/skip-remote-engines-when-scanning-mvs
Open

alexsubota wants to merge 2 commits into
ClickHouse:mainfrom
alexsubota:fix/skip-remote-engines-when-scanning-mvs

Conversation

@alexsubota

@alexsubota alexsubota commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Relation listing no longer opens every MySQL, PostgreSQL, and MaterializedMySQL database engine while looking for materialized views. One unreachable remote database was failing the unfiltered system.tables scan for every model.
  • The associated-view lookup is limited to the model database.
  • source() on a remote schema still queries that schema directly.

Test plan

  • Run dbt against a server that has an unreachable MySQL database engine and confirm relation listing and materialized-view builds succeed for local databases.
  • Confirm a source() on a MySQL or PostgreSQL database engine still reads that schema.
  • Run the materialized view integration tests.

Note

Low Risk
Targeted macro changes to relation-listing SQL; behavior improves resilience with mixed local/remote engines and does not alter explicit remote source() access.

Overview
Relation listing and materialized-view discovery no longer trigger connections to MySQL, PostgreSQL, or MaterializedMySQL database engines. clickhouse__list_relations_without_caching first loads local database names from system.databases (engines excluded without connecting) and restricts the system.tables materialized-view scan to those databases, so a single unreachable remote DB cannot break listing for every schema.

The associated MV lookup in clickhouse__search_associated_mvs_to_target is scoped with database = '<model schema>' instead of scanning all databases. source() on a remote engine schema is unchanged: the outer relation-list query still filters by that schema directly.

Reviewed by Cursor Bugbot for commit f9a4fc5. Bugbot is set up for automated code reviews on this repo. Configure here.

An unreachable MySQL database engine makes an unfiltered system.tables scan fail for every model. Limit the associated-view lookup to the model database, and restrict relation listing to local databases so source() on a remote schema still connects only to that schema.
@alexsubota
alexsubota force-pushed the fix/skip-remote-engines-when-scanning-mvs branch from 13b7c9c to f9a4fc5 Compare November 16, 2026 08:32
select name
from system.tables
where engine = 'MaterializedView'
where database = '{{ relation_schema }}'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is too limiting, a user may create a MV from another database.

where engine = 'MaterializedView'
and create_table_query like '%TO %'
{%- if ns.local_databases %}
and database in ({% for db_name in ns.local_databases %}'{{ db_name | replace("'", "''") }}'{% if not loop.last %}, {% endif %}{% endfor %})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just add the filtering here somehow by calling system.databases and filtering by other values? feels like adding two sequentials queries should not be the way to go and maybe one is enough

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants