Keep the search engine configured on an index when deploying index definitions

[mauroservienti]

Problem

Version 6.20.0 recommends the migration of RavenDB indexes from Corax to Lucene. The documented procedure changes the search engine of one index on the Configuration tab in RavenDB Studio. RavenDB stores this choice as Indexing.Static.SearchEngineType in the index configuration.

At each start-up, ServiceControl deploys its index definitions with IndexCreation.CreateIndexesAsync. These definitions have no search engine. RavenDB sees a different definition and builds a side-by-side ReplacementOf/<index> with the database default. For databases created before 6.20, the default is Corax, because UpdateDatabaseSettings sets it. If the migrated index is not locked, RavenDB resets it to Corax and rebuilds it.

A customer unlocked a migrated MessagesViewIndexWithFullTextSearch because the docs said that a lock is not necessary from 6.20.0. The index was reset to Corax.

Fix

The new IndexDeployment helper in ServiceControl.RavenDB replaces IndexCreation.CreateIndexesAsync in the Primary and Audit DatabaseSetup. Before deployment, it reads the existing index definitions and sets the search engine of each index with these rules:

1. A search engine set in the index definition in code, through SearchEngineType or Configuration, always wins.
2. An existing index keeps the search engine configured on the server. The definitions then match and RavenDB does not rebuild the index. A pending replacement is checked first, because it holds the latest choice of the operator.
3. An index that does not exist on the server is created with Lucene set on the index. This also applies to databases that still default to Corax.
4. An existing index without a search engine of its own continues to use the database default. An upgrade does not start a rebuild.

Effects:

• An index migrated in Studio is not changed. No rebuild and no lock are necessary.
• Changed index definitions still roll out. The rebuilt index keeps the search engine set by the operator.
• Other manual changes to a definition are still reset at start-up, as before.
• A pending Corax replacement created by 6.20.0 or 6.21.0 is discarded at start-up when the migrated index is unlocked. The deployed definition matches the original index again.
• A log line at Info level reports each index that keeps a configured search engine or that is created with Lucene.

Tests

Audit IndexSetupTests:

• Indexes_should_be_reset_on_setup tested the old behavior. Two tests replace it:
  • Search_engine_configured_on_the_index_should_be_preserved_on_setup: no replacement, the index is not recreated.
  • Indexes_should_be_reset_on_setup_keeping_the_configured_search_engine: a changed field is reset, the engine is kept.
• Pending_replacement_using_the_database_default_should_be_discarded_in_favor_of_the_configured_search_engine: reproduces the state of the customer. Indexing is stopped so that the replacement cannot swap.
• The two lock-mode tests now change a field instead of the search engine. A different search engine alone no longer causes an update.
• New_indexes_should_be_created_with_lucene: a deleted index is recreated with Lucene set on it.
• Search_engine_set_in_the_index_definition_should_take_precedence_over_the_one_on_the_server: covers SearchEngineType and Configuration.

Primary IndexSetupTests (new): covers the assembly scan.

Each new Audit test fails without the code that it covers. Both RavenDB persistence suites pass locally: Audit 57/57, Primary 349/349.

Verification with containers

Upgrade paths tested with podman, an external RavenDB container and RabbitMQ. Indexes were migrated in the same way as the Studio does it.

• Fresh 6.21.0: all indexes use Lucene.
• 6.19.3, then 6.21.0: migrated indexes are reset to Corax. The bug is reproduced.
• 6.19.3, then this fix: migrated indexes keep Lucene. The audit MessagesViewIndexWithFullTextSearch is rebuilt once on Lucene, because its definition changed in 6.20.0 (Remove decommissioned endpoints from usage reports #5670).
• 6.19.3, then 6.21.0 with a pending replacement, then this fix: the Corax replacement is discarded and the migrated index is kept.
• 6.19.3, then 6.21.0 with the replacement swapped in, then this fix: the index stays on Corax. The Lucene choice is lost and the migration must be done again.

Related

• Docs: Index locking is still required for ServiceControl 6.20.0 to 6.21.0 docs.particular.net#8521
• Planned for 6.21.1