Query the staging retry batch through the static index

[ramonsmits]

The staging query of the retry processor now uses the static index RetryBatches_ByStatusAndSession. Before, it was a dynamic query. RavenDB created the auto index Auto/RetryBatches/ByStatus for it on every error database.

Analysis

• RetryStagingStore.GetStagingBatch ran session.Query<RetryBatch>() with a filter on Status, without a static index. The query has been dynamic since the first version of group retries in 2015.
• RavenDB serves a dynamic query with an auto index. The retry processor calls GetStagingBatch on every processing cycle. The auto index is created seconds after the database is created, also when the collection is empty.
• An auto index follows the database setting Indexing.Auto.SearchEngineType. It has no per-index search engine setting. The migration procedure in the docs changes the engine per index, so it cannot change this index.
• The Error Database Search Engine custom check and the start-up warning list all indexes, auto indexes included. After all static indexes use Lucene, the check still fails because of Auto/RetryBatches/ByStatus. Found in the smoke test of 6.21.1 for Migrated Lucene indexes are reset to Corax at start-up when the index is not locked #5948.

Changes

• RetryStagingStore.GetStagingBatch queries through RetryBatches_ByStatusAndSession. That index already maps Status.
• New test RetryStagingStoreIndexTests.Getting_the_staging_batch_creates_no_auto_index (RavenDB). It fails on the old query and passes on the new one.

Existing databases

• The code does not delete an existing auto index. RavenDB marks an auto index as idle after 30 minutes without queries and deletes it 72 hours later (Indexing.TimeToWaitBeforeDeletingAutoIndexMarkedAsIdleInHrs). It can also be deleted in the Studio.

Testing

• ServiceControl.Persistence.Tests.RavenDB, filter RetryStagingStore: 14/14 in Release.