English | Français

Retrieval Engine Competition

A search engine that finds the relevant answers to a technical question among 216,041 forum posts, then predicts the question's category.

Built for the Kaggle competition retrieval-engine-competition. Lexical baselines (TF-IDF, BM25+) are compared with dense embeddings from Sentence Transformers, then improved with query tuning, a category classifier, reranking and rank fusion. Encoding only the text of each query raises MRR@10 from 0.27 to 0.46, the largest gain of the project.

Results · How it works · Data · Getting started · Project structure

---

Results

Evaluated on the 327 training queries, at K = 10.

Method	MRR@10	Recall@10
TF-IDF (baseline)	0.151	0.105
BM25+ (baseline)	0.203	0.122
Embeddings (baseline, query = title + text + tags)	0.275	0.196
Embeddings, query = text only	0.464	0.311
Embeddings + category reranking	0.460	0.310
RRF (embeddings + BM25+)	0.415	0.264

• Semantic embeddings clearly beat the lexical methods on this corpus.
• How the query is built matters more than the model: dropping the tags from the query is worth +0.19 MRR@10.
• Category reranking barely helps. It gains a little when the classifier is right (280 queries, +0.003 MRR on average) and loses more when it is wrong (47 queries, -0.039).
• With only 327 queries these scores have high variance (the cross-validated hybrid runs in val/outputs/hybrid_cv_runs.csv show a standard deviation around 0.07), and they measure performance on the training set, not generalization.

---

How it works

flowchart TD
    C[("docs.json<br/>216,041 posts, 5 categories")] --> PRE["01_eda<br/>title + text + tags, cleaning"]
    PRE --> P1
    subgraph P1["Phase 1, baselines"]
        direction TB
        TF["02_tfidf<br/>TF-IDF, cosine"]
        BM["03_bm25<br/>BM25+"]
        EMB["04_embeddings<br/>all-MiniLM-L6-v2"]
    end
    P1 --> EV["05_baseline_results<br/>MRR@10, Recall@10"]
    EV --> P2
    subgraph P2["Phase 2, tuning and fusion"]
        direction TB
        Q["06_parameter_tuning<br/>query = text only"]
        CLF["07_phase2<br/>category classifier, reranking, RRF"]
        ENS["08 / 09<br/>weighted ensembles"]
    end
    P2 --> OUT(["Submission<br/>top 100 docs + category per query"])

Loading

Phase 1. Each document becomes one content field (title, text and tags), lowercased and stripped of punctuation, NLTK stopwords and short tokens. Three retrievers are built on it: TF-IDF with unigrams and bigrams and sublinear_tf, BM25+ from rank_bm25 (k1 = 1.5, b = 0.75, δ = 0.5), and all-MiniLM-L6-v2 embeddings (384 dimensions, normalized and cached). t-SNE and UMAP show the categories as well separated clusters, which motivates a category-aware second phase.

Phase 2.

• Query construction: encoding only the query's text, without its tags, lifts MRR@10 from 0.275 to 0.464.
• Category classifier: logistic regression on TF-IDF features, trained on 183,634 documents. It reaches 98.4% accuracy on 32,407 held-out documents and 85.6% on the 327 training queries. A character n-gram variant (char_wb) trained on queries reaches 98.8% on an 82-query validation split.
• Category reranking: score + α · P(query category = document category), as a soft boost or a hard filter.
• Rank fusion: Reciprocal Rank Fusion of embeddings and BM25+, then weighted ensembles of several runs, routed by the classifier's confidence.

The submission pairs each of the 141 test queries with an ordered list of 100 documents and a predicted category (query_id, relevant_doc_ids, category).

---

Data

Corpus	216,041 documents (questions and answers from technical forums)
Training queries	327, with their relevant documents (9.1 per query on average)
Test queries	141
Categories	android, gaming, programmers, tex, unix

Each document and query has a title, a text and tags. The data comes from the Kaggle competition retrieval-engine-competition and is not versioned in this repository.

---

Getting started

The data, indexes and embeddings (about 1.7 GB) are not in the repository.

1. Install the dependencies:

   git clone https://github.com/Yvngval/Data_Science_Project.git
   cd Data_Science_Project
   pip install -r chatodit/requirements.txt

2. Download the data of the Kaggle competition retrieval-engine-competition and put docs.json, queries_train.json, qgts_train.json and queries_test.json in val/data/ (or chatodit/data/).
3. Run the notebooks in order from the notebooks/ folder. 01_eda writes the preprocessed corpus; 02 to 04 build the indexes and embeddings in models/. Encoding the 216,041 documents takes tens of minutes without a GPU.
4. For the query classifier alone, run python val/predict_category.py. It fills the category column of an existing submission file in val/outputs/.

---

Project structure

The repository holds two workspaces that follow the same pipeline, one per team member.

chatodit/
  notebooks/          01 to 09 (Phase 1, tuning, classification, final ensembles)
  requirements.txt
val/
  notebooks/          01 to 08
  outputs/            figures, results and submission files
  predict_category.py standalone query category classifier
  rapport_projet_ultra.pdf   full technical report (French)

Notebook	Content
01_eda	Data exploration and corpus preprocessing
02_tfidf	TF-IDF index and evaluation
03_bm25	BM25+ index and evaluation
04_embeddings	Sentence Transformers, semantic search, t-SNE and UMAP
05_baseline_results	Baseline comparison and first submission
06_parameter_tuning	Query construction strategies, BM25+ grid search
07_phase2_classification_reranking	Category classifier, reranking, RRF
08_fast_ensemble_submission	Fast fusion of existing submissions
09_deadline_final_ensemble	Weighted RRF, confidence routing, category boost

---

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md.

License

The code is released under the MIT License. The competition data is not included and remains subject to the Kaggle competition rules.

Authors

Valentin Ponnoussamy (@Yvngval) and @chatodit, 2026.