End-to-end neural-network pipeline that tries to decode digits 0–9 from raw EEG signals in the MindBigData dataset, with an honest comparison to the published literature.
Final score: 21.30% test accuracy / 19.80% macro-F1 on the held-out EPOC test set, 19.91% ± 0.63% accuracy under 5-fold cross-validation. Random chance on 10 balanced classes is 10%. Legitimate published results on the same task sit in the 16–22% range (Tsamourgelis & Adamopoulos, 2025).
Real EPOC test event #7190: true label = 7, ensemble prediction = 7 with 84% confidence.
MindBigData is sometimes called the "MNIST of brain digits", but the analogy is misleading:
- The input is a noisy multi-channel EEG signal recorded from a consumer EEG headset, not a clean 28×28 image.
- Data come from a single subject (D. Vivancos, 2014–2015), so there is no between-subject generalization to talk about.
- The dataset has no official train/test split, and a same trial produces several rows in the raw file (one per electrode). Splitting by row instead of by event silently leaks the test set into training and inflates the accuracy reported in many papers.
- A peer-reviewed review of the field (Tsamourgelis & Adamopoulos, Stats 2025) attributes most published 70–98% numbers to data leakage and dimension mishandling. The honest range is closer to 16–22% accuracy on the 10-class task.
The goal of this project is not to chase that headline number but to build a calibrated pipeline that respects the dataset's structure and to quantify what actually works.
| Approach | Test acc | Test macro-F1 |
|---|---|---|
| 1D CNN, 20 epochs | 15.05% | 13.68% |
| Lightweight Transformer | 15.44% | 11.36% |
| CNN+GRU baseline | 16.64% | 15.46% |
| CNN+GRU + temporal derivatives | 17.60% | 16.44% |
| Previous best derivative ensemble | 18.41% | 17.54% |
| Augmented CNN+GRU (single seed) | 20.03% | 18.78% |
| 3-seed augmented ensemble | 20.81% | 19.46% |
| 5-seed augmented + Test-Time Augmentation | 21.30% | 19.80% |
A 5-fold cross-validation of the augmented single-seed recipe gives a mean accuracy of 19.91% ± 0.63% and mean macro-F1 of 18.27% ± 0.76%, confirming the single-split numbers are not a lucky tirage.
A separate binary sanity check (blank screen vs. any digit, 159 vs. 159, balanced) reaches 97.9% test accuracy with the same architecture. Detecting that a stimulus was shown is trivial; identifying which digit is the actual hard problem.
Each row below is an isolated change relative to the row above it.
| Change | Effect |
|---|---|
| Per-event z-score normalization (vs. global) | base (vs. ~chance) |
| FFT band-pass 1–45 Hz / 0.5–45 Hz | neutral |
Clip to [-3, +3] |
neutral |
| Append first temporal derivatives (14 → 28 ch.) | +1.3 pts F1 |
| EEG augmentation (time jitter, gaussian noise, channel dropout, mixup) | +2.3 pts F1 |
| Cosine LR + warmup + label smoothing | small but stable |
| 5-seed equal-weight ensemble | +0.9 pts F1 |
| Test-Time Augmentation (n=8, jitter=6, noise=0.02) | +0.1 pts F1 |
The takeaway: an EEG-specific augmentation pipeline + multi-seed ensembling does most of the lifting; "clever" preprocessing tricks like band-pass and clipping do not.
.
├── README.md (this file)
├── LICENSE MIT
├── requirements.txt
├── src/ data loaders, preprocessing, models, trainers
├── tests/ unit tests for every public function
├── notebooks/
│ ├── Insight_Project_Report.ipynb
│ ├── EPOC_Colab_Pipeline.ipynb GPU pipeline with EEGNet + EEG-Conformer
│ └── build_colab_notebook.py
├── configs/
│ └── epoc_heterogeneous_ensemble.json
├── docs/
│ ├── presentation.tex 10-minute LaTeX/Beamer summary
│ ├── figures/ PDF/JPG/GIF outputs
│ ├── insight/insight_results.md
│ └── epoc/epoc_plan.md
├── results/metrics/ JSON metric files per experiment
└── JOURNAL.md day-by-day project log
Raw and preprocessed data files, .pt model weights, and intermediate
artifacts are intentionally gitignored — the pipeline regenerates them
from scratch.
# 1. Environment
python3 -m venv .venv && .venv/bin/python -m pip install -r requirements.txt
# 2. Download the MindBigData EPOC zip from https://www.mindbigdata.com/opendb/
# and extract it to data/raw/epoc/EP1.01.txt
# 3. Preprocess (resample 256, per-event z-score, stratified split by event)
.venv/bin/python -m src.preprocess_mindbigdata --device EP --exclude-label -1
# 4. Export to NumPy for fast loading
.venv/bin/python -m src.export_events_npz \
--events data/processed/epoc/epoc_events_256.jsonl.gz \
--splits data/processed/epoc/epoc_splits.json \
--output data/processed/epoc/epoc_events_256.npz
# 5. Append temporal derivatives (14 channels -> 28)
.venv/bin/python -m src.add_temporal_derivatives \
--input data/processed/epoc/epoc_events_256.npz \
--output data/processed/epoc/epoc_events_256_with_derivatives.npz \
--summary-output results/metrics/epoc/derivatives_summary.json
# 6. Train the 5 augmented seeds (any order)
for seed in 52 7 21 99 13; do
.venv/bin/python -m src.train_cnn_gru_augmented \
--events data/processed/epoc/epoc_events_256_with_derivatives.npz \
--splits data/processed/epoc/epoc_splits.json \
--metrics-output results/metrics/epoc/cnn_gru_augmented_seed${seed}_metrics.json \
--model-output models/epoc/cnn_gru_augmented_seed${seed}.pt \
--epochs 35 --batch-size 128 --gru-hidden-size 96 --seed $seed
done
# 7. Evaluate the 5-seed ensemble with TTA
.venv/bin/python -m src.evaluate_ensemble_tta \
--members configs/epoc_heterogeneous_ensemble.json \
--output results/metrics/epoc/ensemble_5seeds_augmented_tta8_metrics.json \
--n-passes 8 --time-jitter 6 --noise-std 0.02
# 8. (optional) 5-fold cross-validation sanity check
.venv/bin/python -m src.make_kfold_indices \
--events data/processed/epoc/epoc_events_256_with_derivatives.npz \
--output data/processed/epoc/epoc_kfold5_indices.json
.venv/bin/python -m src.train_kfold_cv \
--events data/processed/epoc/epoc_events_256_with_derivatives.npz \
--folds data/processed/epoc/epoc_kfold5_indices.json \
--output-dir results/metrics/epoc/kfold5A Colab-ready notebook (notebooks/EPOC_Colab_Pipeline.ipynb) reproduces
the augmented CNN+GRU and trains heavier architectures (EEGNet and
EEG-Conformer) on a GPU. It expects a quantized int8 .npz upload
(~205 MB) to keep Colab's web upload short.
- Dataset. MindBigData (David Vivancos), https://www.mindbigdata.com/opendb/. Subject-level EEG recordings during visual digit presentation.
- Calibration benchmark. A. Tsamourgelis & A. Adamopoulos, EEG Signal Analysis for Numerical Digit Classification: Methodologies and Challenges, Stats 8(1):14, MDPI, 2025. Provides a careful review of the field and the legitimate accuracy range used to calibrate this project.
- EEGNet / EEG-Conformer. Lawhern et al. 2018; Song et al. 2022.
Implemented in
src/train_eegnet.pyand in the Colab notebook.
This repository started as a 6-week mini-project for an undergraduate neural-networks course. The methodology, calibration, and write-up are the parts of the work I would defend; the absolute score is, on purpose, where the literature says it ought to be.
MIT — see LICENSE.