MemCon: Memory Confluence for Cross-Imaging-Mode Small Target Detection

Shengjia Chen, Luping Ji, and Mao Ye

Important

The MemCon manuscript is currently under peer review. The results and implementation in this repository are released for research and reproducibility, but they should not be treated as an accepted publication record. A formal citation will be added after publication; no citation is requested at this stage.

Overview

Existing small target detectors are commonly trained for a single imaging mode, such as infrared, satellite, or UAV imagery. When the imaging mode changes, a separate detector is usually retrained, even when the targets and contextual knowledge are related.

This project introduces a new Cross-Imaging-Mode Small Target Detection (CIMSTD) benchmark and provides MemCon as its dedicated framework. CIMSTD incrementally learns a unified detector from a sequence of datasets. At each stage, only the current task data are available; data from previous tasks are not replayed. During inference, the detector predicts from the input image alone and does not require an imaging-mode label or an externally supplied task identity.

The main components are:

Memory Confluence: maintains shared and mode-specific memories and aggregates related memory cells in a unified hyperbolic space.
Memory Reasoning: models group-level relations among recalled memory cells.
Memory Update and Cueing: selectively consolidates new knowledge and uses recalled memory to modulate detection features.
Prototype Routing: identifies the most suitable lightweight task head from input features, enabling task-agnostic inference.

The benchmark contains seven datasets, three imaging modes, and nine dataset-specific tasks:

Infrared: DAUB-R, IRDST-M, and ITSDT-15K.
Satellite: three target tasks from SAT-MTB (train, airplane, and ship).
UAV: RGBT-Tiny, VisDrone-2019-DET, and UAV-10.

CIMSTD Benchmark

CIMSTD should be understood as a new task definition and benchmark suite, rather than a newly collected individual dataset. It combines seven existing or reorganized small-target datasets into nine ordered tasks spanning infrared, satellite, and UAV imaging. The benchmark specifies the task sequence, train/validation/test splits, continual-learning constraint, task-agnostic inference protocol, and evaluation metrics.

Naming: CIMSTD is the task/paradigm, the CIMSTD Benchmark is the standardized evaluation suite, and MemCon is the proposed method for this benchmark.

Figure 1. Comparison between conventional single-imaging-mode STD and the proposed cross-imaging-mode STD benchmark.

Figure 1 explains the key difference:

Single-imaging-mode STD, Figure 1(a): infrared, satellite, and UAV data are treated as three isolated tasks with known identities. Although the target category remains the same, each imaging mode has different target appearances and background characteristics. A separate detector (Model 1, Model 2, or Model 3) is trained for each task, producing independent parameters Θ1, Θ2, and Θ3. Knowledge is fragmented, model training is repeated, and inference requires selecting the correct task-specific detector.
Cross-imaging-mode STD, Figure 1(b): the data arrive sequentially as infrared → satellite → UAV tasks. The same detector is updated at every stage and inherits previously learned knowledge. Its cumulative imaging-mode space expands from {M1} to {M1, M2} and then {M1, M2, M3}, while its target intra-class variability expands from {V1} to {V1, V2} and then {V1, V2, V3}. The detector must learn the current task without accessing previous-task images and must retain its ability on all earlier tasks.

The notation in the figure is:

C: target category. CIMSTD fixes all small targets as one category, target.
M: imaging mode, including infrared, satellite, and UAV imaging.
V: target intra-class variability caused by changes in target appearance, scale, contrast, background, sensor, and observation condition.
Θ: detector parameters.

Therefore, CIMSTD is not class-incremental detection: the category space does not grow. It is also more specific than a conventional domain-shift benchmark because both the physical imaging mechanism and the target's intra-class appearance change across tasks. A method is expected to satisfy all of the following:

Sequential learning: train on one dataset-specific task at a time in the predefined order.
No historical-data access: when learning task t, only the current dataset D_t is available.
Cumulative retention: after each stage, evaluate all tasks observed so far.
Task-agnostic inference: do not provide a task ID or imaging-mode label at test time.
Unified evaluation: report average mAP50, average F1, relative upper-bound ratio (RUBR), and forgetting rate (FR) under the same protocol.

The benchmark contains multiple complementary settings: three 3-task cross-imaging-mode sequences, one 3-task infrared-only sequence, and extended 6-task and 9-task sequences. This supports evaluation of cross-mode knowledge reuse, catastrophic forgetting, task-agnostic routing, and scalability as more imaging modes and target variations are introduced.

Environment

The recommended environment follows the setup used by our previous moving infrared small target detection project:

Ubuntu 20.04
CUDA 11.8
NVIDIA RTX 3090
Python 3.11.8
PyTorch 2.1.1 or 2.4.1
torchvision 0.16.1
NumPy 1.26.4

Dataset Access

All datasets are bounding-box based. The three infrared datasets are publicly available:

Dataset	Imaging mode	Task ID	Access
DAUB-R	Infrared	`I1`	Baidu (code: `jya7`)
IRDST-M	Infrared	`I2`	Baidu (code: `t2ti`)
ITSDT-15K	Infrared	`I3`	Google
SAT-MTB (train, airplane, ship)	Satellite	`R1`, `R2`, `R3`	Alipan
RGBT-Tiny	UAV	`U1`	Alipan
VisDrone-2019-DET	UAV	`U2`	Alipan
UAV-10	UAV	`U3`	Alipan

To request SAT-MTB, RGBT-Tiny, VisDrone-2019-DET, or UAV-10 and the corresponding MemCon split files, contact csj_uestc@126.com.

The public IRDST link above is the source-data archive. IRDST-M denotes the task/split used by the MemCon benchmark; use the accompanying MemCon train.txt, val.txt, and test.json rather than substituting another IRDST partition.

Please follow the license and terms of use of each original dataset. The split files distributed for MemCon define the benchmark protocol and should not be randomly regenerated when reproducing the reported results.

Dataset Preparation

1. Task definitions

The seven datasets form the following nine tasks:

Alias	Dataset/task	Train list	Validation list	Test annotation
`I1`	DAUB-R	`train.txt`	`val.txt`	`test.json`
`I2`	IRDST-M	`train.txt`	`val.txt`	`test.json`
`I3`	ITSDT-15K	`coco_train_ITSDT.txt`	`coco_val_ITSDT.txt`	`test.json`
`R1`	SAT-MTB (`train`)	`train_train.txt`	`train_val.txt`	`train_test.json`
`R2`	SAT-MTB (`airplane`)	`airplane_train.txt`	`airplane_val.txt`	`airplane_test.json`
`R3`	SAT-MTB (`ship`)	`ship_train.txt`	`ship_val.txt`	`ship_test.json`
`U1`	RGBT-Tiny	`train.txt`	`test.txt`	`test.json`
`U2`	VisDrone-2019-DET	`coco_style/train.txt`	`coco_style/val.txt`	`coco_style/test.json`
`U3`	UAV-10	`train.txt`	`test.txt`	`test.json`

SAT-MTB (train) means the train target category. It does not mean the training split. For example, train_train.txt, train_val.txt, and train_test.json are respectively the training, validation, and test files for this target category.

The comments in the current Python files contain the legacy directory strings RGBT-Tiny and UVA-10. They refer to paths only. The benchmark dataset names used in the paper are RGBT-Tiny and UAV-10; set the Python variables to the actual directories on your machine.

2. Training and validation text format

train.py reads one image per line. Each line must have the following form:

/absolute/path/to/image x1,y1,x2,y2,class_id x1,y1,x2,y2,class_id ...

Example:

/path/to/DAUB/images/seq01/000001.bmp 121,84,128,91,0
/path/to/DAUB/images/seq01/000002.bmp 122,85,129,92,0 201,144,205,149,0
/path/to/DAUB/images/seq01/000003.bmp

3. COCO JSON format for evaluation

test.py evaluates one dataset at a time with a COCO-format JSON file. The JSON must contain images, annotations, and categories.

Because model_data/classes.txt contains the single class name target, prepare the test JSON as a single-category detection dataset:

{
  "categories": [
    {"id": 1, "name": "target"}
  ]
}

Requirements:

Remap every evaluated target annotation to the single category named target.
COCO bbox values use [x, y, width, height], unlike the training text files, which use [x1, y1, x2, y2].
Each image_id must be an integer.
Each image's file_name must resolve correctly under dataset_img_path in test.py.
Use the provided MemCon train/validation/test split files to reproduce the paper protocol.

For example:

cocoGt_path      = "DAUB/test.json"
dataset_img_path = "DAUB/"

4. Recommended directory organization

The image subdirectories can retain their original structure because the training text files contain absolute paths. A recommended organization is:


├── DAUB/
│   ├── train.txt
│   ├── val.txt
│   ├── test.json
│   └── images/
├── IRDST-M/
│   ├── train.txt
│   ├── val.txt
│   ├── test.json
│   └── images/
├── ITSDT/
│   ├── coco_train_ITSDT.txt
│   ├── coco_val_ITSDT.txt
│   ├── test.json
│   └── images/
├── SAT-MTB/
│   ├── train_train.txt
│   ├── train_val.txt
│   ├── train_test.json
│   ├── airplane_train.txt
│   ├── airplane_val.txt
│   ├── airplane_test.json
│   ├── ship_train.txt
│   ├── ship_val.txt
│   ├── ship_test.json
│   └── images/
├── RGBT-Tiny/
│   ├── train.txt
│   ├── test.txt
│   ├── test.json
│   └── images/
├── VisDrone-2019-DET/
│   └── coco_style/
│       ├── train.txt
│       ├── val.txt
│       ├── test.json
│       └── images/
└── UAV-10/
    ├── train.txt
    ├── test.txt
    ├── test.json
    └── images/

Incremental Learning Protocols

At task t, only the current dataset is used for optimization. Historical images are not replayed.

Cross-imaging-mode settings

Setting	Ordered tasks
`I1-R1-U1`	DAUB-R → SAT-MTB (`train`) → RGBT-Tiny
`I2-R2-U2`	IRDST-M → SAT-MTB (`airplane`) → VisDrone-2019-DET
`I3-R3-U3`	ITSDT-15K → SAT-MTB (`ship`) → UAV-10

Single-imaging-mode setting

Setting	Ordered tasks
`I1-I2-I3`	DAUB-R → IRDST-M → ITSDT-15K

Imaging-mode scale settings

Setting	Ordered tasks
`I1-R1-U1-I2-R2-U2`	DAUB-R → SAT-MTB (`train`) → RGBT-Tiny → IRDST-M → SAT-MTB (`airplane`) → VisDrone-2019-DET
`I1-R1-U1-I2-R2-U2-I3-R3-U3`	DAUB-R → SAT-MTB (`train`) → RGBT-Tiny → IRDST-M → SAT-MTB (`airplane`) → VisDrone-2019-DET → ITSDT-15K → SAT-MTB (`ship`) → UAV-10

Each setting is an independent experiment and must start from a fresh T0. Do not reuse memory, heads, or prototypes from another setting.

Training

1. Default training configuration

The released configuration in train.py follows the paper:

Option	Value
Input size	`512 × 512`
Epochs per task	`50`
Batch size	`4`
Optimizer	`SGD`
Configured initial learning rate	`0.01`
Configured minimum learning rate	`0.0001`
Momentum	`0.937`
Weight decay	`5e-4`
Random seed	`2024`
Memory top-k (`k`)	`8`
Memory retrieval temperature	`0.07`
Shared-memory update threshold (`epsilon_g`)	`0.60`
Mode-specific novelty threshold (`epsilon_n`)	`0.10`
Hypergraph distance temperature (`tau_e`)	`0.40`

The backbone and feature-fusion adapter are frozen by default.

Note

In the current implementation, utils/utils_fit.py reduces the number of training iterations per epoch to 1/5 of the original value: epoch_step = max(1, epoch_step // 5). This setting is used to reduce repeated optimization on the same training samples and mitigate overfitting, particularly on small-scale datasets.

Use T0 only for the first task of a setting.
Do not rerun T0 inside an unfinished sequence.
Back up a completed run before intentionally restarting it.

2. Per-stage workflow

For every row in the protocol tables below:

Edit TASK_NAME.
Set train_annotation_path and val_annotation_path exactly as listed.

CUDA_VISIBLE_DEVICES=0 python train.py

Evaluation

1. Select the test dataset

Set one COCO JSON and its image root at a time:

cocoGt_path      = "/path/to/DAUB/test.json"
dataset_img_path = "/path/to/DAUB/"

Run:

CUDA_VISIBLE_DEVICES=0 python test.py

For example, after I1-R1-U1, evaluate I1, R1, and U1 separately. This per-task history is required to calculate average mAP50, average F1, and forgetting.

Available test configurations are:

Alias	`cocoGt_path`	`dataset_img_path`
`I1`	`DAUB/test.json`	`DAUB/`
`I2`	`IRDST-M/test.json`	`IRDST-M/`
`I3`	`ITSDT/test.json`	`ITSDT/`
`R1`	`SAT-MTB/train_test.json`	`SAT-MTB/`
`R2`	`SAT-MTB/airplane_test.json`	`SAT-MTB/`
`R3`	`SAT-MTB/ship_test.json`	`SAT-MTB/`
`U1`	`RGBT-Tiny/test.json`	`RGBT-Tiny/`
`U2`	`VisDrone-2019-DET/coco_style/test.json`	`VisDrone-2019-DET/`
`U3`	`UAV-10/test.json`	`UAV-10/`

Single-Image Prediction

predict.py supports task-agnostic single-image prediction.

Run:

CUDA_VISIBLE_DEVICES=0 python predict.py

Parameters and FLOPs

python summary.py

The script reports trainable parameters and FLOPs for a 512 × 512 input.

Results

The following results are reported in the manuscript currently under review.

Main incremental settings

Method	Setting	Average mAP50 (%)	Average F1 (%)	RUBR (%)	FR (%)
MemCon	`I1-R1-U1`	56.13	73.29	79.88	16.13
MemCon	`I2-R2-U2`	55.60	72.14	82.62	19.72
MemCon	`I3-R3-U3`	61.17	77.48	86.47	2.09

RUBR denotes the relative upper-bound ratio based on mAP50, and FR denotes the forgetting rate based on mAP50.

In the main comparison, MemCon uses 3.70 M learnable parameters and approximately 1,022 MB GPU memory under the reported training setup.

Scaling to more tasks

Setting	Average mAP50 (%)	Average F1 (%)	Memory	FPS
`I1-R1-U1-I2-R2-U2`	47.96	67.69	1.75 MB	33.02
`I1-R1-U1-I2-R2-U2-I3-R3-U3`	55.32	72.68	2.50 MB	32.75

Contact

For questions about the code, benchmark splits, or access to the four datasets distributed by request, contact:

Shengjia Chen: csj_uestc@126.com

Navigation Menu

Repository navigation

About

Repository files navigation

MemCon: Memory Confluence for Cross-Imaging-Mode Small Target Detection

Overview

Contents

CIMSTD Benchmark

Environment

Dataset Access

Dataset Preparation

1. Task definitions

2. Training and validation text format

3. COCO JSON format for evaluation

4. Recommended directory organization

Incremental Learning Protocols

Cross-imaging-mode settings

Single-imaging-mode setting

Imaging-mode scale settings

Training

1. Default training configuration

2. Per-stage workflow

Evaluation

1. Select the test dataset

Single-Image Prediction

Parameters and FLOPs

Results

Main incremental settings

Scaling to more tasks

Contact

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
model_data		model_data
nets		nets
utils		utils
README.md		README.md
fig1.png		fig1.png
predict.py		predict.py
summary.py		summary.py
test.py		test.py
train.py		train.py

Latest commit

History

Folders and files

Repository files navigation

MemCon: Memory Confluence for Cross-Imaging-Mode Small Target Detection

Overview

Contents

CIMSTD Benchmark

Environment

Dataset Access

Dataset Preparation

1. Task definitions

2. Training and validation text format

3. COCO JSON format for evaluation

4. Recommended directory organization

Incremental Learning Protocols

Cross-imaging-mode settings

Single-imaging-mode setting

Imaging-mode scale settings

Training

1. Default training configuration

2. Per-stage workflow

Evaluation

1. Select the test dataset

Single-Image Prediction

Parameters and FLOPs

Results

Main incremental settings

Scaling to more tasks

Contact

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages