- 2025-07-15: first submission
- 2025-07-15: pre-review
- 2025-07-15: first rework
- 2025-07-15: second submission
- 2025-07-15: expert review
- 2025-07-16: confirmed acceptance
- 2025-09-03: second rework
- 2025-09-04: final submission
- 2025-12-22: published (🌹congratulations!!!)
- 2026-01-23: EI index updated (🌹10.1145/3773365.3773437)
Material Spectral RCNN: A Multispectral Mask R-CNN for Joint Object Detection and Material Recognition.
This paper proposes a novel multispectral object detection and material recognition network, MatSpecRCNN, which is based on Mask R-CNN and customized for the characteristics of multispectral images. The network backbone uses nine-channel multispectral image as input, and adopts spectral attention and spatial attention mechanism during the feature extraction stage to enhance the representation of spectral and spatial features. Based on the detection and segmentation branches, we designed a material recognition branch that accurately classifies target materials by using roi-pool, head and predictor within the bounding-box prediction results. Experimental results show that our network achieves high-precision in object detection and material recognition on custom multispectral datasets, providing an effective tool for intelligent analysis of multispectral images.
Figure 1. The overall framework of MatSpecRCNN. Compared to Mask R-CNN, MatSpecRCNN replaces the Transform with a fixed Normalizer, modifies the Backbone, and replaces the Keypoint Branch with a new Material Branch.
Figure 2. The structure of Material Head. The processing flow can be divided into three stages: Spatial Attention, Spectral Attention, and Feature Projection. By implementing a Reduce prior to the Multilayer Perceptron (MLP) projection, we achieve a 97% reduction in parameters while maintaining representational capacity.
Figure 3. Combined illustration of our dataset workflow: (a) Imaging System Hardware，RGB camera captures horizontally flipped image; (b) Annotation Example, an “Air Write” with many “Delete”; (c) Affine Example, left shows the RGB image keypoints, right shows the corresponding in grayscale, lines connect the matched keypoints.
Figure 4. A classic example of MatSpecRCNN. Left shows the RGB image of input multispectral data, right shows the predictions of bounding box, object region, object classes with confidence and object material types with confidence.
The main.py script accepts the following command‑line arguments:
# position argument:
mode {train, infer, show} # "train": run training;
# "infer": perform inference;
# "show": print model summary only
# if running in `show` mode, no other args required
# if running in `train` mode, args list:
-e, --epoch <int> Number of epoch, default: 10
-b, --batch <int> Batch size, default: 8
-l, --load <str> A weights filename, will be load before train if not None, default: None
-s, --save <str> The weights filename for save action after train, default: mat-spec-rcnn
# if running in `infer` mode, args list:
-l, --load <str> A weights filename, will be load before infer, required
-b, --batch <int> Batch size, default: 8
# run in NVIDIA Tesla T4
cd /path/to/MatSpecRCNN
# warm-up train
python -u main.py train --epoch 10 --batch 8 --save warm-up > logs/warm-up/train.log 2>&1 &
# fine-turn train
python -u main.py train --epoch 20 --batch 8 --load warm-up --save self-full > logs/self-full/train.log 2>&1 &
# show model
python main.py show > logs/self-full/model.txt
# inference (run in RTX 3060)
python -u main.py infer --load self-full --batch 4
# inference (run in RTX 3060)
python -u main.py track --load self-full --batch 1 --len 200