Simon Roschmann, Quentin Bouniot, Vasilii Feofanov, Ievgen Redko, Zeynep Akata
Adapting vision models for time series analysis is compelling, yet all existing approaches are falling short of dedicated time series foundation models (TSFMs) in classification. In this work, we propose Time Vision Transformer (TiViT), the first framework that successfully unlocks the representational power of frozen Vision Transformers (ViTs) pretrained on large-scale image datasets for time series classification. TiViT achieves state-of-the-art performance without any finetuning by utilizing the hidden representations of OpenCLIP models. We explore the structure of TiViT representations and find that intermediate ViT layers with high intrinsic dimension are the most effective for time series classification. Furthermore, we assess the alignment between TiViT and TSFM representation spaces and identify a strong complementarity, with additional performance gains achieved through feature concatenation. Finally, we unfreeze the ViT backbone of TiViT for continual pretraining and contrastive alignment with TSFMs on time series, enhancing the performance of lightweight TiViT variants. Our findings reveal a new direction for the domain and task adaptation of vision foundation models.
Illustration of TiViT on a time series sample from ECG200. We split the time series into segments and stack them to form a grayscale image. Then, we patch the image in 2D and feed it into a frozen ViT pretrained on large-scale image datasets. We average the representations from a specific hidden layer and pass them to a learnable classification head. Concatenating the representations of TiViT and TSFMs prior to classification further improves accuracy.
This repository works with Python 3.11 and PyTorch 2.7. Please create a conda environment and install the dependencies specified in requirements.txt.
conda create -n tivit_env python=3.11
conda activate tivit_env
python -m pip install -r requirements.txtWe evaluate TiViT on the UCR and UEA benchmark for time series classification. The benchmark datasets can be loaded via the aeon toolkit. For UCR datasets, we apply by default linear interpolation to handle missing values and use padding to accommodate unequal time series lengths. To use the preprocessing functionality provided by Aeon instead, add the --aeon flag.
In our study, we evaluate TiViT with various ViT backbones, as summarized in the following table. These ViTs cover a wide range of pretraining paradigms and model sizes in the vision and vision-language domain.
| Model | Paper | Checkpoints |
|---|---|---|
| CLIP | Radford et al., 2021; Cherti et al., 2022 | ViT-B-16, ViT-B-32, ViT-L-14, ViT-H-14 |
| SigLIP 2 | Tschannen et al., 2025 | SoViT-400m-14 |
| DINOv2 | Oquab et al., 2023 | ViT-S-14, ViT-B-14, ViT-L-14 |
| MAE | He et al., 2021 | ViT-B-16, ViT-L-14, ViT-H-14 |
We compare and optionally fuse TiViT with two state-of-the-art TSFMs exclusively pretrained on time series.
- Mantis (Paper, Checkpoint): Transformer model (8 M parameters) pretrained on 2 million time series with contrastive learning
- MOMENT (Paper, Checkpoint): Transformer models pretrained on 13 million time series with masked modeling. There exist three different sizes: Small (40 M parameters), Base (125 M parameters), Large (385 M parameters).
Below is an example command to perform classification with TiViT on the UCR benchmark using hidden representations from layer 14 of an OpenCLIP ViT-H model:
python main.py --vit_1_name laion/CLIP-ViT-H-14-laion2B-s32B-b79K --vit_1_layer 14 --aggregation mean --patch_size sqrt --stride 0.1 --classifier_type logistic_regression --datasets ucr --data_dir /path/to/your/data --result_dir /path/to/save/results --random_seed 2025Check out arguments.py for a comprehensive overview of all configurable parameters.
To further improve classification accuracy, we propose to concatenate the representations of TiViT and traditional TSFMs by adding:
--mantis--moment [small | base | large]
The following table summarizes the linear classification accuracy of TiViT and TSFMs on the UCR and UEA benchmark.
| Model | UCR | UEA |
|---|---|---|
| Moment | 79.0 | 69.9 |
| Mantis | 80.1 | 72.4 |
| TiViT (Ours) | 81.3 | 72.0 |
| TiViT + Moment (Ours) | 82.5 | 72.6 |
| TiViT + Mantis (Ours) | 83.0 | 73.7 |
To better understand the hidden representations of ViTs, we analyze how their structure evolves across layers. Use the following command to compute the intrinsic dimension of representations from CLIP ViT-H:
python main.py --vit_1_name laion/CLIP-ViT-H-14-laion2B-s32B-b79K --aggregation mean --patch_size sqrt --stride 0.1 --get_intrinsic_dimension --datasets ucr --data_dir /path/to/your/data --result_dir /path/to/save/results --random_seed 2025Alternatively, you may compute the number of principal components necessary to cover 95% of the variance in the representations by setting the flag --get_principal_components.
The alignment of representations from CLIP ViT-H (layer 14) and Mantis on the UCR benchmark can be measured using the mutual k-NN metric.
python main.py --vit_1_name laion/CLIP-ViT-H-14-laion2B-s32B-b79K --vit_1_layer 14 --aggregation mean --patch_size sqrt --stride 0.1 --mantis --measure_alignment --datasets ucr --data_dir /path/to/your/data --result_dir /path/to/save/results --random_seed 2025You may specify different ViTs (--vit_1_name, --vit_2_name) and/or TSFMs (--mantis, --moment). Note that the alignment score can only be computed between two models at a time.
We appreciate the following repositories for their valuable code:
- The Platonic Representation Hypothesis: for assessing model alignment via the mutual k-NN metric
- DADApy: for computing the intrinsic dimension of representations
If you find TiViT useful, please star this repository and cite our work:
@article{roschmann2026tivit,
title={Unlocking Pretrained Vision Transformers for Time Series Classification},
author={Simon Roschmann and Quentin Bouniot and Vasilii Feofanov and Ievgen Redko and Zeynep Akata},
journal={arXiv preprint arXiv:2506.08641},
year={2026}
}If you have any questions, feel free to contact us: simon.roschmann@tum.de