The less the better: Improving cross-family RNA secondary structure generalization with structural-aware subsampling

This repository contains the source code and data for reproducibility of “The less the better: Improving cross-family RNA secondary structure generalization with structural-aware subsampling,” by G. Kulemeyer, L.A. Bugnon, L. Di Persia, G. Stegmayer, D.H. Milone. Research Institute for Signals, Systems and Computational Intelligence,sinc(i).

In this work we propose three subsampling strategies in order to improve the generalization capability of deep learning models for RNA secondary structure prediction. A short summary of each strategy is presented in the table below.

Strategy	Description
sortS	Sorting and iterative removal of sequences based on the smallest pairwise distances within each RNA family.
clusS	Hierarchical clustering within each RNA family using a structural distance matrix, followed by medoid selection of representatives.
randS	Random subsampling within each RNA family, up to a maximum number of sequences.

Installation

These steps will guide you through the process of subsampling a secondary structure dataset.

First:

git clone https://github.com/gkulemeyer/LessIsBetter
cd LessIsBetter

With a conda working installation, run:

conda env create -f environment.yml

This should install all required dependencies. Then, activate the environment with:

conda activate less-is-better

Note: If the subsampling strategy requires a structural distance, (sortS or clusS) the strategy script will expect the distance matrix as a HDF5 file. We provide a script, distances.py, to compute the structural distance matrix, and the ArchiveII distance matrix located at data/ArchiveII_distances.tar.gz, which can be decompressed with the command

tar -xvzf data/ArchiveII_distances.tar.gz  

Dataset subsampling with sortS

To perform a structure-aware subsampling of a dataset using the sortS strategy, main.py executes the algorithm. This script loads a dataset from a CSV file with the columns id and fam, where fam indicates the RNA family and returns the desired subsampled dataset. To perform structural calculations, the script also requires the structural distance matrix as input.

For example, to perform structure-aware subsampling using sortS on the ArchiveII dataset (default) with an upper limit of 100 elements per family, run:

python3 main.py \ 
 --strategy  sortS \
 --dist-path data/ArchiveII_distances.h5 \ 
 --max-sequences 100

The output is saved as outputs/ArchiveII_sortS_100.csv and is formatted as the input file.

Dataset subsampling with clusS

If the selected structure-aware subsampling method is the clusS, the main.py script performs this algorithm and returns the desired subsampled dataset.

For example, to perform subsampling using clusS on the ArchiveII dataset (default) with an upper limit of 100 elements per family, run:

python3 main.py \ 
 --strategy  clusS \
 --dist-path data/ArchiveII_distances.h5 \ 
 --max-sequences 100

The output is saved as outputs/ArchiveII_clusS_100.csv and is formatted as the input file.

Dataset subsampling with randS

In case the desired subsampling strategy is randS, the distance matrix (and the --dist-path option of main.py) is not required.

For example, to perform a randS on the ArchiveII dataset (default) with an upper limit of 100 elements per family, run:

python3 main.py --strategy randS --max-sequences 100

The output is saved as outputs/ArchiveII_randS_100.csv and is formatted as the input file.

In the main.py script, the --file-path option allows to apply the desired subsampling strategy on a custom dataset with required columns id and fam . The output location can be specified with the --save-path option, and the general save pattern is

<save-path>/<input_file>_<strategy>_<max_sequences>.csv

Compute structural distances

To compute the all-vs-all structural distance matrix, the input dataset must contain the fields id and structure, where structure is given in dot bracket format.

To compute the structural distance matrix, run:

python3 distances.py \
        --file-path data/ArchiveII.csv \
        --save-path data

and returns the output data/ArchiveII_distances.h5. If the --save-path is not explicitly provided, the parent directory of --file-path is used by default. WARNING! Computing the all-vs-all structural distance matrix may take a long time.

Reproducibility

The notebooks/ folder contains the code required to reproduce the figures presented in the article.

• Data distribution: Figure 1A shows the family distribution in the dataset and the balance.

• Distance distributions: the notebook Figure 1B contains the changes in the intra-family minimum structural distance produced by each subsampling strategy. Figures 1C and 1D present the mean and minimum inter- and intra-family distances, and also the impact of the different strategies on the train-to-test distance matrices relative to the full dataset. The intra-family distance matrices at a sequence level and the impact of subsampling are presented in Figure 2.

• Impact on training: For the different models trained, the Figure 3 presents the training and validation losses per epoch, and also the F1 score obtained with the validation set and with the family held-out for the different models, and datasets used to train the sincFold model.

• Impact on generalization: The test F1 scores obtained from each model/ strategy/ threshold per family, and a heatmap comparing the performance of each strategy across models and the full dataset is presented in Figure 4 . Also, this interactive notebook allows a better comparison between the baseline models and each strategy employed.