CLAS: Cross-Lingual Activation Steering

Official implementation of Cross-Lingual Activation Steering for Multilingual Language Models by Rhitabrat Pokharel, Ameeta Agrawal, and Tanay Nagar.

Paper · PDF

CLAS is a training-free, inference-time intervention for multilingual causal language models. It categorizes MLP neurons from parallel multilingual inputs, identifies a small set of bridge layers, and rebalances shared and language-specific activations without updating model weights.

Method

For the SwiGLU intermediate activation

h = activation(W_gate x) * (W_up x),

CLAS applies the following intervention only to non-anchor languages:

h_final = h * (1 - lambda * M_shared + alpha * M_specific)
lambda  = number_of_specific_neurons / number_of_shared_neurons

M_shared is the union of partial-shared and all-shared neurons. English is the anchor language and is never modified. The implementation follows Sections 2.3–2.5 of the paper: masks are built from the dataset-level mean activations of 100 parallel XQuAD examples in 12 languages.

Installation

Python 3.10+ and a CUDA-capable environment are recommended.

git clone https://github.com/PortNLP/CLAS.git
cd CLAS
python -m venv .venv
source .venv/bin/activate
pip install -e .

Llama 3.1 is gated on Hugging Face. Accept its model license and authenticate without placing a token in source code:

hf auth login

The datasets are downloaded through the Hugging Face datasets library and cached locally. No benchmark data is stored in this repository.

Quick start

Build or reuse the paper's XQuAD-derived masks and evaluate XNLI in one process:

clas run --model llama --task xnli

The equivalent Qwen run is:

clas run --model qwen --task xnli

The first invocation performs 1,200 forward passes (100 examples × 12 languages) to build masks and writes them to artifacts/<model>_xquad_masks.safetensors. Later runs reuse that artifact.

For a short pipeline check before a complete evaluation:

clas run --model qwen --task pawsx --limit 10

Reproducing the paper setup

Supported task names are xnli, xstorycloze, pawsx, harrypotter, and xquad.

# Build masks once.
clas build-masks --model llama

# Reuse them across tasks.
clas evaluate \
  --model llama \
  --task xquad \
  --masks artifacts/llama_xquad_masks.safetensors

Paper configurations are encoded in src/clas/config.py:

Model	Hugging Face checkpoint	Bridge layers
Llama	meta-llama/Llama-3.1-8B-Instruct	24–29
Qwen	Qwen/Qwen2.5-7B-Instruct	20–25

The task-level steering strengths used by the experiment code are:

Task	Llama α	Qwen α	Metric
XNLI	7.0	0.5	Accuracy
XStoryCloze	-1.0	-1.0	Accuracy
PAWS-X	-3.0	-1.0	Accuracy
HarryPotter Quiz	-3.0	1.0	Accuracy
XQuAD	-1.0	-1.0	Token F1

Use --alpha VALUE to run an ablation without changing the paper presets. Negative values suppress language-specific neurons; positive values amplify them.

Results are written to results/<task>/<model>/:

metrics.json
<language>_predictions.csv

Reported averages

These are the results reported in the paper; parenthesized values are improvements over the unmodified model.

Task	Llama base	Llama CLAS	Qwen base	Qwen CLAS
XNLI	42.42	45.03 (+2.61)	59.08	59.29 (+0.21)
XStoryCloze	83.38	85.58 (+2.20)	78.68	79.70 (+1.02)
PAWS-X	65.70	66.00 (+0.30)	78.58	79.07 (+0.49)
HarryPotter Quiz	51.59	52.59 (+1.00)	53.67	55.50 (+1.83)
XQuAD	26.85	32.10 (+5.25)	32.73	33.94 (+1.21)

Data sources

The evaluation command downloads the public benchmark releases directly:

• XNLI
• XStoryCloze
• PAWS-X
• HarryPotter Quiz
• XQuAD

Dataset downloads, model weights, mask artifacts, predictions, and result files are ignored by Git.

Repository layout

src/clas/
├── artifacts.py      # safe serialization for neuron masks
├── cli.py            # build-masks, evaluate, and run commands
├── config.py         # paper model/task presets
├── data.py           # benchmark downloads and prompts
├── evaluation.py     # accuracy and XQuAD F1 evaluation
├── modeling.py       # model loading and layer access
├── statistics.py     # dataset-level neuron categorization
└── steering.py       # CLAS inference-time intervention
tests/                # lightweight unit tests for the core method

Python API

from clas.artifacts import load_masks
from clas.config import MODEL_PRESETS
from clas.modeling import load_model_and_tokenizer
from clas.steering import inject_clas, set_clas_language

preset = MODEL_PRESETS["llama"]
model, tokenizer = load_model_and_tokenizer(preset)
masks, _ = load_masks("artifacts/llama_xquad_masks.safetensors")

inject_clas(model, masks, preset.bridge_layers, alpha=-1.0, anchor_language="en")
set_clas_language(model, "de")
# Calls to model(...) or model.generate(...) are now steered for German.
set_clas_language(model, None)

Always set the correct input language before inference. Passing None disables steering, and the anchor language follows the original model path exactly.

Tests

pip install -e '.[dev]'
ruff check .
pytest

The tests exercise neuron categorization, the intervention equation, anchor-language bypass, injection/removal, mask serialization, prompt formatting, and metrics without downloading models or datasets.

Citation

@misc{pokharel2026clas,
  title         = {Cross-Lingual Activation Steering for Multilingual Language Models},
  author        = {Rhitabrat Pokharel and Ameeta Agrawal and Tanay Nagar},
  year          = {2026},
  eprint        = {2601.16390},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2601.16390}
}