Explainable AI for Credit Risk Scoring

This project presents a self-explainable AI model for assessing the risk of loan applications. The system leverages a high-performing TabNet model for the initial risk classification and a fine-tuned Mistral-7B language model to generate human-readable explanations for each decision.

Table of Contents

• Project Overview
• Model Performance
• System Architecture
• Installation
• Usage
• Training Details

Project Overview

The goal of this project is to create a transparent and interpretable credit scoring system. While traditional models can provide a risk score, they often lack the ability to explain why a decision was made. This solution addresses that gap by:

1. Predicting Risk: A TabNet model, a deep learning architecture designed for tabular data, is trained on applicant data to predict the probability of loan default.
2. Generating Explanations: A fine-tuned version of the Mistral-7B language model takes the TabNet model's outputs (including feature attributions) and generates a narrative explanation, detailing the key factors that influenced the final decision.

Model Performance

The core TabNet classification model achieved the following performance on the test set:

Metric	Score
AUC-ROC	0.7425
AUPRC	0.2225
F1-Score	0.2543
Precision	0.1563
Recall	0.6824

System Architecture

The final system is composed of two main components:

• TabNet Classifier: A pre-trained model saved as tabnet_model.zip.
• LLM Adapter: A Low-Rank Adaptation (LoRA) of mistralai/Mistral-7B-v0.1, which is fine-tuned to translate model feature importance into natural language.

Installation

1. Fork and Clone the repository:

   git clone [https://github.com/YOUR_USERNAME/explainable-loan-risk-model.git](https://github.com/YOUR_USERNAME/explainable-loan-risk-model.git)
   cd explainable-loan-risk-model

2. Install Git LFS to pull the large model files:

   git lfs install
   git lfs pull

3. Install the required Python libraries:

   pip install torch numpy pandas scikit-learn pytorch-tabnet transformers peft bitsandbytes accelerate

Usage

The ExplainableCreditScorer class provides a simple interface to get a prediction and an explanation from raw applicant data.

import numpy as np
import json
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
from pytorch_tabnet.tab_model import TabNetClassifier
import torch

# NOTE: The generate_llm_payload function needs to be defined here or imported.

class ExplainableCreditScorer:
    def __init__(self, tabnet_path, llm_adapter_path, feature_names, base_llm_name="mistralai/Mistral-7B-v0.1"):
        """
        Initializes the scorer with trained models and feature names.

        Args:
            tabnet_path (str): Path to the saved TabNet model .zip file.
            llm_adapter_path (str): Path to the folder containing the fine-tuned LLM adapter.
            feature_names (list): The ordered list of feature names used during training.
            base_llm_name (str): The name of the base LLM on Hugging Face.
        """
        self.feature_names = feature_names

        self.tabnet_model = TabNetClassifier()
        self.tabnet_model.load_model(tabnet_path)

        bnb_config = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_quant_type="nf4",
            bnb_4bit_compute_dtype=torch.bfloat16,
        )

        base_model = AutoModelForCausalLM.from_pretrained(
            base_llm_name,
            quantization_config=bnb_config,
            device_map="auto"
        )

        self.llm_model = PeftModel.from_pretrained(base_model, llm_adapter_path)
        self.llm_tokenizer = AutoTokenizer.from_pretrained(base_llm_name)
        self.llm_tokenizer.pad_token = self.llm_tokenizer.eos_token

    def predict_and_explain(self, applicant_dict, applicant_id="new_applicant"):
        """
        Takes a raw applicant data dictionary, predicts, and explains the decision.
        """
        ordered_values = [applicant_dict[feature] for feature in self.feature_names]
        applicant_data_processed = np.array(ordered_values).reshape(1, -1)

        payload = generate_llm_payload(
            self.tabnet_model,
            applicant_data_processed,
            self.feature_names,
            applicant_id
        )

        instruction = f"Generate a credit risk explanation based on the following data: {json.dumps(payload)}"
        prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"

        inputs = self.llm_tokenizer(prompt, return_tensors="pt").to("cuda")
        outputs = self.llm_model.generate(**inputs, max_new_tokens=256, temperature=0.7)

        full_text = self.llm_tokenizer.decode(outputs[0], skip_special_tokens=True)
        explanation = full_text[len(prompt):].strip()

        return {
            "applicant_id": applicant_id,
            "decision": payload["prediction"]["decision"],
            "probability": payload["prediction"]["probability"],
            "explanation": explanation
        }

# ==============================================================================
#                               EXAMPLE USAGE
# ==============================================================================

# Assume 'features' is a list of feature names in the correct order
# and 'new_applicant_data' is a dictionary with feature names as keys.

# Example placeholder:
features = ['Ext Source 3', 'Ext Source 1', 'Amt Annuity', 'Name Family Status', 'Ext Source 2'] # Add all your features
new_applicant_data = {
    'Ext Source 3': 0.7295,
    'Ext Source 1': 0.7472,
    'Amt Annuity': 40914.0,
    'Name Family Status': 1.0,
    'Ext Source 2': 0.6052
    # Add all other features with their values
}


# Instantiate the scorer with the correct paths and feature list
scorer = ExplainableCreditScorer(
     tabnet_path='models/tabnet_model.zip',
     llm_adapter_path='models/mistral-lora-adapter',
     feature_names=features
 )

result = scorer.predict_and_explain(new_applicant_data)
print(json.dumps(result, indent=2))

{
  "applicant_id": "new_applicant",
  "decision": "Low Risk (Accept)",
  "probability": 0.8440883159637451,
  "explanation": "The model's recommendation for new applicant is to **Risk** the loan application, with a confidence of 84.41%. This decision of **Low Risk** was primarily influenced by the following factors:\n\n- **Ext Source 3:** A value of `0.7295666907060153` was a significant contributor to the decision.\n- **Ext Source 1:** A value of `0.7472770211998799` was a significant contributor to the decision.\n- **Amt Annuity:** A value of `40914.0` was a significant contributor to the decision.\n- **Name Family Status:** A value of `1.0` was a significant contributor to the decision.\n- **Ext Source 2:** A value of `0.605239221259936` was a significant contributor to the decision.\n\n### Explanation:\nThe model's recommendation of **Low Risk** was largely influenced by the following factors: a high value of `Ext Source 3`, a high value of..."
}

Training Details

The models were trained as follows:

• TabNet Model: The classifier was initially trained with a baseline configuration and then fine-tuned using Optuna for hyperparameter optimization across parameters like n_d, n_a, n_steps, gamma, and lambda_sparse.
• LLM Fine-Tuning: The Mistral-7B model was fine-tuned using a Low-Rank Adaptation (LoRA) approach on a dataset of JSON payloads mapping model predictions and feature attributions to human-readable text. The SFTTrainer from the TRL library was used for this process.

For more details, please refer to the Jupyter Notebook in the /notebooks directory.