giskard

Practical Jupyter notebooks from Andrew Ng and Giskard team's "Red Teaming LLM Applications" course on DeepLearning.AI.

✍️ Collaborate on writing technical content for the Giskard Community

Adversarial LLM red-teaming with Giskard: automated vulnerability scans of DeepSeek-R1, GPT-4o-mini & Llama 3.2 (prompt injection, hallucination, harmful output) plus tabular ML scanning, with per-model HTML reports

Mutant is a red team and behavioral data generation library for LLMs, RAG pipelines, and AI Agents. It provides two powerful capabilities

RAG experiments with LangChain: PDF Q&A over OpenAI and Perplexity models, evaluated with Giskard

A RAG system to answer questions about flights

Develop a system to generate diverse types of subjective questions from PDFs and automate the grading of answers.

Independent SR 11-7-style model validation of an ML trading strategy - deepchecks + giskard, formal report, reproducible. Verdict: not approved for deployment.

LLM evaluation & AI safety suite for a live chatbot — custom LLM-as-a-Judge metrics, jailbreak/prompt-injection red teaming, RAG groundedness (Ragas, Giskard), Langfuse tracing, plus Playwright E2E and k6, gated nightly in CI.

Multi-agent LLM-driven SOC pipeline (n8n + Ollama), adversarially red-teamed against the CSA Agentic AI Red Teaming Guide.

PRML pre-registration for Giskard scenario results: commit an eval claim to a SHA-256 before the run, then verify the result against it.

The complete pipeline for a fine-tuning of a classifier model.

Different approaches to evaluate RAG !!!

Security testing toolkit for AI agents

Red-team assessment of a financial-services chatbot and an e-commerce agent: manual probing, Giskard scanning, and adversarial simulation. 16 findings mapped to the OWASP Top 10 for LLMs, with an audit trail.