Skip to content

About

Local-first UI quality assurance platform using Playwright, Tesseract OCR, and on-device Vision LLMs for privacy-centric visual testing.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

33 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

🔍 QA Vision

Local-First, Privacy-Centric Hybrid UI Quality Assurance Automation Platform

Ingest specs. Extract requirements. Validate visually. Report with confidence.


TypeScript Next.js Playwright Ollama Tesseract Tailwind CSS License i18n


QA Vision Dashboard
QA Vision Dashboard — Reports gallery, New Test modal, and pipeline status at a glance.

Overview

QA Vision is a fully local, privacy-first UI quality assurance platform that bridges the gap between human-readable software requirement specifications and automated visual testing. Zero data leaves your machine — every model call, every screenshot, every validation result is processed entirely on-premises.

Unlike cloud-dependent testing suites, QA Vision operates on a dual-engine validation architecture:

  • Tesseract OCR + DOM BBox mapping for text-based elements (button, text, link)
  • Local Vision LLM via Ollama (LLaVA or compatible) for non-text elements (icon, image, input, search) that OCR cannot detect

The pipeline is orchestrated by a Next.js 16 App Router dashboard that communicates with a standalone TypeScript agent through a secure file-based IPC mechanism — no shared memory, no sockets, no race conditions.


Core Features

📄 Omni-Document Ingestion

Accept requirements in any format your team already uses:

  • PDF — parsed with unpdf (Edge-compatible, zero native dependencies)
  • DOCX — parsed with mammoth (server-external to avoid Turbopack bundling issues)
  • TXT / Markdown — read directly as UTF-8 via Node.js fs
  • Files are securely streamed to the backend, never stored in browser memory

🤖 Strict AI Requirement Extraction

A locally running Ollama LLM (llama3.1) acts as a deterministic requirements parser:

  • Ignores architectural prose, philosophy text, and abstract layout descriptions
  • Extracts only verifiable, discrete UI nodes: button, text, link, input, icon, image
  • Enforces a max 6-word description field — the exact label visible on screen
  • Outputs a strongly-typed JSON array consumed directly by the validation engine

⚡ Dual-Engine Validation

The validation loop routes each requirement to the correct engine automatically:

Requirement Type Engine Method
text, button, link Tesseract OCR + DOM BBox mapping → full-page OCR fallback
image, icon, input, search Vision LLM (LLaVA) Natural language YES/NO visual query

🌐 Hybrid Execution Modes

  • URL Mode — Fires a headless Chromium instance (Playwright), navigates to the target, waits for all images to load, captures a pixel-perfect 1920×1080 screenshot, and extracts live DOM bounding boxes
  • Image Mode — Skips the browser layer entirely; loads a locally uploaded static mockup or screenshot directly for validation

🖼️ Advanced Computer Vision Preprocessing

Before OCR runs, every screenshot is processed by Jimp:

  • Grayscale conversion and binarization for maximum text contrast
  • Per-pixel RGB channel dominance analysis for color verification
  • Bounding box annotation with red rectangles saved as embedded Base64 artifacts in the HTML report

📊 Structured HTML Reports

Every test run produces a self-contained HTML report in agent/docs/ with:

  • Pass/Fail summary badges with counts
  • Per-requirement cards showing description, type, expected color & location
  • Engine badge (🔤 OCR vs 👁 Vision LLM) on every card
  • Embedded annotated screenshot artifacts (Base64, no external file references)
  • Auto-deletion of temp files after each test run

🌍 Full i18n Support

The dashboard UI is fully translated in English, Turkish, and German via next-intl. Language preference is persisted in a cookie for one year. Switch languages from the sidebar at any time — no page reload required.


Architecture & Pipeline

┌─────────────────────────────────────────────────────────────────────┐
│                        QA Vision Pipeline                           │
└─────────────────────────────────────────────────────────────────────┘

  User uploads SRS document  (PDF / DOCX / TXT / MD)
          │
          ▼
┌─────────────────────┐
│  STAGE 01           │  Next.js API Route (dashboard/app/api/run)
│  DOCUMENT INGESTION │  unpdf / mammoth / fs.readFileSync
│                     │  → raw UTF-8 string
│                     │  → written to agent/temp/reqs_[ts].txt  (IPC)
└────────┬────────────┘
         │
         ▼
┌─────────────────────┐
│  STAGE 02           │  agent/src/ai/extractor.ts
│  AI EXTRACTION      │  Ollama llama3.1 (local LLM)
│                     │  → Strict JSON array of typed UI requirements
│                     │  [{ id, type, description, expected_color, location }]
└────────┬────────────┘
         │
         ▼
┌─────────────────────┐
│  STAGE 03           │  agent/src/scraper/browser.ts
│  HYBRID EXECUTION   │  ─── URL Mode ───────────────────────────────
│                     │  Playwright Headless Chromium (1920×1080)
│                     │  waitForFunction(images.every(i => i.complete))
│                     │  DOM BBox extraction → page.evaluate()
│                     │  Full-page screenshot
│                     │
│                     │  ─── Image Mode ─────────────────────────────
│                     │  fs.readFileSync(imagePath) — no browser
│                     │  Empty textBboxes array passed downstream
└────────┬────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────────────┐
│  STAGE 04 — DUAL-ENGINE VALIDATION  (agent/src/vision/analyzer.ts) │
│                                                                     │
│  for each requirement:                                              │
│                                                                     │
│    if type ∈ { text, button, link }                                 │
│      → Primary:  DOM textBboxes lookup (exact / substring match)    │
│      → Fallback: Tesseract OCR PSM-11 (sparse text mode)           │
│      → Color:    Jimp per-pixel RGB channel dominance scan          │
│      → Location: Bounding-box centre vs PAGE_WIDTH/HEIGHT thirds    │
│                                                                     │
│    if type ∈ { image, icon, input, search }                         │
│      → POST http://127.0.0.1:11434/api/generate (Ollama LLaVA)     │
│      → Prompt: "Is '{description}' visible? Answer YES or NO."     │
│      → Graceful fallback if Ollama unreachable (ECONNREFUSED)       │
└────────┬────────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────┐
│  STAGE 05           │  agent/src/reporter.ts
│  IPC & REPORTING    │  Generates standalone HTML report
│                     │  Embeds Base64 annotated artifacts
│                     │  Saves to agent/docs/[slug]_[date]_[hash].html
│                     │  Cleans agent/temp/ directory automatically
└─────────────────────┘
         │
         ▼
  Dashboard polls /api/reports → renders report grid
  User clicks card → iframe viewer inside dashboard

Tech Stack

Frontend Dashboard

Package Version Purpose
Next.js 16.2 App Router framework
React 19.2 UI rendering
Tailwind CSS v4 Utility-first styling
next-intl latest i18n (EN / TR / DE)
unpdf ^1.6.2 Server-side PDF text extraction
mammoth ^1.12.0 Server-side DOCX text extraction

Core Agent

Package Version Purpose
TypeScript ^6.0.3 Strict type safety
ts-node ^10.9.2 Direct TS execution (no compile step)
ollama ^0.6.3 Ollama SDK for LLM extraction
Node.js http built-in Raw HTTP client for Vision LLM calls

Computer Vision & Validation

Package Version Purpose
Playwright ^1.60.0 Headless Chromium, DOM BBox, screenshots
Tesseract.js ^7.0.0 OCR — PSM 11 sparse text mode
Jimp ^0.22.10 Image preprocessing, RGB pixel scanning, artifact annotation

Local AI

Model Engine Role
llama3.1 Ollama SRS document → strict JSON requirement extraction
llava Ollama Screenshot → visual element verification (Vision LLM)

Prerequisites

Ensure the following are installed on your system before proceeding:

Node.js        >= 20.x     https://nodejs.org
Ollama         latest      https://ollama.com

Pull the required Ollama models:

ollama pull llama3.1
ollama pull llava

Note: llava is the default Vision LLM. You can switch to moondream or llava:13b by editing OLLAMA_VISION_MODEL at line 17 of agent/src/vision/analyzer.ts.


Installation

1. Clone the repository

git clone https://github.com/your-username/qa-vision.git
cd qa-vision

2. Install Dashboard dependencies

cd dashboard
npm install

3. Install Agent dependencies

cd ../agent
npm install

4. Install Playwright browsers

cd agent
npx playwright install chromium

Usage

Start the development server

cd dashboard
npm run dev

The dashboard will be available at http://localhost:3000.

Run a test from the dashboard

  1. Open the dashboard and click New Test in the sidebar
  2. Paste your requirements or upload an SRS document (PDF, DOCX, TXT, MD)
  3. Select Test via URL (enter a live URL) or Test via Image (upload a screenshot)
  4. Click Run QA Test — the pipeline executes in the background
  5. When complete, the new report appears instantly in the Reports gallery
  6. Click any report card to open the full inline viewer, or open it in a new tab

Run the agent directly from the CLI

cd agent

# URL mode
npx ts-node src/index.ts --url="https://example.com" --docFile="./temp/reqs.txt"

# Image mode
npx ts-node src/index.ts --image="./temp/upload_screenshot.png" --docFile="./temp/reqs.txt"

CLI flags:

Flag Required Description
--url One of url/image Target web URL to test
--image One of url/image Path to local screenshot or mockup
--docFile ✅ Always Path to the UTF-8 requirements text file

When both --url and --image are provided, --url takes priority.


Project Structure

qa-vision/
│
├── agent/                          # Standalone TypeScript QA engine
│   ├── src/
│   │   ├── index.ts                # CLI entry point; orchestrates both pipelines
│   │   ├── reporter.ts             # HTML report generator (Base64 artifacts, IPC cleanup)
│   │   │
│   │   ├── ai/
│   │   │   └── extractor.ts        # Ollama LLM client; SRS → typed Requirement[]
│   │   │
│   │   ├── scraper/
│   │   │   └── browser.ts          # Playwright headless capture; DOM BBox extraction
│   │   │
│   │   └── vision/
│   │       └── analyzer.ts         # Dual-engine validator (Tesseract OCR + Vision LLM)
│   │
│   ├── temp/                       # IPC staging area — auto-cleaned after each run
│   │   ├── reqs_[timestamp].txt    # Requirements file written by dashboard API
│   │   └── upload_[timestamp].png  # Uploaded screenshot (image mode)
│   │
│   ├── docs/                       # Generated HTML reports (persisted)
│   │   └── [domain]_[date]_[hash].html
│   │
│   ├── output/                     # Annotated bbox artifacts (embedded → deleted)
│   ├── eng.traineddata             # Tesseract English language data
│   ├── tur.traineddata             # Tesseract Turkish language data
│   ├── package.json
│   └── tsconfig.json
│
└── dashboard/                      # Next.js 16 App Router frontend
    ├── app/
    │   ├── layout.tsx              # Root layout with NextIntlClientProvider
    │   ├── page.tsx                # Dashboard — New Test hero + stats bar
    │   │
    │   ├── reports/
    │   │   └── page.tsx            # Reports gallery (filter, sort, search, delete)
    │   │
    │   ├── how-it-works/
    │   │   └── page.tsx            # Pipeline architecture documentation page
    │   │
    │   └── api/
    │       ├── run/route.ts        # POST — triggers agent; handles FormData + IPC write
    │       ├── reports/route.ts    # GET — lists reports; DELETE — removes a report
    │       ├── reports/html/       # GET — serves raw HTML report files
    │       └── locale/route.ts     # POST — sets locale cookie for i18n
    │
    ├── components/
    │   ├── Sidebar.tsx             # Shared navigation sidebar (useTranslations, LocaleSwitcher)
    │   └── LocaleSwitcher.tsx      # EN · TR · DE language toggle (cookie-based)
    │
    ├── i18n/
    │   └── request.ts              # next-intl server config (reads locale from cookie)
    │
    ├── messages/
    │   ├── en.json                 # English translations
    │   ├── tr.json                 # Turkish translations
    │   └── de.json                 # German translations
    │
    ├── next.config.ts              # withNextIntl plugin + serverExternalPackages
    └── package.json

How the IPC Works

The Next.js dashboard and the TypeScript agent run as two completely separate processes. They communicate exclusively through the filesystem:

Dashboard (Next.js API Route)              Agent (ts-node CLI)
──────────────────────────────            ──────────────────────
1. Parse uploaded document                2. Agent starts
2. Write UTF-8 text to                 ┌──► 3. Reads reqs from file
   agent/temp/reqs_[ts].txt  ──────────┘    4. Runs pipeline
3. Spawn:                                   5. Writes HTML to
   npx ts-node src/index.ts                    agent/docs/[name].html
   --url="..." --docFile="..."          ◄── 6. Process exits (code 0)
4. await execPromise(command)
5. On success: clean agent/temp/
6. Return { success: true } to UI

This design guarantees:

  • No shared state between web server and test runner
  • No encoding issues with Unicode/Turkish characters in requirements
  • The agent can be invoked independently from the CLI without the dashboard

Report Filename Convention

{domain-slug}_{YYYY-MM-DD}_{HH-MM-SS}_{8-char-hash}.html

Examples:
  www-google-com_2026-06-02_14-32-10_a3f8b210.html   ← URL test
  upload-1780386783050_2026-06-02_10-53-59_8dc07160.html ← Image test

Reports are parsed back by the dashboard API to extract domain, timestamp, source (url/image), and id (hash) for the gallery view.


Roadmap

Feature Status
Omni-document ingestion (PDF, DOCX, TXT, MD) ✅ Shipped
Strict AI requirement extraction (LLM) ✅ Shipped
Tesseract OCR + DOM BBox validation ✅ Shipped
Playwright headless URL mode ✅ Shipped
Static image upload mode ✅ Shipped
Vision LLM routing (LLaVA) ✅ Shipped
HTML report with Base64 artifacts ✅ Shipped
Next.js dashboard + report gallery ✅ Shipped
File-based IPC + temp auto-cleanup ✅ Shipped
i18n (English / Turkish / German) ✅ Shipped
Report deletion from UI ✅ Shipped
Multi-step sequence testing 🔜 Planned
AI-powered CSS fix suggestions 🔜 Planned
Scheduled / CI-triggered runs 🔜 Planned
Parallel requirement validation 🔜 Planned
Report diffing (run comparison) 🔜 Planned

Contributing

  1. Fork the repository and create a feature branch: git checkout -b feat/your-feature
  2. All code, variable names, comments, and commit messages must be written in strict English
  3. Run npm run lint in dashboard/ before submitting
  4. TypeScript strict mode is enforced in both agent/ and dashboard/ — no any without justification
  5. Open a pull request with a clear description of the change and its motivation

License

Distributed under the ISC License. See LICENSE for more information.


Built with precision, privacy, and purpose.

About

Local-first UI quality assurance platform using Playwright, Tesseract OCR, and on-device Vision LLMs for privacy-centric visual testing.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages