Academic and Engineering Portfolio Project for the Russian Government Scholarship (Open Doors / Россотрудничество)
Field of Study: Computer Science, Artificial Intelligence, and Multimodal Natural Language Processing
Author: Adel Mohamed
Live Deployment: adel-dev-2001.github.io/omniscribe
Repository: github.com/Adel-dev-2001/omniscribe
License: MIT License
Release: v2.4.1
- Executive Summary and Research Motivation
- Candidate Academic Note and Learning Context
- Russian Academic Abstract (Аннотация)
- System Architecture and Data Pipeline
- Forward-Compatible Dynamic Model Discovery
- Multilingual Typography and High-DPI PDF Compilation
- Technical Comparison: Multimodal Vision vs. Classical OCR
- Project Directory Structure
- Security and Client-Side Architecture
- Local Development, Testing, and Deployment
- Release History
OmniScribe is a client-side multimodal document intelligence application engineered to address character degradation and semantic loss in traditional Optical Character Recognition (OCR) workflows.
Conventional OCR engines (such as Tesseract and legacy CNN-based pipelines) rely on isolated glyph segmentation. Consequently, they frequently fail when processing multi-script documents combining Latin, Cyrillic, and right-to-left Arabic scripts, and they systematically strip or corrupt Unicode emojis, Zero-Width Joiner (ZWJ) sequences, mathematical notation, and tabular layouts.
OmniScribe replaces brittle glyph segmentation with direct multimodal vision inference over the Google Gemini REST API:
- Unicode and Emoji Preservation: Captures standard scripts alongside complex emoji sequences, regional indicator flags, skin-tone modifiers, and technical symbols directly into canonical Unicode code points.
- Dynamic Free-Tier Model Discovery: Queries Google's live
/v1beta/modelsendpoint at runtime, filters for active Free-Tier multimodal Flash models, and binds to the newest available generation without hardcoded model IDs. - Live Failover Cascade: Monitors HTTP response codes during batch inference and transitions across verified free-tier models when per-minute rate limits (
HTTP 429) or endpoint deprecations occur. - Consolidated Batch Exports: Processes up to 20 images per batch (via drag-and-drop, file selection, or
Ctrl+Vclipboard capture) and compiles results into Plain Text (.txt), high-DPI multilingual PDF (.pdf), Markdown (.md), or structured JSON (.json).
Note to the Scholarship Evaluation Committee:
I developed OmniScribe as part of an ongoing self-directed study in modern web engineering, cloud API orchestration, and multimodal document processing.Building this application served as a practical exercise in interacting directly with cloud-hosted vision models from a browser client, handling rate-limit failovers cleanly, and solving cross-lingual Unicode rendering challenges across Arabic, Russian, and English scripts. As an applicant for the Russian Government Scholarship in Computer Science and Artificial Intelligence, I present this repository as a transparent sample of my current engineering work. Because software engineering is an iterative craft, I welcome technical feedback and academic mentorship from university faculty in the Russian Federation to strengthen my theoretical foundations further.
Направление подготовки: 09.04.01 Информатика и вычислительная техника / Искусственный интеллект и мультимодальный анализ данных
Автор: Adel Mohamed (github.com/Adel-dev-2001)
Целевые вузы: НИУ ВШЭ, Университет ИТМО, МФТИ, МГУ им. М.В. Ломоносова
Тема проекта: Клиентская мультимодальная система оптического распознавания текста и символов Unicode с динамической адаптацией к актуальным моделям машинного зрения.
- Динамическое обнаружение моделей (Dynamic Model Discovery): Отказ от статической привязки к устаревающим версиям API. Модуль интроспекции выполняет постраничный опрос эндпоинта
GET /v1beta/models(pageSize=1000,nextPageToken) с передачей ключа в HTTP-заголовкеx-goog-api-key, фильтрует бесплатные мультимодальные модели семейства Flash и ранжирует их по номеру поколения (major.minor), сохраняя совместимость с будущими релизами. - Отказоустойчивый каскад и изоляция промпта: При возникновении минутных ограничений квоты (
HTTP 429) или неподдерживаемых параметров система автоматически переключается на резервную модель актуального поколения без прерывания очереди. Инструкции транскрипции передаются черезsystemInstruction(с откатом к одношаговому формату приHTTP 400). - Многоязычный растрово-векторный экспорт: Поддержка кириллицы, арабской вязи (RTL) и латиницы с рендерингом документов в PDF высокого разрешения (192 DPI), а также экспорт в
.txt,.mdи.json.
flowchart TD
subgraph Client [Browser Client Layer]
UI[Three-Section SaaS Interface and Studio]
Dropzone[Batch Ingestion Queue: Up to 20 Images + Ctrl+V]
Canvas[HTML5 Canvas Pre-Scaler and JPEG Optimizer]
Editor[In-Place Text Editor and Unicode Inspector]
end
subgraph Kernel [OmniScribe Core Services]
Discovery[Dynamic Model Discovery Engine]
Cascade[Real-Time Failover Cascade Controller]
Analyzer[Unicode Code Point and Metrics Analyzer]
Exporter[Multi-Format Report Compiler]
end
subgraph Cloud [Google Generative Language API]
Catalog[(GET /v1beta/models)]
Inference[(POST /v1beta/models/model:generateContent)]
end
UI --> Dropzone
Dropzone --> Canvas
UI --> Discovery
Discovery <-->|Live Catalog Handshake| Catalog
Canvas --> Cascade
Cascade <-->|Multimodal Vision Payload| Inference
Cascade --> Analyzer
Analyzer --> Editor
Editor --> Exporter
Exporter --> PDF[High-DPI Multilingual .pdf]
Exporter --> TXT[Structured .txt]
Exporter --> MD[Document .md]
Exporter --> JSON[Dataset .json]
Before transmitting an image to the vision endpoint, src/utils/imageProcessor.ts processes the file locally using an off-screen HTML5 Canvas:
- Images exceeding
1280pxon their longest edge are proportionally scaled down with high-quality bicubic interpolation (imageSmoothingQuality = 'high'). - Payloads are encoded into normalized JPEG streams (
0.85quality), reducing base64 network transmission size while preserving character edge sharpness for sub-second inference.
Hardcoding model identifiers in client applications causes software failure whenever cloud providers retire legacy model versions (such as gemini-1.0 or gemini-1.5).
OmniScribe implements a version-agnostic discovery and inference pipeline in src/services/geminiService.ts:
- Paginated Live Catalog Introspection: Queries
https://generativelanguage.googleapis.com/v1beta/models?pageSize=1000(followingnextPageTokencursors up to 5 pages) directly from the browser using HTTP header authentication (x-goog-api-key), keeping credentials out of request URLs. - Strict Free-Tier Filtering: Retains only models that support
generateContentand belong to the Free-Tier multimodal Flash families (flashandflash-lite). Paid tiers (pro,ultra,argon,carbon,hydrogen), specialized non-OCR models (tts,audio,imagen,veo,embedding,thinking), date-pinned snapshots (versionSuffixmatching so future 4-digit major versions likegemini-1000-flashare preserved), and retired1.xversions are excluded. - Generation Version Parsing: Extracts numerical
majorandminorversion components via regular expression matching (gemini-(\d+)(?:\.(\d+))?). Whether the catalog exposes version2.5,3.8,4.x, or any future generation, the comparator ranks the highest generation first. - Prompt Isolation and Synchronous Failover: Transmits verbatim OCR directives via
systemInstruction(strictly separating instructions from untrusted user image bytes) with an automatic single-turnHTTP 400compatibility fallback and real-timeHTTP 429model cascade failover across verified Free-Tier Flash endpoints.
Standard client-side PDF libraries (such as default jsPDF text methods) rely on 8-bit WinAnsi/Latin-1 encoding, which corrupts Arabic right-to-left (RTL) contextual ligatures, Russian Cyrillic characters, and multi-byte Unicode emojis.
To solve this without server-side dependencies, src/services/exportService.ts implements a hybrid DOM-to-Canvas-to-PDF pipeline:
- Constructs an isolated, print-styled A4 layout container rendered through the browser's native HarfBuzz/DirectWrite text-shaping engine (
Cairo,Inter, and native color emoji fonts). - Uses native HTML5 bidirectional text isolation (
dir="auto"paired with CSSunicode-bidi: plaintextandtext-align: start) so Arabic right-to-left (RTL) paragraphs and Russian Cyrillic or English left-to-right (LTR) blocks shape and align naturally within the same report. - Rasterizes pages at
2xdevice pixel ratio (192 DPI) viahtml2canvasand slices multi-page documents across standard A4 dimensions (210mm x 297mm) with immediate intermediate Canvas buffer reclamation after each page.
| Evaluation Criterion | Classical OCR (Tesseract / Legacy Pipelines) | OmniScribe (Multimodal Vision Architecture) |
|---|---|---|
| Unicode Emojis and Symbols | Omitted or misclassified as punctuation noise | Full preservation of Unicode code points and ZWJ sequences |
| Multilingual Mixed Scripts | Requires manual language pack configuration | Native recognition of English, Russian, and Arabic in a single pass |
| Tabular and Code Layouts | Loses column alignment and indentation | Preserves spatial structure, Markdown tables, and code blocks |
| Model Lifecycle Longevity | Static weights or hardcoded API model names | Dynamic runtime discovery of newest Free-Tier Flash models |
| Batch Report Compilation | Requires external post-processing scripts | Built-in client-side export to PDF, TXT, Markdown, and JSON |
| Data Privacy Boundary | Often requires backend server file uploads | Client-side execution; direct browser-to-Google HTTPS connection |
omniscribe/
├── .github/
│ └── workflows/
│ └── deploy.yml # Automated CI/CD pipeline for GitHub Pages deployment
├── public/
│ └── favicon.svg # Geometric optical aperture vector icon
├── src/
│ ├── __tests__/
│ │ └── coreEngine.test.ts # Automated unit tests for NLP metrics, exports, discovery, and failover
│ ├── components/
│ │ ├── BatchControls.tsx # Queue metrics bar and batch execution controls
│ │ ├── BatchDropzone.tsx # Drag-and-drop and clipboard ingestion component (max 20 files)
│ │ ├── ExportModal.tsx # Multi-format compilation configuration dialog
│ │ ├── Header.tsx # Floating navigation bar, live model pill, and language switcher
│ │ ├── ImagePreviewList.tsx # Queue cards with optical laser scanning overlay
│ │ ├── Logo.tsx # Scalable SVG brand mark component
│ │ ├── ModelSelectorModal.tsx # API key management and live Google catalog scanner
│ │ ├── ResultModal.tsx # Inspection dialog, in-place editor, and Unicode code-point inspector
│ │ ├── SaaSFaq.tsx # Technical architecture and privacy accordion
│ │ ├── SaaSFeatures.tsx # Core architectural capabilities grid
│ │ ├── SaaSFooter.tsx # Minimal single-line attribution footer
│ │ └── SaaSHero.tsx # Hero section and quantitative performance indicators
│ ├── i18n/
│ │ └── translations.ts # Complete trilingual localization dictionaries (EN, RU, AR)
│ ├── services/
│ │ ├── exportService.ts # Client-side PDF, TXT, Markdown, and JSON compilation service
│ │ └── geminiService.ts # Google Gemini REST client, model discovery, and failover engine
│ ├── types/
│ │ └── index.ts # Strict TypeScript interfaces and data contracts
│ ├── utils/
│ │ └── imageProcessor.ts # HTML5 Canvas image downscaling and compression utility
│ ├── App.tsx # Root application state, batch queue controller, and keyboard listeners
│ ├── index.css # Tailwind CSS layers, custom scrollbars, and laser scan keyframes
│ └── main.tsx # React 18 DOM entry point
├── .gitignore # Git ignore rules for dependencies, builds, and environment files
├── LICENSE # MIT Open-Source License
├── README.md # Technical architecture specification and academic documentation
├── index.html # SEO meta tags, OpenGraph cards, and Schema.org JSON-LD graph
├── package-lock.json # Locked deterministic npm dependency tree
├── package.json # Project dependencies and build scripts
├── postcss.config.js # PostCSS pipeline configuration
├── tailwind.config.js # Tailwind CSS theme and typography configuration
├── tsconfig.json # Strict TypeScript compiler configuration
├── tsconfig.node.json # TypeScript compiler settings for Node/Vite build tooling
└── vite.config.ts # Vite bundler configuration with relative base paths
- Direct Browser-to-Google Communication: OmniScribe operates entirely as a client-side Single Page Application (SPA). All API requests are transmitted directly from the user's browser over HTTPS to
generativelanguage.googleapis.com. - HTTP Header Authentication and Secret Redaction: Credentials are transmitted exclusively via the
x-goog-api-keyHTTP request header rather than URL query parameters, preventing key leakage in browser history or server access logs. Any API key strings echoed in upstream error responses are redacted before surfacing diagnostics. - Local Credential Isolation and Output Sanitization: The user's Google Gemini API key is stored locally in
localStoragewith single-click deletion, no third-party proxy ever inspects traffic, uploaded images stay in volatile browser memory (FileReaderandHTML5 Canvas), and PDF DOM templates escape untrusted text (escapeHtml) while sanitizing export filenames (safeFileSlug).
- Node.js: v22.12.0+ or v24.x (required by Vitest 5; verified on Node.js v24.18)
- Package Manager:
npm(v10+) - API Credential: Free-tier Google Gemini API key from Google AI Studio
# 1. Clone the repository
git clone https://github.com/Adel-dev-2001/omniscribe.git
cd omniscribe
# 2. Install dependencies
npm install
# 3. Verify strict TypeScript types
npm run typecheck
# 4. Run automated unit test suite (Vitest)
npm test
# 5. Launch local development server
npm run dev
# 6. Compile production bundle
npm run buildContinuous Integration: Pushing to
maintriggers.github/workflows/deploy.yml, which executesnpm ci,npm run typecheck,npm test, andnpm run buildbefore publishing./distto GitHub Pages.
- Upgraded Gemini REST authentication to the
x-goog-api-keyHTTP header across model discovery and multimodal inference, addednextPageTokencatalog pagination (pageSize=1000), redacted credentials on upstream errors, and fixed auto-discovery key validation so invalid keys do not re-trigger background scans in a loop. - Added
systemInstructionprompt isolation with single-turnHTTP 400fallback, version-suffix regex filtering to protect future multi-digit major releases (such asgemini-1000-flash), and synchronous React ref failover persistence alongside React StrictMode-safe state updates and guardedlocalStorageaccess. - Replaced rigid edge-to-edge borders with subtle
.section-dividerhairlines, updated Tailwind v3 button/card shadows, wiredAbortControllerbatch cancellation withblob:URL cleanup, cleaned up modal/copy timers on unmount, and expanded dialog/tab/group/accordion ARIA accessibility. - Completed trilingual localization (
EN/RU/AR), multi-codepoint Unicode inspection, and inline error reporting for both batch queues and single-item PDF generation. - Refined high-DPI DOM-to-Canvas-to-PDF compilation (
192 DPI) with HTML entity escaping, nativedir="auto"BiDi alignment, dynamic Markdown code fences with TOC bracket escaping, safe filename slugs, Canvas memory reclamation, and updated GitHub Actions CI to Node.js 24 alongside an expanded Vitest unit suite. - Enriched
index.htmlwith Schema.org JSON-LD (SoftwareApplicationandWebSite), strict referrer metadata, accessible<noscript>fallback, and relative./asset paths for GitHub Pages deployment.
- Streamlined the user interface into a focused three-section layout (Hero and Studio Workspace, Core Capabilities, and Technical FAQ) with full dark/light mode persistence.
- Added global
Ctrl+Vclipboard image capture, modal background scroll locking, andEscapekey dismissal. - Integrated non-intrusive quota alert notifications and one-click API key management.
- Added in-place OCR text editing with real-time recalculation of character, word, line, and emoji statistics.
- Added interactive Unicode emoji chip inspector displaying hexadecimal code points (
U+XXXX). - Completed trilingual localization across English (
en), Russian (ru), and Arabic (ar) with automatic RTL/LTR layout switching.
- Initial architecture featuring batch image ingestion (up to 20 files), dynamic Google Gemini API discovery, and consolidated multi-format export (
.txt,.pdf,.md,.json).
Author: Adel Mohamed • Released under the MIT License.