semantic-caching

Semantic cache strategy bench: replay recorded request traces through candidate caching policies and measure which one actually pays off.

Ultra-low-latency LLM gateway with microsecond caching, dynamic routing, budgets, analytics, and forecasting.

Official implementation of "SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching" (COLM 2025). A novel KV cache compression method that organizes cache at sentence level using semantic similarity.

This app leverages Semantic Caching to minimize inference latency and reduce API costs by reusing semantically similar prompt responses.

Semantic caching demo with real-time streaming and a cost & sizing calculator, powered by Azure Managed Redis and Azure OpenAI.

LLM Observability & Evaluation Platform — Live Frontend: https://llm-obs-web-app-9664.web.app

Provider-agnostic LLM observability & reliability SDK — semantic caching, retries, rate limiting, and real-time dashboard via a single @track() decorator.

Semantic caching system using Meta Llama 3.3, Groq, FAISS, and Sentence Transformers to reduce LLM latency and API costs through meaning-based query matching.

Evaluate how a semantic cache performs on your dataset by computing key KPIs over a threshold sweep and producing plots/CSVs:

Enterprise AI infrastructure platform: distributed vLLM inference, agentic workflows, hybrid multimodal RAG, CNN vision, QoS-tiered FastAPI gateway, guardrails and Prometheus/Grafana observability, fully Dockerized.

Rust Local Token Compression Proxy for coding agents, built solo for GenAI Genesis 2026. 🏆 1st Google Sustainability Hack

An intelligent gateway for Claude APIs that dynamically routes requests to the most cost-efficient model, caches responses, and escalates based on confidence signals — reducing LLM spend without sacrificing quality.

Python library for the Semcache API

Semantic caching for LLM responses using Redis Vector DB, LangChain, and HuggingFace embeddings, parses PDFs, generates FAQs with Groq, and serves similarity-based answers without redundant LLM calls.

Semantic memory and caching for LLM agents with classifier-validated equivalence instead of naive cosine thresholds.

A retrieval-augmented generation pipeline in Python with a rigorous offline evaluation harness. Chunks and embeds documents, retrieves by vector similarity, and generates grounded answers — with pluggable LLM providers (including a deterministic local fake for tests) and metrics for retrieval quality and answer faithfulness. No API key required.

Database and data-system foundations for multi-agent collaboration: a topic guide and paper list organized by the ACDS architecture.

Surgical middleware for LLMs. Reduces token waste by 40-60% using Triple-Tier Memory (Anchor + Fact Sheet + Buffer) and Semantic Caching. Built for production-grade branding & cost-recovery.

Code and experimental artifacts for "Synchronous Online Verification Gating in Semantic Caches" — an empirical study of real-time verifier gating for semantic cache hits, evaluated against static/adaptive-threshold baselines on ~210k real requests across three datasets.

Terraform module for a production-hardened Azure AI gateway: APIM in front of Azure AI Foundry with tiered JWT auth, token quotas, content safety, semantic caching, and private networking