See how an AI-powered agentic workflow transforms a public RFP into an evidence-grounded proposal β from automated web extraction and portfolio retrieval to independent AI review and human approval.
Turn an RFP URL into a tailored, evidence-grounded proposal β checked by an independent AI critic and submitted for human review.
Live Demo Β· Windows Quick Start Β· Architecture Β· API
Autonomous RFP Pipeline helps turn software project opportunities into focused proposal drafts. Submit a public RFP, tender, or project-posting URL; the system extracts the page, analyzes its requirements, searches your portfolio knowledge base, writes a proposal, and pauses for a human decision.
| Input | Result |
|---|---|
| Public HTTP/HTTPS RFP or project URL | A tailored proposal draft, normally 100β180 words |
Portfolio files (PDF, TXT, CSV, MD) |
Relevant project context retrieved from ChromaDB |
Core capabilities: browser-based extraction with fallbacks Β· requirement analysis Β· hybrid portfolio RAG Β· proposal writing Β· independent AI critique Β· human approval/revision Β· local run history and proposal export.
The main proposal flow accepts a web URL, not an uploaded RFP PDF. File uploads are for portfolio knowledge only. The system drafts proposals; it does not automatically send them to clients.
The frontend sends requests to FastAPI. Redis and Celery handle background work, while a LangGraph workflow coordinates scraping, analysis, retrieval, writing, critique, and human review. SQLite stores workflow checkpoints; ChromaDB stores portfolio vectors.
%%{init: {'theme':'base','themeVariables':{'fontFamily':'Inter, Segoe UI, Arial, sans-serif','lineColor':'#4FD1BC','textColor':'#CDE7EA','primaryTextColor':'#CDE7EA','edgeLabelBackground':'#0B1822','clusterBkg':'#0A1822','clusterBorder':'#1F4A58'},'flowchart':{'curve':'basis','nodeSpacing':45,'rankSpacing':55}}}%%
flowchart TB
U([User / Browser]) --> FE[Next.js Workspace]
FE -->|HTTP| API[FastAPI Β· backend/main.py]
API <-->|Queue and task state| R[Redis]
R --> W[Celery Worker]
W --> G[LangGraph Workflow]
subgraph AI[LangGraph Workflow Β· top-to-bottom]
direction TB
G[Workflow Orchestrator] --> S[Scraping Engine]
S --> A[Technical Analyst]
A --> Q[Portfolio RAG]
Q --> P[Proposal Writer]
P --> C[Independent Critic]
C --> H{{Human Review Checkpoint}}
end
subgraph MODELS[Model Providers]
O[OpenRouter Β· agent LLMs]
GM[Gemini Β· embeddings]
GR[Groq Β· retrieval fusion]
end
subgraph DATA[Persistent Storage]
DB[(SQLite Β· checkpoints)]
CH[(ChromaDB Β· portfolio vectors)]
end
A --> O
P --> O
C --> O
H -. saves state .-> DB
Q --> GM
Q --> GR
Q --> CH
classDef client fill:#0F2530,stroke:#5FB3C9,color:#CDE7EA,stroke-width:1.5px
classDef api fill:#0B4F4A,stroke:#2DD4BF,color:#D7F7F1,stroke-width:2px
classDef queue fill:#10345A,stroke:#38BDF8,color:#D2EAFB,stroke-width:1.5px
classDef agent fill:#0E3B33,stroke:#34D399,color:#D2F5E6,stroke-width:1.5px
classDef provider fill:#3A2B0C,stroke:#F5B84B,color:#FBE3A8,stroke-width:1.5px
classDef storage fill:#14283B,stroke:#7FA8D6,color:#CFE0F3,stroke-width:1.5px
classDef human fill:#4A3410,stroke:#FFC857,color:#FBE3A8,stroke-width:2px
class U,FE client
class API api
class R,W queue
class G,S,A,Q,P,C agent
class H human
class O,GM,GR provider
class DB,CH storage
style AI fill:#0A1A20,stroke:#1E6B63,color:#7FD9C5
style MODELS fill:#17130A,stroke:#8A6A2B,color:#F0CF8B
style DATA fill:#0B1824,stroke:#3B5F85,color:#A9C5E3
%%{init: {'theme':'base','themeVariables':{'fontFamily':'Inter, Segoe UI, Arial, sans-serif','lineColor':'#4FD1BC','textColor':'#CDE7EA','primaryTextColor':'#CDE7EA','edgeLabelBackground':'#0B1822','clusterBkg':'#0A1822','clusterBorder':'#1F4A58'},'flowchart':{'curve':'basis','nodeSpacing':45,'rankSpacing':55}}}%%
flowchart TB
A([Submit RFP URL]) --> B[FastAPI validates and queues task]
B --> S[Scrape page content]
S -->|Failed / empty| X([Stop with extraction error])
S -->|Usable content| N[Analyze requirements]
N --> R[Retrieve portfolio evidence]
R --> W[Write proposal]
W --> C[Independent critic]
C -->|Revision needed Β· passes remain| W
C -->|Approved / critic error / pass limit| H{{Human review}}
H -->|Approve| F([Reviewed proposal])
H -->|Revise with feedback| W
classDef start fill:#0F2530,stroke:#5FB3C9,color:#CDE7EA,stroke-width:1.5px
classDef process fill:#0B4F4A,stroke:#2DD4BF,color:#D7F7F1,stroke-width:1.5px
classDef review fill:#4A3410,stroke:#FFC857,color:#FBE3A8,stroke-width:2px
classDef stop fill:#3D1A15,stroke:#E5604D,color:#F8CFC4,stroke-width:1.5px
class A,F start
class B,S,N,R,W,C process
class H review
class X stop
| Stage | Responsibility |
|---|---|
| Scraper | Extracts and cleans the public page content; stops if the result is unusable. |
| Technical Analyst | Identifies goals, requirements, deliverables, constraints, and missing information. |
| Portfolio RAG | Retrieves relevant past-project evidence; the workflow can continue without a match. |
| Proposal Writer | Drafts a concise, client-specific proposal without inventing experience or commitments. |
| Independent Critic | Checks technical alignment, omissions, contradictions, and unsupported claims. |
| Human Checkpoint | Pauses for a person to edit, revise, or approve the draft. |
The critic can request revisions. The graph allows up to three writer passes. Even when the critic returns APPROVED, a human review is still required. An unparseable critic response is treated as CRITIC_ERROR and sent for manual review.
The scraper is best-effort: fallbacks improve resilience but cannot guarantee extraction from every site.
%%{init: {'theme':'base','themeVariables':{'fontFamily':'Inter, Segoe UI, Arial, sans-serif','lineColor':'#4FD1BC','textColor':'#CDE7EA','primaryTextColor':'#CDE7EA','edgeLabelBackground':'#0B1822','clusterBkg':'#0A1822','clusterBorder':'#1F4A58'},'flowchart':{'curve':'basis','nodeSpacing':45,'rankSpacing':55}}}%%
flowchart TB
URL([RFP URL]) --> MAIN[Main engine Β· SeleniumBase + Playwright CDP]
MAIN -->|HTML| CLEAN[BeautifulSoup4 + html2text Β· clean text]
MAIN -->|Failure| PW[Fallback Β· Playwright headless Chromium]
PW -->|HTML| CLEAN
PW -->|Failure| ND[Fallback Β· Nodriver isolated process]
ND -->|HTML| CLEAN
ND -->|Failure| ERR([Extraction stops])
CLEAN --> OUT([Text for Technical Analyst])
classDef source fill:#0F2530,stroke:#5FB3C9,color:#CDE7EA
classDef engine fill:#0B4F4A,stroke:#2DD4BF,color:#D7F7F1
classDef fallback fill:#10345A,stroke:#38BDF8,color:#D2EAFB
classDef output fill:#14283B,stroke:#7FA8D6,color:#CFE0F3
classDef fail fill:#3D1A15,stroke:#E5604D,color:#F8CFC4
class URL source
class MAIN,CLEAN engine
class PW,ND fallback
class OUT output
class ERR fail
| Component | File | Role |
|---|---|---|
| Main engine | backend/scraper_engine.py |
SeleniumBase undetected Chrome mode, Playwright/CDP, retries |
| Fallbacks | backend/scraping_retr.py, backend/scraping_retr_2.py |
Playwright and Nodriver extraction |
| HTML cleaning | backend/bs4_engine.py |
BeautifulSoup4 removes page noise; html2text converts content |
Upload portfolio files through Knowledge Admin. A Celery task extracts text, splits it into chunks, creates Gemini embeddings, and stores them in ChromaDB. At retrieval time, vector search and BM25 keyword search are combined by LlamaIndex QueryFusionRetriever (Groq is used in the fusion setup).
%%{init: {'theme':'base','themeVariables':{'fontFamily':'Inter, Segoe UI, Arial, sans-serif','lineColor':'#4FD1BC','textColor':'#CDE7EA','primaryTextColor':'#CDE7EA','edgeLabelBackground':'#0B1822','clusterBkg':'#0A1822','clusterBorder':'#1F4A58'},'flowchart':{'curve':'basis','nodeSpacing':45,'rankSpacing':55}}}%%
flowchart TB
UP[Admin uploads PDF / TXT / CSV / MD] --> TASK[Background indexing task]
TASK --> EX[Extract text]
EX --> CHUNK[Split Β· 500 tokens / 50 overlap]
CHUNK --> EMB[Gemini embeddings]
EMB --> DB[(ChromaDB Β· company_projects_rag)]
ASK[Analyst requirements] --> V[Vector search]
ASK --> K[BM25 keyword search]
V --> F[QueryFusionRetriever Β· Groq]
K --> F
DB --> V
F --> CTX[Relevant portfolio context]
CTX --> WR[Proposal Writer]
classDef input fill:#0F2530,stroke:#5FB3C9,color:#CDE7EA
classDef process fill:#0B4F4A,stroke:#2DD4BF,color:#D7F7F1
classDef store fill:#14283B,stroke:#7FA8D6,color:#CFE0F3
classDef output fill:#4A3410,stroke:#FFC857,color:#FBE3A8
class UP,ASK input
class TASK,EX,CHUNK,EMB,V,K,F process
class DB store
class CTX,WR output
- Embedding model:
gemini-embedding-001; vector collection:company_projects_rag. - Chunking:
SentenceSplitter, size 500, overlap 50; PDFs use PyMuPDFReader. - Storage: ChromaDB persists under
./embeddings_Store; LangGraph checkpoints use SQLite (backend/langgraph_checkpoints.sqliteby default). - Important:
past_projects.md.exampleis only a template.past_projects.mdis not imported automatically. Fill it with real projects, then upload the resulting.mdfile through Knowledge Admin.
| Area | Technologies used in this repository |
|---|---|
| Backend and API | Python 3.12, FastAPI, Uvicorn, Pydantic v2 |
| Agent orchestration | LangGraph, LangChain |
| Retrieval and storage | LlamaIndex, Gemini embeddings, ChromaDB, BM25, SQLite |
| Model providers | OpenRouter (agents), Gemini (embeddings), Groq (retrieval fusion) |
| Background processing | Celery and Redis |
| Browser extraction | SeleniumBase, Playwright, Nodriver, BeautifulSoup4, html2text |
| Frontend | Next.js 15, React 19, Tailwind CSS 4, Framer Motion, GSAP, Lenis |
| Optional deployment | Docker Compose |
The project uses OpenRouter for the Analyst, Writer, and Critic; Gemini for embeddings; and Groq in the RAG query-fusion setup. These providers have different responsibilities.
The native Windows setup is the recommended path. Docker is optional. You need Redis running separately and three PowerShell terminals for the application services.
Option A β Git clone
git clone https://github.com/mrfaizan-dev/Autonomous-RFP-Pipeline.git
cd Autonomous-RFP-PipelineOption B β Download ZIP
- Open the GitHub repository.
- Select Code β Download ZIP, then extract it.
- Open PowerShell in the extracted folder:
cd .\Autonomous-RFP-Pipeline-mainUse the folder name created by extraction if Windows adds a suffix such as
(1).
- Windows 10/11 and Python 3.12 (
py -3.12 --version). - Node.js 22 LTS and npm (
node -v,npm -v). - A running Redis-compatible server reachable at
redis://localhost:6379/0(or your ownREDIS_URL). Redis is required by Celery; this repository does not install Redis for native Windows. - API keys: OpenRouter for proposal agents; Gemini and Groq for portfolio RAG.
- Internet access and browser dependencies for scraping.
Run these commands from the repository root:
Copy-Item .env.example .env
notepad .envAdd or update these exact variable names in .env:
OPENROUTER_API_KEYS=your_openrouter_api_key
GEMINI_API_KEYS=your_gemini_api_key
GROQ_API_KEYS=your_groq_api_key
ADMIN_SECRET_KEY=replace_with_a_long_random_secret
REDIS_URL=redis://localhost:6379/0Optional model overrides: ANALYZER_LLM_MODEL, WRITER_LLM_MODEL, and CRITIC_LLM_MODEL.
Configuration note: The root
.env.examplecurrently uses singular names for Gemini and Groq, but the backend readsGEMINI_API_KEYSandGROQ_API_KEYS(plural). Use the plural names above. Never commit.envor expose provider keys in frontend variables.
Create the frontend environment file:
if (-not (Test-Path .\frontend\.env.local)) { Copy-Item .\frontend\.env.local.example .\frontend\.env.local }Confirm it contains:
NEXT_PUBLIC_API_BASE_URL=http://127.0.0.1:8800Start Redis first. Keep it running, then open three separate PowerShell terminals in the repository folder.
Terminal 1 β FastAPI backend
cd .\backend
if (-not (Test-Path ".venv")) { py -3.12 -m venv .venv }
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -r requirements.txt
python -m playwright install chromium
python main.pyBackend: http://127.0.0.1:8800 Β· Swagger: http://127.0.0.1:8800/docs
Terminal 2 β Celery worker
cd .\backend
.\.venv\Scripts\Activate.ps1
celery -A main.celery_app worker --loglevel=INFO --pool=soloThe worker uses the same .venv and environment as FastAPI. --pool=solo is the intended local Windows setting.
Terminal 3 β Next.js frontend
cd .\frontend
if (-not (Test-Path ".env.local")) { Copy-Item .env.local.example .env.local }
npm install
npm run devFrontend: http://localhost:3000
If PowerShell blocks virtual-environment activation, run this in that terminal only, then activate again:
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass- Make sure Redis, FastAPI, the Celery worker, and Next.js are all running.
- Open
http://localhost:3000. - Paste a public RFP or project URL and submit it.
- Follow the task monitor. When the workflow reaches human review, inspect and edit the draft.
- Copy or download the proposal as Markdown. Approval/revision is available in the UI; see the review integration note.
The workflow pauses with a LangGraph checkpoint. A reviewer can approve, edit, or request a revision with feedback. Checkpoints are stored in SQLite; Redis is used for task queuing/results.
| Method | Endpoint | Purpose |
|---|---|---|
POST |
/api/generate_proposal |
Submit { "url": "https://example.com/rfp" } |
GET |
/api/task-status/{task_id} |
Read Celery task state |
POST |
/api/resume-proposal/{thread_id} |
Queue approval or revision for a paused run |
POST |
/api/admin/upload-rag-file |
Queue portfolio-file indexing; requires X-Admin-Token |
GET |
/docs |
Interactive Swagger documentation |
The status endpoint reports Celery states such as PENDING, STARTED, RETRY, SUCCESS, or FAILURE. PENDING can also mean an unknown task ID. Task monitoring polls about every 1.8 seconds; it does not stream each LangGraph node's progress.
Review integration note: The backend resume endpoint responds immediately with a new Celery task ID. The current frontend resume handler does not visibly poll that new task ID, so the complete browser-based approve/revise cycle needs end-to-end validation before relying on it. The backend route itself is available and can be tested through Swagger.
Open Knowledge Admin (Ctrl + Shift + A on Windows), enter the value matching ADMIN_SECRET_KEY, and upload a .pdf, .txt, .csv, or .md file. The upload response means indexing has been queued; check the worker terminal for processing. Use only real, authorized portfolio evidence.
Autonomous-RFP-Pipeline/
βββ backend/
β βββ main.py # FastAPI, Celery tasks, LangGraph workflow
β βββ rag_engine.py # Portfolio indexing and hybrid retrieval
β βββ scraper_engine.py # Main browser extraction
β βββ scraping_retr.py # Playwright fallback
β βββ scraping_retr_2.py # Nodriver fallback
β βββ bs4_engine.py # HTML cleaning
β βββ requirements.txt
βββ frontend/
β βββ app/ # Next.js page, layout and styles
β βββ components/ # UI and motion components
β βββ lib/ # API and local-storage helpers
β βββ package.json
βββ docs/assets/ # README artwork
βββ .env.example
βββ docker-compose.yml # Optional
βββ past_projects.md.example
βββ project_run_commands
- Some websites block automation, require authentication/CAPTCHA, or return incomplete content. Scraping is not guaranteed.
- Portfolio RAG needs uploaded documents in ChromaDB. The example Markdown file is not auto-indexed.
- Recent run history (up to 10 entries) and the admin token are kept in browser
localStorage, not a central user database. - The frontend currently contains a hard-coded default admin-token value; remove it before public deployment. A browser-stored token is not a hidden secret.
- FastAPI CORS currently allows wildcard origins.
FRONTEND_ORIGINSis parsed but does not currently restrict the middleware. Configure trusted origins before deployment. - Add SSRF protections and rate limits for public RFP URL submission, plus upload-size and stronger file-validation controls for production.
- Treat RFP text, proposals, and portfolio files as sensitive. Never commit real credentials or confidential client data.
- External providers have their own availability, quota, and model-access limits. No automated test suite or benchmark results are claimed here.
| Issue | Check |
|---|---|
| Redis connection error | Redis is running and REDIS_URL is correct. |
| Task remains queued | Terminal 2 worker is running and connected to the same Redis instance. |
| Model/API key error | Check OPENROUTER_API_KEYS; for RAG also check plural GEMINI_API_KEYS and GROQ_API_KEYS. Restart backend and worker after edits. |
| Portfolio search is empty | Upload supported portfolio files and confirm indexing completed in the worker terminal. |
| Scraping fails | Install Playwright Chromium; check network, browser availability, login/CAPTCHA, or site restrictions. |
| Frontend cannot reach API | Check FastAPI on port 8800 and NEXT_PUBLIC_API_BASE_URL; restart Next.js after environment changes. |
| Python activation fails | Confirm py -3.12 --version; if needed, use the process-only PowerShell execution-policy command above. |
Docker is not required for the Windows setup. To use the supplied Compose configuration instead, create the root .env with the correct variable names and run:
Copy-Item .env.example .envThen:
docker compose up --build| Service | Address |
|---|---|
| Frontend | http://localhost:3001 |
| Backend | http://localhost:8800 |
| Redis | Internal Compose network |
Compose persists Redis, SQLite checkpoints, ChromaDB, and uploads in volumes. Current Docker caveat: the upload volume is mounted in the backend but not the worker, which can affect background indexing. Mounting past_projects.md also does not auto-index it. Do not run docker compose down -v unless you intend to delete persistent volumes.
The following is my broader working toolkit; not every item is wired into this repository.
| Focus | Tools and skills |
|---|---|
| AI and agents | Python, LangGraph, LangChain, CrewAI, LlamaIndex, LLMs, Prompt Engineering, OpenAI, OpenRouter, Groq, Gemini, Claude, ChatGPT |
| Evaluation and observability | Langfuse, DeepEval, Guardrails |
| RAG and data | RAG, CRAG (Corrective RAG), ChromaDB, FAISS, Pandas, Pydantic v2 |
| Scraping and automation | Playwright, SeleniumBase, Selenium, Nodriver, BeautifulSoup (BS4), Proxy Rotation, asyncio, web scraping across varied sites |
| Backend and frontend | FastAPI, Redis, Celery, Docker, Next.js, Tailwind CSS, AI-assisted frontend development |
Mr Faizan
Software Developer Β· AI Engineering Β· Agentic Systems Β· RAG Β· Web Automation Β· Full-Stack Development