url-rag

Retrieval-Augmented Generation pipeline built with Apache Airflow, Pinecone, and OpenAI.

Sibling dependency: acidbase

url-rag depends on jurdabos/acidbase — a small, MIT-licensed sibling repo that provides the shared push CLI subcommand (and other ecosystem utilities). It is referenced directly via git in pyproject.toml:

dependencies = [
    "acidbase @ git+https://github.com/jurdabos/acidbase.git",
    # ...
]

For anyone cloning this repo, the install path is just:

uv sync

Nothing private is required. As maintainer, I commit to keeping jurdabos/acidbase public and installable as long as url-rag is published. If that ever stops being practical, I will inline the necessary pieces here and drop the direct git dependency in the same change. — Blai

What it does

DAG	Purpose
ingestionrag	Fetches web pages, chunks them, generates embeddings via OpenAI, and upserts vectors into Pinecone
queryragdag	Accepts a question as a parameter, embeds it, retrieves the top-k relevant chunks from Pinecone, and generates a grounded answer via OpenAI

Quickstart

1. Set up the environment — copy .env.example to .env and fill in your API keys.
2. Install dependencies:

   uv sync

3. Start Airflow (via Astronomer):

   astro dev start

4. Run the ingestion DAG from the Airflow UI to populate the vector store.
5. Trigger the query DAG with your question in the query parameter or use the Click CLI with

uv run url-rag query "What is your real question?"

## Required Airflow variables

- `OPENAI_API_KEY`
- `PINECONE_API_KEY`
- `PINECONE_INDEX_NAME` (defaults to `rag-index`)
- `rag_source_urls` (optional JSON list; defaults are baked into the DAG)

## Project structure

url-rag/ ├── dags/ │   ├── ingestionrag.py # Ingestion pipeline (fetch → chunk → embed → upsert, with CDC) │   └── queryragdag.py # Query pipeline (embed → retrieve → generate) ├── plugins/ │   └── url_helpers_plugin.py # Registers url_rag.* Jinja macros ├── src/url_rag/ │   ├── cli.py # Click CLI (url-rag query) │   ├── rag.py # Library-mode RAG pipeline (used by CLI) │   └── url_filter.py # Pure URL-skip + content-hash helpers ├── tests/unit/ # pytest unit tests ├── include/ # Shared SQL / config / URL list (urls.json is gitignored) ├── .github/workflows/lint.yml # ruff + gitleaks CI ├── .gitleaks.toml ├── .pre-commit-config.yaml ├── Dockerfile ├── airflow_settings.yaml ├── packages.txt ├── requirements.txt ├── pyproject.toml └── .env.example

## Development

```bash
uv sync                       # install runtime + dev deps
uv run pytest                 # run the test suite
uv run ruff check .           # lint
uv run ruff format --check .  # format check
pre-commit run --all-files    # run all pre-commit hooks (uv-lock, ruff, gitleaks, sync-requirements)

CI runs the same ruff + gitleaks checks on every push and PR (see .github/workflows/lint.yml).

Dependency management & Astro image build

The Astronomer Runtime base image (astrocrpublic.azurecr.io/runtime:*) ships Apache Airflow itself and refuses any apache-airflow pin in requirements.txt at build time. To keep both local development (which needs Airflow installed via uv sync) and the Docker build happy, the two dependency files diverge intentionally:

• pyproject.toml lists the full set of direct deps, including apache-airflow>=3.0.6 — this drives uv sync for the local .venv.
• requirements.txt is autogenerated by scripts/sync_requirements.py from the same [project].dependencies block, with the Airflow core packages (apache-airflow, apache-airflow-core, apache-airflow-task-sdk) filtered out. This is what the Astro Runtime image consumes via its ONBUILD install-python-dependencies step.

A pre-commit hook (sync-requirements) keeps the two in sync automatically; never edit requirements.txt by hand. To change deps, edit pyproject.toml and either commit (the hook runs) or run the script manually:

python scripts/sync_requirements.py

Changelog

See CHANGELOG.md — this project follows Keep a Changelog.

Credits

A substantial portion of the original DAG scaffolding and Pinecone wiring in this project is derived from tutorials by the The Data and AI Guy YouTube channel — https://www.youtube.com/@thedataandaiguy. Heartfelt thanks for the clear, runnable walk-throughs. Any bugs, design choices, or production hardening (CDC, skip-lists, OpenRouter fallback, CLI) on top of that base are mine.

License

No license file yet; treat as "all rights reserved" until one is added.