A data science pipeline that scrapes the Anarchy Library and applies modern NLP — BERT-based topic clustering, Cross-Encoder relevance scoring, and sentiment analysis — to surface the dominant philosophical and rhetorical trends across the corpus.
- Overview
- Build Status
- Architecture
- Tech Stack
- Project Structure
- The NLP Pipeline
- Presentation Layer
- API Layer (FastAPI)
- Database Schema (PostgreSQL)
- Update Cadence
- Deployment
- Installation
- Usage
- Results
The Anarchy Library hosts ~17,000 articles, essays, and pamphlets covering anarchist thought across history — ranging from 10-line broadsides to entire books like Thus Spoke Zarathustra. This project treats that library as a corpus and answers two questions:
- What are people writing about? — Surfaced via BERTopic, an end-to-end neural topic modeling framework that combines BERT embeddings, UMAP, HDBSCAN, and class-based TF-IDF (c-TF-IDF) for cluster labeling.
- What's the most relevant quote for a given query? — Surfaced via a two-stage retrieval pipeline: pgvector cosine shortlist over ~3M chunk embeddings, then BAAI/bge-reranker-base scoring of sentence-level passages.
A third layer applies sentiment / tone analysis on top of the topics, so the project doesn't just report what is being discussed, but how — academic, polemical, militant, hopeful — across each thematic cluster.
The project is built as a lakehouse-style data pipeline serving two audiences from the same backend:
- Readers use a lightweight static web page that hits
/quoteto find the most relevant excerpt from the corpus for any natural-language query. - Analysts connect Tableau or PowerBI directly to PostgreSQL materialized views (or load CSV / Parquet exports) to explore topic distributions, tag frequencies, and tone trends across the corpus.
The corpus is updated monthly via incremental scrapes, with the topic model refit quarterly to discover emerging themes. Heavy NLP work runs offline on GPU (Runpod); the API serves on CPU (Railway).
This README documents the complete designed architecture. The repository currently contains the tested offline NLP stages; full-corpus GPU execution, the API, database, and frontend layers are still pending. The split below is honest — recruiters and reviewers can see exactly where the work is.
| Component | File | Status |
|---|---|---|
| Scrapy spider for Anarchy Library | main.py + scrape/organ/ |
Working (sitemap.txt seed, ~17K URLs) |
| Chunking pipeline (LangChain + DuckDB + Parquet shards) | pipeline/chunking.py |
Working and unit-tested |
| Chunk + article embedding pipeline | pipeline/embed.py |
Sharded GPU inference and mean pooling implemented |
| BERTopic topic modeling (PCA + cuML + ngrams) | pipeline/topic.py |
GPU pipeline implemented; full-corpus run pending |
| Four-score zero-shot tone pipeline | pipeline/tone.py |
GPU pipeline implemented; full-corpus run pending |
| Cross-Encoder relevance with spaCy sentence windowing | research/BestWay.py |
Working on subset; needs refactor for serving |
| Centralized config | config.py |
Working |
| Component | Where it'll live | Notes |
|---|---|---|
| PostgreSQL serving layer + pgvector | migrations/, api/models.py |
Chunks, embeddings, topics, tones, tags |
FastAPI service with /quote |
api/ |
Two-stage retrieval (pgvector → bge-reranker) |
| Reader frontend | frontend/index.html |
Vanilla JS + Tailwind CDN, single page |
| BI exports + materialized views | viz/export.py |
CSV / Parquet for Tableau / PowerBI |
| Incremental update orchestration | pipeline/update.py |
Monthly transform + quarterly refit |
| Railway deployment | docker-compose.yml, Railway config |
Postgres + API + static frontend |
The system is structured as a lakehouse architecture: Parquet files act as the immutable raw lake (cheap, append-only, DuckDB-queryable), while PostgreSQL with pgvector acts as the operational serving layer (metadata, chunk embeddings, topic assignments, tone scores). Heavy NLP work runs offline on GPU; the API serves on CPU.
flowchart LR
A["🕷️ Scrapy Spider<br/>(anarchy)"] --> B[("📦 Parquet Lake<br/>data/raw/shard_*.pq")]
B --> C["🧠 NLP Pipeline<br/>(GPU on Runpod)"]
C --> C1["LangChain Chunking<br/>800/160 overlap"]
C1 --> C2A["Chunk Embeddings<br/>(~3M × 384d)<br/>→ retrieval track"]
C1 --> C2B["Mean-pool per article<br/>(~17K × 384d)<br/>→ topic track"]
C2B --> C3["BERTopic w/ cuML<br/>(UMAP + HDBSCAN<br/>+ c-TF-IDF ngrams)"]
C --> C4["Tone Classification<br/>(4-axis, per article)"]
C2A --> D[("🐘 PostgreSQL + pgvector<br/>chunks (retrieval) +<br/>articles.topic_id (topics)<br/>+ tones, tags")]
C3 --> D
C4 --> D
D --> E["⚡ FastAPI Service<br/>(CPU on Railway)"]
D --> G["📊 Tableau / PowerBI<br/>(materialized views)"]
E --> E1["Two-stage retrieval<br/>pgvector → bge-reranker"]
E1 --> F1["🔍 Reader Search Page<br/>(HTML + Tailwind + JS)"]
E --> F2["🗺️ Topic Explorer<br/>(BERTopic viz, embedded)"]
E --> F3["🕸️ Tag-Topic Network<br/>(PyVis, embedded)"]
classDef ingest fill:#1f6feb,stroke:#0d419d,color:#fff
classDef lake fill:#0891b2,stroke:#155e75,color:#fff
classDef nlp fill:#8957e5,stroke:#5a32a3,color:#fff
classDef chunk fill:#0891b2,stroke:#155e75,color:#fff
classDef doc fill:#16a34a,stroke:#14532d,color:#fff
classDef db fill:#336791,stroke:#1a3a52,color:#fff
classDef api fill:#009688,stroke:#00695c,color:#fff
classDef viz fill:#d97706,stroke:#92400e,color:#fff
classDef bi fill:#e11d48,stroke:#9f1239,color:#fff
class A ingest
class B lake
class C,C1,C4 nlp
class C2A chunk
class C2B,C3 doc
class D db
class E,E1 api
class F1,F2,F3 viz
class G bi
Why this shape?
- Two embedding tracks for two different needs. Chunk embeddings (~3M × 384d) power retrieval via pgvector cosine shortlist. Article-level embeddings (~17K × 384d, mean-pooled from chunks) power topic modeling — a large reduction in BERTopic's input size with no signal loss for the per-article question.
- GPU-accelerated topic modeling. With cuML's UMAP and HDBSCAN, the topic fit on ~17K document embeddings completes in ~10 minutes on an A10G instead of multi-hour CPU runtime.
- Parquet is the cold lake, Postgres is the hot layer. Raw article text stays in Parquet (cheap, immutable, can be re-derived). Chunks, embeddings, topic assignments, and tone scores live in Postgres (queryable, vector-indexable, BI-connectable).
- The Cross-Encoder is online but bounded. A pgvector cosine shortlist trims candidates to ~50 chunks per query before bge-reranker-base runs. Reranking is expensive but happens over a small set.
- Two cadences: monthly incremental updates use BERTopic's
transform()to assign existing topics to new articles. Quarterly refits usefit_transform()to discover emerging topics. - The frontend never talks to the model directly. It talks to FastAPI, which is the only thing that talks to the models and the DB.
sequenceDiagram
actor User
participant UI as Search UI
participant API as FastAPI
participant DB as PostgreSQL
participant EMB as MiniLM
participant CE as bge-reranker-base
User->>UI: Submit query + filters
UI->>API: POST /quote {query, tag?, topic_id?}
API->>EMB: Embed query (384-dim vector)
EMB-->>API: query_embedding
API->>DB: pgvector cosine: top-50 chunks<br/>WHERE tag/topic filters match
DB-->>API: Candidate chunks (50)
API->>DB: Fetch articles for candidates
DB-->>API: Article texts
Note over API: Reconstruct text, dedupe overlap,<br/>spaCy sentence split,<br/>generate 1-3 sentence windows
API->>CE: Rerank (query, window) pairs (~200)
CE-->>API: Relevance scores
API-->>UI: Top-N quotes + source articles
UI-->>User: Render results with citations
| Layer | Tool | Why |
|---|---|---|
| Scraping | Scrapy | Used in main.py via CrawlerProcess. Robust polite crawling, easy item pipelines. |
| Raw storage | Parquet shards + DuckDB | Cheap append-only lake for ~17K articles. DuckDB queries shards directly without a DB server. |
| Chunking | LangChain RecursiveCharacterTextSplitter |
800-char chunks with 160-char overlap. Stable, well-tested. |
| NLP — embeddings | sentence-transformers (all-MiniLM-L6-v2) |
384-dim, fast on GPU, runs ~1.5M chunks in ~1 hour on a single A100. |
| NLP — topics | BERTopic + CountVectorizer (ngrams 1-3, min_df=10) | Fitted on 45K document embeddings (mean-pooled from chunks), not on chunks directly — 30x speedup with no signal loss. Trigram c-TF-IDF labels. |
| Topic-model GPU | RAPIDS cuML (UMAP + HDBSCAN) | GPU-accelerated drop-ins for BERTopic. ~27x speedup on UMAP, ~4x on HDBSCAN per published benchmarks. |
| Pre-reduction | scikit-learn PCA (384d → 50d) | Cheap pre-reduction before UMAP — further halves UMAP runtime at no measurable quality cost. |
| NLP — sentence split | spaCy (en_core_web_sm) |
Better than naive splitters for paragraph-level prose. Used at query time, not in pipeline. |
| NLP — reranking | BAAI/bge-reranker-base (Cross-Encoder) |
Best-in-class quality among small open rerankers. ~280M params; 2-5s CPU latency per query. |
| NLP — sentiment | transformers + DeBERTa zero-shot classification |
Independent academic / militant / hopeful / critical scores, aggregated from token windows. |
| Dim. reduction | UMAP via cuML (inside BERTopic) | GPU-accelerated UMAP; preserves global structure better than t-SNE. |
| Clustering | HDBSCAN via cuML (min_cluster_size=50) |
GPU-accelerated variable-density clusters; noise points stay labeled -1. |
| Serving storage | PostgreSQL 15+ with pgvector | Hot layer: chunk embeddings (IVFFlat / HNSW), topic assignments, tones, tags, materialized views. |
| API | FastAPI + Uvicorn | Async, typed, auto-generates OpenAPI docs, plays well with Pydantic. Also serves the static frontend. |
| DB layer | SQLAlchemy 2.0 ORM + asyncpg + Pydantic |
ORM models for type-safe queries and Alembic auto-migrations. Pydantic validates I/O. Raw SQL escape hatch for pgvector similarity and materialized views. |
| Frontend | HTML + Tailwind (CDN) + vanilla JS | Single static page, no build step, served by FastAPI as static files. |
| Plotting | Plotly (via BERTopic's built-in viz) | Interactive topic map, hierarchy, and topics-over-time, exported as HTML. |
| Network graphs | NetworkX + PyVis | Tag ↔ topic bipartite network for the "ideological bridges" view. |
| BI / Analytics | Tableau / PowerBI | Connect directly to Postgres materialized views, or load CSV / Parquet exports for offline use. |
| GPU compute | Runpod | On-demand GPU for the offline pipeline (embedding + BERTopic fitting). Pay-per-hour. |
| Hosting | Railway | Postgres + FastAPI + static frontend, single-region deploy. CPU only at serve time. |
| Design / UX | Figma | Wireframes for the reader page before implementation. |
anarchy_project/
├── scrape/ # Scraper unit (was: main.py + organ/ at root)
│ ├── run.py # CrawlerProcess entry point
│ ├── scrapy.cfg
│ └── organ/ # Scrapy project (spiders, items, pipelines)
│ └── spiders/
│ └── anarchy.py
│
├── pipeline/ # Offline NLP — data transformations (runs on GPU/Runpod)
│ ├── chunking.py # ← was: allsentiment/processingFile.py
│ ├── embed.py # MiniLM embeddings → Parquet → pgvector
│ ├── topic.py # ← was: allsentiment/TopicCategory.py
│ ├── tone.py # NEW: 4-axis tone scoring
│ ├── load_db.py # Persists pipeline output into PostgreSQL
│ └── update.py # NEW: incremental monthly update orchestration
│
├── viz/ # Offline viz — presentation outputs
│ ├── topics.py # Generates BERTopic intertopic + hierarchy HTML
│ ├── tag_network.py # Tag, topic bipartite network HTML (PyVis)
│ ├── topics_over_time.py # Topics-over-time stream graph HTML
│ ├── export.py # CSV / Parquet exports for Tableau / PowerBI
│ └── outputs/ # Generated HTML + export artifacts (gitignored)
│
├── api/ # FastAPI service (runs on CPU/Railway)
│ ├── main.py # app = FastAPI(); routers; static mount; CORS
│ ├── routers/ # HTTP layer — thin
│ │ ├── articles.py
│ │ ├── topics.py
│ │ └── quote.py
│ ├── services/ # Business logic — thick
│ │ ├── retrieval.py # pgvector cosine shortlist (stage 1)
│ │ ├── rerank.py # ← refactored from: allsentiment/BestWay.py (stage 2)
│ │ └── topic_lookup.py # Load saved BERTopic; query topic metadata
│ ├── schemas.py # Pydantic schemas (request/response)
│ ├── models.py # SQLAlchemy ORM models (DeclarativeBase + table classes)
│ └── db.py # Async engine + session factory
│
├── frontend/ # Single static page — no build step
│ ├── index.html # Search UI (search bar + filters + results)
│ └── app.js # fetch() calls to /quote, render results
│
├── config.py # Shared constants (DATA_DIR, model names, etc.)
├── design/ # Figma exports / wireframe screenshots
├── data/ # Pipeline artifacts (gitignored), organized by stage:
│ ├── raw/ # Scrapy output (shard_*.pq Parquet files — the cold lake)
│ ├── cleaned/ # Chunk Parquet shards
│ ├── embeddings/ # Chunk shards + mean-pooled article vectors
│ ├── topics/ # BERTopic outputs (saved model, assignments, probs)
│ ├── tone/ # Sentiment / stance scores
│ └── exports/ # Final CSV / Parquet for offline Tableau / PowerBI use
├── migrations/ # Alembic migrations (Postgres schema + matview DDL)
├── test/
├── docker-compose.yml # Postgres + API in one command (local dev)
├── railway.toml # Railway deployment config
├── pyproject.toml
├── uv.lock
├── .gitignore
└── README.md
The folders mirror the project's actual flow:
scrape/→pipeline/→viz/→api/→frontend/— the data flow, in order. Articles get scraped, processed by the NLP pipeline, turned into visualizations and exports, served by the API, and consumed by the frontend page.data/— the cold data lake, organized by lifecycle stage. Each subfolder is the output of a corresponding pipeline step. Analysts connect Tableau / PowerBI todata/exports/for offline use, or directly to PostgreSQL for live dashboards.api/services/separates HTTP from business logic. Routers do parameter binding and response shaping; services do the actual work (retrieval, reranking, topic lookup). Routers stay thin and testable.design/,migrations/,tests/— supporting folders: UX wireframes, schema migrations, and tests.
The split is intentional: pipeline/ transforms data, viz/ produces presentation artifacts. Both run offline as part of the same end-to-end refresh, but they have different concerns. The pipeline writes to PostgreSQL. The viz layer reads from PostgreSQL (or from the pipeline's intermediate outputs) and writes HTML / CSV / Parquet that the API serves and that analysts consume.
Keeping them separate means you can regenerate just the visualizations (e.g., after tweaking colors or layout) without re-running the embeddings or re-fitting BERTopic.
This pipeline is built around a two-track design: topic modeling runs at the document level (one embedding per article), while retrieval runs at the chunk level (one embedding per 800-char chunk). These two tracks serve different needs and shouldn't share an embedding granularity.
flowchart TD
A["📰 Raw Articles<br/>(Parquet shards, ~17K)"] --> B["✂️ LangChain Chunking<br/>800 chars / 160 overlap"]
B --> C["🔤 MiniLM Embeddings<br/>(~3M chunks, GPU)"]
B --> D["🧮 Mean-pool per article<br/>(~17K doc embeddings)"]
D --> E["🧪 BERTopic<br/>(cuML UMAP + HDBSCAN<br/>+ trigram c-TF-IDF)"]
E --> F["🏷️ One topic per article<br/>+ top-3 secondary topics"]
A --> G["🎭 Tone Classifier<br/>4 axes (per article)"]
C --> H[("🐘 PostgreSQL + pgvector<br/>chunks for retrieval")]
F --> H
G --> H
classDef step fill:#8957e5,stroke:#5a32a3,color:#fff
classDef bertopic fill:#c026d3,stroke:#86198f,color:#fff
classDef db fill:#336791,stroke:#1a3a52,color:#fff
classDef chunk fill:#0891b2,stroke:#155e75,color:#fff
classDef doc fill:#16a34a,stroke:#14532d,color:#fff
class A,B,G step
class C chunk
class D,E,F doc
class H db
Why two tracks? Topic modeling answers "what is this article about?" — that's a per-article question. A 380 page book like Thus Spoke Zarathustra gets one primary topic plus a few secondary themes; running BERTopic on 1.5M chunks would oversegment those themes into noise and burn ~30x more compute. Retrieval answers "which exact passage matches this query?" — that's a per-chunk question. The two needs decouple cleanly.
Note
This is the same pattern the WSJ uses for their production topic model on news articles: document-level topic assignment, with chunks reserved for downstream tasks.
main.py crawl runs a Scrapy spider named anarchy (defined in scrape/organ/). The spider seeds from the site's canonical sitemap.txt (listed in robots.txt), filters to /library/ article URLs (~16,700 texts), and fetches each page. It writes scraped articles as Parquet shards (shard_*.pq, 2,500 records per shard) to data/raw/. Each record contains article_id, url, title, author, published_at (if provided), text, and tags (the human-curated tags pulled from each article's metadata).
The live catalog is ~17,000 articles with extreme size variance — from 10-line broadsides to entire books like Thus Spoke Zarathustra. A full crawl takes several hours depending on throttle settings and network conditions.
Articles are split into overlapping chunks using LangChain's RecursiveCharacterTextSplitter:
text_splitter = RecursiveCharacterTextSplitter(
separators=[" ", ""],
chunk_size=800,
chunk_overlap=160,
)DuckDB reads the Parquet shards directly. The output is ~3M chunks across the full corpus, with article_id, title, idx, and chunk_text as the handoff contract. PostgreSQL loading is a later stage.
Why chunk at all if topic modeling is per-article? Three different granularities serve three different needs in this project:
- Articles — the unit of topic modeling and citation. BERTopic fits on document-level embeddings; users see article titles, authors, and topics alongside their results.
- Chunks (800 chars) — the unit of vector search. pgvector's cosine shortlist runs over chunks to find candidate articles fast. Chunks themselves never reach the user.
- Sentence windows (1-3 sentences) — the unit of reranking and presentation. bge-reranker-base scores sentence windows from the candidate articles, and the user's actual quote comes from this stage. A 1-3 sentence quote is what feels like a "quote."
Each chunk is embedded with sentence-transformers/all-MiniLM-L6-v2 (384-dim) on GPU via Runpod. Input Parquet files are processed one shard at a time and written to data/embeddings/chunks/*.parquet; mean-pooled article vectors are written to data/embeddings/articles.parquet. PostgreSQL ingestion is intentionally deferred to pipeline/load_db.py.
Expected runtime on a Runpod A100: ~1 hour for 1.5M chunks. This is the longest single step of the pipeline.
For topic modeling, chunks are aggregated to document-level embeddings by mean-pooling all chunks belonging to each article:
import numpy as np
import pandas as pd
# Load chunk embeddings and group by article
chunks_df = pd.read_parquet("data/embeddings/")
doc_embeddings = (
chunks_df.groupby("article_id")["embedding"]
.apply(lambda vecs: np.mean(np.stack(vecs), axis=0))
)
# Result: ~17K × 384 array (one embedding per article)Then BERTopic fits on these ~17K document embeddings — a large reduction in input size compared to fitting on millions of chunks. With cuML's GPU-accelerated UMAP and HDBSCAN, the fit completes in 5-10 minutes on an A10G instead of the multi-hour runtime chunk-level fitting would require:
from bertopic import BERTopic
from cuml.manifold import UMAP # GPU-accelerated
from cuml.cluster import HDBSCAN # GPU-accelerated
from sklearn.decomposition import PCA
from sklearn.feature_extraction.text import CountVectorizer
# Pre-reduce 384d → 50d to speed up UMAP further
pca = PCA(n_components=50)
doc_embeddings_reduced = pca.fit_transform(doc_embeddings)
umap_model = UMAP(n_components=5, n_neighbors=15, min_dist=0.0, metric="cosine")
hdbscan_model = HDBSCAN(min_cluster_size=50, metric="euclidean",
cluster_selection_method="eom", prediction_data=True)
vectorizer = CountVectorizer(stop_words="english", ngram_range=(1, 3), min_df=10)
topic_model = BERTopic(
embedding_model=None, # passing pre-computed embeddings
umap_model=umap_model,
hdbscan_model=hdbscan_model,
vectorizer_model=vectorizer,
calculate_probabilities=True,
low_memory=True,
verbose=True,
)
topics, probs = topic_model.fit_transform(article_summaries, doc_embeddings_reduced)
topic_model.save("data/topics/model", serialization="safetensors")Each article gets:
- One primary topic (the cluster its document embedding lands in)
- A
topic_probconfidence score - Top-3 secondary topics via
probs— useful for long, multi-themed articles like full books
Topic IDs land on articles.topic_id in Postgres. Chunks themselves don't carry topic IDs — they're only used for retrieval.
A configurable zero-shot classifier (MoritzLaurer/deberta-v3-base-zeroshot-v1.1-all-33 by default) produces four independent scores per article:
academicmilitanthopefulcritical
Long articles are split into tokenizer-sized windows. Window scores are weighted by token count and aggregated into one row per article in data/tone/scores.parquet. Database materialized-view aggregation remains part of the later load stage.
At query time, the API runs a two-stage retrieval — this is where the chunk-level embeddings earn their keep:
Stage 1 — pgvector cosine shortlist (api/services/retrieval.py):
- Embed the user's query with MiniLM (~5ms).
- pgvector cosine similarity on
chunks.embeddingreturns top-50 candidate chunks, filtered by tag/topic via JOIN toarticlesif specified (~50-100ms with IVFFlat index). - Map chunks back to their parent articles (~10-20 unique articles in the shortlist).
Stage 2 — sentence-window rerank (api/services/rerank.py):
- For each candidate article, fetch its full text (reconstructed from chunks with overlap deduped — see Known Issues).
- spaCy (
en_core_web_sm) sentence-tokenizes the text. - Generate sliding sentence windows of size 1-3 (so a quote can be one sentence or up to three).
BAAI/bge-reranker-basescores(query, window)pairs.- Return top-N windows with their source article, position, and the article's primary topic.
The point of windowing: chunk granularity (800 chars) is right for retrieval but wrong for presentation. A user wants a clean, citation-worthy passage — windowing gives that. The topic context shown alongside each quote comes from the article's topic_id, set during the topic modeling stage.
Latency budget on Railway CPU:
- Stage 1 (pgvector): ~100ms
- Stage 2 reconstruction + spaCy: ~200ms
- Stage 2 bge-reranker-base over ~200 windows: 2-4 seconds
- Total: ~3 seconds per query
If this proves too slow in practice, the swap to ms-marco-MiniLM-L12-v2 drops total to ~500ms at slightly lower reranking quality. Tradeoff documented.
| Stage | Where | Hardware | Runtime per full pass |
|---|---|---|---|
| Chunking | Runpod | CPU | ~10 min |
| Chunk embedding (1.5M) | Runpod | A100 GPU | ~1 hour |
| Document-level BERTopic | Runpod | A10G GPU | ~10 min |
| Tone scoring (45K) | Runpod | A10G GPU | ~30 min |
| Load into Postgres | Runpod → Railway | network | ~15 min |
| Full pipeline total | ~2 hours, ~$1 | ||
| Monthly incremental (~5% new) | ~15 min, ~$0.10 |
The project serves two audiences from the same backend, with very different surfaces.
The reader frontend is intentionally tiny: a single static index.html served by FastAPI's StaticFiles mount, styled with Tailwind via CDN. No build step, no framework, no separate dev server.
The page is wireframed in Figma before implementation — search bar, optional topic/tag filters, results panel, and a "topic context" sidebar. The Figma file lives in design/ for reference.
What the page does:
- Search bar — user types a natural-language query like "the role of mutual aid in revolution."
- Optional filters — narrow by topic or by one of the human-curated tags from the library.
- Results panel — top quote(s) returned by the Cross-Encoder, each with the source article title, author, and a link out.
- "Topic context" sidebar — for each result, the page shows which BERTopic topic the quote belongs to and the topic's top c-TF-IDF terms, so the reader sees not just the quote but where it sits in the corpus.
- "Explore topics" view — embedded BERTopic interactive HTML (
visualize_topics(),visualize_hierarchy()) plus the PyVis tag-topic network graph, all pre-generated during the offline pipeline so the runtime cost on the page is zero.
Analysts don't touch the API at all. They connect directly to PostgreSQL using the native Tableau or PowerBI connector and pull from a set of materialized views the pipeline maintains:
topic_summary— per-topic size, dominant tone, and average tone scorestag_frequency— most common human tags across the corpustopics_over_time— topic prevalence by publication period (BERTopic dynamic topic modeling output), important to note that this is based on publication date on the Anarchy Library, not article release date in the real world, which may differ significantly for older works.articles_flat— denormalized article + topic + top-tags + tone for easy filtering
For analysts working offline or sharing dashboards, the same data is exported as CSV and Parquet to the exports/ folder by pipeline/export.py.
The offline pipeline produces several interactive HTML visualizations that are reused by both audiences:
- Intertopic distance map —
visualize_topics(), embedded in the reader page and viewable as a standalone HTML. - Hierarchical topic tree —
visualize_hierarchy(), a dendrogram showing parent → child topic relationships. - Topics over time —
topics_over_time(), stream graph showing how topics rise and fall across decades. Drives a Tableau dashboard and an embedded panel in the reader page. - Tag-to-topic network graph — bipartite network where one node-set is the explicit human tags and the other is the BERTopic topics, edges weighted by co-occurrence. Tags that bridge multiple distant topics become visually obvious — the ideological connectors of the corpus.
| Method | Path | Purpose |
|---|---|---|
| GET | / |
Serves frontend/index.html (the reader page). |
| POST | /quote |
Body: {query, tag?, topic_id?, top_k?} → top sentences + sources. |
| GET | /topics |
All BERTopic topics with size, top c-TF-IDF terms, dominant tone. |
| GET | /topics/{id} |
Topic detail + sample articles (used by the "topic context" panel). |
| GET | /topics/over-time |
Topics-over-time series, used by the embedded stream graph. |
| GET | /articles/{id} |
Full article record + topic + tone scores (for the "view source" page). |
| GET | /tags |
List of all available tags (populates the filter dropdown). |
| GET | /stats |
Tag frequencies, topic sizes, tone distribution (mirrors the views). |
OpenAPI docs are auto-generated at /docs.
PostgreSQL was chosen over SQLite to enable richer analytical queries — window functions, CTEs, JSONB for flexible metadata, full-text search via tsvector, and pgvector for native vector similarity search on embeddings.
erDiagram
articles ||--o{ article_tags : has
tags ||--o{ article_tags : tagged_by
topics ||--o{ articles : contains
articles ||--o{ chunks : split_into
articles ||--o{ tone_scores : scored_by
articles {
int id PK
text url UK
text title
text author
date published_at
text body
tsvector body_tsv
int topic_id FK
float topic_prob
int_array secondary_topics
jsonb metadata
timestamptz created_at
}
tags {
int id PK
text name UK
}
article_tags {
int article_id PK
int tag_id PK
}
topics {
int id PK
text label
text_array top_terms
int size
text dominant_tone
jsonb tone_distribution
int parent_topic_id FK
}
chunks {
int id PK
int article_id FK
int position
text text
vector embedding
}
tone_scores {
int article_id PK
float academic
float militant
float hopeful
float critical
}
Important: The schema reflects the two-track design:
- Topics live at the article level.
articles.topic_idis the article's primary topic from BERTopic's document-level fit (mean-pooled chunk embeddings).secondary_topicsholds the next 2-3 best matches for long, multi-themed pieces. Thechunkstable has notopic_id— chunks exist only for retrieval. - Chunks are storage units only. Sentence-level windows are generated at query time by spaCy from reconstructed article text — they are never stored. This keeps the table count small and avoids massive duplication.
The schema is defined as SQLAlchemy 2.0 ORM models. Alembic auto-generates migrations by diffing these against the live database. The DeclarativeBase subclass is the shared registry every model inherits from.
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
from sqlalchemy import ForeignKey, Text, Integer, Date, REAL, TIMESTAMP, ARRAY
from sqlalchemy.dialects.postgresql import JSONB, TSVECTOR
from pgvector.sqlalchemy import Vector
from datetime import date, datetime
class Base(DeclarativeBase):
"""Shared model base. All tables inherit from this."""
class Topic(Base):
__tablename__ = "topics"
id: Mapped[int] = mapped_column(primary_key=True)
label: Mapped[str | None]
top_terms: Mapped[list[str] | None] = mapped_column(ARRAY(Text))
size: Mapped[int | None]
dominant_tone: Mapped[str | None]
tone_distribution: Mapped[dict | None] = mapped_column(JSONB)
parent_topic_id: Mapped[int | None] = mapped_column(ForeignKey("topics.id"))
class Article(Base):
__tablename__ = "articles"
id: Mapped[int] = mapped_column(primary_key=True)
url: Mapped[str] = mapped_column(unique=True)
title: Mapped[str | None]
author: Mapped[str | None]
published_at: Mapped[date | None]
body: Mapped[str | None] # full text (for reconstruction at query time)
topic_id: Mapped[int | None] = mapped_column(ForeignKey("topics.id"))
topic_prob: Mapped[float | None] # confidence of primary topic assignment
secondary_topics: Mapped[list[int] | None] = mapped_column(ARRAY(Integer))
# top-3 secondary topic IDs for multi-themed articles
metadata_: Mapped[dict | None] = mapped_column("metadata", JSONB)
created_at: Mapped[datetime] = mapped_column(TIMESTAMP(timezone=True))
topic: Mapped["Topic | None"] = relationship()
tags: Mapped[list["Tag"]] = relationship(secondary="article_tags")
chunks: Mapped[list["Chunk"]] = relationship(back_populates="article", cascade="all, delete-orphan")
class Tag(Base):
__tablename__ = "tags"
id: Mapped[int] = mapped_column(primary_key=True)
name: Mapped[str] = mapped_column(unique=True)
class Chunk(Base):
"""800-char chunk used ONLY for retrieval. No topic_id — topics live at article level."""
__tablename__ = "chunks"
id: Mapped[int] = mapped_column(primary_key=True)
article_id: Mapped[int] = mapped_column(ForeignKey("articles.id", ondelete="CASCADE"))
position: Mapped[int | None] # ordinal position within the article
text: Mapped[str | None]
embedding: Mapped[list[float] | None] = mapped_column(Vector(384))
article: Mapped[Article] = relationship(back_populates="chunks")
class ToneScore(Base):
__tablename__ = "tone_scores"
article_id: Mapped[int] = mapped_column(ForeignKey("articles.id", ondelete="CASCADE"), primary_key=True)
academic: Mapped[float | None]
militant: Mapped[float | None]
hopeful: Mapped[float | None]
critical: Mapped[float | None]
# article_tags is an association table — defined via Table() since it has no class behavior
from sqlalchemy import Table, Column
article_tags = Table(
"article_tags", Base.metadata,
Column("article_id", ForeignKey("articles.id", ondelete="CASCADE"), primary_key=True),
Column("tag_id", ForeignKey("tags.id", ondelete="CASCADE"), primary_key=True),
)For reference — what Alembic produces when migrating Base.metadata against an empty database. The topics_over_time table and the materialized views are managed manually (Alembic doesn't auto-generate matviews):
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE topics (
id SERIAL PRIMARY KEY,
label TEXT,
top_terms TEXT[], -- c-TF-IDF top terms from BERTopic
size INTEGER,
dominant_tone TEXT,
tone_distribution JSONB,
parent_topic_id INTEGER REFERENCES topics(id) -- BERTopic hierarchy
);
CREATE TABLE articles (
id SERIAL PRIMARY KEY,
url TEXT UNIQUE NOT NULL,
title TEXT,
author TEXT,
published_at DATE,
body TEXT,
body_tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
topic_id INTEGER REFERENCES topics(id), -- PRIMARY topic from BERTopic (document-level fit)
topic_prob REAL, -- BERTopic confidence for primary topic
secondary_topics INTEGER[], -- top-3 secondary topic IDs (multi-themed articles)
metadata JSONB DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_articles_body_tsv ON articles USING GIN (body_tsv);
CREATE INDEX idx_articles_topic ON articles (topic_id);
CREATE TABLE tags (
id SERIAL PRIMARY KEY,
name TEXT UNIQUE NOT NULL
);
CREATE TABLE article_tags (
article_id INTEGER REFERENCES articles(id) ON DELETE CASCADE,
tag_id INTEGER REFERENCES tags(id) ON DELETE CASCADE,
PRIMARY KEY (article_id, tag_id)
);
-- The retrieval unit: ~1.5M of these for a 45K-article corpus.
-- Note: chunks do NOT carry topic_id. Topics live at article level (document-level BERTopic fit).
CREATE TABLE chunks (
id SERIAL PRIMARY KEY,
article_id INTEGER REFERENCES articles(id) ON DELETE CASCADE,
position INTEGER, -- ordinal position within the article
text TEXT,
embedding VECTOR(384) -- MiniLM, used for pgvector cosine shortlist
);
CREATE INDEX idx_chunks_article ON chunks (article_id);
CREATE INDEX idx_chunks_embedding ON chunks USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 1000); -- ~sqrt(1.5M); tune based on actual count
CREATE TABLE tone_scores (
article_id INTEGER PRIMARY KEY REFERENCES articles(id) ON DELETE CASCADE,
academic REAL,
militant REAL,
hopeful REAL,
critical REAL
);
-- Topics-over-time (BERTopic dynamic topic modeling output)
CREATE TABLE topics_over_time (
topic_id INTEGER REFERENCES topics(id) ON DELETE CASCADE,
period DATE,
frequency INTEGER,
top_terms TEXT[],
PRIMARY KEY (topic_id, period)
);
-- Materialized views consumed directly by Tableau / PowerBI
CREATE MATERIALIZED VIEW topic_summary AS
SELECT t.id, t.label, t.size, t.dominant_tone,
AVG(ts.academic) AS avg_academic,
AVG(ts.militant) AS avg_militant,
AVG(ts.hopeful) AS avg_hopeful,
AVG(ts.critical) AS avg_critical
FROM topics t
JOIN articles a ON a.topic_id = t.id
JOIN tone_scores ts ON ts.article_id = a.id
GROUP BY t.id;
CREATE MATERIALIZED VIEW tag_frequency AS
SELECT t.name, COUNT(*) AS n
FROM tags t
JOIN article_tags at ON at.tag_id = t.id
GROUP BY t.name
ORDER BY n DESC;
CREATE MATERIALIZED VIEW articles_flat AS
SELECT a.id, a.title, a.author, a.published_at,
t.label AS topic_label, a.topic_prob,
ts.academic, ts.militant, ts.hopeful, ts.critical,
ARRAY_AGG(tg.name) AS tags
FROM articles a
LEFT JOIN topics t ON t.id = a.topic_id
LEFT JOIN tone_scores ts ON ts.article_id = a.id
LEFT JOIN article_tags at ON at.article_id = a.id
LEFT JOIN tags tg ON tg.id = at.tag_id
GROUP BY a.id, t.label, ts.academic, ts.militant, ts.hopeful, ts.critical;The chunks table is the central piece of the retrieval architecture, but only the retrieval architecture — it has no role in topic modeling. Topics are assigned to articles during the document-level BERTopic fit, with chunks aggregated up via mean-pooling. This keeps BERTopic's input at 45K data points (fast on GPU) while still supporting fine-grained chunk-level retrieval via pgvector.
At query time, Stage 1 (api/services/retrieval.py) runs SELECT * FROM chunks ORDER BY embedding <=> $query_vec LIMIT 50 (optionally joined to articles for topic/tag filtering). Stage 2 (api/services/rerank.py) then reconstructs the candidate articles' text and runs bge-reranker-base over sentence windows. The topic shown alongside each result comes from the article's topic_id, not from any per-chunk assignment.
The corpus updates monthly as new articles are added to the Anarchy Library. Re-running the full pipeline from scratch every month would be wasteful (and would invalidate stable topic IDs). Instead, the project uses a two-tier cadence:
- Scrape runs and writes new Parquet shards to
data/raw/. - Diff new shards against existing article URLs in Postgres → only new articles enter the pipeline.
- Chunk + embed new articles → append to
chunkstable. BERTopic.transform()— assigns existing topic IDs to new chunks. Fast, deterministic, preserves topic stability across months.- Tone scoring runs on new articles.
- Refresh materialized views (
REFRESH MATERIALIZED VIEW CONCURRENTLY topic_summary, tag_frequency).
This pipeline runs in ~10-30 minutes on Runpod GPU depending on how many new articles arrived.
Every ~3 months:
- Re-fit BERTopic on the entire chunk corpus with
fit_transform(). - New topic IDs may emerge; existing ones may merge or shift.
- A simple alignment step matches new topic IDs to old ones by c-TF-IDF term overlap, preserving continuity in the UI where possible.
- All article topic assignments are recomputed.
topics_over_timeis regenerated from scratch.
The full refit takes ~1-2 hours on a single A100. It's the only time topic IDs change.
gantt
title Pipeline cadence
dateFormat YYYY-MM
axisFormat %b
section Monthly
Scrape + chunk + embed :a1, 2024-01, 1d
BERTopic.transform :a2, after a1, 1d
Tone scoring :a3, after a2, 1d
Refresh matviews :a4, after a3, 1d
Scrape + chunk + embed :b1, 2024-02, 1d
BERTopic.transform :b2, after b1, 1d
Scrape + chunk + embed :c1, 2024-03, 1d
BERTopic.transform :c2, after c1, 1d
section Quarterly
Full BERTopic refit :crit, d1, 2024-04, 3d
Topic ID alignment :d2, after d1, 1d
The pipeline and the API run in different environments because their hardware needs are opposite.
| Environment | Where | Hardware | Runs |
|---|---|---|---|
| Offline pipeline | Runpod | A100 (embedding) / A10G (topics, tone) | Chunking, chunk embedding, document-level BERTopic with cuML, tone scoring |
| Online serving | Railway | CPU | FastAPI + Postgres + pgvector + static frontend |
The pipeline pushes results into the Railway-hosted Postgres at the end of each run (over the network — Railway exposes a Postgres connection string).
Railway provides Postgres as a managed addon. pgvector is enabled via a custom Docker image (pgvector/pgvector:pg15 or ankane/pgvector). The FastAPI service deploys from Dockerfile, with environment variables for DATABASE_URL, MODEL_PATH, etc.
A railway.toml config defines the build and run commands. The static frontend is served by FastAPI itself via StaticFiles mount — no separate CDN or hosting tier needed for portfolio-scale traffic.
| Stage | Time |
|---|---|
| MiniLM query embedding | ~5ms |
| pgvector cosine shortlist (50) | ~50-100ms |
| Fetch candidate articles + reconstruct | ~100ms |
| spaCy sentence-tokenize | ~100ms |
| bge-reranker-base (~200 pairs) | 2-4s |
| Total per query | ~3s |
Acceptable for a portfolio demo. If 3 seconds is too slow once real users try it, the swap to ms-marco-MiniLM-L12-v2 (~32M params instead of ~278M) drops total latency to ~500ms at slightly lower reranking quality. One-line model swap.
bge-reranker-base is ~1.1GB on disk and needs ~1.5GB RAM when loaded. Railway's hobby tier provides 8GB RAM, which is enough but tight. Loading the model on every request would be a disaster — it must load once at app startup via FastAPI's lifespan context manager and stay resident.
git clone https://github.com/Humbertxx/anarchy_project.git
cd anarchy_project
uv syncOn a Linux Runpod host with a compatible NVIDIA CUDA image, install the RAPIDS GPU extra:
uv sync --extra gpuThe simplest path is docker-compose:
docker-compose up -d postgresOr install PostgreSQL 15+ locally and enable the vector extension:
CREATE DATABASE anarchy_db;
\c anarchy_db
CREATE EXTENSION vector;Then set your connection string in .env:
DATABASE_URL=postgresql+asyncpg://anarchy:anarchy@localhost:5432/anarchy_db
alembic upgrade headuv run python main.py crawlThis runs the anarchy Scrapy spider: it downloads sitemap.txt, queues every /library/ article URL (~16,700), and writes Parquet shards to data/raw/. Monitor progress in the terminal or via shard count:
ls data/raw/shard_*.pq | wc -l # ~7 shards at 2,500 articles eachExpect several hours for a full crawl. Scrapy settings in scrape/organ/settings.py enable ROBOTSTXT_OBEY and AutoThrottle.
After transferring data/raw/ to the GPU machine, run the ordered workshop:
uv run python main.py pipelineThis runs chunking, embedding, topic fitting, tone scoring, and PostgreSQL loading in their required order. Resume a failed workshop from an existing artifact boundary with, for example:
uv run python main.py pipeline --from embedFor development or a small corpus on one machine, crawl and then run the pipeline with:
uv run python main.py allRuntime paths, models, batch sizes, and devices are centralized in config.py.
Monthly updates and quarterly topic-ID reconciliation are deferred.
Local tests mock model inference and do not require CUDA. On Runpod, opt into the real GPU smoke tests:
RUN_GPU_TESTS=1 uv run pytest -m gpuuvicorn api.main:app --reloadOpen http://localhost:8000/docs for the interactive Swagger UI.
python -m viz.topics # writes intertopic + hierarchy HTML to viz/outputs/
python -m viz.tag_network # writes tag-topic network HTML to viz/outputs/
python -m viz.topics_over_time # writes topics-over-time stream graph HTML to viz/outputs/
python -m viz.export # writes CSV / Parquet to data/exports/ for Tableau / PowerBIThe HTML outputs are served by FastAPI at /static/viz/* and embedded in the page's "Explore topics" panel via <iframe>. The CSV / Parquet exports feed the analyst dashboards. Tableau / PowerBI can also connect directly to PostgreSQL and read the materialized views (topic_summary, tag_frequency, articles_flat, topics_over_time) — no export step needed for live dashboards.
docker-compose upThis brings up Postgres, runs migrations, executes the pipeline if the DB is empty, and starts the API on :8000.
Full-corpus scrape and pipeline runs are in progress. The chunking, BERTopic, tone, and API layers are implemented in-repo; a complete pass (~17,000 articles → ~3M chunks) populates data/ and PostgreSQL for production retrieval.
Expected findings, based on subset runs and corpus structure:
- BERTopic should identify a stable set of topics spanning mutual aid, anti-state critique, labor organizing, ecological / green anarchism, insurrectionary writing, and historical retrospectives. With
min_cluster_size=50the count should land in the 40-80 topic range; lower values will produce too many micro-topics. - Topics-over-time analysis should show ecological / green anarchism rising in later periods, mutual-aid discourse persisting throughout, and historical retrospectives concentrating around specific anniversaries.
- Tag-to-topic network analysis should surface a small number of bridge tags (likely solidarity, direct action, autonomy) connecting otherwise distant clusters.
- The tone overlay should reveal that similar topics diverge in rhetorical posture — academic vs. militant treatment of the same subject matter.
- The two-stage retrieval design (pgvector shortlist → bge-reranker sentence windows) should surface more on-topic, citation-worthy passages than article-level matching alone. This is the central UX bet of the project.
Once the full-corpus pipeline runs, this section will be replaced with actual numbers, screenshots of the topic map, and example queries showing the search experience.
This project is licensed under the MIT License. See LICENSE for details.