arXiv Scraper for Node.js: preprint metadata as JSON for your services

This repo calls the arXiv Papers Scraper from Node.js through the official apify-client package. A backend, queue worker or serverless handler sends an arXiv query string, say cat:cs.CR for new security preprints, and gets back an array of objects with stable keys: identifier, abstract page and PDF links, title, abstract, author names, category codes, DOI, journal reference, comment and both timestamps. Apify runs the request loop and turns the Atom feed into JSON, so your code never touches an XML parser. Running node index.js also saves results.json and results.csv for a quick look.

Actor: arXiv Scraper — Search & Export Paper Metadata · Publisher: Devil Scrapes · Also in Python: arxiv-scraper-python

🔥 What you get

• arXiv's own query language, passed through: field prefixes like cat:, ti: and abs:, the operators AND, OR and ANDNOT, and submittedDate:[... TO ...] ranges all reach the official export API untouched.
• Paging you do not write: set maxResults up to 5,000 and pageSize up to 2,000; the Actor requests consecutive pages and stops early when a page comes back short.
• Flat JSON instead of Atom XML: namespaces are resolved for you, titles have their line breaks collapsed and authors arrive as an ordered list of names.
• Optional fields stay honest: doi, journal_ref and comment are null when the paper's metadata does not carry them, never an empty guess.
• Transient failures retried with backoff: a temporary error or a maintenance page is retried a few times before the run gives up and reports the failure.
• Identifies itself: requests carry a Devil Scrapes user agent, in line with arXiv's request that automated clients say who they are.

🎯 Use cases

A daily preprint digest for a team

Schedule a run each morning for the categories your team follows, render the new titles into Markdown or HTML and post them to Slack, Teams or email. Keep the last seen arxiv_id per query so only fresh papers go out.

Paper search inside your own product

Expose an endpoint that forwards a user's query to the Actor and returns the objects as they are. The categories array maps directly to filter chips in a frontend.

Keep a papers table current

Upsert rows into Postgres or SQLite keyed on the abstract URL without its version suffix. When updated changes, store the new version and keep the history for your users.

Trigger downstream jobs from new submissions

Point an Apify webhook at a serverless function. When the run succeeds, the function reads the dataset and enqueues each pdf_url for a parsing or summarisation worker.

⚡ Quick start

You need an Apify account and an API token from Console → Settings → Integrations. Apify's free plan includes monthly usage credit; this Actor's event prices are listed under Pricing below.

git clone https://github.com/devilscrapes/arxiv-scraper-javascript.git
cd arxiv-scraper-javascript
npm install
export APIFY_TOKEN=your_token_here
node index.js                       # uses input.json
node index.js my-input.json --out data --max-charge 2

The script starts the arXiv Scraper — Search & Export Paper Metadata, waits for it to finish, prints a link to the run in Apify Console and saves every result to results.json and results.csv. It exits with code 1 and the run's status message if the run fails.

🧩 The core call

The integration itself is the snippet below. call() waits until arXiv has been paged through, and listItems() returns the papers from the run's dataset.

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('DevilScrapes/arxiv-papers-scraper').call({"searchQuery": "cat:cs.AI", "sortBy": "submittedDate", "sortOrder": "descending", "maxResults": 10, "proxyConfiguration": {"useApifyProxy": false}});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

📥 Input

index.js submits input.json unchanged. The example requests ten of the latest cs.AI submissions in descending date order, enough to see every field without spending much. Change searchQuery to any arXiv expression and raise maxResults once you trust the query.

{
  "searchQuery": "cat:cs.AI",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxResults": 10,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Field	Type	Required	Default	Allowed values
searchQuery	string	yes	"cat:cs.AI"	—
sortBy	string	no	"submittedDate"	relevance, lastUpdatedDate, submittedDate
sortOrder	string	no	"descending"	ascending, descending
maxResults	integer	no	50	—
pageSize	integer	no	50	—
proxyConfiguration	object	no	{"useApifyProxy": false}	—

📤 Output

Every object stands for one paper version. doi, journal_ref and comment hold null when arXiv has no value, and authors and categories are arrays. Timestamps are UTC strings that new Date() parses without help.

Field	Type	Always present
arxiv_id	string	yes
url	string	yes
pdf_url	string	no
title	string	yes
summary	string	no
authors	array	no
primary_category	string	no
categories	array	no
doi	string or null	no
journal_ref	string or null	no
comment	string or null	no
published	string	yes
updated	string	no
scraped_at	string	no

💰 Pricing

Event	Price (USD)
Actor start	$0.2
Result emitted	$0.0015

Prices as configured on 2026-08-20. The Actor page always shows the current price. Runs are billed to the Apify account that owns the token. Use --max-charge to cap what a single run can cost.

🛠️ Working with the data in JavaScript

This script turns the saved objects into a Markdown digest grouped by primary category, newest first, with author lists shortened to three names. It needs nothing beyond Node built-ins.

import { readFile, writeFile } from 'node:fs/promises';

const papers = JSON.parse(await readFile('results.json', 'utf8'));
const byCategory = Map.groupBy(
    papers.toSorted((a, b) => new Date(b.published) - new Date(a.published)),
    (p) => p.primary_category || 'uncategorised',
);

const shortAuthors = (names = []) => (names.length > 3 ? `${names.slice(0, 3).join(', ')} et al.` : names.join(', '));
const sections = [...byCategory].map(([category, list]) => {
    const items = list.map((p) => {
        const day = p.published.slice(0, 10);
        const tag = p.journal_ref ? ' (published)' : '';
        return `- [${p.title}](${p.url}) ${day}${tag} - ${shortAuthors(p.authors)} - [PDF](${p.pdf_url})`;
    });
    return `## ${category} (${list.length})\n\n${items.join('\n')}`;
});

await writeFile('digest.md', `# arXiv digest\n\n${sections.join('\n\n')}\n`);
console.log(`digest.md: ${papers.length} papers in ${byCategory.size} categories`);

❓ FAQ

Why not call export.arxiv.org from Node directly?

You can. You would then own the paging loop, the backoff on temporary errors and an Atom parser with two namespaces. This client trades that code for one call and a small per-paper fee.

Does this work in TypeScript?

Yes. apify-client includes type definitions; declare an interface for the paper fields you use and cast items to it.

Which Node version do I need?

Node 20 or newer for the client. The digest recipe uses Map.groupBy and toSorted, which need Node 21 or later; swap in a reduce on older runtimes.

How do I keep a test run cheap?

Leave maxResults at 10 and pass --max-charge 0.3 to index.js. The cap stops the run from billing beyond that amount whatever the input says.

🧯 Troubleshooting

• Set APIFY_TOKEN first: export the token in the same shell, or put it in a local .env file your shell loads. Never commit it.
• Run finished with FAILED or ABORTED: open the printed Console link; the run log says why. Input validation errors name the field.
• Fewer rows than expected: a --max-charge cap also limits how many results can be charged. Raise the cap or lower the limits in your input.
• Something looks wrong in the data: open an issue on the Actor page so the maintainers see it.

🔗 More from Devil Scrapes

• arXiv Scraper — Search & Export Paper Metadata on Apify Store
• Python version of this example
• All Devil Scrapes Actors
• Apify web scraping cookbook

Related Actors:

• Crossref Works Scraper — DOIs, Authors & Citations — DOI-registered journal versions of the same research.
• OpenAlex Works Scraper — Papers, Citations & Authors — citation counts and institutions across all disciplines.
• ORCID Researcher Profiles Scraper — public researcher records to enrich author lists.

---

MIT licensed example code. The Actor itself runs on Apify and is maintained by Devil Scrapes. Not affiliated with arXiv or Cornell University.