Microproduct data guideintermediate

arXiv Preprint Metadata

Open preprint descriptive metadata for building topic alerts and research discovery tools that link back to arXiv.

Start the guide Open official source

At a glance

Difficulty
Intermediate — some data preparation helps
Size
Small · ≤0.1 GB
Formats
Atom XML
Access
API
API key
Not required
Provider
arXiv
Updates
Daily
Data terms
arXiv API terms; descriptive metadata under CC0 1.0

From source to product signal

Test a product idea in four steps

The arXiv API returns Atom feeds of e-print metadata. Start with one category and a handful of recent results. Metadata is CC0; full-text PDFs remain under each paper's license, and a preprint is not a peer-reviewed journal record.

1

Check the setup

  • Python 3.10 or newer
  • A notebook environment such as Jupyter or Google Colab
  • An internet connection
2

Access the data

  1. 1.Read the arXiv API terms, including the one-request-every-three-seconds limit.
  2. 2.Query one category with max_results set to a small number.
  3. 3.Link users to arXiv abstract pages rather than mirroring PDFs.
Open official source
3

Run the Python example

Install the packages, then run the notebook cell.

python -m pip install pandas requests

import xml.etree.ElementTree as ET
import pandas as pd
import requests

response = requests.get(
    "https://export.arxiv.org/api/query",
    params={
        "search_query": "cat:cs.LG",
        "start": 0,
        "max_results": 5,
        "sortBy": "submittedDate",
        "sortOrder": "descending",
    },
    timeout=30,
)
response.raise_for_status()
ns = {"atom": "http://www.w3.org/2005/Atom"}
root = ET.fromstring(response.text)
rows = []
for entry in root.findall("atom:entry", ns):
    rows.append({
        "id": entry.findtext("atom:id", default="", namespaces=ns),
        "title": " ".join((entry.findtext("atom:title", default="", namespaces=ns) or "").split()),
        "published": entry.findtext("atom:published", default="", namespaces=ns),
    })
preprints = pd.DataFrame(rows)
preprints["retrieved_at_utc"] = pd.Timestamp.now(tz="UTC")
print(preprints)
4

Test a useful signal

Watch recent machine-learning preprints

List five recent cs.LG submissions and their arXiv identifiers.

  1. 01Parse the Atom feed and confirm each entry has an id, title, and published date.
  2. 02Keep only metadata fields and link to the abstract URL rather than downloading PDFs.
  3. 03State that arXiv metadata is CC0 while paper files follow each submitter's license, and that preprints are not peer-reviewed publication records.

Dataset details

arXiv is an academic source. Last verified 2026-08-14. Temporal coverage: 1991-present.

Theme

Research & Reference

Domains

Scholarly CommunicationPreprintsResearch

Data types

Bibliographic DataText

Tasks

Literature MonitoringTopic ResearchAlerting

Geography

Global

Formats

Atom XML

Provider

arXiv

Data terms

arXiv API terms; descriptive metadata under CC0 1.0