Microproduct data guideintermediate

GitHub Archive

Hourly public GitHub event archives for building repository-activity monitors without using BigQuery.

Start the guide Open official source

At a glance

Difficulty
Intermediate — some data preparation helps
Size
Small · 0.02–0.2 GB
Formats
JSON, GZIP
Access
Download
API key
Not required
Provider
GH Archive
Updates
Near real time
Data terms
GitHub Terms of Service for public events, with GH Archive access terms

From source to product signal

Test a product idea in four steps

GH Archive stores hourly gzip JSON of public GitHub events. Start with one UTC hour and stream the first events rather than loading the whole file. Do not use BigQuery for this guide, and public events still fall under GitHub's terms.

1

Check the setup

  • Python 3.10 or newer
  • A notebook environment such as Jupyter or Google Colab
  • An internet connection
2

Access the data

  1. 1.Open gharchive.org and note the hourly URL pattern.
  2. 2.Stream one recent hourly gzip.
  3. 3.Keep event type and repository name from the first records.
Open official source
3

Run the Python example

Install the packages, then run the notebook cell.

python -m pip install pandas requests

import gzip
import json
import pandas as pd
import requests

response = requests.get(
    "https://data.gharchive.org/2026-08-17-15.json.gz",
    stream=True,
    timeout=60,
)
response.raise_for_status()
response.raw.decode_content = True
rows = []
with gzip.GzipFile(fileobj=response.raw) as archive:
    for line in archive:
        event = json.loads(line)
        rows.append(
            {
                "type": event.get("type"),
                "repo": (event.get("repo") or {}).get("name"),
            }
        )
        if len(rows) >= 50:
            break
events = pd.DataFrame(rows)
print(events.head())
4

Test a useful signal

Count event types in one GitHub hour

Test whether a streamed hour can power a bounded activity monitor.

  1. 01Stream the first 50 events from one hourly gzip and keep type plus repository.
  2. 02Count event types in that sample.
  3. 03Explain that the sample is not the full hour, BigQuery is out of scope here, and GitHub terms apply to public event data.

Dataset details

GH Archive is a community source. Last verified 2026-08-18. Temporal coverage: 2011-present hourly public GitHub events.

Theme

Technology & Cybersecurity

Domains

Open SourceSoftwareAttention Measurement

Data types

Event DataJSON Lines

Tasks

Activity MonitoringTrend AnalysisRepository Research

Geography

Global

Formats

JSONGZIP

Provider

GH Archive

Data terms

GitHub Terms of Service for public events, with GH Archive access terms