🤖 OpenRouter Model Census

A complete analysis tool for all 316+ AI models on OpenRouter—featuring fast scraping, Python-style data analysis, and beautiful Observable Plot visualizations. Get pricing insights, provider comparisons, and model capabilities in an interactive dashboard.

🚀 Quick Start

# Clone and install
git clone https://github.com/your-username/openrouter-scrape
cd openrouter-scrape
npm install

# Run complete pipeline (scrape + analyze + serve dashboard)
npm run full-pipeline

That's it! Your dashboard opens at http://localhost:8080 with fresh data on all 316+ models.

✨ Features

⚡ Fast scraping - All 316 models in under 1 second
📊 Beautiful charts - 7 interactive Observable Plot visualizations
💰 Pricing insights - Find the cheapest models, price vs context analysis
🏢 Provider analysis - Market share, strategy mapping, capabilities
📱 Responsive design - Works great on mobile and desktop
🔄 Easy updates - Re-run anytime for fresh data

What gets collected

Layer 1 — Models (`GET /api/v1/models`)

Identity: id, name, canonical_slug, created, description
Architecture: input_modalities, output_modalities, tokenizer, instruct_type
Aggregates: context_length (model-wide), supported_parameters (union across providers)
Pricing (as published): prompt, completion, request, image, web_search, internal_reasoning, input_cache_read, input_cache_write
Top provider summary: context_length, max_completion_tokens, is_moderated Notes: supported_parameters at model level is a union, not guaranteed at any single endpoint. (OpenRouter)

Layer 2 — Provider Endpoints (`GET /api/v1/models/:author/:slug/endpoints`)

Per provider that hosts the model:

provider_name, name (endpoint), status, uptime_last_30m
Limits: context_length, max_prompt_tokens, max_completion_tokens, quantization
Capabilities: endpoint-level supported_parameters
Pricing overrides: prompt, completion, request, image Use these to resolve real availability and price for your route. (OpenRouter)

Layer 3 — Optional Generation Telemetry (`GET /api/v1/generation` or inline usage accounting)

If you have generation IDs or enable usage accounting in your workloads:

Identity & routing: model, provider_name, origin, upstream_id
Tokens (native): native_tokens_prompt, native_tokens_completion, native_tokens_reasoning
Tokens (gateway): tokens_prompt, tokens_completion
Cost: total_cost, cache_discount, upstream_inference_cost
Timings: latency, generation_time, moderation latency
Termination: finish_reason, native_finish_reason
Media/search counters; cached token counts (reads) This enables empirical latency/cost analyses grounded in the provider’s native tokenizer. (OpenRouter)

Outputs (immutable, analysis-ready)

artifacts/YYYY-MM-DD/
- models.jsonl (one model per line; raw API objects)
- endpoints.jsonl (provider-expanded rows)
- joined.parquet (denormalized: model ⨯ endpoint)
- telemetry.jsonl (optional generation/usage rows)
- schema/*.json (JSON Schema for each file)
- MANIFEST.sha256 (file → SHA-256)
- FETCH_METADATA.json (request URLs, headers, etags, timestamps, versions)
- CITATION.cff (how to cite this snapshot)
- NOTICE.txt (license + terms provenance)
All files compressed with zstd if --zstd is set.

Install

# Node 20+ recommended
pnpm i -g openrouter-census   # or: npx openrouter-census

Local dev:

git clone https://github.com/yourorg/openrouter-census
cd openrouter-census
pnpm install

Configuration

# Optional: required only if you fetch protected usage for your own generations
export OPENROUTER_API_KEY=... 

# Rate limits (Bottleneck): default is conservative; override as needed
export CENSUS_RESERVOIR=100        # max requests per window
export CENSUS_INTERVAL_MS=60_000   # window length
export CENSUS_MAX_CONCURRENCY=8    # parallelism

Tip: tune Bottleneck’s reservoir + interval for burst/steady control. Keep headroom for Cloudflare/DDOS protections and any per-account limits. (OpenRouter, DEV Community)

Usage (CLI & TUI)

openrouter-census tui

TUI features

Live counters: queued / inflight / completed / failed
Throttles: adjust concurrency & reservoir at runtime
Filters: by provider, family, modality, supports(param)
Export panel: write JSONL/Parquet + schemas + manifest
Retry controls: backoff/rehydrate failed requests

Non-interactive

# Full scrape → immutable snapshot directory
openrouter-census run --out artifacts/$(date +%F) --zstd

# Fetch with telemetry (ids from stdin) and merge
cat ids.txt | openrouter-census usage --append artifacts/2025-08-21/telemetry.jsonl

Data model (schemas)

schema/models.schema.json – exact API fields from /models with permissive additionalProperties: true
schema/endpoints.schema.json – rows keyed by (model_id, provider_name, endpoint_name)
schema/telemetry.schema.json – union of /generation + inline usage accounting fields
schema/joined.schema.json – normalized and namespaced (model.*, endpoint.*, price.*, limits.*)

Schemas are versioned; each run stores the exact schema copy used to validate outputs.

Reproducibility & provenance

Deterministic traversal: stable sort of model IDs; endpoint requests queued in lexical order.
Version pins: pnpm-lock.yaml checked in; run logs emit Node + package versions.
Raw preservation: we persist unmodified API payloads alongside normalized tables.
Integrity: SHA-256 digests and FETCH_METADATA.json (timestamps, etags, cache status).
Citations: CITATION.cff includes title, version, DOI placeholder, and snapshot date.
License/Terms capture: NOTICE.txt records the doc URLs and access dates for API docs.

Limits, caveats, ethics

supported_parameters at model level is union; intersect with endpoint capabilities for truth in UX or routing guarantees. (OpenRouter)
Endpoint uptime_last_30m is a short pulse, not an SLA. Prefer aggregates if you collect telemetry. (OpenRouter)
Pricing may differ per endpoint; do not assume model-level pricing applies to your chosen provider. (OpenRouter)
Respect rate limits & terms; don’t stress shared free endpoints. Consider backoffs and caching. (OpenRouter)

Programmatic API (library usage)

import { census } from "openrouter-census";

const run = await census({
  outDir: "artifacts/2025-08-21",
  concurrency: 8,
  reservoir: 100,
  intervalMs: 60_000,
  fetchTelemetry: false,
});
console.log(run.summary);

Minimal pipeline spec

Models pass
- GET https://openrouter.ai/api/v1/models → write raw → validate → extract rows. (OpenRouter)
Endpoints pass
- For each {author}/{slug}, GET /api/v1/models/:author/:slug/endpoints → write raw → validate → rows. (OpenRouter)
(Optional) Telemetry pass
- For provided generation IDs or inline usage payloads, normalize and append to telemetry.jsonl. (OpenRouter)
Join & export
- Create joined.parquet + schemas + manifest + checksums.

What you can visualize later (suggestions)

Price vs context (per endpoint)
Parameter support heatmap (models × params; intersected, not union)
Short-window reliability (endpoint uptime)
Observed latency & effective $/1k from telemetry (native tokens)

References

Models API & schema. (OpenRouter)
List models (union of supported_parameters). (OpenRouter)
Per-model endpoints (provider-specific limits/pricing/uptime). (OpenRouter)
Usage accounting (native tokens, cached tokens, reasoning tokens). (OpenRouter)
Precise token accounting & /generation. (OpenRouter)
Reasoning tokens overview. (OpenRouter)
OpenRouter limits (guidance for throttling). (OpenRouter)
Bottleneck patterns for Node rate limiting. (DEV Community)

License

MIT for code. Data artifacts inherit upstream terms; see NOTICE.txt.