t1k:datastore-cost
| Field | Value |
|---|---|
| Module | t1k-devops |
| Version | 4.1.3 |
| Effort | low |
| Tools | — |
Keywords: aggregate, cloudflare, d1, D1 pricing, d1-cost, datastore design, datastore-cost, event pipeline, hot counter, index review, ingestion, object storage, pre-deploy cost, r2, raw event, rollup, rows-read, rows-written, scan cost, workers
How to invoke
Section titled “How to invoke”/t1k:datastore-coststore design, schema review, ingestion pipelineDatastore Cost Discipline — Workers / D1 / R2
Section titled “Datastore Cost Discipline — Workers / D1 / R2”Prevent the $430/month class of surprise: writing raw rows to a row-billed managed datastore (D1), one scan away from the next surprise on the read side. Decides what a row costs before it is ever written.
Apply when: architecting an ingestion/data path, reviewing a D1 schema or a query that scans, or pre-deploy cost-checking any Worker that touches D1/R2.
Pre-flight — announce before you write or scan
Section titled “Pre-flight — announce before you write or scan”Mandatory, before executing any command that writes to or scans a managed datastore
(D1 INSERT/UPDATE/DELETE/batch, cf d1 query, a migration, a bulk
R2 put/list):
state the estimated row/operation count and its dollar cost at the current published rate,
then act. Unknown volume is itself the warning — say “volume unknown, could be unbounded”
rather than staying silent.
Rates to do the arithmetic inline (retrieved 2026-08-29 — full table + source URLs in the Pricing reference under Pre-deploy cost check below):
| Operation | Included | Overage |
|---|---|---|
| D1 rows written | 50M/mo | $1.00 / million rows |
| D1 rows READ — scanned, not returned | 25B/mo | $0.001 / million rows |
| R2 Class A (write/list) | — | $4.50 / million requests |
| R2 Class B (read) | — | $0.36 / million requests |
Reads are not exempt. D1 bills rows the query planner scans, not rows it returns, so an
unindexed WHERE is a cost event too — estimate the table’s row count, not the expected
result-set size. If the volume can’t be bounded, smoke a small N first and surface the
measured per-row cost before scaling (modules/t1k-base/rules/preview-first-batch.md).
Mandatory Cost Report — every change, every time
Section titled “Mandatory Cost Report — every change, every time”Standing instruction (user, 2026-08-31): report the cost impact of every change you make. The Pre-flight announcement covers a command you are about to run. This covers a change you are about to ship — a schema edit, a new query, a sync cadence, an index, a mirrored table. Emit it unprompted; never wait to be asked.
Four elements, all required:
- Before/after rows + dollars. Rows written and rows scanned per run, before vs after, priced at the rates above. Cannot estimate? Say “volume unknown, potentially unbounded” — that is the report, not a reason to skip it.
- Monthly run-rate projection. Extrapolate per-run to $/day and $/month. A per-tick figure is not a cost signal on its own: a 448.8K rows/hour floor reads as trivial per tick and is ~$323/month (measured 2026-08-31). Always project before judging a number small.
- Read-scan cost, not just writes. Rows the planner scans for any new or changed query —
unindexed
WHERE,COUNT(*)on a growing table, a wide date range over a large fact table. Per Read-scan review; the number to report is the table’s row count, not the result-set size. - Post-deploy verification. After any cost-affecting deploy, measure actual rows/hour against
what you predicted and report the delta. Never claim a cost fix worked without this
measurement — a daily total cannot tell “the fix landed” apart from “yesterday had spikes”.
Method:
references/measuring-d1-cost.md.
A partial replace still owes elements 2 and 4. The skip-gate in Recurring full-replace mirror/sync jobs removes writes it proves unnecessary, but leaves a floor wherever the source genuinely changed each tick; that floor is the residual to project and then measure, not to assume away. Closing it further is step 3 of that section (upsert genuine deltas) — a separate change, owing its own report.
Decision Tree
Section titled “Decision Tree”| Intent | Path |
|---|---|
| “I’m about to write/scan a managed datastore” | Pre-flight — mandatory, do this first |
| “I changed something — what does it cost?” | Mandatory Cost Report — all four elements, unprompted |
| “Where should this data live?” | Choose the right store |
| “Design the ingestion pipeline” | Aggregate-then-write pipeline |
| “I have a scheduled mirror/sync/refresh job” | Recurring full-replace mirror/sync jobs |
| “Is this query/schema going to cost me?” | Read-scan review |
| “The read bill jumped” / “a warmer, poller or report reads D1” | Read-scan review → references/read-multipliers.md |
| “Before I deploy, estimate the bill” | Pre-deploy cost check |
| “Size a Cloudflare backend for a game (per-player DO vs D1)” | references/per-player-do-game-backend.md |
| “What is requests-per-DAU?” (before any such costing) | references/request-volume-audit.md |
Choose the right store
Section titled “Choose the right store”- Raw events / payload / blobs (analytics, logs, fails) → object store (R2-class) or compute at query time. If a row is not reduced, it does not belong in a row-billed store.
- Rollup / aggregate / KPI / config → D1-class row store. Config that grows with the
product (allow-lists, audiences, feature flags, tenants) is the case for a table rather than env +
redeploy — see
skills/t1k-cloudflare/references/config-as-data.md. - Hot counter (~user count, DAU, fails) → maintained counter column/row, never
COUNT(*)a growing table on a hot path. - Ephemeral → compute locally; do not persist at all.
Aggregate-then-write pipeline
Section titled “Aggregate-then-write pipeline”Do NOT write one row per event. Insert late; insert reduced.
- Buffer → batch → aggregate in the Worker (in-memory, or KV if state must survive).
- Roll up to fixed windows (5m / 1h / 1d) with a stable bucket key.
- Upsert the bucket row (
INSERT … ON CONFLICT DO UPDATE). - Stream the raw event to object storage keyed by
id/timestamp(R2 puts, cheap), for backfill and audit. - Write only the aggregate count / sum / distinct-set size / last-seen. If the raw row must exist to answer a future question, it lives in R2 — not D1.
Recurring full-replace mirror/sync jobs
Section titled “Recurring full-replace mirror/sync jobs”Aggregating the input is not enough if the write is a full-refresh cron: a periodic job that
strips and rebuilds an entire partition on every tick, whether or not the source moved. This is
its own cost anti-pattern — the data is already aggregated (a Postgres → D1 mirror of rollups),
so “aggregate-then-write” lets it through. The cost is not wrong granularity; it is
rewriting an unchanged partition on every tick. Because DELETE and INSERT both bill as
rows written, a job re-inserting ~1.2K rows to replace ~287K deletes reached 149.47M rows/day
(~$4,434/mo) within a week of going live.
Gate the replace step behind a cheap content comparison before paying the DELETE+INSERT cost:
- Compare, don’t assume. Before replacing a partition, read a cheap fingerprint of what the
target already holds — count plus summed measures, or a watermark (max timestamp).
If the source fingerprint equals the target’s,
skip. - Keep the comparison indexed. A covering index makes the pre-check a handful of rows read against ~287K rows written avoided — reads bill ~1000× cheaper than writes.
- Use an upsert for genuine deltas, not an unconditional wipe-and-reload — write only the rows that actually changed.
Why this is safe by construction: the comparison can only remove writes it proves unnecessary — worst case is zero savings, never a regression. It also correctly skips partitions the target has never held (a new partition generation), which is correct behaviour, not a gap. Measured outcome: 100% skip on every already-synced partition across two production ticks, ~87% total reduction, $4,434/mo → ~$550/mo. Treat “full-refresh cron” as its own anti-pattern alongside raw-event ingestion, not a variant of it.
Read-scan review
Section titled “Read-scan review”Row-billed stores charge rows scanned, not returned. An unindexed WHERE over an N-row
table bills N rows read per query.
- Index every filtered column — the D1 index write is offset, in nearly all cases, by the scan it avoids.
LIMITevery scan; neverSELECT *over wide tables. Reading 1 wide row is 1 row scanned, but a fan-out SELECT scans the whole table.- Never
COUNT(*)a growing table on a hot path. Maintain a counter row/column and read that. - Check the query
meta: every D1 response carriesrows_readandrows_written(many SDKs exposemeta.rows_read). Total up your daily row-read spend during load tests. - Cost a read as runs/hour × rows/run, per caller. A 200M-row query is a rounding error once and ~$2,800/month under a 7-minute warmer fanned out per account (see references). Multiply cadence × fan-out × rows/run × attempts before shipping any scheduled, warm, poll or report caller.
- Never recompute a closed period. A past window is immutable: compute it once, store the result row, and recompute only the open window (plus the previous one when a delta carries across the boundary).
- Bound the scan to what is shown. A 98-day floor feeding a view of the current week bills for the ~93% it discards.
- A
LEFT JOINonto a CTE or subquery can scan once per outer row. SQLite keeps nosqlite_stat1until something runsANALYZE, and a D1 database is no exception (none existed on the incident database), so the planner may skip the automatic index and read N × M rows. Look forSCAN (subquery-N) LEFT-JOINinEXPLAIN QUERY PLAN(it does not execute), and prefer shapes that need no automatic index (aUNION ALLor one window pass). AMATERIALIZEDhint adds no index. CI gate, equivalence proof and the window trick:references/plan-shape-ci-gate.md. - Assume a client timeout does not stop D1. Cloudflare does not document what an aborted REST request costs; in the recorded incident, runs the client had timed out still showed in analytics with no error and full rows read (observed, not documented). Price each retry as a full scan. A slow query needs a cheaper shape, not a retry.
Incident, arithmetic, equal-result patterns and the alert/budget-test mechanisms:
references/read-multipliers.md.
Pre-deploy cost check
Section titled “Pre-deploy cost check”Estimate before you deploy. A Worker that writes a raw row per event on a paid plan with something like 50M write rows included is a bill waiting to be discovered.
- Estimate daily event volume × rows written per event. This is the write spend.
- Estimate rows scanned by the hottest query × queries/day. An unindexed
WHEREon your largest table is this number. Keep it under the included read budget, then pad. - Verify the running query’s actual cost in staging by reading
meta.rows_read. - For every scheduled, warm, poll or report caller, multiply cadence × fan-out × rows/run × attempts. This product is what surprises, not the per-request figure.
- Raise the estimate by 3× for a freshly-shipped ingestion path. Fat-fingered loops and bot traffic are what the included budget does not survive.
- Re-check quarter-over-quarter — a growing table makes every unindexed scan linearly more expensive. Indexes are the fix, not a “we’ll see” attitude.
Pricing reference (retrieved 2026-08-29):
- D1 rows written: first 50M/mo included, then $1.00/M; rows read: first 25B/mo included, then $0.001/M — writes ~1000× reads https://developers.cloudflare.com/d1/platform/pricing/
- D1 indexes best-practices (scan billing + index mitigation) https://developers.cloudflare.com/d1/best-practices/use-indexes/
- R2: no egress; Class A $4.50/M · Class B $0.36/M (per million requests) · storage $0.015/GB-mo https://developers.cloudflare.com/r2/pricing/
Worked example
Section titled “Worked example”A Worker ingesting 6M events/day wrote 6M raw rows/day to D1 → 180M rows/mo → ~$130/mo in
writes alone. Rework: aggregate to 5-minute buckets (~8.6K bucket rows/day → ~260K rows/mo →
well inside the included 50M), raw payloads streamed to R2. Month-end R2 cost for ~180M small
objects: roughly $25–60 (Class A puts) + a few cents storage. D1 read spend near zero because
every lookup is by an indexed bucket key under LIMIT.