Retail product prices sampled and trended with Python.

Treat a large retail price CSV as an evidence package, not a spreadsheet: verify its manifest and hash, stream it once, keep every price in its source currency, and export small dated samples instead of forcing evaluation through the full file. This worked example derives a bounded market sample and a barcode price history from a 238 MB community-contributed dataset using only the Python standard library.

The example uses the tagged and SHA-256-pinned August 8, 2026 WebTruffle retail product prices release, the latest published edition checked on August 19. Its normalized CSV holds 270,334 observed prices across 129,150 distinct barcodes, 115 reported countries, and 85 reported currencies. The edition is a cumulative snapshot with an August 8 cutoff — eleven days before this article's publication — so every result is labeled with that date rather than presented as live shelf pricing.

The default worked example selects one currency market — France, EUR — cuts a 2,550-row breakfast-category sample observed in 2026, and builds the France/EUR price history of barcode 3017620422003 (Nutella): 148 observations across 37 retailer names between February 26, 2021 and August 7, 2026.

Retail product price data with Python: the short answer

Use this sequence:

1. Pin the release tag, target date, manifest byte count, and manifest SHA-256. Do not begin from a moving latest URL.
2. Verify the manifest bytes and hash first; only then trust the file declarations inside it.
3. Verify the CSV's 238,009,759 bytes and SHA-256, then require all 40 headers in the exact published order.
4. Read the CSV with utf-8-sig and stream it row by row. Never load a quarter-gigabyte file into a dataframe to answer a filterable question.
5. Parse price with Decimal, observed_date with date.fromisoformat, and categories_tags with a JSON parser. Keep barcodes and identifiers as text.
6. Treat price_is_discounted as the exact strings True and False; a truthiness check counts every row as discounted.
7. Select one country and one currency before computing any price statistic. A country filter alone does not create a single-currency market.
8. Audit metadata coverage before filtering on metadata: 21,664 rows carry a barcode and price but no catalog name.
9. Export small, dated, source-linked samples. Keep source_url, source_license, and the edition date on every row.
10. Build trends per barcode inside one currency; report the observation count, retailer count, and date spread beside every min/median/max.
11. Never convert currencies, infer unit prices, or blend observation dates with pipeline timestamps.
12. Write a provenance receipt with the filter values, input hashes, output hashes, and runtime identity last.

If you want to browse or download the current normalized product, start with the retail product catalog and prices dataset. Use the recipe when a sample, comparison, or handoff must be repeatable.

Pin the edition before opening the CSV

A release tag is a convenient address, but GitHub currently reports these releases as mutable. The hash is the mutation detector. Pin both the manifest and the file selected through that verified manifest.

The current edition for this example is:

release tag       2026-08-08
target date       2026-08-08
generated at      2026-08-09T13:32:23Z
schema version    1.0
manifest bytes    8,608
manifest SHA-256  1d9c13c7e8cc506873b6ff0a7e7f3ae28c860ac23f10c52f3001cf6c7e13021e
CSV bytes         238,009,759
CSV SHA-256       49e2a5a9944c5e2c905aef968bfa7d4d58dc6c9ef6d558c28916cf4a25a29f0e

The manifest declares the edition's identity: 270,334 records, 129,150 products, 6,153 locations, 115 countries, 85 currencies, 23,999 discounted observations, and an observation window from 2010-07-08 to 2026-08-08. It also records the upstream extract — the Open Prices prices.jsonl.gz dump of 19,225,729 bytes with its own SHA-256 — and the documented exclusions applied to reach the retained population: 9,556 non-product observations, 6,885 duplicates, 171 invalid currencies, 15 invalid dates, 11 invalid prices, and 1 missing identity, out of 286,973 raw rows.

Those clocks mean different things. The target date is the edition's analytical cutoff. generated_at records when the release was built. A contributor-reported observation date, an Open Prices record timestamp, a pipeline first-seen time, and the moment your script runs are four separate clocks, and the time register section keeps them apart.

Run on macOS, Linux, or WSL

curl -fsSLO https://www.webtruffle.com/examples/retail-product-prices-data-python.py
python3 retail-product-prices-data-python.py \
  --data-dir ./evidence/retail-product-prices-2026-08-08

Run on Windows PowerShell

Invoke-WebRequest `
  -Uri https://www.webtruffle.com/examples/retail-product-prices-data-python.py `
  -OutFile retail-product-prices-data-python.py
py -3.11 .\retail-product-prices-data-python.py `
  --data-dir .\evidence\retail-product-prices-2026-08-08

The default market sample is France, EUR, category en:breakfasts, observed in 2026, and the default barcode history is 3017620422003. Pass different filters when the question changes:

python3 retail-product-prices-data-python.py \
  --country "United States" --currency USD \
  --category "en:beverages" --sample-year 2025 \
  --product-code 0292475000000

A filter that matches zero rows fails instead of producing an authoritative-looking empty sample. When non-default filters are active, the recipe keeps its global oracles — record counts, identity uniqueness, discount reconciliation — and skips the default-filter oracles.

Verify the manifest, file, and schema

The trusted starting point is a manifest hash supplied outside the manifest itself. Once the manifest matches, its files declaration can authenticate the CSV:

import hashlib
import json
from pathlib import Path

def sha256_and_size(path: Path) -> tuple[str, int]:
    digest = hashlib.sha256()
    size = 0
    with path.open("rb") as source:
        for chunk in iter(lambda: source.read(1024 * 1024), b""):
            digest.update(chunk)
            size += len(chunk)
    return digest.hexdigest(), size

manifest_path = Path("manifest-2026-08-08.json")
actual_hash, actual_bytes = sha256_and_size(manifest_path)
assert actual_bytes == 8_608
assert actual_hash == (
    "1d9c13c7e8cc506873b6ff0a7e7f3ae2"
    "8c860ac23f10c52f3001cf6c7e13021e"
)

manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
assert manifest["target_date"] == "2026-08-08"
assert manifest["record_count"] == 270_334

For network downloads, the recipe creates a uniquely named temporary file in the destination directory, hashes and counts while streaming, aborts as soon as the bytes exceed the declared 238,009,759, and uses os.replace only after both checks pass. A predictable .part filename would allow concurrent runs to interfere with each other.

The CSV uses 40 normalized fields. Validate the header before interpreting a column:

import csv

with open(
    "retail-product-prices-2026-08-08.csv",
    encoding="utf-8-sig",
    newline="",
) as source:
    reader = csv.DictReader(source)
    assert tuple(reader.fieldnames or ()) == manifest["record_fields"]

utf-8-sig is not optional here: the published file begins with a byte-order mark, and a plain utf-8 read turns the first header into \ufeffid, silently breaking every id lookup.

All CSV values begin as strings. Keep product_code, source_price_id, and id that way. Parse prices with Decimal and observation dates with date.fromisoformat. Do not let a dataframe infer away leading zeros in a barcode or round a published decimal through a binary float.

Three fields deserve special handling:

import json

def parse_categories(value: str) -> list[str]:
    text = value.strip()
    if not text:
        return []
    parsed = json.loads(text)
    if not isinstance(parsed, list) or not all(
        isinstance(item, str) for item in parsed
    ):
        raise ValueError("expected a JSON array of strings")
    return parsed

def is_discounted(value: str) -> bool:
    return value == "True"

categories_tags, brands_tags, and labels_tags are JSON arrays encoded inside CSV cells; comma-splitting corrupts tags that legitimately contain punctuation. In this edition every non-empty array cell parses cleanly, and 48,382 rows carry an empty array. price_is_discounted is serialized as the capitalized strings True and False; a truthiness test such as if row["price_is_discounted"] evaluates the string "False" as true and would mark all 270,334 rows discounted.

Stream the 238 MB file instead of loading it

The file is large enough to punish casual loading, and small enough that a disciplined single pass is fast. The recipe streams the CSV with csv.DictReader, keeps only bounded aggregates plus the rows that match the requested filters, and finishes the full pass — verification, coverage audit, sample, and barcode history — in seconds on ordinary hardware.

from collections import Counter, defaultdict
from decimal import Decimal

seen_ids: set[str] = set()
product_codes: set[str] = set()
coverage: dict[str, Counter] = defaultdict(Counter)
sample_rows: list[dict[str, str]] = []
history_rows: list[dict[str, str]] = []

with open(csv_path, encoding="utf-8-sig", newline="") as source:
    for row in csv.DictReader(source):
        if row["id"] in seen_ids:
            raise RuntimeError("duplicate id breaks the grain statement")
        seen_ids.add(row["id"])
        product_codes.add(row["product_code"])
        price = Decimal(row["price"])
        # ...bounded aggregates and filter-matched rows only...

Two identifiers are both unique in this edition and serve different purposes. id is the normalized record identity. source_price_id is the Open Prices price identifier, and it is the key embedded in each row's source_url — for example https://prices.openfoodfacts.org/api/v1/prices/84616. Keep both in exports.

The stream pass also enforces the declared grain: one retained Open Prices PRODUCT observation for one barcode, location, and observation date. Every row must carry a barcode, a parseable price, and a parseable observation date; the recipe halts on the first violation rather than averaging over it. record_type is product_price_observation for all 270,334 rows, and location_type splits into 269,560 OSM physical locations and 774 ONLINE observations.

One field is systematically empty: price_per. The product preserves source currencies and reported prices without conversion or inferred unit pricing, so any unit-price analysis is your computation, with your quantity-parsing assumptions, stated explicitly.

Respect the currency boundary

The edition reports prices in 85 distinct currency codes. Summing or averaging across them produces a number with no economic meaning, and converting them requires an exchange-rate model the dataset does not provide. The working rule is simple: one currency per statistic.

The boundary matters even inside a single country because contributors report the currency they saw. France contributes 183,396 observations: 183,320 in EUR, plus 76 rows in six other codes — 38 XPF, 33 ADP, 2 BGN, and one each of MAD, BAN, and CHF. Some of those codes are legacy or unusual; the dataset preserves them as reported rather than silently reclassifying them. That preservation is a feature: you decide whether a row belongs in your market, with the evidence visible.

def in_market(row: dict[str, str]) -> bool:
    return row["country"] == "France" and row["currency"] == "EUR"

Select the market before any aggregation, carry the currency code into every exported row, and label every published statistic with both the country and the currency that produced it.

Audit metadata coverage before analyzing

Prices in this product are joined to community-maintained catalog metadata by exact barcode, and the join is deliberately disclosed rather than papered over. The August 8 edition reports 21,664 observations — 8.0% — with no matched catalog metadata at all: the barcode, price, retailer, and dates remain available, but there is no product name, brand, or category to filter on.

The practical consequences:

• A filter on product_name or categories_tags silently excludes every unmatched row. State that denominator when the filtered set is presented as “the market.”
• quantity is present on 208,783 of 270,334 rows (77.2%). Size-comparable pricing — per kilogram, per liter — is only possible on the dated subset where the quantity text parses, and the community text is multilingual and inconsistent.
• Retailer context is far denser than catalog context: 269,960 rows carry retailer_name and 265,737 carry city. Retailer-level analysis survives the unmatched gap; product-attribute analysis does not.
• brands and categories_tags are community labels, not a controlled taxonomy. Category analysis means exact tag matching — en:breakfasts, not “breakfast” — with the knowledge that tagging is uneven.

The recipe writes this audit as coverage-audit.csv, one row per catalog source group, with observation counts, unique barcodes, field-presence counts, and name coverage as an explicit fraction.

Cut a small filtered market sample

The point of the workflow is to stop forcing evaluation through 238 MB. A scoped sample — a few thousand rows, one market, one category, one year — is something a colleague can open, review line by line, and argue with.

The default sample selects France, EUR, observations dated in 2026 whose categories_tags include en:breakfasts, and produces 2,550 rows. For scale, the same market with en:dairies yields 8,666 rows in 2026.

from datetime import date

def matches_sample(row: dict[str, str]) -> bool:
    observed = date.fromisoformat(row["observed_date"])
    return (
        row["country"] == "France"
        and row["currency"] == "EUR"
        and observed.year == 2026
        and "en:breakfasts" in parse_categories(row["categories_tags"])
    )

The exported sample keeps twenty columns per row: the edition date, observation date, barcode, product name, brands, quantity, price, currency, discount fields, retailer name/brand/type, city, country, location type, source_price_id, source_url, and source_license. Every row is therefore checkable against the authoritative Open Prices record, and the file carries ODbL attribution with it.

Sort deterministically — observation date, then source_price_id — so the exported bytes are stable across runs and hashable for provenance.

Build a price history for one barcode

A trend view answers a different question than a sample: how has the observed price of one product moved, and across whom? The default example follows barcode 3017620422003 — Nutella — inside the France/EUR market.

The history contains 148 observations on 111 distinct dates, across 37 distinct retailer names, from 2021-02-26 to 2026-08-07, with prices from 2.58 to 5.02 EUR and an overall median of 3.51 EUR. The recipe derives three products from it:

• barcode-price-history.csv — one row per observation, sorted by date, with retailer, city, discount fields, and source_url.
• barcode-monthly-summary.csv — 38 observed months with observation count, min, median, max, discounted count, and distinct retailer count.
• barcode-retailer-summary.csv — 37 retailer names with observation counts, date spreads, and price ranges.

Three interpretation rules keep the trend honest:

1. Observation density varies. January 2025 holds 27 observations from 19 retailer names; most months hold one to four. A monthly median from one observation is a single data point, not a market measurement.
2. Retailer mix drives spread. The series-wide retailer comparison runs from a 3.03 EUR median at Centre Commercial E.Leclerc (18 observations) to 4.59 EUR at Proxi (9 observations). A within-month price jump can be a different store, not a price change.
3. Retailer names are free text. The top French retailers in this edition include Centre Commercial E.Leclerc (31,865 observations), E. Leclerc (7,783), and E.Leclerc (5,982) as three separate strings. The dataset does not resolve names to legal entities; any retailer consolidation is your documented mapping.

from statistics import median

monthly: dict[str, list[Decimal]] = {}
for row in history_rows:
    monthly.setdefault(row["observed_date"][:7], []).append(
        Decimal(row["price"])
    )

summary = [
    {
        "month": month,
        "observations": len(values),
        "min_price": min(values),
        "median_price": Decimal(median(values)),
        "max_price": max(values),
    }
    for month, values in sorted(monthly.items())
]

Read discount fields as observed events

The edition contains 23,999 observations flagged price_is_discounted = True — exactly the manifest's discounted_count. Of those, 21,702 carry a price_without_discount reference value; 2,297 discounted observations have no reference price. A discount analysis must report both numbers.

discount_type is present on 17,151 rows, with the observed vocabulary:

LOYALTY_PROGRAM   8,412
SALE              6,711
QUANTITY          1,484
OTHER               300
EXPIRES_SOON        217
SEASONAL             25
SECOND_HAND           2

Three boundaries follow:

• A discount flag describes one observed price event on one date, not a current promotion. The product is not a live promotion feed.
• price_without_discount is the contributor-reported reference price for that event. It is not a verified shelf price from another day.
• Discounted and non-discounted observations of the same barcode belong in separate lanes of any trend. Mixing them silently is how a loyalty-card price becomes a fake price cut.

Keep the four clocks apart

The cumulative window is the most misread fact in the file. Observations dated 2010 through 2023 account for 21,312 rows; 2024 adds 45,466; 2025 adds 93,614; and 2026 through the August 8 cutoff adds 109,942. Relative to the cutoff, only 8,577 observations fall within the final 30 days and 25,507 within the final 90 days. Any “current price” view must therefore state its recency window — for example, observations within 90 days of the edition cutoff — instead of implying the whole file describes today's shelves.

Freeze recency calculations to the edition cutoff. Using date.today() makes the same input produce a different “recent” set tomorrow and destroys reproducibility.

Export results with a provenance receipt

The downloadable recipe stages every CSV in a unique staging directory, calculates hashes there, and promotes the complete result set only after all row-count assertions and a final input recheck pass. It removes any old provenance receipt before promoting files, then writes provenance last through an atomic replace.

That ordering prevents a failed run from leaving a new partial CSV beside an old receipt that appears to authenticate it.

The query-provenance.json receipt includes:

• recipe version and SHA-256;
• Python implementation, runtime version, and platform;
• the exact filter values: country, currency, category, sample year, barcode;
• release tag, target date, generation time, schema version, and record grain;
• manifest and CSV byte counts and hashes;
• manifest-declared counts beside the streamed totals that reproduced them;
• exact output filenames, row counts, byte counts, and hashes;
• interpretation notes for currencies, retailer names, missing metadata, discounts, and clocks; and
• the ODbL attribution and source-license URL.

The expected outputs for the default run are:

Hashes establish byte identity. They do not prove that a contributor-reported price was actually charged, that a retailer name is a legal entity, or that a barcode's history is complete.

Choose a published release or source APIs

Use the normalized public release when the job needs:

• a dated, cumulative observation snapshot with one documented schema;
• CSV, JSON, or JSONL with a manifest and SHA-256 declarations;
• a repeatable local sampling or trend workflow; or
• exports that retain official source URLs and ODbL attribution.

Use the official Open Prices API when the job needs interactive queries, prices added after this edition's cutoff, contributor-facing detail, or a live collection your team will retain from now on. Open Prices also publishes the raw bulk dumps — the same prices.jsonl.gz this edition is built from — via its data page. Use the Open Food Facts data exports when you need the full community catalog beyond the joined price fields.

Whichever route you choose, respect the ODbL 1.0 terms: attribute the sources, share adapted databases alike, and keep the OpenStreetMap attribution for location data.

This article owns post-download analysis. It does not promise a public WebTruffle API, convert currencies, or turn community observations into an official price index. Use the product price monitoring dataset guide to design the separate product, variant, listing, offer, and observation layers needed for recurring collection. If you need a defined retail market — named retailers, matched assortments, availability, promotions, history, and managed delivery — that is a scoped feed conversation, not a CSV download.

Retail prices Python checklist

Before sharing a sample or trend, verify all of these:

• Pin the release tag and verify the trusted manifest byte count and SHA-256.
• Verify the CSV against the already-verified manifest before parsing.
• Record target date, generation time, schema version, source extract identity, and record grain.
• Require all 40 headers in their declared order; read with utf-8-sig.
• Keep barcodes, price IDs, and record IDs as text; parse prices with Decimal.
• Parse JSON-array cells with a JSON parser, not comma splitting.
• Compare price_is_discounted against the exact strings True and False.
• Select one country and one currency before any price statistic; carry the code into exports.
• Report metadata coverage — including unmatched-barcode rows — before filtering on metadata.
• State the observation-count, retailer-count, and date-spread beside every trend figure.
• Keep discounted observations in a separate lane from full-price observations.
• Freeze recency windows to the edition cutoff, never date.today().
• Sort exports with complete tie-breakers before hashing them.
• Recheck inputs after processing and promote provenance only after every output passes.
• Retain source_url, source_license, exact filters, input hashes, output hashes, and runtime identity.

Limitations and interpretation boundaries

This worked result has explicit limits:

• It analyzes one cumulative community snapshot, not a complete retailer catalog, a live price feed, a stock signal, or a representative price index.
• Coverage follows community contributions to Open Prices and the Open Food Facts ecosystem: global but uneven across products, retailers, countries, and dates. France contributes 67.8% of this edition's rows.
• The edition cutoff is August 8, eleven days before publication. Results are intentionally reproducible, not live.
• Prices are contributor-reported observations. A row does not prove the product remains available or that the amount appears at checkout.
• Currency codes, retailer names, cities, and product text are source-reported free text. There is no entity resolution, no currency conversion, and no inferred unit pricing.
• 21,664 observations carry no catalog metadata; name- or category-based filters exclude them by construction.
• Discount fields describe observed events, not current promotions; 2,297 discounted rows lack a reference price.
• Barcode histories are as complete as community observation density: 148 observations over five years is a sparse, uneven sample of one product's market.
• A matching hash proves artifact identity, not factual accuracy, source completeness, or analytical fitness.
• GitHub currently marks the releases as mutable. The pinned manifest hash detects changed release bytes.
• The normalized product deliberately excludes contributor and owner identifiers, proof and receipt records, exact addresses, coordinates, and arbitrary website URLs.

Frequently asked questions