Build a price monitoring dataset around offers, not pages.

A dependable product price monitoring dataset keeps the product family, sellable variant, retailer listing, seller-and-market offer, dated observation, and detected change as related but different records.

A product family is not a sellable variant. A variant is not an offer. Price is a dated offer claim, not a product attribute. A missing page is not proof of an out-of-stock offer. These boundaries prevent false product matches, mixed pack comparisons, invented stock events, and price histories that silently combine different sellers or purchase conditions.

The useful output is a versioned evidence system, not a one-price-per-URL spreadsheet. It records the sellable item, seller, channel, market, terms, observation time, and unresolved evidence.

Price monitoring dataset: the short answer

Build the dataset in this order:

1. Define the decision, named markets, included sellers and channels, eligible products, comparison basis, cadence, and tolerated delay.
2. Separate product family, sellable variant or trade item, retailer listing, offer, observation, and change-event grains; derive each change event from observations under a versioned policy.
3. Register each source with its access basis, seller identity, regional scope, rights, fields, cadence, and known blind spots.
4. Assign stable internal IDs while retaining source-scoped listing keys, GTINs, MPNs, SKUs, URLs, and every identity decision.
5. Preserve source-native titles, attributes, pack text, prices, availability, shipping terms, and timestamps before normalization.
6. Match variants with positive evidence and hard conflicts; do not let a similar title override incompatible size, count, condition, or model.
7. Model multipacks, bundles, subscriptions, refurbished items, and marketplace sellers explicitly rather than hiding them in a product name.
8. Store active, reference, sale, member, coupon, subscription, installment, tax, and shipping amounts as separate typed claims.
9. Normalize currency and units only through documented rules, retaining the original value and the inputs behind every derived comparison.
10. Derive availability from explicit source evidence and successful observations, never from a request failure or one disappeared page.
11. Keep append-only observations and reproducible events so corrections do not masquerade as retailer price changes.
12. Test collection, extraction, identity, offer semantics, availability, history, and delivery against representative labeled cases.
13. Publish dated snapshots, history, deltas, manifests, quality results, and limitations with stable replay semantics.
14. Approve access, licensing, attribution, media, personal-data, retention, and comparison-claim controls before recurring collection.

Use this guide when you are designing the operational model. If you need a maintained feed for specified competitors, products, markets, fields, refresh rates, and delivery, use the managed product price and stock monitoring service. If you want to analyze one already published public data edition in Python, the retail product prices walkthrough is a snapshot-analysis recipe, not the recurring collection and history model described here.

Define the decision, market, and source universe

“Monitor competitor prices” is a theme, not a specification. A merchandising team may compare current advertised prices for directly substitutable variants. A procurement team may follow distributor pack prices. A brand may audit channel availability. A promotion analyst may need reference-price and sale-window evidence. Each question creates a different eligible population and a different definition of a meaningful change.

Write a decision contract before selecting sources:

Describe the universe as an intersection: named sources, sellers, storefronts, countries or delivery regions, store locations, product taxonomy, brands, identifier coverage, item condition, fulfillment modes, customer eligibility, and time window. “All online prices” has no measurable denominator. “Public offers observed on five named storefronts for successfully collected catalog partitions” is narrower but auditable.

A source partition is the smallest collection slice whose success and completeness can be assessed independently—for example, one storefront, seller type, region, category, pagination path, and crawl window. Record caps, login gates, geolocation behavior, cookie state, dynamic availability, personalization, and delivery-postcode assumptions. An empty partition can mean no eligible offers, an access denial, a source redesign, or a collector defect. Preserve those states separately.

Store-level projects need a stable seller and location model. A retailer name, marketplace seller, franchise, branch, fulfillment warehouse, pickup location, and delivery market are not interchangeable. The business location data guide explains how to preserve source identity and location evidence; use that discipline where an offer changes by store or delivery area.

Separate product family, variant, listing, offer, and observation

The central modeling decision is record grain:

• Product family: a conceptual model or group whose members vary by declared dimensions such as color, size, capacity, flavor, or finish.
• Sellable variant or trade item: the exact configuration a customer can select, order, or receive, including condition and declared pack configuration.
• Retailer listing: one publisher's product page, catalog entry, feed row, or source record, identified within that publisher's namespace.
• Offer: commercial terms from a seller through a channel for a variant, region, customer class, fulfillment mode, and time.
• Observation: what the collector successfully saw about that listing and offer at a recorded time, with source evidence and collector context.
• Change event: a typed difference or source event derived from observations under a versioned policy.

One family can contain many sellable variants. One variant can appear in many retailer listings. A marketplace listing can expose offers from several sellers. One seller can quote different prices for delivery, pickup, membership, subscription, or regions. One observation may see only part of that state. Model the relationships instead of flattening them into one mutable row.

Schema.org ProductGroup provides hasVariant, variesBy, and productGroupID, while a Schema.org Product can point back with isVariantOf. Its Offer vocabulary can represent commercial terms including price, price currency, availability, seller, and eligible conditions. These are useful web-publishing concepts, not evidence that every page implements them, that visible markup is current, or that one offer covers every seller and region.

Keep separate tables or equivalent event-stream entities:

• product_families hold the conceptual grouping and the attributes on which members vary;
• variants hold stable internal identity, manufacturer assertions, exact attributes, condition, and pack definition;
• source_listings hold source, source key, URL history, source-native identifiers, title, and represented variant match;
• sellers, channels, and markets define who offers what, where, and under which fulfillment context;
• offers hold stable seller, variant, market, channel, fulfillment, quantity, condition, and eligibility identity;
• offer_claims hold each typed price, tax, shipping, promotion, or availability assertion with source text, observation, eligibility, and stated effective interval;
• identifier_aliases hold scoped GTIN, MPN, SKU, offer-ID, and URL claims with issuer, namespace, validity, verification, and evidence;
• match_decisions hold candidate pairs, compared evidence, hard conflicts, rule or model version, outcome, reviewer, and decision time;
• collection_attempts hold attempted time, request context, outcome, and failure evidence;
• observations hold successfully extracted source claims, linked collection attempt, raw-evidence reference, and parser version;
• offer_events hold price, promotion, availability, seller, URL, or identity changes with derivation provenance.

Do not require every listing to resolve to a variant before publication. Retain an unresolved or ambiguous source listing with its evidence. A false match blends two products' price histories; an unresolved record can be revisited when a better identifier or attribute arrives.

Google's product variant structured-data guidance illustrates a publisher-side family and variant distinction: a group declares its varying properties, variants carry unique identifiers in markup, and each variant must be directly selectable through a distinct URL state for Google's feature. Treat that as Google-specific publishing guidance. It does not prove that a URL outside that implementation represents one variant or that every visible selection was collected.

Register sources, rights, and coverage limits

A source register should state what the source asserts, how it may be accessed, and which uses are permitted. Record the owner, entry point, authentication, approved method, expected cadence, rate limits, geographic behavior, seller model, subject grain, identifiers, field scope, terms, license, attribution, retention, redistribution, media rules, and operational owner.

Standards and structured-data vocabularies are not product feeds or reuse licenses. Schema.org describes a vocabulary. GS1 defines identification rules. Google documents requirements for its products. None grants access to a retailer's data or establishes rights to retain, combine, publish, or use the source for automated comparison.

Assign field authority locally. A manufacturer page may be strongest for a model specification. A brand-issued GTIN can identify a trade item within GS1's scope. A retailer directly asserts its listing, advertised price, promotion text, and seller attribution. A delivery quote may supply a destination-specific charge. A marketplace page may aggregate several sellers. One source rarely establishes all fields.

For every run, preserve enough evidence to distinguish:

• a successfully observed listing and offer;
• an explicitly unavailable, out-of-stock, preorder, or backorder claim;
• a successful catalog partition in which a listing was absent;
• a changed URL, redirected identifier, or seller substitution;
• a login wall, consent block, throttle, timeout, partial render, parser failure, or challenge;
• a source-wide change in totals, filters, pagination, markup, or regional behavior.

Only apply absence logic to successfully covered partitions. If a retailer caps category results or personalizes delivery availability, “not returned” is not equivalent to “not offered.” Retain the source-native payload or the minimum permitted evidence, plus source, URL or key, attempted time, request market, response state, content hash where available, collector or extractor version, and extracted values only when an observation succeeded. A hash supports byte comparison; it does not prove truth, completeness, identity, or permission.

Design stable product and offer identifiers

Create opaque internal IDs for families, variants, listings, sellers, markets, offers, observations, and events. Identity persistence is grain-specific: family and variant IDs may survive title, URL, and image changes; source-listing IDs can retain URL history; offer IDs may survive price, promotion, availability, and parser changes only while seller, variant, channel or market, fulfillment, condition, eligibility, and quantity identity remain stable. Do not build a permanent variant ID from mutable presentation text.

Store every external identifier as a scoped alias: scheme or issuer, namespace, entity grain, value, source, market if relevant, observed interval, verification state, and evidence. A manufacturer part number is manufacturer-scoped. A retailer SKU is retailer-scoped and may refer to a listing, an assortment, or a fulfillment item under that retailer's rules. A marketplace offer ID may identify a seller's terms rather than the product. A URL is a locator whose query parameters can encode variant, region, or campaign state.

The GS1 GTIN Management Standard describes a GTIN as identifying a trade item that may be priced, ordered, or invoiced in a supply chain. It requires new GTINs for new products and gives variant and pack-level cases where distinct identification is needed; its rules also address changes such as declared net content and pack quantity. A GTIN whose structure, GS1 assignment, represented trade item, and packaging level have been verified is strong evidence for that documented scope; a correct length and check digit alone establish only that the number is correctly composed. Verified by GS1 can return assignment and available product or company information, but the GTIN is not a universal product-family, seller-offer, or listing ID, and it does not guarantee that a retailer's claim is correct.

Never silently “repair” identifiers by dropping leading zeros, converting them to numbers, or accepting a check-digit failure. Preserve the source string and a validation outcome. Store any scheme-defined canonical form separately and compare canonical forms before declaring a GTIN conflict; never overwrite the exact source representation. Keep multiple claimed GTINs, MPNs, and SKUs when sources disagree. Resolve the current preferred alias through a versioned decision without rewriting historical observations.

An offer identity needs more than variant ID. At minimum consider seller, channel or storefront, market or delivery region, fulfillment mode, item condition, customer eligibility, quantity basis, and source listing. Treat those dimensions as offer-identity constraints. Preserve a verified seller rename or alias through a resolution event; otherwise create a related offer when the seller or another identity-defining dimension changes. Keep redirects, replacements, and listing lineage explicit.

Match products with evidence and hard conflicts

Product matching should produce an auditable decision, not merely a similarity score. Start with blocking rules that generate plausible candidates, then compare evidence at the sellable-variant grain. Candidate features may include validated GTIN, manufacturer and MPN pair, brand, model, normalized title tokens, exact variant attributes, dimensions, declared net content, pack count, compatible category, source-native structured data, and permitted image fingerprints.

Apply hard conflicts before fuzzy similarity. Examples include different verified, canonically unequal GTINs for the same claimed packaging level with no documented replacement or hierarchy relationship, incompatible model numbers, different size or color where those define a variant, new versus refurbished condition, single item versus multipack, conflicting voltage or region, and a bundled accessory absent from the candidate. A high title score must not override a conflict that changes what the buyer receives.

Use explicit match states:

1. Accepted: the evidence meets a versioned rule for the same sellable variant.
2. Rejected: the records refer to distinct variants or violate a hard constraint.
3. Ambiguous: a plausible match exists, but accepting it would be unsafe.
4. Unreviewed: the candidate has not passed an automated rule or human review.

Store both compared records, source values, normalized features, missingness, candidate rule, hard conflicts, score components, decision rule or model version, threshold, outcome, reason codes, reviewer, and decision time. A single confidence number is not enough to reproduce why two listings were joined.

Use identity inheritance cautiously. If a retailer family page contains several selectable variants, a family-level brand or model may apply to all children, while price, availability, GTIN, seller SKU, image, or pack text may change with the selected option. Record which page state was selected and which claims were inherited. Do not copy one observed variant's offer onto its siblings.

Evaluate matching on labeled pairs that reflect the real catalog: near-identical titles, missing GTINs, regional model suffixes, multipacks, refurbished items, private-label equivalents, marketplace duplicates, and variant selectors. Report precision and recall by category, brand, source pair, identifier coverage, and conflict type. Optimize false merges aggressively because one false merge fabricates a price comparison; track false splits too because they hide genuine competitor coverage.

Model bundles, packs, and variants explicitly

A shopper can buy a 500-millilitre bottle, a two-pack of that bottle, a mixed bundle, a refill, or a subscription. They may share a brand and marketing name while differing in quantity, included items, packaging hierarchy, condition, or purchase commitment. Treat these distinctions as data, not title cleanup.

For a simple item, store the base product kind, declared amount, unit, and count. For a multipack, store units per pack, amount per unit, total amount, and whether the packaging itself has an identifier. For a bundle, model component variant, quantity, required or optional status, and bundle-level seller terms. For a set with unknown components, retain the source description and mark the composition unresolved rather than inventing a total.

Make variant-defining dimensions category-aware. Apparel may vary by size and color. Electronics may vary by capacity, connectivity, voltage, or regional model. Food may vary by flavor, net content, count, or formulation. Furniture may vary by finish and dimensions. Keep source-native attributes, then map them through a versioned taxonomy with exact, broader, narrower, related, and unmapped outcomes.

Condition and fulfillment deserve first-class fields. New, used, refurbished, open-box, rental, digital, and physical items are not direct equivalents. Delivery, pickup, marketplace fulfillment, backorder, preorder, and made-to-order can change both price and availability meaning. If the decision contract excludes a class, retain the observed class as an exclusion reason rather than deleting the evidence silently.

Comparisons should state their basis: exact variant, compatible substitute, family-level range, identical declared quantity, or normalized unit basis. Never label a family minimum as the price of every variant, and never compare a bundle total with a single item without making the quantity transformation visible.

Separate active, reference, member, and promotional prices

One offer can display several amounts simultaneously. Store each as a typed, dated claim rather than choosing one generic price column:

• Active public price: the currently advertised amount for the stated market and quantity, before any separately modeled charges.
• Reference or regular price: a crossed-out, list, was, recommended, or prior amount with its exact source label.
• Sale price: a promotional active amount, with stated start and end times when available.
• Member or loyalty price: an amount conditional on enrollment, tier, account, or other eligibility.
• Coupon price or discount: a code, clip, basket, rebate, or post-purchase condition whose redemption is not guaranteed.
• Subscription price: an amount conditional on recurring purchase, schedule, or cancellation terms.
• Installment terms: payment count, amount, deposit, financing conditions, and any separately stated total.

Preserve the source display text and label. “From,” “starting at,” “up to,” “save,” and “as low as” do not become exact prices through parsing. Keep ranges as lower and upper bounds. Keep percentage, fixed-amount, multi-buy, and gift promotions in separate structures with prerequisites and affected quantities.

Google's Merchant Center product data specification separates price, sale price, sale-price effective dates, loyalty programs, subscriptions, installments, and unit-pricing measures. Those are Google platform semantics and requirements, not a universal ontology. Use them as useful examples, while preserving each retailer's native labels and the rules in the target market.

Google's merchant listing structured-data documentation places an Offer under a Product and requires the merchant to be the seller for eligibility. That reinforces the product-versus-offer boundary, but markup remains a publisher claim rather than proof of checkout terms.

Price changes need typed causes. A retailer may change the active amount, start or end a sale, alter a member condition, replace a seller, change tax display, or correct a page. The pipeline may also fix a parser. A source-observed change and a processing correction must never share one event type.

Normalize currency, tax, shipping, and unit price

Keep the original amount string, parsed decimal, currency claim, amount type, seller, market, quantity basis, source, observation time, and parse status. Use decimal arithmetic. Do not infer a currency solely from a symbol when several currencies share it; use page locale, explicit code, storefront, and source evidence under a documented rule, or leave the value unresolved.

Tax treatment varies by jurisdiction, customer, and channel. Record whether the source says tax is included, excluded, estimated, or unknown, and which tax or region that claim concerns. Google Merchant Center itself documents different price-tax submission rules for the United States and Canada versus other target countries. That is a Google-specific platform rule, but it demonstrates why one global “tax included” assumption is unsafe.

Shipping is not one scalar property of a product. It may depend on destination, seller, cart total, membership, speed, weight, pickup choice, surcharge, and time. Store shipping method, destination context, amount, currency, threshold, eligibility, estimated window, and observed time. If the page only says “calculated at checkout,” preserve that state. Do not turn a product-page price into a delivered or checkout price without a reproducible quote for the same context.

Unit price is derived from a compatible item amount and declared measure. Retain the source measure, count, unit, denominator, conversion rule, and resulting value. The UNECE code-list recommendations include Recommendation 20 codes for units of measure used in international trade. A controlled code can improve interchange, but it does not repair ambiguous source text or determine whether gross, net, drained, usable, or per-item quantity is intended.

For every conversion, retain the original values. A volume-to-mass conversion requires a justified density and normally should not be assumed. Count-based and weight-based prices are not interchangeable. “Each,” “pair,” “sheet,” “dose,” and “serving” need category context. A pack of six 330-millilitre cans can yield a price per litre only when both count and per-unit volume are established.

Currency conversion is another derived layer. Record the rate provider, rate timestamp or date, base and quote currencies, rate type, rounding policy, and calculation version. Do not overwrite the observed amount. For historical comparisons, decide whether to use observation-date rates, period-average rates, or no conversion at all, and label the result.

Publish at least three comparison fields when needed: observed item price, derived unit price, and a separately constructed delivered-price estimate. Each needs eligibility and quality flags. None should be described as the checkout total unless an actual checkout-context quote establishes that claim.

Track availability without inventing stock

Availability is an offer claim under a specific context, not a warehouse count. Preserve source-native text and map it conservatively to classes such as available, unavailable, preorder, backorder, limited, made to order, pickup only, delivery only, unknown, and not observed. Keep seller, variant, market, destination, store, fulfillment mode, and observation time attached.

Google Merchant Center's specification uses in_stock, out_of_stock, preorder, and backorder values and tells participating merchants to align availability across their feed, landing page, checkout, and structured data. Those values are Google-specific submission semantics. A monitored page may use different labels, and a collector usually cannot verify the seller's physical inventory.

Distinguish four evidence states:

1. Explicit availability: the source successfully displayed a recognizable state for the selected variant and context.
2. Successful unknown: the page loaded, but no supported availability claim could be extracted.
3. Successfully absent: the listing was not present within a demonstrably completed source partition.
4. Collection unavailable: access, rendering, selection, request, or parsing failed.

A missing page is not proof of an out-of-stock offer. The item may have moved, been delisted, changed identifiers, become region-restricted, disappeared from a capped result set, or encountered a collection failure. Record a disappearance or absence event. Only map it to an inactive offer under a versioned, source-specific rule with successful coverage and an appropriate observation window.

Do not infer inventory quantity from “in stock,” delivery speed, purchase limits, urgency text, or a successful add-to-cart interaction. “Only three left” is a source claim if explicitly shown; it is not independently verified stock. Preserve the phrase, quantity claim, context, and timestamp, then decide whether the intended use permits publication.

Reappearance should reopen the prior offer or create a linked replacement under the identity policy. Preserve the gap: observations can bound a state, but they cannot establish what happened between checks.

Preserve price, promotion, and availability history

An overwrite-only table answers “what does the latest parser believe?” but cannot answer when an offer changed, how long a promotion appeared, or whether a correction altered history. Keep append-only observations as the evidence layer and derive current views and events from them.

Every collection attempt should carry source, listing or partition, market and destination context, attempted timestamp, request outcome, failure evidence, and collector version. A successful observation should link to that attempt and carry the selected variant, seller, observed timestamp, source timestamps if present, raw-evidence reference, parser version, extracted claims, and content or field hashes. Deduplicate retries through stable attempt and observation keys without collapsing genuinely distinct observations.

Keep the clocks separate:

• source time: when a retailer says a price, promotion, listing, or availability state applies;
• observed time: when the collector successfully saw the claim;
• processed time: when extraction and normalization ran;
• decision time: when a match or derived event was accepted;
• effective time: the interval a derived current view assigns under its policy;
• publication time: when a dataset release became available to consumers.

A first observation is not necessarily a promotion start. A last observation is not necessarily its end. If the source supplies an effective interval, retain it as a source claim and compare it with observations. Otherwise label first-seen and last-seen bounds; do not manufacture exact campaign dates.

Create typed events only when a versioned comparison rule supports them: active-price claim changed, reference price first observed, sale first observed, sale no longer observed, member terms changed, availability claim changed, seller offer appeared or disappeared, listing redirected, variant match corrected, parser corrected, or record suppressed. Reserve sale_started and sale_ended for explicit source-effective intervals; otherwise retain first-seen and last-seen bounds. Store the before and after observations, compared fields, tolerance, policy version, reason, and event-generation time.

Numeric equality alone can hide semantic change. The same amount can move from a public price to a member price, tax display can change, or the pack count can change. Conversely, a presentation rewrite or rounding change may not meet the project's material-change rule. Compare the complete typed offer context.

The monitoring runbook describes schedules, incidents, staleness, and recovery states. Apply those operational controls per source partition before deriving business events. Reprocessing old observations under an improved parser should emit a correction or restatement, not a fictional historical retailer change.

Distinguish market observations from price indices

A recurring dataset of observed offers is not automatically a market price index. Its records reflect the included retailers, successfully covered partitions, matched variants, visible sellers, regions, observation schedule, and price-eligibility rules. Assortment changes and collection incidents can move a simple average even when no continuing item's price changed.

Before computing an index, define the target population, sampling frame, item and outlet weights, comparable item specification, substitutions, new and disappearing products, promotions, seasonal products, quality change, missing observations, tax and shipping treatment, aggregation formula, base period, revision policy, and uncertainty. Those are statistical design decisions beyond collection.

Publish descriptive measures with their denominators: exact-variant offer count, matched retailer count, observation window, coverage rate, unresolved-match rate, and promotion inclusion. Label an unweighted median of observed offers exactly that. Do not call it representative of a country, category, or consumer experience without a defensible sampling and weighting design.

The retail product prices Python guide demonstrates how to analyze a specific public snapshot while keeping its edition, source currency, coverage, and observation dates visible. It does not convert that snapshot into a live feed or a representative price index, and it should not be used as a substitute for the operational evidence model in this guide.

Measure quality at each boundary

Quality metrics should reveal where evidence or processing failed:

1. Collection: expected partitions attempted and completed, response states, throttling, caps, variant-selection success, and evidence retention.
2. Extraction: required fields found, structured-versus-visible disagreements, parser errors, source-native fidelity, and fixture regressions.
3. Identity: candidate coverage, accepted and ambiguous rates, precision, recall, hard-conflict escapes, false merges, and review age.
4. Variant and pack semantics: attribute-map coverage, condition, pack count, declared measure, bundle composition, and unmapped reason codes.
5. Offer semantics: seller, market, eligibility, active and reference price labels, promotion prerequisites, currency, tax, and shipping completeness.
6. Availability: explicit versus inferred states, successful-unknown rate, absence evidence, transition validity, reappearance, and stale records.
7. History: ordered observations, event replay, clock completeness, correction separation, and policy-version reproducibility.
8. Delivery: schema validation, referential integrity, counts, hashes, atomic publication, delta replay, and consumer acknowledgments.

Every rate needs a numerator, denominator, time window, filters, and exclusions. “Prices are 99% complete” is meaningless unless the eligible offer population and required components are stated. Segment metrics by source, seller type, market, category, brand, identifier coverage, variant complexity, pack form, parser version, and collection path.

Build representative gold sets for product matching, pack parsing, seller attribution, price classification, availability mapping, and change detection. Include shared family names, regional model suffixes, size and color selectors, multipacks, mixed bundles, member prices, coupon conditions, marketplace sellers, out-of-stock pages, redirects, source failures, and parser corrections. Keep evaluation examples separate from tuning.

Measure false merges and false out-of-stock classifications on high-risk segments, not only globally. Track false splits and missed availability changes separately because their harms differ.

Monitor distributions alongside pass rates: offer counts, prices by currency, active-to-reference ratios, discount depths, pack quantities, unit-price tails, seller mix, state transitions, observation gaps, match-score bands, ambiguous queues, and partition duration. Sudden shifts can expose a selector failure, consent wall, taxonomy redesign, or promotion-parser error before consumers act on bad data.

The web-scraping data-quality framework covers completeness, validity, fidelity, duplicates, freshness, and release gates. Quality must pass per source before cross-retailer comparisons are released; a perfect schema cannot compensate for a failed source partition.

Publish snapshots, history, and deltas

Publish convenient views without discarding evidence:

• a current family and variant view with stable IDs, source aliases, and match state;
• a current listing and offer view with seller, market, terms, availability claim, and last successful observation;
• append-only observations and typed offer events;
• match decisions, listing lineage, redirects, replacements, and corrections;
• versioned snapshots, deltas, manifests, schemas, code lists, and limitation notes.

A release manifest should name the dataset and release IDs, schema and policy versions, covered source partitions, collection cutoff, files, record counts, byte counts, hashes, quality-gate results, and known incidents. Publish files to versioned locations, validate the complete release, and then atomically update the current pointer.

Define delta semantics. State whether an update replaces a complete record or named fields, how source absence differs from a deletion, how offer closure differs from item unavailability, how ID merges and splits are represented, and whether corrections restate previous releases. Use stable event IDs and sequence rules so retries are idempotent and consumers can detect gaps.

The W3C PROV overview describes provenance through the entities, activities, and people involved in producing data. Apply that discipline pragmatically: connect a published offer assertion to source evidence, the extraction and normalization activity, the matching or event decision, and the software or reviewer responsible.

Document covered sellers, markets, channels, destination assumptions, successful partitions, field freshness, match thresholds, promotion rules, missing-price reasons, availability limits, and degraded incidents in every release. The custom data feed guide covers contracts, manifests, atomic delivery, snapshots, and deltas in more detail.

Govern licenses, media, and retention

Public visibility does not settle permission to collect, retain, combine, compare, republish, or commercialize data. Review source terms, contracts, licenses, database rights, copyright, access controls, permitted fields, recipients, territories, retention, attribution, and automated-use restrictions with qualified counsel for the intended use.

Different source elements may be governed by different contractual, database, trademark, copyright, privacy, or other restrictions. Minimize collection to the decision purpose. Store identifiers, images, text, reviews, seller profiles, badges, promotional creative, or derived fingerprints only where permitted and necessary, and do not assume that a right to view a page includes a right to redistribute its contents.

Account-based or localized collection can expose personal data, tokens, addresses, saved locations, or personalized prices. Use controlled service identities where allowed, separate secrets from evidence, restrict raw captures, encrypt sensitive fields, log exports, and define deletion procedures. Label a personalized offer with its eligibility context or exclude it.

Comparison outputs require their own review. A technically accurate observation can still become misleading when tax, shipping, pack size, condition, seller, membership, time, or coverage is omitted. Define which claims may be automated, what evidence appears beside them, when human review is required, and how corrections reach consumers. This guide is an engineering framework, not legal advice or a guarantee that a comparison claim complies with a particular jurisdiction.

Set source-specific retention for raw pages, structured payloads, screenshots, media, normalized observations, and published releases. When rights change or access ends, know which artifacts must stop, expire, be suppressed, or remain solely as permitted audit evidence. WebTruffle's responsible data and security principles describe the boundaries applied to managed work.

Price monitoring dataset checklist

Before production, confirm all of the following:

• [ ] The decision, sellers, channels, markets, product universe, customer context, cadence, delay, and exclusions are explicit.
• [ ] Product family, sellable variant, retailer listing, seller-and-market offer, observation, and event grains are separate.
• [ ] Internal IDs survive title, URL, price, promotion, seller, availability, and parser changes.
• [ ] GTIN, MPN, SKU, offer ID, and URL aliases retain issuer, namespace, grain, evidence, and valid interval.
• [ ] Accepted, rejected, ambiguous, and unreviewed product matches remain reproducible.
• [ ] Hard conflicts cover variant attributes, pack size, bundle composition, condition, model, voltage, and region where relevant.
• [ ] Source-native attributes, quantity text, prices, availability, seller, shipping, tax, and timestamps survive normalization.
• [ ] Active, reference, sale, member, coupon, subscription, installment, tax, and shipping claims use separate fields.
• [ ] Unit prices retain item quantity, denominator, unit code, conversion rule, and eligibility status.
• [ ] Missing listings, collection failures, unknown availability, explicit out-of-stock claims, and offer closure remain distinct.
• [ ] Source, observed, processed, decision, effective, and publication times are not substituted.
• [ ] Price and availability events can be replayed from observations under a named policy version.
• [ ] Matching and quality metrics publish denominators, windows, segments, exclusions, and support counts.
• [ ] Snapshots, deltas, manifests, redirects, corrections, tombstones, replay, and atomic publication are documented.
• [ ] Descriptive comparisons are not labeled checkout prices, inventory quantities, or representative indices without supporting evidence.
• [ ] Source access, licenses, media rights, personal data, security, retention, attribution, correction, and claim review have owners.
• [ ] Every published product link, offer, price, availability state, and event can be traced to source evidence and transformation history.

Frequently asked questions