Data quality beginsbefore production.

“Accurate data” is not a useful acceptance criterion on its own. Accurate compared with which source state? Complete across which pages? Fresh within what window? Matched to which entity? Normalized according to whose rule?

Web scraping data quality becomes measurable only after the source scope, record definition, required fields, identity rules, cadence, transformations, destination, and failure actions are explicit. The purpose is not to award a universal quality score. It is to prove that a delivered dataset is fit for the decision it must support.

The W3C Data Quality Vocabulary takes the same contextual view: quality dimensions and metrics are selected according to the dataset and consumer’s use. (W3C) For a managed web feed, that context belongs in the specification before production.

Quality begins with the decision the data must support

Start by writing down the business outcome. A daily product feed supporting pricing decisions needs different tolerances from a monthly research dataset. A job-posting aggregator may care about first-seen, last-seen, and stale-record rules, while a directory feed may care most about stable company identity and duplicate resolution.

Define at least these inputs:

• the agreed sources, regions, categories, and page types;
• what constitutes one record or entity;
• required and optional fields;
• normalisation and mapping rules;
• stable-identity and deduplication logic;
• collection and delivery windows;
• acceptable missing, invalid, or late values;
• the destination schema and version;
• the action for each failed rule;
• the evidence the customer receives.

These definitions prevent a common conflict: the collector optimizes for technical completion while the customer judges business usefulness. Both sides need the same definition of done.

For product, price, and stock monitoring, “price” may mean the visible sale amount excluding delivery, including tax, for a specific variant and seller in a specific currency. For job postings and labor-market data, “active” may mean present on the source at the observation time rather than guaranteed to accept applications. Quality depends on those distinctions.

Six useful dimensions for web-scraped data

The following framework is practical rather than universal. A project can add or change dimensions, but it should not collapse all of them into one percentage.

1. Coverage

Did collection reach the agreed scope: expected sources, categories, entities, pages, locales, and time windows? Coverage is broader than a successful request count. It requires a model of what should have been observed.

For competitor and market intelligence, coverage might be expressed by tracked entities and signal types. A silent gap in one country site should not disappear inside a global record total.

2. Completeness

Are required fields populated on the records where they apply? Measure completeness by field, source, record type, and run—not only across the entire dataset. A global 99% can hide one source with a broken price field.

Distinguish a true source absence from an extraction failure. A blank optional field, an explicit “not available” state, and a parser that failed to find the value are different facts.

3. Validity

Do values conform to the agreed type, format, range, enumeration, and cross-field rules? JSON Schema can enforce structural constraints such as required properties, types, numeric bounds, and patterns. (JSON Schema)

Structural validity is necessary but insufficient. A value of 99.99 may be a valid decimal while representing the wrong variant, currency, or interpretation.

4. Uniqueness and identity

Does each record represent the intended entity without accidental duplicates or false merges? Retries, pagination overlap, tracking parameters, and several URLs for one item can all create duplicates. Conversely, aggressive deduplication can collapse two genuine variants.

Identity is particularly difficult in real estate listing data, where one property can appear across agencies, disappear, and return with a new listing identifier. Keep raw source identifiers and confidence evidence instead of pretending every match is certain.

5. Freshness

Was each source observed and the output delivered within the agreed window? A successful daily file containing three-day-old observations is not fresh. Track collection time and delivery time separately.

Freshness rules should reflect the source and decision. Do not market a feed as real time unless the source, implementation, and operational design support that claim.

6. Source fidelity

Do representative outputs reflect what the source showed and the interpretation defined in the schema? Source-to-output sampling helps detect plausible parser errors that pass every type check.

Sampling is targeted evidence, not proof that every record is correct. Use a risk-based sample across sources, record types, edge cases, transformations, and changes. Preserve source URLs and observation timestamps so a reviewer can reproduce the comparison where permitted.

Turn requirements into a data-quality contract

A quality contract connects every important field with its meaning, validation, and failure action. It can be a schema document, acceptance table, or machine-readable specification supported by narrative rules.

The example below is deliberately small and hypothetical.

The contract should also define dataset-level rules. Examples include expected source coverage, record-count ranges, duplicate candidate thresholds, maximum age, delivery deadline, and the minimum source-sample plan.

Version the contract. When a field changes meaning or a new status is introduced, downstream consumers need to know which schema produced each delivery. Do not silently repurpose an existing field.

For public business and directory data, a quality contract should explain how aliases, locations, categories, and stable company identities are handled. “Deduplicated” is not enough; the customer needs to know the entity boundary and what evidence is retained.

Automated checks at every stage

Run checks where they can prevent a bad result, not only after the final file exists.

Source and access checks

Validate that each source returned an expected content state. Detect error templates, consent pages, blocked responses, redirects, empty application shells, unexpected locale changes, and missing pagination paths.

Record-level checks

Validate required fields, types, formats, ranges, enumerations, URLs, timestamps, and cross-field relationships. Keep the failed rule and original evidence for quarantined records.

Transformation checks

Test currency, unit, date, category, and status mappings. Preserve raw source values when they help trace a normalization decision. A normalized value should not destroy the evidence needed to explain it.

Dataset-level checks

Compare source coverage, record counts, null rates, duplicate rates, identity confidence, category distribution, and change rates with the accepted baseline. Segment the metrics so one healthy source cannot mask another.

Delivery checks

Include the schema version, run ID, record manifest, observation window, delivery timestamp, and checksum where the delivery method supports it. Remember the boundary: a checksum proves that transferred bytes match; it does not prove that a price or address is correct.

Validation tools can return more than pass or fail. Great Expectations, for example, documents result formats that can include failed values and row-level evidence. (Great Expectations) Whatever tool is used, the output should support diagnosis without exposing unnecessary sensitive data.

Where experienced manual QA adds value

Automation is strong at repeating known rules. Manual QA is valuable where meaning, ambiguity, and change are difficult to encode.

Use targeted manual review for:

• representative source-to-output sampling;
• first acceptance of a new source or schema;
• ambiguous statuses and values;
• uncertain product, property, company, or job identity matches;
• regression testing after a source redesign;
• locale, variant, pagination, and edge-case coverage;
• anomalies that are statistically unusual but may be legitimate.

Manual QA should not mean inspecting every record by hand. It should challenge the assumptions that automation cannot yet prove. Findings should become reusable examples, regression fixtures, or new validation rules when possible.

This is also why source fidelity must stay separate from schema validity. Automation can prove that a currency is a three-letter string. A reviewer may discover that the parser captured the crossed-out list price when the specification requires the current offer.

Do not treat every failure the same

The quality contract should connect failed rules to operational behavior.

Retries are appropriate for transient collection or delivery failures, not for an invalid business value that will be identical on every attempt. Quarantine preserves questionable records and their evidence without contaminating the accepted output. A blocked delivery protects downstream systems when a critical rule fails. An exception report supports an agreed tolerance without pretending the issue did not happen.

Never silently drop a failed record merely to improve the published pass rate. The report should show how many records were received, accepted, quarantined, rejected, and delivered.

What a useful validation report contains

A customer should be able to understand the run without reading crawler logs. A useful report includes:

• feed name, run ID, and schema version;
• collection and delivery timestamps;
• expected and observed source coverage;
• records received, accepted, quarantined, and delivered;
• required-field completeness by important field and source;
• validation failures grouped by rule;
• duplicate, identity, and anomaly results;
• freshness results by source or partition;
• representative source-sampling outcome;
• known exceptions and corrective action;
• destination, manifest, and transfer-integrity evidence.

WebTruffle publishes a synthetic sample data quality report that separates acceptance rules, observed results, quarantined exceptions, and delivery evidence. It is not a customer case study; it is a concrete specimen of the artifact a project can adapt.

For a recurring operational data feed, the report also creates continuity. Teams can compare runs, identify drift, and explain why a delivery was held or corrected.

How to evaluate a provider’s quality process

Ask questions that reveal the operating model:

1. Is a representative source and output sample approved before production?
2. Are source scope, field definitions, identity rules, and acceptance thresholds documented?
3. Can the provider explain coverage, completeness, validity, freshness, identity, and source fidelity separately?
4. Are schema changes versioned and communicated?
5. What is automated, and where is manual QA used?
6. What happens to rejected or uncertain records?
7. Which rule failures stop a delivery?
8. How are corrections and backfills handled?
9. What evidence accompanies each run or incident?
10. Can the provider discuss limitations without promising impossible universal accuracy?

A provider should be comfortable saying that a field is unavailable, a source is unsuitable, or a match is uncertain. Honest boundaries are part of data quality.

Frequently asked questions