Real estate data · Schema and lifecycle
Build a property listing dataset that survives relistings.
Design a property listing dataset with separate property and listing IDs, normalized prices and units, relisting logic, lifecycle history, provenance, and QA.
A dependable property listing dataset keeps the real-world asset, its addressable units, each advertisement, each offer, each advertiser, every dated observation, and any completed transaction as related but different records.
A listing is not the property. A disappearance is not a transaction. An asking price is not an achieved price. Those three boundaries prevent a large share of false duplicates, invented sales, broken price histories, and misleading market comparisons.
The useful output is not one row per URL. It is a versioned evidence system that can answer which properties or units were advertised, by whom, on what terms, at what time, under which coverage rules, and with what uncertainty.
Property listing dataset: the short answer
- 01
property_idProperty
The land, building, or durable physical asset.
Not the advert and not proof that a unit is available.
- 02
unit_idUnit
A separately marketable space inside or on the property.
Unit 4 and the whole building must not share one identity.
- 03
listing_idListing
One publisher's source-scoped advert record.
Syndicated copies keep separate listing identifiers.
- 04
offer_idOffer
The commercial proposition represented by one or more listings.
A price change can update an offer without creating a new asset.
Observation observation_id
What a named source showed at a capture time: price text, status, agent, images, fields, and evidence reference. An observation can fail without changing the real-world offer state.
Transaction transaction_id
A completed sale, lease, or other recorded outcome with its own authority and effective date. Listing removal alone does not create a transaction.
- Property → unit
- contains · valid_from · valid_to
- Listing → offer
- publishes · source · first_seen
- Offer → observation
- observed_as · observed_at · evidence_id
Build the dataset in this order:
- Define the market question, covered geography, property segment, channels, and time window.
- Separate property, addressable unit, listing, offer, advertiser, observation, and completed-transaction grains.
- Register each source with its access basis, subject grain, field scope, cadence, rights, and blind spots.
- Assign stable internal IDs while retaining source-scoped identifiers and every match decision.
- Preserve source-native values before parsing names, addresses, property types, areas, prices, or dates.
- Detect exact duplicates, syndicated copies, materially revised listings, and genuine relistings as different relationships.
- Store asking price, rent period, deposit, fees, incentives, taxes, and derived comparable values separately.
- Build listing states from successful dated observations and explicit source events, never from one missing page.
- Retain source, posted, observed, changed, effective, withdrawn, transaction, registration, and publication times.
- Test identity, extraction, normalization, lifecycle, history, and delivery with representative labeled examples.
- Publish reproducible current views, append-only history, deltas, manifests, and documented limitations.
- Approve licenses, media use, personal-data handling, security, corrections, and retention before collection.
Use this guide to design and operate the data model. If you need a maintained output for specified markets, fields, history, cadence, and delivery, use the managed real estate listing data service. For publication mechanics, continue with the custom data feed guide.
Start with the market question and coverage universe
“Collect property listings” is a topic, not a specification. A search product may need active residential sale offers. A rental analyst may need advertised monthly rent and listing duration. A development team may study newly marketed units. A valuation workflow may compare current asking evidence with registered transactions. Each question creates a different universe, denominator, and acceptable lag.
Write a decision contract before choosing sources:
| Contract item | Question | Bounded example |
|---|---|---|
| Decision | What action will the data support? | Compare newly advertised apartments within selected districts |
| Universe | Which listings are eligible? | Public long-term rental offers from named channels |
| Grain | Property, unit, listing, offer, or observation? | One source listing per observation, resolved to a unit where supported |
| Coverage | Which places, types, and price bands? | Named municipal boundaries and residential categories |
| Freshness | How quickly must change appear? | Daily observations with a documented publication cutoff |
| Evidence | What supports identity and state? | Source key, URL, observed payload, match decision, and status evidence |
| Exclusions | What must not be inferred? | Do not infer a sale, tenancy, or achieved price from disappearance |
A source partition is the smallest collection slice whose completeness can be evaluated independently—for example, one source, geography, property type, query, and page range. Describe the universe as an intersection: named sources, successful source partitions, geographic boundaries and versions, transaction mode, property classes, advertiser types, visibility rules, price or area eligibility, and observation window. “All listings in a city” is indefensible unless an authoritative universe exists and coverage is measured against it. “Listings observed on three named portals in successfully collected partitions” is narrower but auditable.
Record exclusions and unknowns. Off-market homes, private networks, unindexed pages, gated inventory, source caps, query-result limits, regional outages, and delayed updates all shape the data. A zero count can mean no eligible listings, no access, a failed collector, a changed interface, or an overly narrow filter. Keep those states separate.
Geography also needs operational meaning. A postal label, administrative area, sales territory, neighborhood, and map polygon are not interchangeable. Store source-native labels and coordinates, then derive membership under a named boundary version. The approach parallels the evidence layers in the business location data guide, but a residential address can identify several units and may require stricter exposure controls.
Separate property, unit, listing, offer, and observation
The central modeling decision is record grain:
- Property: the persistent real-world parcel, building, structure, or other asset defined for the project.
- Addressable unit: a separately identifiable flat, suite, lot, room, or sub-property that can be marketed independently.
- Listing: one publisher's advertisement or listing record, normally tied to a source-scoped identifier or URL.
- Offer: the commercial terms presented through a listing, such as sale, lease, auction, or short-term rent.
- Advertiser: the organization, branch, agent, owner, developer, or other party attributed by the source.
- Observation: the exact listing state successfully seen at a time, including source fields and evidence.
- Completed transaction: a separately sourced recorded event with its own source identifier and the fields made available under that source's scope.
A property can contain many addressable units. One unit can have sequential or overlapping listings. One listing can expose more than one offer, such as sale and rental alternatives, or a price range for several units. One advertiser can publish listings across offices, and a listing can name several participants. Model those relationships instead of flattening them into a mutable row.
The Schema.org RealEstateListing type makes one useful distinction: it represents an overall listing as manifested on a web page and says that listing describes one or more real-estate offers. The type is also marked as being in Schema.org's “new” area. Treat it as developing web vocabulary, not as proof that every publisher uses the same semantics or that the markup is complete.
Start with separate tables or equivalent event-stream entities:
propertiesandunitshold stable internal identities and typed relationships;listingshold publisher, source key, canonical source reference, represented subject, and listing lineage;offershold transaction mode, currency, price components, rent period, availability, and terms;advertisershold source-attributed party identity and role;observationshold observed time, response state, source-native payload reference, and collector version;listing_eventshold detected or explicit state changes with evidence and policy version;transactionshold independent completed-event evidence and any cautious resolution to property or unit.
Do not require every listing to resolve to a property before publication. Publish a source listing with an unresolved or ambiguous match state when the evidence is insufficient. A false merge corrupts history for two assets; an unresolved listing can be revisited when new evidence arrives.
Keep relationship validity intervals. An advertiser can change while the property remains the same. A building may be converted into several units. Two previously separate addressable records may later be proven to be one, or one ambiguous record may be split. Retire IDs through redirects and resolution events rather than rewriting old observations.
Register sources, rights, and coverage limits
A source register should answer what a source asserts, what it cannot establish, how it may be accessed, and how its evidence can be retained or redistributed. For each source, record the owner, entry point, approved method, authentication, rate limits, geography, subject grain, identifiers, update behavior, rights, attribution, field-level restrictions, media rules, expected partitions, and operational owner.
Standards are not data licenses. The RESO Data Dictionary defines standard resources, fields, and lookups for exchanging real-estate data, while the RESO specifications page publishes current technical materials. RESO can help align names and meanings, but its Web API guidance explicitly says that RESO creates standards rather than providing MLS data or credentials. Access rights and permitted uses still come from the relevant MLS, broker, publisher, agreement, or licensed feed.
Register field authority rather than declaring one globally preferred source. A portal may directly assert its listing state and advertised price. An advertiser's own page may clarify terms. An address reference can support a normalized location identifier. A land registry can supply a recorded transaction. A geocoder supplies a derived coordinate claim. None automatically answers all the others' questions.
For every collection run, preserve enough evidence to distinguish:
- a successful observation of a listing;
- a successful observation that the source explicitly marks inactive;
- a successful source partition in which the listing was absent;
- a redirect or replacement key;
- an access denial, throttle, timeout, parser failure, or partial response;
- a source-wide change in totals, fields, pagination, or visibility.
Coverage must be measured at the same partition used for absence logic. If the source exposes results through geography and property-type searches, record which queries completed and whether caps or pagination made them exhaustive. Never translate “not returned by one query” into “withdrawn” when the query was partial.
Retain source-native values before transformations where permission and proportionality allow. At minimum keep the source and source key, request or page reference, observed time, response state, content hash, parser version, and extracted values. A hash shows byte equality, not truth, legality, completeness, or identity.
Design stable identifiers and match states
| Evidence | Supports | Permitted weight | Does not prove | Hard conflict |
|---|---|---|---|---|
| 01Authoritative parcel or building ID | Property | Strong when namespace and geography agree | A unit, listing, offer, or occupancy | Different live authoritative IDs for the same claimed asset |
| 02Normalized address | Property candidate | Supporting; stronger with address components and country rules | A unique property or unit | Incompatible country, postcode, or locality |
| 03Unit, floor, area, and access | Unit | Strong when several structured attributes agree | The parent property's identity | Different unit number plus incompatible floor or area |
| 04Publisher listing ID | Listing | Deterministic only inside that publisher's namespace | Cross-publisher identity or a durable offer | One source ID simultaneously represents incompatible units |
| 05Agent, price, text, and images | Syndication candidate | Corroborating fingerprint when captured close in time | Property ownership or transaction completion | Same creative reused across incompatible locations |
| 06Coordinates or geometry | Property candidate | Supporting at declared precision and reference system | Unit identity or physical entrance | Point falls outside the declared parcel or locality |
| 07Temporal overlap | Offer or relisting | Required context for duplicate and succession decisions | A match by itself | Two supposedly identical units advertised concurrently as distinct stock |
Store the decision state
accepted, candidate, rejected, or unresolved—with evidence IDs, rule version, reviewer, and effective time.
Merge rule
Calibrate thresholds against reviewed pairs for the actual geography and asset class. A high score never overrides a hard conflict.
Create opaque internal IDs for properties, addressable units, listings, offers, advertisers, observations, and transactions. They should survive changes to a title, URL, agent, price, photography, description, address formatting, or coordinates. Do not construct the permanent property key from mutable presentation fields.
Store external identifiers as scoped aliases with issuer, namespace, entity type, source, jurisdiction, valid interval, observation, and verification state. A portal listing ID identifies a publisher record. An agent's reference can be reused under that agent's rules. A cadastral, parcel, title, building, or address key has a jurisdiction-specific definition. These are strong features only within their documented scope.
The UK government's property and street identification standard calls UPRNs the unique identifiers for every addressable location in Great Britain and says property and street identifiers should remain the same over time and be easy to exchange. That makes a validated UPRN valuable for GB records. It is GB-specific, not a universal property identifier, and an addressable location is not automatically identical to the project's property or marketed-unit grain.
Generate match candidates, then decide. Candidate features can include exact scoped IDs, country-aware address components, unit designators, spatial distance, property type, room and area attributes, development names, floor level, advertiser references, image fingerprints where permitted, and temporal compatibility. Apply hard conflicts—different verified unit IDs, incompatible addresses, or overlapping evidence for distinct units—before fuzzy similarity.
Use explicit decision states:
- Accepted: evidence meets the versioned rule for this entity grain.
- Rejected: evidence supports distinct entities or violates a hard constraint.
- Ambiguous: a plausible link exists but is unsafe to accept.
- Unreviewed: a candidate has not passed a rule or human review.
Store the compared records, feature values, missingness, rule or model version, threshold, outcome, reason codes, reviewer, and decision time. A 2026 NIST Grant/Contractor Report uses entity resolution for data integration as an illustrative example. It frames the core task as deciding whether records from multiple sources refer to the same entity and names subtasks including cleaning, labeling and annotation, feature extraction, reduction, and manipulation. In property data, match evaluation must also be segmented by unit density, address quality, source pair, language, geography, and property type.
Optimize for the cost of each error. A false property merge fabricates a price history and can blend two units. A false split inflates inventory and hides syndication. Measure precision and recall on representative labeled pairs, publish support counts, and keep an ambiguous queue. One opaque confidence score is not provenance.
Model duplicates, syndication, and relisting explicitly
| Observed | Source event | Evidence bundle | Ledger decision | Interpretation boundary |
|---|---|---|---|---|
| 1D01 · 09:00 | Portal A · A-731 appears | Unit, agent, asking rent, photos | Create listing A-731 and offer O-18 | New source record; first observation |
| 2D01 · 11:20 | Portal B · B-844 appears | Same unit, agent, rent, and image set | Keep listing B-844; link to offer O-18 | Syndicated duplicate, not a second unit |
| 3D09 · 08:40 | A-731 asks 2,550 GBP pcm | Prior captured value was 2,400 GBP pcm | Append price-change event to O-18 | Same listing and offer; value changed |
| 4D21 · 07:30 | A-731 and B-844 absent | Two successful captures; no completion source | Close source-active intervals; offer status = unconfirmed | Removal is not a transaction |
| 5D58 · 10:10 | Portal A · A-992 appears | Same unit; new source ID; 37-day gap; revised price | Create offer O-31; link relisted_from O-18 | Relisting candidate preserves both histories |
| 6D76 · effective date | Registry records completed lease | Authoritative transaction reference and date | Create transaction T-07; link to unit | Outcome remains distinct from listing lifecycle |
- Duplicate
- Two source listings represent one offer at overlapping times; retain both source records.
- Relisting
- A later offer succeeds an earlier one for the same unit; create a new offer and link them.
- Transaction
- A separately sourced completed outcome; never derive it from disappearance alone.
“Duplicate” covers several phenomena that need different treatment:
- the same source record collected twice;
- two URLs or IDs from one source representing the same listing;
- a syndicated copy of one underlying advertisement across publishers;
- several agents concurrently advertising the same property or unit;
- a listing withdrawn and later republished;
- a new offer for the same unit with materially different terms;
- two distinct units that share an address, plan, price, or photographs.
First deduplicate observations by source, source key, observed time, and payload identity. Then resolve source listings to listing lineages and, separately, to property or unit identities. A syndicated pair may share one underlying marketing instruction yet remain two source listings because each publisher has its own state, price display, timestamps, and rights.
Preserve directed relationships such as syndicated_from, mirrors, replaces, relisted_as, same_marketing_episode, and same_unit. Record the evidence and valid interval. Do not collapse the records and discard the publisher-level history.
Relisting requires a policy, not a single inactivity-gap threshold. Useful evidence includes a stable source or agent reference, exact address or unit ID, advertiser continuity, unchanged photographs or text where permitted, compatible property attributes, price continuity, explicit prior-listing links, and the elapsed gap. A later listing can represent the same marketing episode, a new instruction, a changed tenure, a new unit in the same development, or a genuinely new sale cycle.
Create a marketing_episode_id only when the project needs that concept, and retain the match decision behind it. Define whether a material transaction-mode change, advertiser change, long gap, property conversion, or verified intervening transaction starts a new episode. Different analytical questions may require different episode policies, so version the policy and make the raw listing lineage recoverable.
Exact and near-duplicate text or image hashes are features, not final proof. Shared templates, staged photos, development renders, virtual tours, and copied descriptions can span several units. Perceptual fingerprints can also be sensitive to crop, watermark, compression, and rights. Use them only under approved terms and alongside address, unit, source, and temporal evidence.
Report both source-listing counts and resolved-unit counts. The difference is not automatically an error: it may be legitimate syndication or simultaneous marketing. For each published metric, state whether the denominator is observations, listings, marketing episodes, units, or properties.
Normalize location, property types, and area units
Preserve four address layers: source-native text and components, parsed components, normalized components, and a use-case-specific resolved address. Keep unit, building, street, locality, postal, and country fields distinct. Store alternative names, scripts, entrances, and geocoding inputs without replacing the source value.
A standardized address does not prove property identity. Apartment numbers can be omitted, reformatted, translated, or intentionally generalized. New developments may use plot numbers before postal allocation. Rural properties may rely on names. Several units can share a coordinate. Generate candidate links from address evidence, but require the match policy to handle these cases.
Store coordinates as derived assertions with provider, input, method, result time, coordinate reference system, precision class, native confidence, and license. RFC 7946 specifies GeoJSON positions in longitude, latitude order and uses WGS 84 decimal degrees. Record that order explicitly; a swapped pair can still be numerically valid and geographically wrong.
Keep source property types and map them through a versioned crosswalk. “Apartment,” “flat,” “condominium,” “studio,” “maisonette,” “unit,” and “residential property” can encode different legal, regional, structural, or marketing ideas. Map exact, broader, narrower, related, and unmapped states instead of forcing every term into one global enum.
Area values need the original number, original unit, area basis, included spaces, measurement standard if stated, conversion method, and derived values. Gross internal area, net internal area, usable floor area, plot area, living area, and lettable area are not synonyms. The International Property Measurement Standards aim to create a uniform approach to measuring buildings, but a listing should only be labeled with an IPMS basis when the source actually asserts or supports it.
Convert square feet and square metres through a documented rule without discarding the original. Do not infer precision from display decimals. Keep ranges as ranges, and distinguish “from” values from exact claims. Run plausibility tests by property type and geography, but route suspicious values to review rather than silently clipping them.
Separate asking price, rent, fees, and derived values
Asking sale price
amount=425000 · currency=GBP · type=asking_saleNot the completed consideration, valuation, or seller's net proceeds.
Asking rent
amount=2400 · currency=GBP · type=asking_rent · period=monthDo not compare with weekly or annual rent until period rules are explicit.
Fees and charges
amount=285 · type=service_charge · period=monthKeep deposit, tax, service charge, incentive, and utilities as separate claims.
Price qualifier
qualifier=offers_over · negotiable=unknownThe qualifier changes meaning even when the numeric amount is identical.
Price per area is a calculation, not a source value.
Keep the source price and source area intact. Publish the normalized result with formula version, area basis, period conversion, rounding rule, and input evidence IDs.
2,400 GBP / month ÷ 86.7 m²
27.68 GBP / m² / month
derived=true · area_basis=internal · formula=v3 · rounded=2dp
Unknown is a valid value. Do not silently assume tax inclusion, rent period, furnished state, area standard, exchange-rate date, or whether a displayed range is inclusive.
An asking price is not an achieved price. Store every price as a typed assertion, not one generic numeric column:
- amount, currency, and amount type: exact, from, guide, range, auction guide, price on application, or unknown;
- transaction mode: sale, long-term rent, short-term rent, shared accommodation, auction, or another defined class;
- rent period and payment basis: per week, month, quarter, year, night, room, unit, or area;
- tax treatment, service charges, ground rent, association fees, utilities, deposits, premiums, incentives, and concessions;
- furnished or unfurnished state and any occupancy or lease conditions that affect comparability;
- source, listing, offer, observed time, stated effective time, and source-native display text.
Retain the source string because qualifiers carry meaning. “Offers over,” “guide price,” “from,” and “per calendar month” cannot be recovered safely after reducing the field to a number. A price range should have lower and upper values, not an average invented by ingestion.
Derived comparable values belong in separate fields with a method version. Monthly rent derived from weekly rent needs an explicit annualization convention. Price per area requires a compatible area basis, nonzero value, and known currency. Currency conversion requires the rate source, rate date, base and quote currencies, and rounding rule. Never overwrite the source amount with the converted amount.
Distinguish a source-observed price change from a pipeline correction. If yesterday's parser misplaced a decimal, correcting the parsed value is not evidence that the advertiser changed the price. Preserve the source observations, correction record, parser versions, and current resolved view so analysts can exclude processing artifacts.
For missing price, use reason codes such as not published, price on application, source redaction, access restriction, parse failure, not applicable, or unresolved range. A null without reason mixes commercial behavior with data failure.
Map listing lifecycle without inventing transactions
A listing-state model describes visibility and source claims. It does not describe the legal or commercial completion of a property transaction unless independent evidence supports that event.
Use source-native statuses plus a conservative project vocabulary:
- candidate or first seen;
- active under a successful observation;
- changed with typed field differences;
- source-marked pending, under offer, reserved, contingent, withdrawn, expired, rented, or sold, preserving the native term;
- not observed under successfully covered scope;
- stale after a documented observation window;
- unavailable because access or collection failed;
- inactive under explicit evidence or a versioned absence rule;
- replaced, redirected, or relisted;
- reopened after a later active observation.
A disappearance is not a transaction. It may indicate withdrawal, expiry, sale, let, suppression, URL migration, publisher policy, search-result cap, access failure, or source redesign. Record the disappearance as an observation or absence event, and preserve any source-marked sold or closed status as a source claim. Publish a completed transaction only when a separately scoped transaction source—or other evidence whose meaning and limitations have been validated for the jurisdiction—supports it.
Separate source status from project interpretation. RESO's StandardStatus lookup defines listing-contract states such as Active, Pending, Closed, Expired, and Withdrawn, with distinct meanings. A portal's “sold subject to contract,” “pending,” or “under offer” label still has source-specific meaning. Retain the exact value, then map it to a broader analytical class through a versioned crosswalk. Avoid translating provisional marketing states into a legal completion.
Require successful coverage before absence logic runs. A listing can become “not observed in completed partition X at time T”; it cannot become “withdrawn” because a request timed out. Use repeated absences only where the source behavior and decision risk justify them, and state the window. Reappearance should reopen the listing or link a relisting without erasing the intervening evidence.
Keep lifecycle derivation reproducible: input observations, partition coverage, explicit status, rule version, decision time, outcome, and reviewer if applicable. Reprocessing under a better rule may change the derived history; publish that as a correction or restatement, not as a fictional source event.
Preserve every relevant time axis
Property listing data contains several clocks:
- source time: when the publisher says a record was posted, modified, available, withdrawn, or updated;
- observed time: when the collector successfully saw a representation;
- processed time: when extraction, normalization, matching, or geocoding ran;
- effective time: when an offer or relationship is believed to apply;
- decision time: when a rule or reviewer accepted a match or state;
- transaction time: the event date supplied by a completed-transaction source;
- registration time: when that transaction entered an official register;
- publication time: when the dataset release became available to consumers.
Do not substitute one clock for another. “Days on market” can mean source-posted to source-withdrawn, first observed to last observed, or marketing-episode start to end. Publish the formula, included states, timezone, gap treatment, relisting policy, and censoring rule beside the metric.
Store timestamps with timezone or a documented source-zone assumption. Keep date-only values as dates when the source supplies no time. A collection timestamp does not add precision to a source date. Retain both first-seen and source-posted times because neither is guaranteed to equal the true start of marketing.
Use bitemporal thinking where corrections matter: when the event was believed effective and when the system learned or decided it. That lets a current view improve without pretending the improved knowledge existed in earlier releases.
Compare listing data with completed transactions cautiously
Listing history and transaction history answer different questions. Asking data describes marketed terms within observed channels. Transaction data describes events recorded under the transaction source's rules and timetable. Their populations, dates, identifiers, values, and coverage can differ.
HM Land Registry's Price Paid Data covers property sales in England and Wales that were sold for value and lodged for registration. It is transaction data, not listing or asking-price data. HM Land Registry says its reports use data collected when a transaction is registered and may not reflect the most recent information.
Resolve a transaction to a property or unit through a separate decision. Address similarity alone can confuse flats, new-build plots, renamed properties, and multiple transfers at one address. Keep accepted, rejected, ambiguous, and unreviewed transaction matches, with the transaction source ID and all compared evidence.
When calculating an asking-to-achieved difference, publish the exact pairing rule: which listing or marketing episode, which price observation, which transaction, how multiple offers or transactions are handled, the maximum time interval, currency and area assumptions, exclusions, and match-confidence requirement. Do not label an unmatched disappearance as a sale or impute an achieved price from the last ask.
Report selection limits. Listings without a matched registered transaction may be withdrawals, rentals, out-of-scope transactions, delayed registrations, source gaps, or unresolved identities. A matched subset is rarely representative without analysis.
Measure quality at each boundary
- 01
Source
- Pass oracle
- Expected publishers, partitions, rights, and capture cadence are declared.
- Review trigger
- Blocked page, missing partition, changed terms, or unexplained coverage drop
- 02
Grain
- Pass oracle
- Property, unit, listing, offer, observation, and transaction keys satisfy the schema.
- Review trigger
- Source row collapsed across grains or required parent relation missing
- 03
Identity
- Pass oracle
- Every link has evidence, a method version, and accepted or unresolved state.
- Review trigger
- Hard conflict, ambiguous unit, or suspected syndication cluster
- 04
Lifecycle
- Pass oracle
- New, changed, removed, duplicate, and relisted events follow temporal rules.
- Review trigger
- One failed capture, impossible sequence, or removal treated as completion
- 05
Price
- Pass oracle
- Amount, currency, type, period, unit, qualifier, and derivation lineage validate.
- Review trigger
- Mixed periods, fee folded into rent, or unsupported normalization
- 06
Delivery
- Pass oracle
- Edition counts, deltas, manifests, evidence links, and destination receipt reconcile.
- Review trigger
- Partial batch, orphan relation, stale snapshot, or missing acknowledgement
- Pass
- Release the scoped edition with gate results and evidence references.
- Review
- Hold only the affected records; store the question, owner, and disposition.
- Quarantine
- Exclude unsafe scope and disclose the gap instead of inventing continuity.
Quality should identify where evidence or processing failed:
- Collection: expected partitions attempted and completed, status distributions, throttling, pagination, caps, and evidence retention.
- Extraction: required fields found, parser errors, source-native value fidelity, and fixture regressions.
- Identity: candidate coverage, match precision and recall, false-merge rate, unresolved rate, and review backlog.
- Normalization: address parse outcomes, type crosswalk coverage, unit conversion validity, and unmapped reason codes.
- Offer semantics: currency, amount type, rent period, fee separation, and derived-value eligibility.
- Lifecycle: explicit versus inferred states, absence evidence, transition validity, and reappearance handling.
- History: ordered observations, replayable derivations, corrections, relisting lineage, and time-axis completeness.
- Delivery: schema validation, referential integrity, counts, hashes, atomic publication, and consumer replay.
Every rate needs numerator, denominator, window, filters, and exclusions. “Ninety-nine percent complete” is meaningless without the eligible field set and population. Segment metrics by source, geography, property type, transaction mode, advertiser class, parser version, and difficult identity cases.
Use representative gold sets for property and listing resolution, relisting, price extraction, area interpretation, and lifecycle transitions. Include multi-unit buildings, shared addresses, new developments, rural names, reused photographs, multi-agent listings, long gaps, source redirects, ranges, weekly rents, auctions, and records with missing unit numbers. Keep evaluation examples separate from tuning.
Protect against false merges first where they would fabricate price or transaction histories. Then measure recall so syndicated copies and relistings do not remain fragmented. For lifecycle classification, measure false inactive or sold labels separately from missed transitions because their harms differ.
Monitor distributions as well as pass rates: listing counts, new-to-inactive ratios, null reasons, price changes, area units, price-per-area tails, coordinate precision, match-score bands, source-status maps, observation gaps, and review age. A sudden shift can expose a source redesign before consumers notice.
The web-scraping data-quality framework covers completeness, validity, duplicates, freshness, fidelity, and release gates. The monitoring runbook covers schedules, incident states, and recovery. Apply those controls per source before trusting cross-source identity or market metrics.
Publish current views, history, and deltas
Publish convenient projections without discarding evidence:
- a current property and unit view with explicit resolution policy and as-of time;
- a current listing and offer view with source status and last successful observation;
- append-only observations and typed listing events;
- listing lineage, syndication, relisting, and identity decisions;
- separately sourced completed transactions and their match decisions;
- versioned snapshots, deltas, manifests, data dictionaries, and limitation notes.
A release manifest should name the dataset and release IDs, schema and policy versions, covered source partitions, observation cutoff, files, record counts, byte counts, hashes, quality status, and known incidents. Publish to versioned locations, validate the complete release, then atomically update the current pointer.
Define delta semantics. State whether updates replace a whole record or named fields, how deletions differ from source inactivity, how redirects and merged IDs work, how corrections restate earlier decisions, and how long consumers can replay changes. Use stable event IDs so retries are idempotent.
The W3C PROV overview describes provenance as information about entities, activities, and people involved in producing data. A property dataset can use the same discipline pragmatically: identify the source evidence, the extraction or resolution activity, and the software or reviewer responsible for the published assertion.
Document limitations in every release: covered sources and geographies, excluded inventory, source caps, degraded partitions, field freshness, matching thresholds, relisting policy, and transaction lag. A consumer should be able to reproduce why a listing appeared in a snapshot and why it later changed state.
Govern licenses, media, privacy, and retention
Publicly visible does not mean unrestricted to collect, combine, republish, or retain. Review the exact source, access method, contract or license, database rights, copyright, permitted fields, recipients, geography, retention, and attribution with qualified counsel where appropriate.
Property media needs explicit handling. Photographs, floor plans, virtual tours, descriptions, agent branding, and thumbnails can have rights distinct from structured facts. Store media references, hashes, or derived fingerprints only when permitted and necessary. Do not assume that a feed right includes public redistribution or model-training rights.
Listings can expose personal data: owner-listed contact details, occupied-home photographs, names, direct numbers, precise location, access instructions, or free text. Minimize fields to the decision purpose, classify access, encrypt data, restrict raw evidence, log exports and corrections, and set source-specific retention. Consider whether a generalized public coordinate can coexist with a restricted precise coordinate.
The UK Information Commissioner's accuracy guidance says organizations should keep the source and status of personal data clear, consider accuracy challenges, and correct or erase inaccurate data as appropriate. Build a correction and suppression process; retain only the minimum decision provenance that is lawful and necessary, without continuing unnecessary exposure.
Separate facts from inference. Predicted property type, inferred unit match, estimated area, occupancy guess, likely sale, and automated image classification should be labeled derived, with input evidence, method version, uncertainty, and review state. Do not let an inference become indistinguishable from a source claim.
Document incident response, access review, deletion, backup expiry, and source retirement. When a source right ends, know which raw captures, media, derived fields, and published outputs must stop or be removed. WebTruffle's responsible data and security principles describe the boundary used for managed work. This section is an engineering checklist, not legal advice.
Property listing dataset checklist
Before production, confirm all of the following:
- [ ] The market question, covered sources, geography, types, transaction modes, observation window, and exclusions are explicit.
- [ ] Property, addressable unit, listing, offer, advertiser, observation, event, and completed-transaction grains are separate.
- [ ] Internal IDs survive URL, advertiser, status, price, address-format, and coordinate changes.
- [ ] External IDs retain issuer, namespace, entity grain, jurisdiction, observation, and verification state.
- [ ] Accepted, rejected, ambiguous, and unreviewed matches remain auditable.
- [ ] Source listings, syndicated copies, marketing episodes, relistings, properties, and units are not counted interchangeably.
- [ ] Source-native address, property type, area, price, qualifier, status, and time values survive normalization.
- [ ] Geocodes retain input, provider, method, precision, coordinate order, reference system, and rights.
- [ ] Asking prices, rents, periods, fees, deposits, incentives, currencies, and derived values use separate fields.
- [ ] Absence, collection failure, source inactivity, withdrawal, sale, and completed transaction remain distinct.
- [ ] Source, posted, observed, processed, effective, decision, transaction, registration, and publication times are not substituted.
- [ ] Entity-resolution and lifecycle metrics publish denominators, windows, segments, and support counts.
- [ ] Current views can be rebuilt from observations, events, and versioned policies.
- [ ] Snapshots, deltas, manifests, redirects, tombstones, corrections, and replay behavior are documented.
- [ ] Source access, licenses, media rights, privacy, security, retention, attribution, correction, and suppression have owners.
- [ ] Every published property, listing state, price, or transaction relationship can be traced to evidence and transformation history.
Frequently asked questions
What is a property listing dataset?
A property listing dataset records advertisements and offers for properties or addressable units, together with advertisers, dated observations, lifecycle events, source evidence, and resolution decisions. A dependable design keeps those records separate from completed transactions and publishes its source coverage and time window.
Is a property listing the same as a property?
No. A listing is one publisher's representation of one or more offers. The property is the real-world asset, and an addressable unit may be the specific part being marketed. One property can have many units and many listings over time; one listing can also describe several units or offers.
How should syndicated property listings be deduplicated?
Retain each source listing, then link likely syndicated copies through evidence such as scoped IDs, address and unit compatibility, advertiser references, attributes, permitted media fingerprints, and timing. Publish source-listing and resolved-unit counts separately rather than deleting the publisher-level history.
How can I detect a relisted property?
Use a versioned rule combining stable identifiers, address or unit evidence, advertiser continuity, attributes, price history, permitted text or image features, explicit links, and the inactive gap. Preserve ambiguous cases and distinguish a reused source record, new listing, new offer, and new marketing episode.
Does a disappeared listing mean the property sold or rented?
No. Disappearance can mean withdrawal, expiry, completion, suppression, a new URL, search-result limits, access failure, or a source change. Record successful absence evidence and source statuses. Only publish a completed transaction when separately scoped evidence supports it.
What is the difference between asking price and achieved price?
An asking price is an advertised offer term observed on a listing. An achieved price is a value supplied by a completed-transaction source under its own rules. Linking them requires a defensible property or unit match, a pairing policy, compatible dates and currencies, and explicit selection limits.
What identifier should a property listing database use?
Use opaque internal IDs for properties, units, listings, offers, advertisers, observations, and transactions. Store portal keys, UPRNs, parcel numbers, agent references, and other external IDs as scoped aliases with issuer, jurisdiction, entity grain, evidence, and validity rather than treating one scheme as universal.
How should rental prices and fees be normalized?
Keep the source amount, currency, qualifier, payment period, subject, deposit, service charges, utilities, incentives, and source text. Store annualized or monthly equivalents separately with the exact conversion method. A missing fee should remain unknown unless the source explicitly says it is included or absent.
How often should property listing data be refreshed?
The cadence follows the decision and source behavior. Fast-moving search or alerting may require frequent observations; research may tolerate daily or weekly releases. Define separate freshness targets for collection, source status, price, availability, matching, and publication, then measure each against successful source coverage.
Which quality metrics matter for property listing data?
Measure collection coverage, extraction fidelity, property and listing match precision and recall, unresolved candidates, type and unit normalization, price-semantic completeness, lifecycle error rates, observation gaps, history replay, and release integrity. Publish numerators, denominators, windows, segments, exclusions, and support counts.
Can public property listings be reused freely?
Not necessarily. Visibility alone does not settle contract terms, copyright, database rights, media permissions, privacy, attribution, or redistribution. Review each source and field, minimize personal data, restrict raw evidence, define retention and correction procedures, and obtain qualified legal advice for the intended jurisdictions and uses.