Build a custom data feed that survives production.

A production custom data feed is a versioned contract between a publisher and a consumer—not merely a scheduled export. Define the decision it supports, the record grain and stable key, the coverage and cutoff, the meaning of nulls and deletions, the artifact format, the delivery protocol, the validation gates, and the recovery promise before building the recurring job.

The safest general pattern is to publish immutable, edition-addressed artifacts plus a small machine-readable manifest. The publisher validates an edition in staging, makes its files available, publishes the manifest, and updates a mutable latest pointer last. The consumer verifies the manifest, bytes, schema, identity rules, and reconciliation totals in its own staging area before promoting the edition.

This guide is source-neutral. The input can be a database, partner API, internal application, sensor, public dataset, survey, or authorized collection workflow. It is also format-neutral: a custom feed may be a file, API, event stream, warehouse share, or a combination. The design starts with the consumer's required behavior, not the transport.

How to build a custom data feed: the short answer

Use this sequence:

1. Name the consumer decision or process that fails without the feed.
2. List the authorized sources and the observable coverage universe.
3. Declare one row or object grain and one stable record key.
4. Separate source event time, observation time, data cutoff, generation time, and delivery time.
5. Decide whether each delivery is a full replacement, an upsert/delete delta, an event ledger, or a replayable combination.
6. Define required fields, types, units, vocabularies, null reasons, and source-versus-derived provenance.
7. Choose CSV, JSON/JSONL, Parquet, an API, or events from the consumer's tools and scale.
8. Publish a manifest with edition identity, schema version, cutoff, files, byte counts, row counts, hashes, coverage, and known gaps.
9. Validate to a unique staging location, publish edition files, publish the manifest, then update the latest pointer last.
10. Make the consumer verify and stage every edition before a transactional replacement or delta application.
11. Measure accepted delivery, data lag, coverage, validity, and recovery with explicit denominators.
12. Version breaking schema or behavior changes, dual-publish when necessary, and retain enough history to replay or roll back.

The W3C's Data on the Web Best Practices reaches the same broad design boundary from a standards perspective: publish descriptive and structural metadata, provenance, licensing and quality information; give datasets and versions persistent identifiers; provide bulk access; preserve identifiers; explain unavailable data; and avoid breaking consumers. A production feed turns those publication practices into a repeatable operating contract.

Start with the consumer outcome

“Send us the data every day” is not a sufficient requirement. It names a cadence but not the job.

Start with the downstream action:

• refresh an executive dashboard before 08:00;
• update product availability without deleting unchanged products;
• train a model from an approved, reproducible snapshot;
• route newly qualified records into a CRM;
• reconcile yesterday's transactions against a ledger; or
• reproduce the state that supported a regulated decision.

The action determines the acceptable latency, history, delivery mode, failure behavior, and evidence. A dashboard may tolerate a full nightly replacement. An operational system may need idempotent deltas and explicit deletion events. An audit workflow may need immutable editions and retained source evidence even when a current-state API would be faster.

Write one outcome sentence before discussing technology:

By 07:30 Europe/Bucharest on each scheduled business day, the consumer can replace its catalog-status table with a complete, accepted snapshot representing authorized source observations through 07:00, while retaining the previous 90 editions for rollback.

That sentence exposes decisions the vague request hid: timezone, calendar, data cutoff, full replacement, acceptance, coverage, and retention.

Also define the boundary. A custom feed is not automatically:

• real time because it is delivered through an API;
• complete because every delivered row passed validation;
• current because a file was generated recently;
• a stream because it updates frequently;
• authoritative because it has normalized fields; or
• accurate because its bytes match a checksum.

Each property needs its own definition and evidence.

Write the feed contract before the pipeline

The feed contract should be readable by the people who operate, consume, approve, and pay for it. It can be represented in code later, but begin with a short specification containing these sections:

Do not hide variable scope behind words such as “all,” “latest,” or “complete.” If the real universe cannot be enumerated, describe the collection or selection process that creates the observable denominator. For example, “all active accounts in the approved account registry at the cutoff” is measurable; “all relevant companies” is not.

The contract should also say who owns each decision. The publisher may own extraction and normalization, while the consumer owns match rules, business exclusions, or the final eligibility decision. A custom data collection service cannot repair an undefined business rule by adding more rows.

Choose snapshots, deltas, and replay deliberately

A feed's update model controls how consumers interpret absence.

Full snapshot

A snapshot represents the complete admitted population under a named scope, observation window, and cutoff. It may be assembled over minutes or hours; it is a release view governed by those rules, not necessarily the source's state at one instant. The consumer normally stages and replaces the prior snapshot as one transaction. A missing record can mean deletion only when the publisher guarantees that the snapshot is authoritative for the declared population and the edition completed successfully.

Snapshots are easy to reconcile and recover from, but they can be expensive to transfer and load. They work well for moderate datasets, low-frequency refreshes, and consumers that value simple replacement semantics.

Upsert/delete delta

A delta contains only changes relative to a named base edition or checkpoint. Every row needs an operation such as upsert or delete, a stable record key, an event or observation time, and an edition or sequence boundary. The manifest should declare base_edition_id as well as the new edition_id. Absence from a delta means “unchanged or not represented,” never “deleted.” Deletions require tombstones.

Deltas reduce transfer volume but move more responsibility to both sides. The publisher must capture every relevant change; the consumer must apply operations idempotently and in the right scope. A later retry must not create duplicate state or resurrect an older version.

Event ledger

An event ledger keeps immutable observations or domain events rather than only current state. It supports audit, temporal analysis, and reconstruction, but consumers need deterministic rules to build a current view. Distinguish a source event from “first observed by this feed”; the latter does not prove when the source fact originally changed.

Backfill and replay

Backfill establishes history before the recurring feed begins. Replay reconstructs a delivery interval after failure or reprocesses retained source evidence under a corrected rule. Both need a named window, deduplication key, precedence rule, and a way to distinguish replayed facts from newly observed facts.

Many dependable feeds publish a periodic full snapshot plus frequent deltas. The snapshot provides a reconciliation anchor; deltas provide low-latency change. Define how a consumer detects a missed delta and when it must reset from the next snapshot.

Fix record identity and grain

State the grain in one sentence: “One row is one location as observed at the edition cutoff,” or “One event is one change to one source item.” If that sentence contains “and,” the feed may be mixing grains.

Keep these identities separate:

• feed_id: the long-lived dataset or product series;
• edition_id: one published delivery or snapshot;
• record_id: one stable entity inside the feed's declared scope;
• version_id or event_id: one immutable change or observation of that record;
• source-native identifiers: retained with their system and scope; and
• relationship identifiers: child keys scoped to their parent where required.

A name, title, URL, or array position is rarely a safe stable key. In JSON, identifiers should usually remain strings so leading zeroes and values outside universally safe integer ranges survive across implementations. Reject duplicate JSON object member names rather than relying on parser-specific behavior. If the source has no durable ID, document the composite or matching rule, collision behavior, and effect of source corrections. Never silently merge uncertain matches into one master entity.

When identity spans organizations, brands, products, aliases, and changing source evidence, the competitive intelligence database guide develops this boundary into a reviewable entity, observation, assertion, change, and signal model. For the domain-specific distinction between an organization, brand, physical site, publisher listing, and repeated observation, use the business location data guide.

For real estate, the property listing dataset guide separates the property, addressable unit, source listing, commercial offer, observation, and completed transaction before defining duplicate, relisting, price-history, and lifecycle rules.

Time fields also need separate names:

• source_updated_at: time declared by the source, if trustworthy and available;
• observed_at: time the publisher actually observed the source fact;
• data_cutoff_at: latest admitted observation for the edition;
• generated_at: time the artifact was built;
• published_at: time it became available; and
• accepted_at: time the consumer completed acceptance.

Use offset-aware timestamps. RFC 3339 defines an interoperable Internet timestamp profile with Z or a numeric UTC offset; an unqualified local timestamp is unsafe across systems. A date-only business field can remain a date when an instant would invent precision.

Choose format and delivery from consumer needs

Format and transport are different decisions. CSV can travel over HTTPS, SFTP, or object storage; JSON can be a file, API response, or event payload.

The W3C's tabular-data model and CSV guidance recommends text/csv, UTF-8, a header row, consistent field counts, and metadata capable of describing and validating the table. Apache Parquet is a column-oriented format designed for efficient storage and retrieval of complex bulk data, but producers and consumers still need compatibility tests for the features and logical types they use. Neither is universally “best”; consumer capability and record shape decide.

JSON Lines is a convention that needs an explicit framing and media-type agreement. Do not label newline-delimited JSON as application/json-seq unless it actually uses the record-separator framing defined by RFC 7464.

For HTTP APIs, an OpenAPI description provides a language-agnostic, machine-readable interface contract. For portable event envelopes, CloudEvents defines common event metadata. These standards document the wire interface; they do not replace the feed's business grain, coverage, or acceptance rules.

Select delivery by operational fit:

• versioned HTTPS or object-storage files for simple bulk editions;
• SFTP where an established enterprise transfer boundary requires it;
• an API for consumer-selected subsets or low-volume lookups;
• a queue or webhook for low-latency events; and
• a warehouse load or share when the destination is already the analytical system of record.

Avoid a bespoke protocol when a documented standard path works. Whatever the route, define authentication, paths or topics, filenames, compression, maximum size, timeouts, retry policy, acknowledgement, retention, and the publisher's behavior when the destination is unavailable.

Make the schema semantic, not only syntactic

A type checker can prove that amount is numeric. It cannot prove whether the amount is estimated, transacted, annual, gross, net, or safe to sum.

For every field, document:

• name and human definition;
• source-native field or derivation rule;
• type, format, unit, and precision;
• required, optional, or conditionally applicable status;
• allowed vocabulary and casing;
• null, blank, zero, unknown, redacted, and not-applicable behavior;
• normalization and rounding rules;
• source and observation provenance; and
• whether consumers may aggregate, compare, or use it as an identifier.

For JSON, JSON Schema provides Core and Validation specifications; its current published dialect is 2020-12. In a schema document, $schema identifies the JSON Schema dialect and $id identifies the schema resource; neither replaces the feed's explicit schema_version. For CSV, W3C's Metadata Vocabulary for Tabular Data can describe columns, datatypes and constraints and support validation. A custom schema can also work if it is versioned, complete, testable, and published with the feed.

Preserve raw source values where interpretation matters. Put normalized values in separate fields and name the transformation. If an empty source value could mean several things, do not convert it to zero. Use a reason code such as not_published, not_applicable, redacted, parse_failed, or not_collected, with the raw evidence retained according to policy.

Treat derived fields as governed code. Record the rule version, inputs, timezone, reference vocabulary, and fallback. A field called is_active is not reproducible until the exact rule and cutoff are known.

Publish a machine-readable manifest

The manifest is the handoff between the publisher's build and the consumer's acceptance. It should be small enough to retrieve first and complete enough to decide what to retrieve next.

An illustrative manifest—not a universal standard—might look like this:

{
  "feed_id": "catalog-status",
  "edition_id": "2026-08-18T06:00:00Z",
  "generated_at": "2026-08-18T06:12:34Z",
  "data_cutoff_at": "2026-08-18T06:00:00Z",
  "observation_window": {
    "from": "2026-08-17T06:00:00Z",
    "through": "2026-08-18T06:00:00Z"
  },
  "schema_version": "2.1.0",
  "schema_uri": "https://data.example.com/schemas/catalog-status/2.1.0.json",
  "record_grain": "one source item at the edition cutoff",
  "delivery_mode": "full_snapshot",
  "previous_edition_id": "2026-08-17T06:00:00Z",
  "base_edition_id": null,
  "release_status": "accepted_with_known_gap",
  "files": [
    {
      "path": "records.parquet",
      "media_type": "application/vnd.apache.parquet",
      "bytes": 4819231,
      "rows": 125308,
      "sha256": "<64 lowercase hexadecimal characters>",
      "hash_scope": "exact stored object bytes"
    }
  ],
  "coverage": {
    "expected_partitions": 24,
    "accepted_partitions": 23,
    "missing_partitions": ["partner-17"]
  },
  "quality": {
    "accepted": true,
    "quarantined_rows": 17
  },
  "known_gaps": [
    "partner-17 unavailable at the cutoff; excluded rows remain unknown"
  ],
  "provenance_uri": "https://data.example.com/feeds/catalog-status/provenance",
  "license_uri": "https://data.example.com/feeds/catalog-status/terms"
}

The edition ID should be stable and unique. The cutoff and observation window state what the data can represent; generated_at does not substitute for either. A delta uses base_edition_id to name the state against which its operations apply. File declarations bind a path to exact byte and row expectations. Coverage, release status, and gaps prevent a technically valid partial feed from presenting itself as complete. Versioned schema, provenance, license, and terms links make the artifact interpretable beyond the team that produced it.

A SHA-256 value detects whether retrieved bytes differ from the declared artifact. It does not establish semantic accuracy, completeness, or authenticity on its own. If an attacker can replace both file and manifest, an untrusted hash in that manifest proves nothing. Publish the manifest through an authenticated channel, use trusted HTTPS and access control, or add a verifiable signature when the threat model requires independent authenticity.

For HTTP delivery, RFC 9530 defines Content-Digest and Repr-Digest integrity fields. A manifest-level hash remains useful because it travels with edition metadata and can be retained in the consumer receipt.

Make publication a logical commit

Never build directly into the path consumers treat as current.

Use this publication order:

1. Freeze the source window and data_cutoff_at.
2. Write every artifact to a unique staging location.
3. Validate structure, schema, keys, semantics, coverage, and reconciliations.
4. Calculate final byte counts, row counts, and hashes from the staged bytes.
5. Move or upload artifacts to an edition-addressed path that will not be mutated in place.
6. Publish the edition manifest only after its referenced files are readable.
7. Update the mutable latest pointer last, using the destination's conditional or atomic primitive where one exists.
8. Notify consumers only after the pointer is valid.

This sequence is a logical commit-marker pattern; it does not make a multi-file publication universally atomic. Visibility and consistency guarantees depend on the storage service, so test the actual destination. If the pointer store supports entity tags or generation preconditions, use them to prevent concurrent publishers from overwriting a newer edition. HTTP's conditional request model uses validators such as ETag with If-Match or If-None-Match; RFC 9110 defines the semantics. An ETag is an opaque representation validator and can be weak—it is not a replacement for the separately declared SHA-256 of exact published bytes.

Do not overwrite an accepted edition to “fix it.” Publish a corrected edition with a new ID, link it to the superseded edition, state the reason, and preserve the earlier bytes according to retention policy. Consumers then have an auditable correction path.

Validate before promoting the feed

The consumer should distrust transport success until acceptance completes.

A safe consumer loop is:

1. Read the pointer and manifest; reject an already committed edition ID.
2. Validate the manifest version and required fields.
3. Download to a unique staging location with size limits and bounded retries.
4. Verify the declared byte count and hash before parsing.
5. Validate the artifact schema and supported schema version.
6. Check stable-key uniqueness, operation vocabulary, timestamp formats, units, and applicable null rules.
7. Reconcile rows, partitions, totals, quarantines, and known gaps to the manifest.
8. Load into a staging table or namespace.
9. Replace the snapshot or apply the delta in one database transaction.
10. Commit a receipt containing edition ID, input hashes, validator versions, counts, timestamps, and result.

Retry around the edition ID. If a consumer crashes after loading but before recording the receipt, it must be able to determine whether the edition was committed and safely repeat or roll back. For deltas, store the last committed sequence or edition in the same transaction as the applied changes.

Validation should have layers:

• transport: expected object, bytes, digest, compression, and media type;
• contract: manifest and schema version, headers or properties, and supported types;
• identity: stable keys, scoped child keys, duplicates, and operation rules;
• semantic: dates, ranges, vocabularies, units, applicability, and derived-rule checks;
• reconciliation: partitions, row counts, totals, source controls, and snapshot-versus-delta agreement; and
• historical: unexpected volume, null-rate, duplicate, category, or change-rate shifts against comparable editions.

Threshold breaches should quarantine or halt according to business consequence. Silent fallback—dropping a column, coercing malformed values, reusing yesterday's file, or accepting a partial population—converts an observable failure into plausible bad data.

Measure the data service, not only the job

A green scheduler and HTTP 200 response do not prove that an accepted, current, complete feed reached the consumer.

Define indicators before targets:

• accepted-delivery rate: accepted scheduled editions / scheduled editions;
• delivery timeliness: accepted editions delivered by the deadline / scheduled editions;
• data lag: accepted_at - data_cutoff_at, reported as a distribution and by critical partition;
• coverage: accepted applicable partitions or records / expected applicable denominator;
• validity: observations passing a named rule / observations to which the rule applies;
• change capture: represented known source changes / known source changes in an external control set, where such a set exists; and
• recovery: time from detected missed outcome to safe redelivery and reconciled backfill.

Google's Site Reliability Engineering guidance separates a service level indicator (the quantitative measure), an SLO (the target), and an SLA (an agreement with consequences). Apply that separation to the delivered data. Do not call an internal target a contractual guarantee or select a percentage before defining its numerator, denominator, exclusions, clock, and evidence.

Measure both timeliness and freshness. A delivery can arrive on time while repeating stale source observations; a late delivery can contain current data. Measure coverage separately from field completeness so an easy, small subset cannot hide an unobserved population.

When a source or partition is unavailable, use a reason-coded state such as accepted, expected_quiet, source_unavailable, publisher_failed, consumer_destination_failed, or excluded_by_scope. Keep failures and unresolved records visible in the denominator unless the contract defines a valid exclusion.

For a deeper measurement framework, continue to what a data-feed SLA should measure and the data-quality QA framework. Those guides use web collection as their operating context, but the acceptance, freshness, coverage, validity, fidelity, and recovery distinctions apply to recurring feeds more broadly.

Apply least privilege and data minimization

Only deliver fields the consumer is authorized to receive and needs for the stated purpose. More columns create more exposure, ambiguity, storage, and future compatibility burden.

At minimum:

• document source rights, license, consent, and permitted uses;
• exclude unnecessary personal or sensitive data;
• use separate identities and credentials per consumer and environment;
• grant read-only access to only the required paths, topics, tables, or API scopes;
• rotate and revoke credentials without changing the data contract;
• treat signed download URLs as bearer credentials: keep lifetimes short, restrict scope, provide a refresh and revocation path, and keep them out of logs and analytics;
• encrypt network transport and protect retained artifacts according to their classification;
• keep secrets out of URLs, filenames, manifests, logs, and sample code;
• log access, publication, acceptance, and administrative changes; and
• define retention and deletion for artifacts, raw evidence, logs, and consumer copies.

NIST defines least privilege as restricting users or processes to the minimum resources and authorizations necessary for their assigned tasks. Apply it on both sides: the publisher should not receive write access to the consumer's production database merely to deliver a file, and the consumer should not receive access to unrelated publisher data.

Webhooks require special care because they cause the publisher to connect to a consumer-supplied address. Validate destinations, prevent server-side request forgery, sign messages, set short timeouts, retry with backoff, and provide replay without allowing one slow consumer to block others. An object-store inbox or queue can be a safer boundary when low latency does not justify webhook complexity.

Version and migrate without surprising consumers

Track at least four versions independently:

1. data edition: the facts represented at one cutoff;
2. schema or semantic version: the fields and their meanings; and
3. source schema or format version: the upstream representation the publisher interpreted; and
4. pipeline or parser version: the transformation code that produced the edition.

Track the transport or API version separately as well when retrieval behavior changes independently of the data artifacts.

A new daily edition is not a schema change. A new API path is not proof that field semantics changed. A renamed enum or altered null rule can be breaking even when the JSON type stays string.

Classify proposed changes by consumer effect:

• additive optional field;
• new enum value;
• wider precision or longer text;
• changed requiredness or null behavior;
• renamed, removed, or retyped field;
• changed record grain, key, deletion rule, ordering, cutoff, or derivation; and
• changed authentication, path, pagination, or acknowledgement behavior.

Even an additive CSV column can break a consumer that loads by position or rejects unknown headers. Test actual consumers rather than relying only on a “non-breaking” label.

For a breaking change:

1. publish the proposal, examples, schema, and migration mapping;
2. provide a realistic test edition;
3. dual-publish old and new versions for an agreed window;
4. require consumer acceptance evidence for the new version;
5. announce a dated retirement policy and fallback;
6. monitor both paths and reconcile their intended differences; and
7. retire the old version only after the agreed gate passes.

Maintain a changelog that explains semantic as well as structural differences. The W3C best practices explicitly recommend a version indicator, version history, persistent identifiers for versions, and avoidance of breaking API changes. Versioning is a communication and migration discipline, not merely a number in a filename.

Custom data feed implementation checklist

Before production, verify that the following statements are true:

• The consumer outcome and accountable owner are named.
• Every source is authorized and its terms are recorded.
• The included universe, partitions, exclusions, and cutoff are measurable.
• One record grain and stable-key rule are written in plain language.
• Snapshot, delta, delete, event, backfill, and replay semantics are unambiguous.
• Every delta names its base edition or checkpoint and ordering rule.
• Source time, observation time, cutoff, generation, publication, and acceptance clocks are distinct.
• Every field has a definition, type, unit, applicability rule, null behavior, and provenance.
• Format, compression, filenames, protocol, authentication, size limits, acknowledgement, and retries are specified.
• The manifest declares edition, schema, files, bytes, rows, hashes, coverage, quarantine, and known gaps.
• Publication uses staging and updates the current pointer last.
• The consumer validates and promotes transactionally.
• Retries are idempotent and missed deltas have a reset or replay path.
• Service indicators have formulas, denominators, clocks, evidence, and owners.
• Security follows data minimization and least privilege.
• Retention, correction, rollback, support, version migration, and exit are tested.

A small feed that satisfies this checklist is more production-ready than a sophisticated pipeline whose consumer behavior is implicit.

Frequently asked questions