Build a competitive intelligence database from evidence, not noise.

A competitive intelligence database should preserve the path from source evidence to a reviewed business signal. Build it as several related records—tracked entities, governed sources, immutable observations, normalized assertions, changes, and review decisions—not as one wide table of “latest competitor facts.”

That distinction is what lets an analyst answer three different questions without manufacturing certainty:

• What did the source show?
• What changed in the source?
• What does that change mean for the decision we care about?

The first is evidence, the second is a comparison, and the third is an interpretation. A production design keeps them connected but never interchangeable.

Competitive intelligence database: the short answer

Use this sequence:

1. Write the decisions and actions the system must support before listing competitors or sources.
2. Define the tracked universe: organizations, brands, products, offers, locations, people, or another explicit entity type.
3. Register every source with its scope, authority, access basis, cadence, expected behavior, and owner.
4. Give each internal entity a stable ID. Store source-native IDs, domains, aliases, and match evidence separately.
5. Capture immutable observations with the exact source reference, retrieval time, content hash, and parser or method version.
6. Derive normalized facts from observations while retaining the raw value and transformation provenance.
7. Compare compatible observations to produce candidate changes. Do not call every page diff a business event.
8. Preserve effective, published, observed, processed, and delivered times as separate fields when they exist.
9. Model confidence by decision: identity match, extraction validity, source authority, and event relevance are different judgments.
10. Route ambiguous identity, conflicting evidence, and high-impact changes through review states rather than forced automation.
11. Publish a current view for convenient queries and retain the underlying history for reconstruction and audit.
12. Measure source health, applicable-field completeness, match precision and recall, event precision, review backlog, and latency with visible denominators.
13. Put an API, feed, or portal on top only after the evidence and identity layers are stable.
14. Review source rights, contracts, privacy, security, and retention before collection begins and whenever scope changes.

This article owns the competitive-data model. For the cross-domain delivery contract—manifests, atomic publication, replay, and recovery—use the custom data feed production guide. For a managed collection scope, go to competitor and market intelligence data feeds.

Begin with the decision, not the database

“Track our competitors” is not a usable data requirement. It names a subject, not a decision. The same company could be monitored for price changes, product launches, geographic expansion, hiring direction, regulatory activity, partnerships, or messaging. Each question needs different entities, sources, clocks, and evidence.

Write a decision contract before choosing technology:

The contract prevents an expensive failure mode: collecting everything visible and only later asking what the records mean. It also makes missingness interpretable. A missing field that is irrelevant to the stated decision is not a quality defect; a missing field required to approve a high-impact signal is.

Model the grains before the fields

A competitive intelligence database usually needs at least six record grains. Flattening them into one row creates duplicate facts, lost evidence, and ambiguous timestamps.

The core grains are:

• Entity: the internal organization, brand, product, offer, location, or other thing being tracked.
• Source: one governed origin, such as a company newsroom, product page, public filing system, marketplace listing, or careers site.
• Observation: what one source showed at one collection time, including a content or payload hash.
• Assertion: one normalized fact derived from a specific observation, such as a displayed plan price or a stated launch market.
• Change: a typed comparison between compatible assertions or observations.
• Signal: a change or evidence bundle that passed the business rule and review policy for a named decision.

Keep review decisions at their own grain too. A reviewer may accept the entity match but reject the event interpretation, or approve a change while lowering its urgency. If review columns live only on the entity or observation, one decision overwrites another.

Relationships are also first-class. An organization can own a brand, a brand can market a product, a product can have market-specific offers, and a location can belong to a subsidiary. Store a relationship with its own ID, type, validity interval, source evidence, and review state. Do not encode the entire graph in an entity-name string. The business location data guide develops the location case into separate organization, brand, physical-site, listing, address, category, and lifecycle records.

Build a governed source registry

A URL list is not a source registry. The registry should explain why a source exists in the system, what it can support, and how its behavior will be assessed.

For each source, record:

• source_id, canonical URL or endpoint, source type, geography, language, and owning entity;
• decision purpose and expected signal types;
• access method and the authority or permission for that method;
• terms, license, attribution, personal-data posture, and review date;
• expected cadence, normal quiet periods, and freshness target;
• source-native identity fields and timestamp semantics;
• capture mode: structured payload, document, rendered page, or another approved representation;
• parser or mapping version, schema profile, and known limitations;
• owner, escalation route, and active/paused/retired state.

Source authority is field-specific. A company newsroom may be authoritative for its own announcement but weak evidence for a product's current local price. A marketplace can be direct evidence of one seller's displayed offer without proving the manufacturer's list price. Store source_role at the assertion level when that nuance matters; do not give an entire domain one universal truth score.

Define an expected schedule carefully. “No records today” can be normal for a newsroom and suspicious for a catalog expected to contain thousands of products. Source health therefore needs an expectation—scheduled publication, expected partitions, stable watch list, or historical range—not merely an HTTP success.

The W3C PROV model describes provenance around entities, activities, and agents so users can assess quality, reliability, or trustworthiness. You do not need to implement the full ontology to borrow its discipline: identify the evidence entity, the transformation activity, and the software or reviewer responsible for the derived result.

The W3C's Data on the Web Best Practices also separates provenance, version history, persistent identifiers, coverage, and licensing into explicit publication concerns. That is a useful registry audit: a source entry should not be considered governed merely because its URL resolves.

Resolve entity identity without forcing uncertain matches

Competitive intelligence becomes unreliable when aliases are treated as entities or similar names are merged automatically. Build identity as a separate, reviewable process.

Use an internal stable ID that never depends on the current display name. Around it, store scoped evidence:

• official or source-native identifiers, including their scheme and issuer;
• registered domains and source accounts;
• legal names, trading names, brands, former names, transliterations, and local aliases;
• parent, subsidiary, brand, product-family, and location relationships;
• effective dates and the evidence for an alias or relationship;
• candidate-match features, rules, score components, model version, and review outcome.

A matching outcome should be one of at least four states:

1. Accepted: the evidence is sufficient for this use case.
2. Rejected: a hard conflict proves the records refer to different entities.
3. Ambiguous: evidence exists on both sides or does not meet the acceptance rule.
4. Unreviewed: a candidate has been generated but not evaluated.

Do not collapse those into match/no-match. An ambiguous candidate is useful operational state; it is not an error that must be hidden.

Hard conflicts should outrank fuzzy similarity. Different official identifiers, incompatible markets, pack sizes, legal entities, or product variants may invalidate a high name-similarity score. Conversely, a shared identifier is only as strong as its scope: a seller SKU can be unique inside one seller, not across the market.

The retail product price Python workflow shows the deliberately narrower alternative: it analyzes exact-barcode observations and does not expand them through fuzzy product resolution. That boundary is safer than silently treating similar catalog names as the same product.

NIST's 2026 discussion of entity resolution notes that combining datasets depends on determining whether records refer to the same entity and that the work involves cleaning, labeling, feature extraction, and evaluation. Treat matching as a measured subsystem, not a preprocessing footnote. Evaluate accepted matches against a representative labeled set with both precision and recall: high precision reduces false merges, while recall shows how many true links the system misses.

Store observations, facts, and evidence separately

An observation is what the source showed. A normalized assertion is what your system says that observation means. Keep both.

A practical minimum schema looks like this:

The evidence layer should be content-addressed where practical. Store the raw response, document, or approved capture outside the analytical tables, then retain its SHA-256, byte count, media type, storage reference, and collection metadata. A hash proves byte identity, not truth, completeness, or legal permission—but it lets a later reviewer verify which exact evidence supported the assertion. The IETF's Digest Fields specification makes the same boundary for HTTP content digests: integrity metadata does not provide authentication, authorization, or privacy.

Normalized assertions need explicit value types. A price, percentage, date, boolean, identifier, and free-text claim should not all live in one untyped string. Preserve the source-native value beside the typed value and record the transformation version. That lets you repair a mapper and re-derive results without pretending the source changed.

JSON Schema is one option for declaring JSON structure and constraints. Declare the dialect with $schema; otherwise a validator may assume a different specification version. Also remember that formats such as date-time can be annotations unless the chosen validator is configured to assert them. Passing schema validation is necessary, not proof that the business meaning is correct.

Turn observations into change without inventing events

The safest change pipeline has four steps:

1. Compare compatible observations. Same source, same source-native object, same field semantics, and compatible parser versions.
2. Produce a candidate change. Record before and after assertion IDs, the comparison rule, and the detected difference.
3. Classify the cause. New source fact, source correction, late discovery, normalization correction, extraction error, or unresolved.
4. Apply the decision rule. Decide whether the change becomes a signal, enters review, or remains an informational history row.

That separation matters. A page title changing from “Acme Pro” to “Acme Pro Platform” is a source observation. It may be a product rename, an editorial correction, an experiment, or merely a template change. The database should not announce a rebrand until the evidence and rule support that conclusion.

Use stable change IDs so reprocessing does not create duplicate alerts. A useful key can combine the decision contract, entity, source object, field or event type, normalized before/after values, and rule version. Store the key inputs rather than relying only on a hash, so operators can explain a duplicate or collision.

For downstream event envelopes, the CNCF CloudEvents specification offers a useful vocabulary: id, source, type, and specversion are required core attributes, while time, subject, dataschema, and content type add context. Adopting CloudEvents is optional. The design lesson is not: every signal needs a stable identity, origin, type, schema reference, and clearly defined time semantics.

Keep a current-state table or materialized view for fast queries, but rebuild it from accepted assertions and changes. Historical attributes can follow a Type 2 slowly changing dimension pattern: the previous row is expired and a new version is inserted. Microsoft describes that pattern as retaining old rows rather than overwriting them. It is useful for attributes, while a separate event ledger remains better for reviewed business signals.

Preserve effective, published, observed, and processed time

One timestamp column cannot answer competitive-intelligence questions reliably.

Store these clocks separately when available:

• Effective time: when the fact or event applies in the real world.
• Published time: when the source says it published or revised the information.
• Observed time: when your system retrieved the evidence.
• Processed time: when parsing, normalization, or change detection completed.
• Reviewed time: when a person or policy accepted the interpretation.
• Delivered time: when the accepted signal reached its consumer.

Only observed_at is always under your control. Do not copy it into published_at when the source supplies no publication time. Use null plus a reason or availability state.

This is a bitemporal problem whenever facts arrive late or are corrected. Microsoft describes bitemporal history as separating business time—when something happened—from system time—when the system recorded it. Martin Fowler's bitemporal example makes the same distinction between actual and record dates. That separation lets an analyst ask both “what do we now believe happened on Monday?” and “what did our system believe when Tuesday's decision was made?”

Never infer a launch date from first observation alone. The safe label is “first observed on.” Never infer discontinuation from one missing capture. The safe state is “not observed in this accepted run,” followed by a source-specific absence policy and, often, repeated confirmation.

Handle conflicts, corrections, and missingness

Two sources can disagree without either record being corrupt. They may refer to different markets, currencies, products, effective dates, or concepts. Preserve both assertions and resolve them for a named use case.

Use three layers:

• Source assertions: immutable claims tied to observations.
• Resolution decisions: rules or reviews that select, combine, or leave claims unresolved for a purpose.
• Current views: replaceable projections built from accepted resolution decisions.

Do not overwrite a source assertion when a mapping was wrong. Add a correction activity, supersede the derived assertion, and retain the previous result. The source did not change; your interpretation did.

Missingness needs reason codes rather than one null bucket:

• not applicable to this entity or signal;
• not published by the source;
• not captured because the source or partition failed;
• extraction or validation failure;
• withheld or removed;
• not yet reviewed;
• unknown.

Those states drive different actions. “Not published” may be an accepted limitation; “partition failed” blocks a completeness claim; “not yet reviewed” belongs in a queue.

Confidence should also be decomposed. Store identity confidence, extraction validity, source authority, change relevance, and review state independently. A single score of 0.87 hides which uncertainty matters and encourages downstream users to invent thresholds disconnected from risk.

Measure quality at each decision boundary

Use a scorecard with numerators, denominators, windows, and exclusions:

Overall accuracy is not enough for an imbalanced review problem. If only one in a thousand candidate diffs is important, a system that labels everything “not important” appears 99.9% accurate and finds nothing. Report precision, recall, and the reviewed support counts for identity and event decisions.

Sample across sources, languages, entity types, and hard cases—not only random easy records. Keep the gold set outside training and threshold tuning where a model is involved. Re-review a portion over time because product naming, site structures, markets, and decision policies drift.

The general data-quality QA framework explains completeness, validity, duplicates, freshness, fidelity, and publication gates in depth. The monitoring runbook covers schedules, incidents, and recovery. This database adds domain-specific identity, evidence, event, and review tests on top.

The UK Government Data Quality Framework is a useful external reference for this approach: it frames quality as fitness for purpose and calls for continuous monitoring, reporting, and communication of limitations. It is guidance, not proof that a particular competitive-intelligence dataset is accurate.

Choose a database, feed, or portal view

These are consumption modes, not competing sources of truth:

• Database: best for analysts, joins, longitudinal queries, reconciliation, and model training. Provide current and historical views with documented grains.
• Feed: best for systems that need new accepted signals or bounded snapshots on a schedule. Include event IDs, schema versions, manifests, and replay rules.
• Portal: best for human triage, exploration, evidence review, saved views, and annotation. Show why a signal exists and what remains uncertain.

Build the evidence database first. A portal directly over page diffs becomes a noise dashboard; a feed directly over a mutable current table loses replay and correction semantics.

A useful portal result should display:

• entity and exact identity evidence;
• signal type and review state;
• before and after values with units, currencies, and languages;
• source URL and evidence reference;
• every relevant clock with its label;
• rule or model version and reason codes;
• related signals and conflicts;
• an explicit unknown or limitation state.

Do not advertise the interface as “the market.” It represents the registered sources, tracked universe, collection windows, and accepted rules. Show those coverage boundaries beside charts and counts.

Set legal, ethical, and security guardrails

Competitive intelligence is not an exemption from source terms, privacy law, confidentiality, intellectual-property rules, access controls, or internal ethics policies. Review the exact sources, jurisdictions, fields, purposes, methods, retention, and recipients with qualified counsel where needed.

Minimum governance controls include:

1. document the legitimate business purpose and approved sources;
2. use public or properly authorized access and never misrepresent identity or authorization;
3. respect access restrictions and do not bypass technical controls;
4. record terms, license, attribution, and legal-review dates by source;
5. minimize personal data and sensitive content rather than collecting it because it is visible;
6. separate sourced fact from analyst opinion and automated inference;
7. restrict evidence access, encrypt storage and transport, and log review or export activity;
8. define retention and deletion for raw evidence, personal data, assertions, and derived signals;
9. provide a correction and challenge process;
10. re-review governance when sources, purposes, models, fields, or destinations change.

The UK Information Commissioner's Office states that personal data should be adequate, relevant, and limited to what is necessary, and that records should keep their source and status clear. Its data-minimisation guidance and accuracy guidance are useful design references even when another legal regime applies. A joint regulator statement on data scraping likewise warns that publicly accessible personal information remains subject to privacy and data-protection law in most jurisdictions. Public visibility alone does not remove obligations.

WebTruffle's responsible data and security principles describe the operating boundary for managed work. This section is an engineering checklist, not legal advice.

Competitive intelligence database checklist

Before production, confirm all of the following:

• [ ] Every tracked decision has an owner, action, signal definition, evidence requirement, cadence, and exclusion rule.
• [ ] Entity, source, observation, assertion, change, signal, relationship, and review grains are documented.
• [ ] Stable internal IDs are separate from names, URLs, and source-scoped identifiers.
• [ ] Ambiguous matches remain ambiguous; hard conflicts override fuzzy similarity.
• [ ] Every assertion points to an immutable observation and evidence reference.
• [ ] Source-native values survive normalization.
• [ ] Schema, mapper, comparison rule, model, and review-policy versions are stored.
• [ ] Effective, published, observed, processed, reviewed, and delivered times are not conflated.
• [ ] Correction, deletion, absence, and missingness states are explicit.
• [ ] Current views can be rebuilt from retained accepted history.
• [ ] Match and event precision/recall use representative labeled examples and visible support counts.
• [ ] Source health and completeness use expected, applicable denominators.
• [ ] Review queues have owners, severity rules, age targets, and escalation paths.
• [ ] Feeds and portals display evidence, uncertainty, and coverage boundaries.
• [ ] Source rights, privacy, security, retention, and correction processes have been approved.
• [ ] A consumer can trace a chart, alert, or briefing claim back to the exact evidence and transformation path.

Frequently asked questions