Government tender data verified and queried with Python.

The safest way to analyze government tender data with Python is to pin one dated release, verify its manifest and file hash, choose the product whose grain matches the question, and query the verified local file with explicit casts. Do not begin with a moving latest URL or load a large CSV into a dataframe before proving which bytes and records it contains.

This guide uses the tagged and SHA-256-pinned WebTruffle government-tenders-rfps release dated August 8, 2026. The worked product is current-opportunities.csv: 38,210 deterministic actionable candidates across TED, SAM.gov, and Contracts Finder. It is an edition-dated analytical product, not a claim that all 38,210 notices remain open today.

The companion script was executed with Python and DuckDB 1.5.5 against the downloaded release before publication. It verifies the manifest and input files, checks the schema/export boundary, reproduces the expected source counts, and writes four bounded CSV results plus a provenance JSON file.

Government tender data with Python: the short answer

Use this sequence:

1. Choose a tagged release and record its tag, target date, generation time, schema version, and manifest SHA-256.
2. Read the manifest before selecting a data file. Its top-level record_count is not the count of every derivative product.
3. Choose the file by declared grain: edition-day records, current candidates, observations, changes, or analytical summaries.
4. Download the manifest and selected assets to local storage. CSV over HTTP is still a full download in most DuckDB queries.
5. Verify the pinned manifest hash, then verify each selected file's exact byte count and SHA-256 from that manifest.
6. Read CSV headers with UTF-8 BOM handling. Require all core schema fields, but report additive product/export fields separately.
7. Materialize the verified local CSV in DuckDB through Python and initially keep source columns as VARCHAR.
8. Cast only the values needed by one query with TRY_CAST; preserve source strings and identifiers.
9. Reproduce known source, stage, and deadline-bucket counts before trusting a new analytical result.
10. Freeze every time-relative filter to the edition target date. Never let CURRENT_DATE silently turn a reproducible snapshot into a different query tomorrow.
11. Keep lifecycle stages, currencies, value bases, CPV, NAICS, PSC, and source-local buyer identities separate.
12. Export the result with its release tag, target date, input hashes, query method, row count, and runtime version.

If you only need to browse or filter a few records, start with the government tenders and RFP dataset. Use this workflow when the result needs to be repeatable, reviewable, or handed to another analyst.

Pin one release before writing SQL

A reproducible query needs an exact input identity. A convenient latest link answers “what file would I get now?” It cannot answer “which file produced last week's result?” A GitHub release tag alone is not immutable; the pinned manifest hash detects mutation.

For this example, pin all of these values:

release tag       2026-08-08
target date       2026-08-08
generated at      2026-08-09T13:18:11Z
schema version    2.0
manifest bytes    33,479
manifest SHA-256  cbbe78fe20b38cb94ebbf21144638207ba796f2b546eee2f4785b51565a9d2b1

The dates have different meanings. The target date defines the edition's analytical cutoff. generated_at records when the release was built. A source publication date, source update time, observation date, tender deadline, award date, and the time your script runs are separate clocks.

The release manifest declares product grains, windows, counts, filenames, byte counts, hashes, source health, schema version, and governance assets. Verify the pinned manifest hash before trusting the file hashes inside it. Otherwise, a changed manifest could simply declare a new hash for changed bytes.

Do not mistake 270 for every product count

The top-level manifest has record_count: 270. That count belongs to the edition-day tenders product: the latest normalized source records observed on August 8. The same release declares 38,210 current candidates, 91,952 latest stable records in its rolling observed history, 23,337 change observations, and 41 material changes.

There is no contradiction. The products have different grains and windows.

Install the tested Python environment

The recipe requires Python 3.10 or newer and pins DuckDB 1.5.5, the stable release used for this article. DuckDB's official Python documentation supports installation with pip, local CSV relations, SQL queries, and direct CSV export. (DuckDB Python API; CSV reader)

On macOS, Linux, or WSL:

curl -fsSLO https://www.webtruffle.com/examples/government-tender-data-python.py
curl -fsSLO https://www.webtruffle.com/examples/government-tender-data-requirements.txt

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r government-tender-data-requirements.txt
python government-tender-data-python.py

On Windows PowerShell:

Invoke-WebRequest `
  -Uri https://www.webtruffle.com/examples/government-tender-data-python.py `
  -OutFile government-tender-data-python.py
Invoke-WebRequest `
  -Uri https://www.webtruffle.com/examples/government-tender-data-requirements.txt `
  -OutFile government-tender-data-requirements.txt

py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r government-tender-data-requirements.txt
python government-tender-data-python.py

The default data directory is government-tender-data-2026-08-08. Use an explicit location when the evidence must live with a project:

python government-tender-data-python.py \
  --data-dir ./evidence/government-tenders-2026-08-08

A successful run ends with this tested checkpoint:

verified checkpoint: 38,210 candidates; 8,646 in the edition's first two deadline buckets; 2 material deadline changes

If an existing input has the wrong size or hash, the script stops rather than silently replacing it. Move or remove the unexpected file deliberately, then rerun. A missing input downloads into a unique temporary file in the destination directory; the script aborts as soon as the stream exceeds the declared byte count and promotes it only after both size and SHA-256 match.

Choose the artifact that matches the question

“The tender dataset” is not one interchangeable table. The release exposes products for state, observations, changes, current candidates, lifecycle relationships, buyers, summaries, and governance.

Use the minimum product that preserves the needed grain:

• tenders.csv answers which source records were observed in the target-day edition.
• current-opportunities.csv is a convenience view for deterministic actionable candidates as of the edition.
• history-7d.csv supports a dated state join such as (id, observation_date).
• changes.csv is the broader seven-day event ledger and includes new, updated, cancelled, awarded, and closed deltas.
• material-changes.csv narrows the change ledger to updates classified as material.
• rolling-observed-history.csv provides one latest row per stable record seen in the retained rolling window.
• buyer-intelligence.csv is already aggregated at period × source-local buyer × value basis × currency.
• market-summary.csv is one count or aggregation-safe value metric per declared dimension—not one notice per row.

The product name is not enough. Read grain, window_start, window_end, observed_days, and unique_records from the manifest. The so-called 90-day history product in this release contains 26 observed days because retained public history began July 14. A maximum calendar window is not the same as observed coverage.

Verify the manifest, bytes, and SHA-256

The downloadable recipe uses Python's standard library so verification does not depend on the analytical engine. The core pattern is:

import hashlib
from pathlib import Path

def sha256_and_size(path: Path) -> tuple[str, int]:
    digest = hashlib.sha256()
    size = 0
    with path.open("rb") as source:
        for chunk in iter(lambda: source.read(1024 * 1024), b""):
            digest.update(chunk)
            size += len(chunk)
    return digest.hexdigest(), size

actual_hash, actual_bytes = sha256_and_size(
    Path("current-opportunities.csv")
)
assert actual_bytes == 88_200_754
assert actual_hash == (
    "e15c63a4d0fc8c5076bdafe1baa6b4d8"
    "7d81335b9de002a217209c3291c51bc9"
)

For a general release, do not hard-code every product hash. Hard-code or otherwise pin the trusted manifest hash, verify that manifest, then read the selected product's declarations:

import json

manifest = json.loads(Path("manifest.json").read_text())
assert manifest["schema_version"] == "2.0"
assert manifest["target_date"] == "2026-08-08"

declared = manifest["files"]["current-opportunities.csv"]
assert actual_bytes == declared["bytes"]
assert actual_hash == declared["sha256"]
assert manifest["products"]["current_opportunities"]["record_count"] == 38_210

A cryptographic hash establishes byte identity. It does not prove that the publisher, source, normalizer, classification, or analytical interpretation is correct. Those need separate validation.

Validate the schema and CSV boundary

The August 8 current-opportunities.csv has 142 unique headers. schema.json and data-dictionary.json describe 141 core normalized record fields. The additional CSV header is observation_date, a product/export field outside the core record schema.

That means strict header equality would reject the genuine release. The useful contract is “every core field is present; additive export fields are retained and reported.”

This validator expresses that boundary:

import csv
import json

with open(
    "current-opportunities.csv",
    encoding="utf-8-sig",
    newline="",
) as source:
    headers = next(csv.reader(source))

schema = json.loads(Path("schema.json").read_text())
schema_fields = set(schema["properties"])
header_fields = set(headers)

assert len(headers) == len(header_fields)
assert schema_fields <= header_fields
assert header_fields - schema_fields == {"observation_date"}

Use utf-8-sig with the standard-library CSV reader because the file begins with a UTF-8 byte-order mark. DuckDB handled that header in the tested scan, but the explicit Python encoding avoids turning the first name into \ufeffid in another parser.

Several CSV cells retain structured source values as JSON text. Examples include classification-code lists, document and resource URLs, change fields, field-level change evidence, countries, and quality flags. Keep the raw cell; parse it only when the query needs its elements.

Do not let a CSV type sniffer decide the permanent type of a source identifier. Numeric-looking IDs can have leading zeros, dates can carry offsets or incomplete precision, and blank cells can change inferred types across releases.

Load the verified CSV with DuckDB

The recipe uses Python for download and evidence checks, then materializes the verified local CSV in a DuckDB temporary table:

from pathlib import Path
import duckdb

data_dir = Path("government-tender-data-2026-08-08")
connection = duckdb.connect()

connection.read_csv(
    str(data_dir / "current-opportunities.csv"),
    header=True,
    all_varchar=True,
).create_view("opportunities_source")
connection.execute(
    "CREATE TEMP TABLE opportunities AS "
    "SELECT * FROM opportunities_source"
)
connection.execute("DROP VIEW opportunities_source")

all_varchar=True is intentional. DuckDB documents that option as skipping type detection and reading every CSV column as VARCHAR. The query can then apply TRY_CAST to a date, integer, boolean, or decimal without changing the retained source value.

Materialization matters because DuckDB relations are evaluated lazily. The recipe verifies the file, materializes that verified population, rechecks every input with the same read recorded as evidence, and only then exports queries from the temporary tables. The result therefore cannot silently switch to different file bytes during a later lazy scan.

Querying the release URL directly can be convenient for exploration, but it is not the reproducibility path used here. DuckDB's HTTP documentation says CSV files are downloaded entirely in most cases because the format is row-based, while Parquet can support partial reads. Downloading locally first also gives the hash verifier a durable file. (DuckDB HTTP support)

The 88 MB current-candidate CSV and 96 MB buyer-intelligence CSV do not require a pandas copy for these groupings. DuckDB can scan and export them directly.

Reproduce the source, stage, and deadline counts

Before answering a new question, reproduce a few independent checkpoints from the pinned artifact.

SELECT source, count(*) AS records
FROM opportunities
GROUP BY source
ORDER BY records DESC, source;

SELECT record_stage, count(*) AS records
FROM opportunities
GROUP BY record_stage
ORDER BY records DESC, record_stage;

SELECT deadline_bucket, count(*) AS records
FROM opportunities
GROUP BY deadline_bucket
ORDER BY records DESC, deadline_bucket;

The three source rows establish composition: TED contributes 29,406 of the 38,210 records, SAM.gov 8,685, and Contracts Finder 119. Find a Tender contributes no opportunity-stage rows to this public schema 2.0 product. That is a known mapping and observed-population boundary, not evidence that the UK source published no opportunities.

The stage and deadline totals catch different failures. A row-count match with a shifted stage distribution can reveal a schema or parsing change. A source-count match with missing deadline buckets can reveal that a derived column was read incorrectly.

Keep the expected values tied to this artifact. A newer edition is supposed to change; it should pass its own manifest and product assertions rather than be forced to match August 8 forever.

Build a deadline watchlist as of the edition

The current-candidate file stores deadline buckets calculated for the edition. To build a reproducible near-term watchlist, use those frozen buckets and carry the target date into the output:

SELECT
    '2026-08-08' AS edition_target_date,
    source,
    id,
    source_id,
    title,
    buyer_name,
    deadline_at,
    deadline_bucket,
    try_cast(days_to_deadline AS INTEGER) AS days_to_deadline,
    classification_primary_scheme,
    classification_division,
    classification_division_label,
    actionability_basis,
    source_license,
    source_url
FROM opportunities
WHERE deadline_bucket IN ('0-7 days', '8-14 days')
ORDER BY
    try_cast(days_to_deadline AS INTEGER),
    source,
    id;

This query returns 8,646 rows: 4,860 in the 0–7 day bucket and 3,786 in the 8–14 day bucket. It means “candidate deadlines in the first two buckets as of the August 8 edition.” It does not mean 8,646 opportunities are open on August 14.

Avoid this reproducibility trap:

-- Moving result: do not use for a dated release reproduction.
WHERE try_cast(deadline_at AS TIMESTAMPTZ) > current_timestamp

A live operational watchlist can compare deadlines with the current time, but then its evidence must include the query execution time and a fresh source state. That is a different analytical contract.

Find material deadline changes

Materialize the second verified input the same way:

connection.read_csv(
    str(data_dir / "material-changes.csv"),
    header=True,
    all_varchar=True,
).create_view("material_changes_source")
connection.execute(
    "CREATE TEMP TABLE material_changes AS "
    "SELECT * FROM material_changes_source"
)
connection.execute("DROP VIEW material_changes_source")

Then select material updates whose retained changed_fields array includes the deadline field:

SELECT
    '2026-08-08' AS edition_target_date,
    observation_date,
    source,
    id,
    source_id,
    title,
    buyer_name,
    previous_deadline_at,
    deadline_at,
    change_summary,
    changed_fields,
    field_changes,
    source_license,
    source_url
FROM material_changes
WHERE changed_fields LIKE '%"deadline_at"%'
ORDER BY source, id;

The pinned file contains 41 material update observations and this filter returns two material deadline changes, both from Contracts Finder. The query retains the previous and current deadline plus field-level evidence rather than reducing the event to a boolean “changed.”

For a broader change monitor, use changes.csv, whose grain is one observed new or updated stable record per observation date. Its delta_type values also distinguish new, updated, cancelled, awarded, and closed states. Do not count every change row as a new procurement.

Keep procurement taxonomies separate

The normalized product retains a primary classification vocabulary and division. A safe category summary groups inside the scheme and source:

SELECT
    source,
    classification_primary_scheme,
    classification_division,
    classification_division_label,
    source_license,
    count(*) AS records
FROM opportunities
GROUP BY
    source,
    classification_primary_scheme,
    classification_division,
    classification_division_label,
    source_license
ORDER BY
    classification_primary_scheme,
    source,
    records DESC,
    classification_division,
    classification_division_label,
    source_license;

This yields separate CPV, NAICS, PSC, and source-native residual rows. Do not make one league table by sorting every code together:

• CPV describes procurement subject matter and dominates TED and Contracts Finder rows.
• NAICS classifies U.S. business establishments by economic activity and dominates the SAM.gov product.
• PSC describes U.S. products, services, and research-and-development purchases; it is not a NAICS synonym.
• SOURCE identifies a retained source-native classification that was not mapped into one of those three schemes.

When grouping NAICS at sector level, apply official multi-code sectors such as 31–33 Manufacturing and 48–49 Transportation and Warehousing. When one record has several codes, decide whether the analysis counts a retained primary division once or explodes every code into a many-to-many table. Publish that decision with the denominator.

The dated August procurement demand report demonstrates a source- and scheme-separated category analysis. The broader government contract analysis guide owns market measures and disclosure rules.

Analyze buyers without inventing global identities

buyer-intelligence.csv has 61,203 rows. That does not mean 61,203 unique buyers. Its declared grain is:

rolling period × source-local buyer identity × value basis × currency

The file contains 7-, 30-, and 90-day period rows, and a buyer can repeat across value partitions. Before ranking buyers, test whether that declared grain is unique:

WITH declared_grain AS (
    SELECT
        period_days,
        source,
        buyer_key,
        value_basis,
        currency,
        count(*) AS rows_at_declared_grain
    FROM buyer_intelligence
    GROUP BY
        period_days,
        source,
        buyer_key,
        value_basis,
        currency
)
SELECT
    count(*) FILTER (
        WHERE rows_at_declared_grain > 1
    ) AS duplicate_grain_keys,
    sum(rows_at_declared_grain - 1) FILTER (
        WHERE rows_at_declared_grain > 1
    ) AS surplus_rows,
    max(rows_at_declared_grain) AS maximum_rows_per_key
FROM declared_grain;

The August 8 file fails that test: 547 declared-grain keys repeat, contributing 583 surplus rows, with as many as five rows at one key. A seemingly defensive 30-day SELECT DISTINCT still produces 25,360 rows for 25,009 (source, buyer_key) keys; 328 keys repeat, and many repeated keys disagree on unique_record_count or opportunity_count. That makes a one-row-per-buyer ranking unsafe. This is a measured product-quality finding, not a reason to hide duplicates with DISTINCT, MAX, or an arbitrary last row.

Until the aggregate is corrected, retain the rows as published variants or calculate buyer measures from an appropriate record-level product with an explicit source-local key and deduplication rule. Keep source with buyer_key. Some keys are official identifiers; others are normalized published-name keys. A matching name across TED, SAM.gov, and UK data is evidence for entity resolution, not permission to merge automatically. Preserve identity_method and the source-local occurrence before building any cross-source organisation master.

The 90 period label is also a maximum window, not proof of 90 observed days in this release. The manifest says the rolling input contains 26 observed days from July 14 through August 8.

Keep values and currencies aggregation-safe

A row with estimated_value still needs both its currency and its value_basis. This query produces separate partitions:

WITH parsed AS (
    SELECT
        source,
        value_basis,
        currency,
        estimated_value,
        try_cast(estimated_value AS DECIMAL(38, 6)) AS parsed_value
    FROM opportunities
    WHERE
        estimated_value IS NOT NULL
        AND estimated_value <> ''
        AND currency IS NOT NULL
        AND currency <> ''
)
SELECT
    source,
    value_basis,
    currency,
    count(*) AS nonblank_value_strings,
    count(parsed_value) AS valued_records,
    count(*) - count(parsed_value) AS value_cast_failures,
    sum(parsed_value) AS total_value
FROM parsed
GROUP BY source, value_basis, currency
ORDER BY source, value_basis, currency;

valued_records counts successfully parsed decimals, while value_cast_failures exposes nonblank strings excluded from the sum. Six decimal places retain every value string's observed fractional precision in this pinned release; a future schema or release still needs the same scale audit before reuse. Do not add the resulting EUR, GBP, PLN, RON, CZK, USD, and other rows. Do not add tender estimates to award values or contract values. Currency conversion requires an exchange-rate source, valuation date, method, and disclosure. Even after conversion, unlike value bases can remain non-additive.

SAM.gov has no normalized estimated values in this current-candidate schema 2.0 slice. A pooled value ranking would therefore exclude the entire SAM population while appearing cross-source. Report valued-record coverage by source before any total.

For prepared aggregates, market-summary.csv already partitions count and value metrics by its declared dimensions, value basis, and currency. Its 38,667 rows are metric rows, not 38,667 tenders.

Export results with provenance

The recipe writes:

Every output CSV carries release_tag and edition_target_date. The provenance sidecar records:

• release tag, URL, target date, generation time, and schema version;
• recipe version, recipe-file byte count and SHA-256, and the exact SQL for each output;
• Python version, implementation, platform, DuckDB version, and query execution time in UTC;
• byte count and SHA-256 for the manifest and every selected input;
• distinct row-level source-license references and an explicit missing-license flag for each input product and source;
• core-schema field count, CSV header count, missing fields, and additive export fields;
• manifest-declared product counts and reproduced source counts;
• output row counts, byte counts, and SHA-256 hashes; and
• the current-candidate grain and deadline-watchlist interpretation.

Keep the query text with the project as well. A result is not fully reproducible from a CSV and input hash alone if nobody can recover its filter, casts, grouping, ordering, or null rules. The recipe writes every result into a unique staging directory, validates all expected row counts, and promotes query-provenance.json last. An interrupted promotion cannot leave old provenance beside newly written result files and make the mixed set look complete.

Keep source_license in record-level extracts. In the pinned inputs it resolves to the GSA API terms for SAM.gov, TED's legal notice for TED rows, and Open Government Licence 3.0 for the UK sources. The manifest's source-license summary is not a substitute for row-level evidence: in this release the summary omits TED even though the current-candidate rows retain a TED legal-notice URL. The recipe profiles every selected product/source/license combination and marks a missing value explicitly.

Use deterministic ordering before export. SQL tables are unordered unless ORDER BY defines the output order, and byte-identical result hashes depend on stable order and serialization.

Use Python alone or DuckDB

Use the Python standard library when:

• verifying downloads and hashes;
• reading a header or a small edition file;
• streaming a narrowly filtered extract with csv.DictReader;
• parsing a few JSON-shaped cells with json.loads; or
• running in an environment where another analytical dependency is unnecessary.

Use DuckDB through Python when:

• scanning the 88 MB and larger release products;
• grouping or sorting without loading a pandas dataframe copy;
• applying explicit SQL casts and null rules;
• joining observations on composite keys such as (id, observation_date);
• exporting a bounded CSV or Parquet result; or
• keeping the verification and analytical steps in one executable workflow.

Pandas, Polars, and Arrow are reasonable downstream choices, but they are not prerequisites for this workflow. DuckDB's Python API can also return those result forms if a notebook or visualization needs them.

For a pure-Python category count, stream rows and use collections.Counter. For a change-to-history join or buyer grouping, a SQL engine is clearer and less memory-sensitive. Choose by the operation, not by habit.

Choose downloads, source APIs, or managed delivery

Use the public release when the job needs:

• one normalized schema across the published U.S., EU, and UK source paths;
• dated CSV or JSONL products with manifests, hashes, governance, observations, and changes;
• a local analytical workflow without operating four source collectors; or
• a reproducible input for a report, prototype, or qualification model.

Use a native source API when the job needs source-specific fields, exact source identities, complete source lifecycle semantics, or a retrieval cadence the normalized release does not provide. The dedicated implementation guides cover the SAM.gov Opportunities API, TED Search API v3, and Find a Tender OCDS API. The source comparison explains when those systems cover different jurisdictions and notice populations.

Use a tailored recurring delivery when the market definition, additional sources, history, taxonomy mapping, qualification evidence, or delivery schedule needs to differ from the public edition. No public WebTruffle API is being promised by this article; the free route is the published download and explorer.

Reproducibility checklist

Before publishing or handing off a result, confirm all of these:

• Record the release tag and verify the trusted manifest SHA-256 before using its declarations.
• Confirm every selected artifact matches both declared bytes and SHA-256.
• Record schema version, target date, generation time, product grain, and observation window.
• Require every core schema field; retain and report additive product/export fields.
• Handle the CSV byte-order mark and JSON-shaped cells deliberately.
• Preserve source identifiers as text and keep numeric, date, and boolean casts query-local.
• Reproduce manifest product counts and independent source, stage, and deadline checkpoints.
• Freeze every time-relative predicate to the edition cutoff or disclose a live execution time.
• Keep lifecycle stages, taxonomies, currencies, value bases, and buyer identity scopes separate.
• Carry row-level source-license references into bounded extracts and report missing references.
• Apply deterministic SQL ordering before hashing or comparing exports.
• Include input hashes, exact query text, recipe and runtime identity, row count, and output hash.
• Label the result “observed,” “candidate,” “current as of,” or “material change” according to its actual grain.

Limitations and interpretation boundaries

This worked example has explicit limits:

• It analyzes WebTruffle's normalized public release, not recall against an official universe of everything that should have been published.
• The August 8 edition was generated August 9 and is intentionally stale relative to the August 14 publication date of this guide.
• current-opportunities is a derived convenience product. Source status, deadline type, documents, corrections, and the official response channel remain decisive.
• The file includes no normalized Find a Tender opportunity-stage rows under schema 2.0. That is a product mapping/population limitation, not a UK-market conclusion.
• Source composition is uneven: TED represents 76.96% of current-candidate rows. Pooled results inherit that composition.
• History products cannot supply observations before the public retention start. A “90-day” label does not create 90 observed days.
• Presence is not accuracy. A checksum cannot detect an upstream semantic error that was faithfully published into the artifact.
• Missing values can be structural for a source or lifecycle stage, absent upstream, or caused by a mapping limitation.
• Stable records, source notices, procedures, lots, opportunities, awards, contracts, and suppliers are different grains.
• Buyer name normalization does not establish a cross-source legal entity.
• Estimated values are incomplete and span several currencies. No pooled spending estimate is calculated here.
• A later release, schema migration, source correction, or backfill can change the result. The pinned edition makes that change visible rather than wrong.

Read the August data-quality benchmark before generalizing from field presence or source mix. It reports source health, stage-conditioned completeness, and the known schema 2.0 mapping boundaries behind this same public release.

Frequently asked questions