US federal procurement forecasts turned into a verified watchlist.

Use a federal procurement forecast as an early planning signal, not as a solicitation. Pin one accepted release, verify its four-source health, load all 62 CSV fields as text, and route current records, normalized-content updates, and source absences into different review queues. A projected value is not spend, a forecast date is not a deadline, and not_in_current_snapshot is not evidence of cancellation or award.

This worked example uses WebTruffle's US federal procurement forecasts release tagged August 27, 2026. Its current product contains 2,501 planning records from DHS, DOE, Education, and DOJ. Its separate change product contains 347 observations: 136 new, 12 updated, and 199 not in the current snapshot.

The release's most useful warning is empirical: 337 of those 347 change rows—97.12%—came from one DOJ source refresh. That concentration does not prove a DOJ market surge or contraction. The source document changed, and some rows rely on derived identity. A defensible workflow keeps the 12 stable-ID content updates apart from additions and absences, then verifies every actionable lead in the official agency source and SAM.gov Contract Opportunities.

The downloadable Python recipe independently pins the manifest, verifies schema.json, source-health.json, the current CSV and changes.csv, requires an atomic four-source release, derives a comparison-only six-digit NAICS key without replacing the raw field, and writes bounded DuckDB results with a provenance receipt.

US federal procurement forecasts with Python: the short answer

Use this sequence:

1. Pin a tagged release, its exact manifest byte count, and an independently recorded SHA-256. Do not build evidence from a moving latest URL.
2. Verify the manifest before trusting any hash declared inside it.
3. Verify schema.json, source-health.json, us-federal-procurement-forecasts.csv, and changes.csv before opening DuckDB.
4. Require atomic_complete=true, all four required sources, and the declared 2,501 current rows and 347 change rows.
5. Load all 62 fields as text. Preserve identifiers, projected timing, value ranges, set-aside labels, and blanks exactly as published.
6. Use the current product for a current-source watchlist. Use the change product for review since the prior accepted edition. Do not append them as interchangeable event rows.
7. Derive a leading six-digit NAICS comparison key while retaining naics_code. DOJ preserves code-and-description source text in that field.
8. Measure field presence by source before applying a filter. A blank set-aside or timing field is unknown or structurally unavailable, not “no.”
9. Keep projected exact dates, quarters, and timing text in separate lanes. Keep value ranges as source labels; do not invent numeric midpoints or total “pipeline value.”
10. Send new and updated rows to an active review queue. Send not_in_current_snapshot rows to a separate absence investigation.
11. Treat derived-ID remove/add pairs as possible identity changes, not automatically as two procurement events.
12. Verify the current agency record and look for a published SAM.gov notice before acting on scope, competition, set-aside, value, or deadline.
13. Export only a bounded, deterministically sorted watchlist and keep its release tag, filters, input hashes, row count, and output hash together.

If you only need the current files and metadata, start with the free US federal procurement forecasts dataset. Use this guide when the post-download query and its interpretation must survive review.

Pin the 2026-08-27 release

A GitHub tag locates an edition; an independently recorded manifest digest establishes the bytes you intended to trust. Verify that root first. Only then accept the asset declarations inside it.

The worked evidence package is:

release tag          2026-08-27
generated at         2026-08-27T14:03:32Z
schema version       1.0
current rows         2,501
change rows          347
public fields        62

manifest bytes       17,403
manifest SHA-256     507f08dd3f2ddcac46d0612fa130c3dc3718babfbbfc0e7c3f2fcbe2ad4cdd02

schema bytes         5,576
schema SHA-256       5b41eb451d476b0f4bf1dbcccec4731d2edefd63233b1b70084c1b773540cb8d
source-health bytes  10,079
source-health SHA    15d5092829fed68d8969cbfc30c7aea1d0e17720209f4549181af9207934fae6

current CSV bytes    2,985,263
current CSV SHA      1076227ca33fcb46810f4dc0ae68039690db96c0b8b662ac4070ae42ca599c3d
changes CSV bytes    377,363
changes CSV SHA      3f2f8be431f76a400134f74a6dc2dfa649ebc2cb0cc4d87e1e057333d2fd0707

The pinned manifest declares the two product grains, exact file identities, source counts, change counts, field order, identity policy, and limitations. The tagged schema documentation defines every field, and the verified known-gaps artifact explains the four-agency coverage boundary.

The prior accepted edition had 2,564 current records. The latest release reconciles exactly:

2,564 prior current records
+ 136 first observed in this edition
− 199 not observed in the current source snapshots
= 2,501 current records

That equation proves internal membership reconciliation. It does not say why any source row appeared, changed, or disappeared.

Install the tested Python environment

The recipe requires Python 3.10 or newer and pins DuckDB 1.5.5. DuckDB can read local CSV files with all columns held as strings and execute the bounded exports without loading the full data into pandas. (DuckDB Python API; CSV reader)

On macOS, Linux, or WSL:

curl -fsSLO https://www.webtruffle.com/examples/us-federal-procurement-forecasts-python.py
curl -fsSLO https://www.webtruffle.com/examples/us-federal-procurement-forecasts-python-requirements.txt

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r us-federal-procurement-forecasts-python-requirements.txt
python us-federal-procurement-forecasts-python.py

On Windows PowerShell:

Invoke-WebRequest `
  -Uri https://www.webtruffle.com/examples/us-federal-procurement-forecasts-python.py `
  -OutFile us-federal-procurement-forecasts-python.py
Invoke-WebRequest `
  -Uri https://www.webtruffle.com/examples/us-federal-procurement-forecasts-python-requirements.txt `
  -OutFile us-federal-procurement-forecasts-python-requirements.txt

py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r us-federal-procurement-forecasts-python-requirements.txt
python .\us-federal-procurement-forecasts-python.py

The default evidence directory is us-federal-procurement-forecasts-2026-08-27. The default example filter is NAICS prefix 5415, with at most 50 watchlist rows. Both are explicit CLI inputs:

python us-federal-procurement-forecasts-python.py \
  --data-dir ./evidence/federal-forecasts-2026-08-27 \
  --naics-prefix 5415 \
  --watchlist-limit 50

The recipe accepts a two- through six-digit NAICS prefix and caps the export at 500 rows. An existing input with the wrong size or hash causes a hard failure; it is never silently replaced.

Verify the manifest, assets, and source health

Start outside the analytical engine so DuckDB never decides which bytes count as evidence:

import hashlib
from pathlib import Path

def sha256_and_size(path: Path) -> tuple[str, int]:
    digest = hashlib.sha256()
    size = 0
    with path.open("rb") as source:
        for chunk in iter(lambda: source.read(1024 * 1024), b""):
            digest.update(chunk)
            size += len(chunk)
    return digest.hexdigest(), size

manifest_path = Path("manifest.json")
actual_hash, actual_bytes = sha256_and_size(manifest_path)
assert actual_bytes == 17_403
assert actual_hash == (
    "507f08dd3f2ddcac46d0612fa130c3dc"
    "3718babfbbfc0e7c3f2fcbe2ad4cdd02"
)

After that root passes, require its release contract:

import json

manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
assert manifest["dataset_id"] == "us-federal-procurement-forecasts"
assert manifest["target_date"] == "2026-08-27"
assert manifest["schema_version"] == "1.0"
assert manifest["record_count"] == 2_501
assert manifest["change_count"] == 347
assert len(manifest["record_fields"]) == 62

source-health.json is a release gate, not optional decoration:

health = json.loads(Path("source-health.json").read_text(encoding="utf-8"))
assert health["atomic_complete"] is True
assert health["all_required_sources_succeeded"] is True
assert health["required_sources"] == [
    "dhs_apfs",
    "doe_acquisition_forecast",
    "education_acquisition_forecast",
    "doj_acquisition_forecast",
]
assert health["record_counts_by_source"] == {
    "dhs_apfs": 841,
    "doe_acquisition_forecast": 823,
    "education_acquisition_forecast": 370,
    "doj_acquisition_forecast": 467,
}

The files begin with a UTF-8 byte-order mark. Use encoding="utf-8-sig" when validating the ordered CSV header, then require all 62 unique fields exactly as declared by the verified manifest.

A passing hash establishes byte identity. A passing source-health file establishes that this collector accepted all four source partitions together. Neither one guarantees complete government-wide coverage, a correct agency forecast, or the future procurement outcome.

Keep forecasts separate from notices, awards, and grants

The federal lifecycle has different records for different questions.

A forecast says an agency is planning or considering an acquisition. A SAM.gov notice publishes a procurement-stage record such as a sources-sought notice, presolicitation, or solicitation. A contract award or transaction records a downstream result. A grant opportunity belongs to financial assistance, not procurement.

Federal Acquisition Regulation 5.404-1 says long-range acquisition estimates can help industry plan, but must be described as based on the best information available, subject to modification, and non-binding. It also says a set-aside determination is made when the individual acquisition is initiated—not by the forecast alone. (FAR 5.404-1)

GSA gives the same operational boundary for its participating-agency forecast tool: use forecasts to prepare early, but make final competition, small-business, value, and other procurement decisions only if and when a solicitation is posted to SAM.gov. (GSA forecast guidance)

Coverage is fragmented. Acquisition.gov's procurement-forecast directory points to many agency-specific products. This release normalizes four: DHS APFS, DOE's acquisition forecast, the Department of Education forecast, and the DOJ forecast. It is not a government-wide forecast.

Use these claims:

• “2,501 current planning records in four covered agency source snapshots.”
• “136 first-observed and 12 content-updated records in the August 27 accepted edition.”
• “Projected source values and timing, subject to agency revision and official verification.”

Avoid these claims:

• “2,501 upcoming federal contracts.”
• “199 cancelled procurements.”
• “The government plans to spend the sum of these ranges.”
• “A forecasted set-aside or date is final.”

Choose the current snapshot or change product

The current CSV has one retained record per stable dataset id across the four accepted source snapshots. Use it for questions such as “Which visible records match this NAICS prefix now?” It includes 2,353 unchanged, 136 new, and 12 updated rows.

The change CSV has one row for every membership or normalized-content change since the prior accepted edition. It includes the 136 new, 12 updated, and 199 not_in_current_snapshot rows. Use it for review routing, not as a second copy of current state.

Keep these semantics strict:

• new means first observed by this dataset history in this edition. It does not prove a newly approved acquisition.
• updated means the same dataset identity has a changed normalized content hash. It does not identify which source business event caused the edit.
• unchanged appears in the current product, not the change product.
• not_in_current_snapshot means a previously observed identity was absent from the accepted current source snapshot. It does not prove cancellation, deletion, award, or loss of funding.

Do not left-join the change product to the current product and discard unmatched rows: every absence is expected to be unmatched by definition. Keep active and absence queues separate.

Load all 62 columns as text with DuckDB

Verify first, materialize second:

from pathlib import Path
import duckdb

data_dir = Path("us-federal-procurement-forecasts-2026-08-27")
connection = duckdb.connect()

for view, filename in {
    "forecasts": "us-federal-procurement-forecasts.csv",
    "changes": "changes.csv",
}.items():
    relation = connection.read_csv(
        str(data_dir / filename),
        header=True,
        all_varchar=True,
    )
    relation.create_view(f"{view}_source")
    connection.execute(
        f"CREATE TEMP TABLE {view} AS SELECT * FROM {view}_source"
    )
    connection.execute(f"DROP VIEW {view}_source")

all_varchar=True is deliberate. A source range such as R2 – $250K–$7.5M is not a number. A NAICS string can carry a description. A quarter is not an exact date. An empty field is not zero or false. Preserve each original string and derive typed comparison columns only inside a bounded query.

Then prove the declared grains:

SELECT count(*) AS rows, count(DISTINCT id) AS ids
FROM forecasts;

SELECT change_type, count(*) AS rows
FROM changes
GROUP BY change_type
ORDER BY change_type;

Expect 2,501 rows and 2,501 distinct current IDs. Expect 136 new, 12 updated, and 199 absent rows in the change product.

Measure source-specific field coverage before filtering

Across the full current product, title, naics_code, and estimated_value_range are present on all 2,501 records. Other fields are structurally source-specific:

• set-aside text is present on 1,176 rows, or 47.02%;
• competition strategy is present on 1,678 rows, or 67.09%;
• an exact estimated solicitation date appears on 841 rows, or 33.63%;
• only 19 rows, or 0.76%, carry a solicitation URL.

Those are composition facts, not quality grades. DHS supplies exact solicitation and award dates for its 841 rows. DOJ supplies timing text for its 467 rows. Education supplies an award quarter for 370 rows. DOE supplies none of those forecast-timing fields in this edition, but supplies a performance end date. PSC appears only in DOJ. The current-fiscal-year value range appears only in Education.

Measure presence by source before a business filter:

SELECT
  source,
  count(*) AS rows,
  count(*) FILTER (WHERE nullif(trim(naics_code), '') IS NOT NULL)
    AS naics_present,
  count(*) FILTER (WHERE nullif(trim(set_aside_type), '') IS NOT NULL)
    AS set_aside_present,
  count(*) FILTER (WHERE nullif(trim(competition_strategy), '') IS NOT NULL)
    AS competition_present,
  count(*) FILTER (WHERE nullif(trim(estimated_solicitation_date), '') IS NOT NULL)
    AS exact_solicitation_date_present,
  count(*) FILTER (WHERE nullif(trim(estimated_solicitation_timing), '') IS NOT NULL)
    AS solicitation_timing_present
FROM forecasts
GROUP BY source
ORDER BY source;

A query for “set-aside is blank” will disproportionately select sources that do not publish that normalized lane. It cannot be interpreted as “not set aside.”

Build a bounded NAICS watchlist without converting unknown to false

All 2,501 current rows have a naics_code string, but only 2,034 values are exactly six digits. DOJ preserves 467 values in a code-plus-description form such as 541512--Computer Systems Design Services. Retain that evidence and create a comparison key alongside it:

CREATE TEMP VIEW forecast_query AS
SELECT
  *,
  CASE
    WHEN regexp_matches(trim(naics_code), '^[0-9]{6}([^0-9]|$)')
      THEN regexp_extract(trim(naics_code), '^([0-9]{6})', 1)
    ELSE NULL
  END AS naics_6
FROM forecasts;

Then apply an explicit two- through six-digit prefix and deterministic limit:

SELECT
  edition_date,
  id,
  source,
  agency_name,
  title,
  naics_code,
  naics_6,
  competition_strategy,
  small_business_program,
  set_aside_type,
  business_size_selection,
  estimated_value_range,
  estimated_solicitation_date,
  estimated_solicitation_timing,
  anticipated_award_date,
  anticipated_award_timing,
  award_quarter,
  source_url,
  source_identifier_method,
  change_type
FROM forecast_query
WHERE naics_6 LIKE '5415%'
ORDER BY source, agency_name, naics_6, title, id
LIMIT 50;

The result is a review queue, not a ranked lead score. Preserve every blank. If a user truly requires a final set-aside, exact response deadline, or live notice, make that a verification condition rather than filling the forecast field by assumption.

The recipe writes this bounded result to forecast-naics-watchlist.csv. It rejects malformed prefixes and unbounded limits before constructing the query, so arbitrary shell text cannot become SQL.

Keep value ranges and projected timing source-native

The four sources publish different value vocabularies. DHS uses labels such as $2M to $5M; DOE uses bands such as R2 – $250K–$7.5M; Education uses labels such as >= $1M and < $5M; DOJ has many narrower formatted ranges.

Do not:

• parse all four into a single midpoint and rank it as expected value;
• sum upper bounds and call the result forecast spend;
• treat a blank value as zero;
• infer currency or period beyond what the source and schema support.

Keep estimated_value_range as a categorical source label with source. If a decision requires numeric bounds, create a versioned source-specific mapping, retain the original string, disclose open-ended bands, and publish unmapped counts. Even then, the result describes planning ranges—not obligations, awards, or payments.

Timing needs the same discipline:

• keep DHS exact projected dates as dates only after validation;
• keep DOJ quarter or timing text as text;
• keep Education award_quarter separate;
• do not fill missing DOE solicitation timing from another field;
• keep collection timestamps separate from agency publication or update dates.

An exact-looking forecast date is still projected. The deadline that governs a response belongs to the current official notice and documents.

Reconcile new, updated, and absent records

The change feed is highly concentrated:

DOJ  132 new + 12 updated + 193 absent = 337 change rows
DHS    4 new +  0 updated +   6 absent =  10 change rows
DOE    0
ED     0

That means DOJ accounts for 97.12% of all changes. The 199 absence rows are 57.35% of the change file. Reporting “federal procurement forecasts fell” from those counts would confuse one source-document refresh with the full federal planning market.

Identity makes the boundary sharper. Of 2,501 current records, 841—33.63%—use a documented derived identity method: all 823 DOE records because the source has no row ID, plus 18 DOJ rows whose tracking number is missing, placeholder-like, or duplicated. An edit to a derived identity component can surface as one old ID absent and one new ID added.

Route the records as follows:

• new: review the current source record and look for related SAM.gov notices; do not call it newly approved.
• updated: compare retained normalized fields under the same ID; the 12 stable-ID updates are the strongest direct content-change candidates.
• not_in_current_snapshot: preserve the prior record in an absence queue, inspect source health and the current source, and record an investigated resolution separately.
• possible derived-ID pair: compare source, agency, title, NAICS, value, and timing as evidence, but do not automatically merge or assert the reason.

The recipe writes active changes to forecast-new-updated.csv, absences to forecast-not-in-current-snapshot.csv, and the source/type reconciliation to forecast-change-counts-by-source.csv.

Hand candidates to the official source and SAM.gov

The normalized dataset is a discovery and review layer. Decision evidence remains upstream.

Use the row's source_url and source_document_url first. The current product has source_url on every record, but only 844 unique values: DHS supplies 841 record-level URLs while DOE, Education, and DOJ each reuse an agency landing page. Only 19 rows have a solicitation_url. A nonblank link is useful; a blank link is not evidence that no solicitation exists.

For a candidate:

1. Open the official forecast source and confirm that the record is still present.
2. Search SAM.gov Contract Opportunities using the agency, title, NAICS, office, tracking reference, and any solicitation identifier.
3. Confirm notice type, status, posted date, response deadline, set-aside, attachments, and amendments in the current official record.
4. Store the forecast ID and SAM.gov noticeId as separate keys.
5. Record the match rule, evidence URL, checked-at time, reviewer, and confidence. Do not overwrite the forecast with the notice.

If the task is systematic SAM.gov collection rather than post-download forecast analysis, use the SAM.gov Opportunities API guide. If it is downstream award and supplier research, use the US federal contract awards Python guide.

Export bounded results with a provenance receipt

The companion recipe writes seven outputs:

• forecast-release-checkpoints.csv
• forecast-change-counts-by-source.csv
• forecast-source-field-coverage.csv
• forecast-naics-watchlist.csv
• forecast-new-updated.csv
• forecast-not-in-current-snapshot.csv
• forecast-query-provenance.json

The CSVs are deterministically ordered and bounded where they expose row-level records. The final JSON receipt records the release tag, manifest identity, verified asset identities, recipe version, selected NAICS prefix and limit, query names, row counts, output bytes, and SHA-256 digests.

A receipt should be the final write, after every query and output passes its checks. It proves what the local workflow consumed and produced. It does not elevate a forecast into an authoritative solicitation.

Choose the public dataset, official sources, or a managed feed

Use the public forecast dataset when the four covered agency sources, current snapshot, observed changes, public fields, and daily collector cadence fit the decision. Pin the release you actually analyze.

Use the official agency pages when a single record, current source document, contact route, or source-native context matters. Use SAM.gov when the question is whether a notice is published, active, amended, or open for response.

Use a managed feed when the requirement needs more agencies, a longer observation history, source-specific numeric range mapping, forecast-to-notice or notice-to-award matching, account-specific qualification, reviewed alerts, or delivery into a warehouse or CRM. Define those additions as a new scope; do not imply they are already in the free edition.

Federal procurement forecasts Python checklist

• [ ] Pin the release tag, manifest byte count, and independent manifest SHA-256.
• [ ] Verify every selected asset's declared bytes and hash before analysis.
• [ ] Require an atomic edition and all four declared source partitions.
• [ ] Require 2,501 unique current IDs and the declared source-count equation.
• [ ] Read all 62 columns as text and handle the UTF-8 BOM.
• [ ] Keep current state and the change product at their declared grains.
• [ ] Preserve raw naics_code; derive a comparison key alongside it.
• [ ] Measure field presence by source before filtering on set-aside, timing, PSC, or URLs.
• [ ] Keep blanks distinct from false, zero, “not set aside,” or “no solicitation.”
• [ ] Keep source value-range labels; do not sum or manufacture midpoints.
• [ ] Keep exact dates, quarter labels, timing text, and collection timestamps separate.
• [ ] Route new/updated and not_in_current_snapshot to different queues.
• [ ] Review derived-ID remove/add ambiguity instead of inferring a business event.
• [ ] Verify the official agency record and SAM.gov notice before acting.
• [ ] Export a deterministic bounded watchlist and final provenance receipt.

Limitations and interpretation boundaries

• The release covers DHS, DOE, Education, and DOJ only. It is not government-wide; DoD and many civilian-agency forecasts remain outside the edition.
• The public repository has two accepted releases as of this article. That is not enough history to establish long-run reliability, seasonality, or source-update behavior.
• Daily collection does not mean an agency source changes daily. Agency publication cadences differ.
• Forecasts are non-binding planning records. Requirements, funding, dates, values, acquisition strategy, competition, and set-asides can change or disappear.
• A forecast is not a SAM.gov solicitation, response deadline, contract award, obligation, payment, or grant opportunity.
• new means first observed by this pipeline; updated means changed normalized content under one ID; not_in_current_snapshot does not prove cancellation, deletion, award, or funding loss.
• DOE has no source row ID. Some DOJ tracking values require a derived fallback. Identity-component edits can appear as remove/add pairs.
• Source value, organization, competition, contract-type, set-aside, timing, and NAICS vocabularies are not fully harmonized.
• Field-presence percentages describe the composition of this four-source release, not intrinsic source quality.
• Structured names, emails, and phone numbers are excluded. The product is not a contact list.
• Only 19 current rows contain a solicitation URL. Follow the official source and search SAM.gov; do not infer absence of a notice from a blank field.
• The GSA centralized Forecast of Contracting Opportunities service is a documented coverage gap because its public data service was not reliably reachable during implementation.
• Public releases do not retain the raw downloaded source files. Hash verification proves the published derivative's identity, not end-to-end reconstruction from a raw archive.
• No blanket CC0 grant applies. Review source notices, agency terms, and field-specific rights before redistribution.

Frequently asked questions