Federal grants data screened and compared with Python.

Use a dated federal-grants release as an evidence package: verify its manifest and CSV hash, load every source field as text, separate forecasted plans from posted announcements, and treat applicant-type labels as a shortlist—not a legal eligibility decision. Compare two pinned snapshots by stable source ID when you need pipeline movement. Never infer that a disappeared record was cancelled.

This worked example uses the tagged and SHA-256-pinned August 11, 2026 WebTruffle federal-grants release, the latest published edition checked on August 17. Its normalized CSV has 1,691 active opportunities: 1,152 posted and 539 forecasted. The edition is intentionally six days behind this article's publication date, so every result is labeled with the August 11 cutoff rather than presented as a live Grants.gov count.

For a change view, the downloadable recipe also verifies the July 22, 2026 baseline release. Between those two active snapshots, 207 source IDs entered, 266 exited, and 30 retained records moved from forecasted to posted. “Exited” is a set-difference observation, not a cancellation reason.

Federal grants data with Python: the short answer

Use this sequence:

1. Pin release tags, target dates, manifest byte counts, and manifest SHA-256 values. Do not begin from a moving latest URL.
2. Verify each manifest before trusting the file hash declared inside it.
3. Verify the CSV's bytes and SHA-256, then require 43 unique headers in the exact published order.
4. Load identifiers, dates, and money as strings. Parse only the fields required by one result.
5. Keep forecasted and posted populations separate. A forecast is planning evidence, not an open notice of funding opportunity.
6. Filter eligible_applicant_types to find explicit label matches, then read the official announcement and application instructions before deciding legal eligibility.
7. Report the denominator beside every funding measure. In the August 11 snapshot, only 952 of 1,691 rows carry an estimated total program-funding value.
8. For a posted-opportunity watchlist, use close_date. For forecast planning, use forecasted_close_date and label it forecasted.
9. Compare snapshots on source_opportunity_id, not title or opportunity number alone.
10. Classify set differences as entered, exited, retained, or status transition. Do not manufacture a reason that the source data does not contain.
11. Sort every export deterministically and write its release dates, input hashes, script hash, runtime, filter, row count, and output hash to provenance.
12. Open source_url and read the current NOFO before an application decision. The dataset is for discovery and analysis, not submission or award research.

If you want to browse or download the current normalized product, start with the US federal grants dataset. Use the recipe when a shortlist, comparison, or handoff must be repeatable.

Pin the editions before filtering grants

A release tag is a convenient address, but GitHub currently reports these releases as mutable. The hash is the mutation detector. Pin both the manifest and the file selected through that verified manifest.

The current edition for this example is:

release tag       2026-08-11
target date       2026-08-11
generated at      2026-08-11T13:32:02Z
schema version    1.0
manifest bytes    4,627
manifest SHA-256  d92af401bf7cc547457c595ef2c57a28fda2091c62d836af1541248e22603cb5
CSV bytes         1,951,757
CSV SHA-256       3759daf0fca66d8708f6c8dcbc786b48b8c41a715b3144316438055ccd383d4a

The comparison baseline is:

release tag       2026-07-22
target date       2026-07-22
manifest bytes    4,625
manifest SHA-256  0555203a7723ea7da586bc8a2e7c9317d4ac9d4134bfc02195b35cc4d4b55db7
CSV bytes         1,987,836
CSV SHA-256       c70ab8bdddea9c8724317dd6f3d949d8b1a6f105aa3ce30e29c7cc605e50b844

The target date, generation time, source extract date, opportunity posted date, close date, forecasted dates, last-updated date, and time the recipe runs are different clocks. A result derived from this pair should say “July 22 to August 11 snapshots,” not “the last 20 days” without the dates.

posted_date is when the Grants.gov source record—forecast or synopsis—was posted. On a forecast row, it is not the future official-announcement date. forecasted_post_date is the agency-estimated future synopsis or FOA post date. Keep both fields and the status beside them.

The August 11 manifest points to GrantsDBExtract20260811v2.zip. Grants.gov says its database is exported to XML once per day and that the v2 format includes forecast information. (XML extract guide) The public dataset applies documented lifecycle rules to that extract and uses the Grants.gov API only as an advisory validation path.

The edition's validation found the same 1,691 active total through XML and API, but their status split differed by one record: XML produced 1,152 posted and 539 forecasted; the API check produced 1,151 and 540. That is why the normalized release declares which source representation governs its records instead of silently blending the two.

Run on macOS, Linux, or WSL

curl -fsSLO https://www.webtruffle.com/examples/federal-grants-data-python.py
python3 federal-grants-data-python.py \
  --data-dir ./evidence/federal-grants-2026-08-11

Run on Windows PowerShell

Invoke-WebRequest `
  -Uri https://www.webtruffle.com/examples/federal-grants-data-python.py `
  -OutFile federal-grants-data-python.py
py -3.11 .\federal-grants-data-python.py `
  --data-dir .\evidence\federal-grants-2026-08-11

The default applicant-type filter is Small businesses. Pass a different exact normalized label when the screening question changes:

python3 federal-grants-data-python.py \
  --applicant-type "Nonprofits with 501(c)(3) status, other than institutions of higher education"

The filter is case-insensitive after trimming, but the output keeps the normalized, decoded label and its official code array. A typo that matches zero records fails rather than producing an authoritative-looking empty shortlist.

Verify the manifest, file, and schema

The trusted starting point is a manifest hash supplied outside the manifest itself. Once the manifest matches, its files declaration can authenticate the CSV:

import hashlib
import json
from pathlib import Path

def sha256_and_size(path: Path) -> tuple[str, int]:
    digest = hashlib.sha256()
    size = 0
    with path.open("rb") as source:
        for chunk in iter(lambda: source.read(1024 * 1024), b""):
            digest.update(chunk)
            size += len(chunk)
    return digest.hexdigest(), size

manifest_path = Path("manifest-2026-08-11.json")
actual_hash, actual_bytes = sha256_and_size(manifest_path)
assert actual_bytes == 4_627
assert actual_hash == (
    "d92af401bf7cc547457c595ef2c57a28"
    "fda2091c62d836af1541248e22603cb5"
)

manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
assert manifest["target_date"] == "2026-08-11"
assert manifest["record_count"] == 1_691

For network downloads, the recipe creates a uniquely named temporary file in the destination directory, hashes and counts while streaming, aborts as soon as bytes exceed the declaration, and uses os.replace only after both checks pass. A predictable .part filename would allow concurrent runs to interfere with each other.

The CSV uses 43 normalized fields. Validate the header before interpreting a column:

import csv

with open(
    "funding-opportunities-2026-08-11.csv",
    encoding="utf-8-sig",
    newline="",
) as source:
    reader = csv.DictReader(source)
    assert reader.fieldnames == manifest["record_fields"]
    rows = list(reader)

assert len(rows) == manifest["record_count"]
assert len({row["source_opportunity_id"] for row in rows}) == len(rows)

utf-8-sig handles a possible byte-order mark without corrupting the first header. Exact header order is useful here because the manifest declares the complete product schema, not merely a subset.

All CSV values begin as strings. Keep source_opportunity_id, opportunity_number, agency codes, Assistance Listing numbers, and versions that way. Parse dates with date.fromisoformat, integer counts with a guarded conversion, and money with Decimal. Do not let a dataframe infer away leading zeros or round a published decimal through a binary float.

Fields such as eligible_applicant_types, funding_activity_categories, funding_instrument_types, and assistance_listing_numbers are JSON arrays encoded inside CSV cells:

import json

def parse_string_array(value: str) -> list[str]:
    if not value.strip():
        return []
    parsed = json.loads(value)
    if not isinstance(parsed, list) or not all(
        isinstance(item, str) for item in parsed
    ):
        raise ValueError("expected a JSON array of strings")
    return parsed

Treat malformed structured cells as validation failures. Splitting on commas corrupts labels that legitimately contain punctuation.

Separate forecasted from posted opportunities

The August 11 active snapshot contains two different operational states:

• 539 forecasted records: agency planning notices that may change and may never become official announcements.
• 1,152 posted records: synopses that WebTruffle derives as active from the edition's XML dates at the snapshot cutoff.

Grants.gov explicitly says a forecast is planned, is not yet an official funding opportunity announcement, and is not guaranteed to become one. Its search help describes posted records as announced opportunities currently accepting applications. (Search status definitions; forecast help)

In this normalized product, opportunity_status is derived from the edition's XML dates and the row carries status_basis: derived_from_snapshot_dates; it is not copied from a live status endpoint. “Posted” therefore means derived as active at the August 11 cutoff. It does not prove the record is still open when this article or script is read.

Keep status in every aggregation. This is safer than reporting one unqualified “active grants” number:

from collections import Counter

status_counts = Counter(row["opportunity_status"] for row in rows)
assert status_counts == {"posted": 1_152, "forecasted": 539}

The status field is snapshot state, not an award outcome. The file contains opportunities, not successful applications, recipients, obligations, payments, or award transactions.

The recipe also avoids treating change_type as a reliable semantic event. In the August 11 manifest, 1,679 of 1,691 records are marked updated, even though many decision-relevant fields may be unchanged. For analysis, compare named fields across pinned rows and record exactly which field changed.

Screen applicant types without declaring eligibility

The eligible_applicant_types array is valuable for reducing a review queue. It is not sufficient for an eligibility decision. Grants.gov tells applicants to read the application instructions attached to each opportunity because the NOFO contains the legal requirements. (Grant eligibility guidance; applicant eligibility)

For the default Small businesses label, the August 11 file produces 763 explicit label matches: 437 posted and 326 forecasted. The filter counts an opportunity once even if the array contains several labels.

This exact-label file is deliberately narrower than a complete small-business review queue. The edition also has 300 Unrestricted records; 74 overlap the explicit Small businesses matches. Selecting official code 23 or code 99 would therefore yield 989 potential records: 646 posted and 343 forecasted. The downloadable recipe does not silently add those 226 unrestricted-only rows. Run a separate Unrestricted filter, retain both code arrays, and review the omitted narrative conditions before merging the two queues.

requested_type = "Small businesses".casefold()

label_matches = [
    row
    for row in rows
    if requested_type
    in {
        label.strip().casefold()
        for label in parse_string_array(row["eligible_applicant_types"])
    }
]

assert len(label_matches) == 763
assert Counter(row["opportunity_status"] for row in label_matches) == {
    "posted": 437,
    "forecasted": 326,
}

The exported explicit-label review file retains:

• the edition cutoff and opportunity status;
• stable source ID and opportunity number;
• title, agency, and Assistance Listing numbers;
• posted, close, and forecast dates;
• every applicant-type label, not only the match;
• funding estimates and cost-sharing flag;
• source_url and source_license.

That last group makes the review file auditable. An exact label match must still pass program-specific geography, project, organization, size-standard, registration, cost-share, and application-package rules. A row that includes “Small businesses” does not prove the company meets the applicable SBA size standard or every NOFO condition.

Treat broad labels carefully. Grants.gov code 99 means Unrestricted across the listed entity types but can still be narrowed by narrative rules. Code 25 means Other and requires additional eligibility information. The public normalized file omits unrestricted narrative descriptions, so both cases require opening source_url; “Other” is not a wildcard. (Grants.gov status and eligibility codes)

Group by agency and Assistance Listing

The recipe writes 192 agency-code/name groups for the August 11 edition. Both values are retained because a display name is not a durable key and a code is not a readable label.

Each agency row separates:

• total, posted, and forecasted records;
• the selected applicant-type label-match count;
• posted records closing within 30 days;
• records with a reported funding estimate; and
• estimate coverage, with numerator and denominator.

The largest agency group in this edition is the National Institutes of Health (HHS-NIH11) with 684 records—40.4% of the snapshot. That is an observed inventory share in one normalized active snapshot, not a statement that NIH supplies 40.4% of federal grant dollars or awards.

Assistance Listing numbers are also arrays. Explode them only when the analytical grain is one opportunity × one listing membership, then say that totals overlap. An opportunity with two listings should remain one row in an opportunity count and two rows in a listing-membership table.

The recipe keeps its primary agency summary at one agency-code/name row and carries the listing array into record-level outputs. That avoids accidentally converting a discovery table into an overlapping program total.

Measure funding-estimate coverage before summing

Only 952 of 1,691 records (56.3%) in the August 11 snapshot report estimated_total_program_funding_usd. Coverage is 657 of 1,152 posted records and 295 of 539 forecasted records. A total calculated from the present values would describe the reported subset, not the full opportunity population.

The recipe therefore writes a 952-row funding-estimate-watchlist.csv and reports coverage before any aggregation. Each row retains:

• status and edition cutoff;
• estimated total program funding as a decimal string;
• expected number of awards;
• award floor and award ceiling;
• the award_ceiling_unlimited flag;
• agency, Assistance Listings, applicant types, and funding categories; and
• official source and license links.

Parse money with Decimal and retain the original string:

from decimal import Decimal, InvalidOperation

def parse_decimal(value: str) -> Decimal | None:
    if not value.strip():
        return None
    try:
        return Decimal(value)
    except InvalidOperation as error:
        raise ValueError(f"invalid published decimal: {value!r}") from error

funded = [
    row for row in rows
    if parse_decimal(row["estimated_total_program_funding_usd"]) is not None
]
assert len(funded) == 952

Do not turn blank into zero. A missing total estimate, a zero estimate, a missing ceiling, and an explicitly unlimited ceiling have different meanings. Keep the unlimited flag beside the numeric ceiling.

This distinction is observable, not theoretical: 51 rows in the edition explicitly report an estimated total of zero. They belong in the 952-value coverage numerator. A truthiness check such as if amount would wrongly discard them.

The release manifest describes funding and expected-award counts as source-reported planning values, not payments or guaranteed awards. They are also not obligations, outlays, recipient amounts, or a forecast of the amount one applicant will receive. Post-award analysis belongs to an award-data source, not this opportunity snapshot.

For a funding-focused agency comparison, publish at least these columns together:

agency_code
opportunity_records
records_with_estimated_funding
funding_estimate_coverage
reported_estimated_funding_sum_usd

The last measure should be labeled “reported planning estimates” and never shown without the first three. A rank based only on the sum rewards agencies that publish estimates more often and may reflect a small number of unusually large programs.

Use the right deadline for each status

Posted and forecasted records have different date contracts:

• close_date is the application deadline for a posted opportunity.
• forecasted_close_date is a planning date for a forecast that may change or never become a posted announcement.

The August 11 manifest reports 313 effective deadlines within 30 days. That combines 301 posted close_date values with 12 forecasted forecasted_close_date values. Within seven days, all 111 records are posted in this edition.

Report date coverage with the window count: 1,028 of 1,152 posted rows carry close_date, leaving 124 without one; 479 of 539 forecasts carry forecasted_close_date, leaving 60 without one. The 301 and 12 are near-term subsets of those dated populations, not complete counts over all records.

The operational output deliberately narrows the population to posted opportunities closing from August 11 through September 10, inclusive:

from datetime import date

cutoff = date.fromisoformat("2026-08-11")

def parse_date(value: str) -> date | None:
    return date.fromisoformat(value) if value.strip() else None

posted_deadlines = []
for row in rows:
    if row["opportunity_status"] != "posted":
        continue
    close_date = parse_date(row["close_date"])
    if close_date is None:
        continue
    days_to_close = (close_date - cutoff).days
    if 0 <= days_to_close <= 30:
        posted_deadlines.append((days_to_close, row))

assert len(posted_deadlines) == 301
assert sum(days <= 7 for days, _ in posted_deadlines) == 111

Freeze the calculation to the edition cutoff. Using date.today() makes the same input produce a different watchlist tomorrow.

Sort by days_to_close, then close_date, agency code, source ID, and opportunity number. The export stays byte-stable when two records share a deadline. Retain blank dates outside this view instead of inventing a far-future sentinel.

A forecast-planning view can be useful, but it should be a separate file with “forecasted” in the filename and label. Combining the two date fields into an unlabeled deadline column would hide the most important uncertainty in the data.

Compare two active snapshots with stable identifiers

An active snapshot tells you which normalized records met the edition's active rules on one date. It does not contain the complete event history between two dates. Comparing July 22 with August 11 can establish set membership and retained status, but not the cause of every change.

Use source_opportunity_id as the comparison key:

baseline_by_id = {
    row["source_opportunity_id"]: row for row in baseline_rows
}
current_by_id = {
    row["source_opportunity_id"]: row for row in current_rows
}

baseline_ids = set(baseline_by_id)
current_ids = set(current_by_id)

entered = current_ids - baseline_ids
exited = baseline_ids - current_ids
retained = baseline_ids & current_ids

assert (len(entered), len(exited), len(retained)) == (207, 266, 1_484)

The exact comparison is:

Among retained records, 995 stayed posted, 459 stayed forecasted, and 30 moved from forecasted to posted. That accounts for all 1,484 retained IDs. The active stock fell by 59, but stock change alone says nothing about application volume, award volume, or funding awarded.

The recipe writes one event row for each entered ID, exited ID, and forecast-to-posted transition: 503 rows in total. A retained status transition is a separate analytical event; it is not counted as entering the snapshot.

Use conservative event names:

• entered_snapshot: absent from the baseline active set and present in the current active set.
• exited_snapshot: present in the baseline active set and absent from the current active set.
• forecasted_to_posted: retained in both and moved from planning to the derived posted status.

Do not rename exited_snapshot to closed or cancelled. An opportunity can leave an active product because its deadline passed, it was archived, it was removed or corrected upstream, or the normalizer's lifecycle rule changed. The two active rows do not reveal which reason applies.

Likewise, “entered” means first present in this comparison, not necessarily first published after July 22. A record missing from the baseline because of collection or normalization behavior can appear later without being newly created at the source.

For field-level changes among retained records, compare named raw values such as version, close_date, forecasted_close_date, last_updated_date, funding values, and applicant arrays. Record the old value, new value, source ID, both release cutoffs, and normalization rule. A generic content-hash difference can tell you bytes changed, but not which decision changed.

Export results with a provenance receipt

The downloadable recipe stages every CSV in a unique result directory, calculates hashes there, and promotes the complete result set only after all row-count assertions and a final input recheck pass. It removes any old provenance receipt before promoting files, then writes provenance last through an atomic replace.

That ordering prevents a failed run from leaving a new partial CSV beside an old receipt that appears to authenticate it.

The query-provenance.json receipt includes:

• recipe version and SHA-256;
• Python implementation, runtime version, and platform;
• selected applicant-type filter;
• both release tags, cutoffs, filenames, byte counts, and hashes;
• manifest schema, generation time, source extract identity, and record grain;
• exact output filenames, row counts, byte counts, and hashes;
• interpretation notes for eligibility, funding, deadlines, and snapshot exits; and
• the required source attribution and source-license URL.

The expected outputs for the default run are:

Hashes establish byte identity. They do not prove that an agency entry is accurate, a source population is complete, an applicant is eligible, or a funding estimate will become an award.

Choose a published release, XML, or an API

Use the normalized public release when the job needs:

• an active posted-and-forecasted opportunity snapshot with one documented schema;
• CSV, JSON, or JSONL with a manifest and SHA-256 declarations;
• a repeatable local screening or comparison workflow; or
• a shortlist that retains official source URLs for review.

Use the official Grants.gov daily XML extract when the job needs the source's bulk representation, fields outside the public normalized schema, an independent reconciliation path, or a feed that your team will retain from now on. Grants.gov describes the XML export as a once-daily service for power users and database owners and notes that blank elements are omitted. Its public index exposes a short recent window, so users must archive editions themselves; the official route is not a general historical-snapshot service. (XML extract; field guide)

Use an official search API when the job needs interactive queries or source-specific detail without downloading the full XML. The classic search2 and fetchOpportunity endpoints remain documented, while Simpler.Grants.gov provides the forward-looking API route. The Grants.gov API migration guide owns endpoint selection, keys, pagination, field mapping, detail retrieval, and dual-run design.

This article owns post-download analysis. It does not promise a public WebTruffle API, teach application submission, or turn opportunity records into grant-award data.

Whichever route you choose, follow the Grants.gov API terms. They permit searching, displaying, analyzing, retrieving, and viewing grants data; require attribution for products using the API; prohibit implying HHS endorsement; and describe the service as provided as-is and as-available.

Federal grants Python checklist

Before sharing a shortlist or trend, verify all of these:

• Pin each release tag and verify a trusted manifest byte count and SHA-256.
• Verify every selected CSV against the already verified manifest.
• Record target date, generation time, schema version, source extract, and record grain.
• Require all 43 headers in their declared order and unique source_opportunity_id values.
• Read CSV with utf-8-sig; preserve source identifiers and opportunity numbers as text.
• Parse JSON-array cells with a JSON parser, not comma splitting.
• Keep posted and forecasted rows separate in counts, dates, and labels.
• Call applicant-type results explicit label matches, not eligibility decisions; confirm legal eligibility in the NOFO and instructions.
• Report funding-estimate numerator, denominator, and coverage before a sum.
• Preserve blanks and the unlimited-ceiling flag rather than converting them to zero.
• Freeze deadline calculations to the edition cutoff and select the date field by status.
• Compare snapshots on source ID; label set differences entered or exited without inventing causes.
• Treat change_type and content hashes as signals to inspect named fields, not self-explanatory events.
• Sort exports with complete tie-breakers before hashing them.
• Recheck inputs after processing and promote provenance only after every output passes.
• Retain source_url, source_license, exact filter, input hashes, output hashes, and runtime identity.

Limitations and interpretation boundaries

This worked result has explicit limits:

• It analyzes two normalized active-opportunity snapshots, not every Grants.gov record or an official-universe recall denominator.
• The product is named for federal grants, but its source instrument vocabulary also includes cooperative agreements, Other, and procurement contracts. In this edition, 1,661 records include Grant or Cooperative Agreement and 30 include neither; filter funding_instrument_codes before making a grant-only claim.
• The main edition cutoff is August 11, six days before publication. It is intentionally reproducible, not live.
• Forecasts are plans that can change or never become posted announcements.
• Posted is a WebTruffle-derived active-synopsis status at the edition cutoff, not a live source-status lookup. Always confirm the current official notice, application package, deadline, and submission route.
• Applicant-type arrays support screening but omit or simplify program-specific legal conditions found in the NOFO and instructions.
• Funding values and expected award counts are planning data, not obligations, payments, recipient amounts, or guaranteed awards.
• Estimate coverage is incomplete, and source-published values can be corrected later.
• One opportunity can have several applicant types, funding categories, instruments, and Assistance Listings. Membership tables overlap.
• An agency name is not an organization-resolution system; use the code/name pair and preserve source evidence.
• An exited ID is not automatically closed or cancelled, and an entered ID is not automatically newly published.
• A 20-day endpoint comparison does not reveal every intermediate edit or status transition.
• A matching hash proves artifact identity, not factual accuracy, completeness, legal eligibility, or analytical fitness.
• GitHub currently marks the releases as mutable. The pinned manifest hash detects changed release bytes.
• The normalized product deliberately excludes narrative descriptions, contact details, applications, award recipients, obligations, and payments.

Frequently asked questions