FPDS API replacement: migrate to the SAM.gov Contract Awards API.

Replace an FPDS Atom integration with the SAM.gov Contract Awards API by mapping the request and response contracts, preserving base awards and modifications at source grain, running current and deleted-record syncs separately, and proving a parallel run before switching readers. Do not treat this as an XML-to-JSON serialization change.

The production endpoint is:

GET https://api.sam.gov/contract-awards/v1/search

The timing is no longer theoretical. GSA decommissioned FPDS ezSearch on 24 February 2026, says the FPDS Atom Feed will retire later in fiscal year 2026, and tells consumers to begin using the replacement API now. The active announcement does not give an exact Atom retirement day. Build the cutover around evidence, not an assumed final date. (SAM.gov FPDS transition; active retirement announcement)

Endpoint names, quotas, result limits and retirement wording in this guide were checked against GSA's API page, OpenAPI file, variance document and active announcement on 6 August 2026.

FPDS API replacement: the short answer

Use this migration sequence:

1. Inventory every FPDS Atom query, credential, downstream field, identity rule, amount calculation, deletion rule and service-level expectation.
2. Map supported Atom request parameters to the Contract Awards API v1 filters using GSA’s official variance document.
3. Obtain the appropriate SAM.gov personal or system-account API key, then remove the key from logs, stored URLs and exception messages.
4. Store the replacement response at source-action grain: issuing subtier, PIID, modification number, transaction number and referenced-IDV context.
5. Query narrow lastModifiedDate ranges with an explicit limit=100, zero-based page indexes and selected response sections.
6. Partition any query that approaches the synchronous 400,000-result ceiling; use asynchronous extracts for larger bounded backfills.
7. Run deletedStatus=yes as a separate collection lane because ordinary absence is not proof of deletion.
8. Revisit recent defense periods because public DoD contract data can have a 90-day disclosure delay.
9. Dual-run FPDS Atom and SAM.gov long enough to reconcile counts, identifiers, action obligations, negative actions, modifications, IDV relationships, nulls and late records.
10. Switch downstream readers only after written acceptance gates pass and a tested rollback checkpoint exists.

The replacement API’s v1 base version is dated 5 December 2025. It returns JSON synchronously, offers asynchronous JSON or CSV extracts, supports deleted-contract retrieval and can summarize a PIID family. Those additions are useful, but none removes the need to understand the procurement-action model. (GSA Contract Awards API)

This guide owns the engineering migration and cutover. If the job is to research one award manually, understand SAM.gov versus USAspending, or interpret obligations, use the awarded-government-contracts guide. If the source is a pre-award notice rather than a reported contract action, use the SAM.gov Opportunities API guide.

Starting a new integration rather than replacing FPDS? Skip to the first request and the collector.

What FPDS functionality is retiring

Two different FPDS transitions are easy to collapse into one headline.

ezSearch has already retired

FPDS.gov ezSearch was the public human-search interface. GSA decommissioned it on 24 February 2026 and moved public contract-award search into SAM.gov. GSA says all public-facing FPDS functionality has transitioned and attempts to access FPDS.gov redirect to SAM.gov. A SAM.gov account is required to view public Contract Awards search results, but a special role is not required. (SAM.gov Contract Awards transition)

That browser transition is complete. It is not the same as an Atom integration cutover.

The Atom feed remains temporary

The active SAM.gov announcement says the FPDS.gov Atom Feed will retire later in FY 2026. It does not publish an exact day. The transition page says the Atom feed will be replaced by the SAM.gov Contract Awards API and explicitly tells consumers to begin testing and updating their consumption methods now.

Do not infer that “later in FY 2026” guarantees service until a particular date. A safe project plan uses the remaining overlap to prove parity, then makes the legacy feed a rollback source rather than a production dependency.

Contract-writing submission is a separate workflow

Federal agencies’ Contract Writing Systems continue to submit data through the SAM.gov Contract Awards management environment, formerly the FPDS database. The public read API does not replace agency creation, approval or correction workflows. An organization that both submits and consumes contract data should separate those projects, credentials and acceptance criteria.

The scope statement for a public migration should read:

Replace the read-only FPDS Atom consumption path with the revealed-data SAM.gov Contract Awards API v1, while preserving source action identity, modification history, relationships, lifecycle evidence and the downstream business contract.

That statement prevents the project from quietly expanding into authoring, acquisition workflow or USAspending analytics.

Treat this as a data contract migration

GSA’s active transition identifies the retiring consumption path as the FPDS Atom Feed. Its official variance document describes the wider legacy web-service side as SOAP/XML and the replacement as REST/JSON with API-key authentication. The document also maps request parameters, renamed response fields, nested response sections and a different representation for codes and descriptions. (GSA FPDS Atom versus SAM Contract API variance document; active FPDS transition announcement)

A migration inventory needs more than a list of URLs. For every legacy job, capture:

• the complete FPDS query and its intended population;
• schedule, expected latency, timeout and retry behavior;
• credential owner and runtime identity;
• XML namespaces, fields and attributes actually consumed;
• source identity and deduplication logic;
• how base awards, modifications, orders and IDVs are represented;
• whether the consumer uses action obligations, total obligations, current value, potential value or a derived amount;
• how updates, corrections, closures and deletions are detected;
• null, missing, empty-string and code/description behavior;
• downstream schemas, dashboards, alerts, joins and retention requirements; and
• current reconciliation reports and known legacy defects.

For each item, classify the replacement as one of four states:

1. Direct mapping — the API has a documented request or response equivalent.
2. Structural mapping — the meaning exists but moved into a nested object or separate code/name pair.
3. Behavioral redesign — pagination, authentication, extraction or lifecycle handling must change.
4. No direct equivalent — the consumer needs a tested alternate filter, client-side step or business decision.

GSA lists three legacy request parameters as deprecated and unsupported by the replacement API: DESCRIPTION_OF_REQUIREMENT, CONTRACTING_OFFICE_NAME and FUNDING_OFFICE_NAME. The new API has a free-text q filter and office-code filters, but the variance document does not say these are exact substitutes. Measure their result populations before rewriting a legacy query around them.

Get and protect the right SAM.gov API key

Every Contract Awards API request requires api_key. Public revealed data are available through an eligible personal key or a SAM.gov system-account key. Unrevealed data, including recent DoD records inside the disclosure window, require appropriate federal DoD access. A public migration should document that it collects revealed data unless the organization has explicitly authorized access. (GSA API user requirements)

Choose the account for the workload

GSA publishes these default daily request limits for the Contract Awards API:

• non-federal personal key with no SAM.gov role: 10 requests per day;
• non-federal personal key with a role: 1,000 requests per day;
• federal personal key: 1,000 requests per day;
• non-federal system-account key: 1,000 requests per day; and
• federal system-account key: 10,000 requests per day.

Ten calls can return at most 1,000 synchronous rows when each page requests the maximum 100. That is a diagnostic allowance, not a credible broad migration budget. A recurring organizational integration should use the account type, permission and approved network configuration that match its operating model.

SAM.gov system accounts require the Contract Awards read permission and registered invoking IP addresses. Treat system-account approval as a migration dependency rather than a task to begin after code is finished.

Keep the key out of the evidence

The key is sent in the URL query string. URLs are routinely captured by proxy logs, tracing agents, command history, exception messages, monitoring tools and copied support tickets.

Use these controls:

• keep the key in a server-side secret manager or protected environment variable;
• never call the API directly from public browser code;
• redact api_key before any URL or parameter map is logged;
• do not use generic HTTP exception text that echoes the prepared request URL;
• disable full-query capture in reverse proxies and observability systems where possible;
• retain only a non-secret key label and account class in run manifests;
• do not store a completed extract download URL after inserting the API key; and
• follow the SAM.gov Terms of Use: update API keys and system-account passwords every 90 days, never share an individual key, and apply the stated organization boundary to system credentials; and
• test credential revocation without rebuilding the collector.

The evidence store needs to prove which account class ran a job without reproducing the credential.

Preserve awards, IDVs and modifications

The Contract Awards API can return delivery or task orders, purchase orders, definitive contracts, BPA calls, other transaction orders, IDVs, Federal Supply Schedules, blanket purchase agreements and other documented contract vehicles. A result is not automatically one economically independent “contract.” (GSA API overview)

At minimum, preserve these source fields before deriving internal IDs:

• issuing or PIID subtier code;
• PIID;
• modification number;
• transaction number when present;
• referenced IDV subtier, PIID and modification context;
• award-versus-IDV classification and type code;
• action obligation and total reported obligation as different measures;
• signed, approved, created, modified, effective and completion dates as separate facts;
• reason for modification;
• contracting and funding organizations;
• awardee UEI, CAGE and raw legal name; and
• the complete raw response that produced the normalized record.

Check third-party awardee-data rights

“Publicly revealed” does not mean every returned field is unrestricted for every downstream use. GSA's Contract Awards API page includes a D&B data-use limitation. It says SAM.gov contract-award records can contain D&B-supplied data; all base awards dated before 4 April 2022 are in the affected population when the outbound Entity Validation Service source is D&B. The notice requires attribution for defined D&B Open Data, restricts bulk access, use or dissemination that could substitute for the licensed product, and does not authorize other D&B data for commercial, resale or marketing uses. (GSA Contract Awards API — Additional Information)

Preserve the awardee data-source marker, segregate affected fields and obtain an appropriate rights decision before bulk publication, resale, marketing use or redistribution. The collector below requests awardeeData to demonstrate schema preservation during a controlled migration; the example does not grant a data license.

PIID alone is not the action key

A PIID can identify a procurement instrument while modifications identify reported actions against it. Orders can reference a parent IDV. Agency namespace and transaction context can matter. Deduplicating on PIID alone destroys action history; deduplicating on a title, supplier name or obligation amount is weaker still.

Keep two layers:

1. an immutable source-action layer with the complete returned contractId, oldContractId aliases, raw hash and a collision-tested candidate key; and
2. a derived award-family layer that groups related actions and vehicles for a stated analytical purpose.

The API’s piidAggregation feature can summarize a contract family, modifications and referencing orders or calls. It is useful for validation and lookup. It should not replace the underlying actions when the consumer needs modification history, dates, descriptions, negative obligations or an auditable total.

Public DoD data have a disclosure delay

GSA defines revealed data to include contracts either funded or awarded by a civilian subtier, plus contracts both funded and awarded by DoD when the signed date is at least 90 days before the current date. Appropriate DoD users can access unrevealed data. A revealed-data collector can therefore be internally correct and still appear to “miss” recent defense actions.

Record the access class and retrieval time. Revisit the disclosure horizon, and label late public arrivals separately from ordinary source modifications. Do not fill the gap with estimates.

Make the first Contract Awards API request

Start with a narrow, known range and an alpha API key in GSA's alpha environment. Send every operational control explicitly:

GET https://api-alpha.sam.gov/contract-awards/v1/search
  ?api_key=REDACTED
  &lastModifiedDate=[08/04/2026,08/05/2026]
  &limit=100
  &offset=0
  &includeSections=contractId,coreData,awardDetails,awardeeData

After the alpha contract tests pass, promote the same explicit request shape to the production endpoint:

GET https://api.sam.gov/contract-awards/v1/search

The documented date syntax is MM/DD/YYYY for a single date or [MM/DD/YYYY,MM/DD/YYYY] for a range. The API page describes offset as a page index beginning at zero, while the linked OpenAPI file declares a default of one. Always send the value explicitly and prove in alpha that pages zero and one are distinct before rollout. Synchronous JSON defaults to 10 rows and allows at most 100 per page. (GSA request parameters; Contract Awards OpenAPI file)

The first contract test should assert:

• HTTP 200 with JSON or the documented HTTP 204 no-data response;
• an awardSummary array for a successful non-empty result;
• parseable totalRecords, limit and offset, even when the source serializes them as strings;
• returned offset matching the requested page index;
• the selected response sections being present at the expected nested paths;
• every accepted row having a usable source-action key or an explicit quarantine reason;
• no key in logs, filenames, manifests or exception messages; and
• a raw body hash, retrieval timestamp and redacted request contract for every accepted page.

GSA documents HTTP 400 responses for malformed dates, dollar ranges and unknown parameters. For synchronous search, do not retry an unchanged 400. Extract polling also uses 400 for some operation-specific states, including a file still being generated, so branch on the operation and sanitized response contract rather than applying one global 400 rule. The response-code table uses 204 for no data. Treat that exact documented status as an empty result for a validated query; do not reinterpret 401, 403, 404 or an HTML proxy response as zero records.

Map FPDS Atom queries to REST filters

The official variance document should be versioned with the migration. It maps legacy request names such as LAST_MOD_DATE, AGENCY_CODE, MODIFICATION_NUMBER, REF_IDV_PIID, CAGE_CODE and VENDOR_UEI to the new request parameters.

Test semantics, not spelling

A name match in a crosswalk does not prove the same boundary behavior. For every production filter, build fixtures that test:

• single dates and closed date ranges;
• beginning-only and end-only ranges if the legacy query used them;
• date-boundary inclusion;
• null and not-null searches where supported;
• partial versus complete name matching;
• ~ OR lists and ! NOT conditions;
• code filters versus name filters;
• negative and positive dollar values;
• base records versus specific modifications;
• awards versus IDVs; and
• civilian versus delayed DoD results under the actual account class.

The API documentation says &, |, {, }, ^ and backslash are not allowed in parameter values. Use a real URL encoder and reject unsupported values before dispatch. Do not concatenate an untrusted filter string into a URL.

Use codes where identity matters

Names change, collide and vary in punctuation. Prefer department, subtier and office codes for stable partitions. Retain the returned names as source labels. If a legacy job filtered by an office name that no longer has a direct request equivalent, resolve and version the intended codes rather than silently broadening the population.

Limit sections deliberately

includeSections can request contractId, coreData, awardDetails, awardeeData and nasaSpecific. Smaller responses reduce transfer and parsing work, but a minimal section set can omit evidence a downstream consumer relies on.

Create a field-to-section manifest. A job may use contractId for a lightweight existence check, yet a production action feed will usually require core classification, amounts, dates, organization and awardee identity. Test the selected sections against every downstream field before optimizing payload size.

Paginate and partition synchronous search

The response provides totalRecords, limit, offset and awardSummary. Advance page indexes 0, 1, 2 and so on. Do not multiply the page by the limit unless a future version explicitly changes the contract.

The synchronous API exposes only the first 400,000 matches. A response can report a total larger than that ceiling, as GSA’s own examples show. A loop that stops at the cap has not completed the query.

For every bounded partition:

1. request page zero with limit=100;
2. parse and record the reported total;
3. reject the partition for synchronous collection if the total exceeds the accessible ceiling;
4. store the exact response bytes and page metadata;
5. hash the ordered source-action keys;
6. advance the page index and reject a repeated non-empty page hash;
7. compare each page’s total with the first page’s total;
8. stop only when the collected row count reaches the stable reported total or a documented empty response is consistent with that total; and
9. compare raw rows, unique source-action keys and expected rows before marking the page walk reconciled; and
10. repeat the closed partition and require the sorted key-and-content fingerprint to converge before accepting it for staging.

Partition before the ceiling

Do not plan around exactly 400,000 rows. A moving source can grow during a page walk. Partition by short lastModifiedDate ranges, then by stable codes such as contracting department, subtier, award/IDV type or another source fact when needed. Keep the partition manifest so every slice is provably covered once.

Large date ranges also increase the chance that page membership shifts during collection. Narrow ranges shorten a retry, reduce quota consumption after failure and make count changes easier to explain.

Deduplication does not repair a bad walk

If 10,000 rows were expected, 9,998 rows arrived and 9,996 source keys are unique, the accepted count is not 9,996. The run has missing and repeated evidence. Retain the page observations, fail the partition and retry or subdivide it.

Stable totals are not snapshot isolation

Stable totals and unique keys do not prove that every page came from one source snapshot. If page membership changes from A/B/C to B/C/D during the walk, an unlucky traversal can collect A/B/D with the same expected count and no duplicate key.

Call the first result page_walk_reconciled, not accepted. Repeat the same closed partition and compare a sorted fingerprint of every source key plus its canonical raw record. Convergence is a practical stability gate, not proof that GSA supplied snapshot isolation. A high-risk cutover can require several converged replays after a quiet period or use a bounded asynchronous extract with its own reconciliation evidence.

Build a migration-safe Python collector

The following skeleton uses the synchronous JSON path for a bounded modified-date partition. It checks each response for a reflected credential before writing raw bytes, redacts the key from durable metadata, distinguishes the documented 204 no-data result, rejects result sets beyond the synchronous ceiling and produces a walk-evidence manifest. A second converged walk is required for accepted: true.

It is intentionally a collector core rather than a complete scheduler or warehouse loader. Add your organization’s secret manager, metrics, object storage, schema registry and alerting around it.

from __future__ import annotations

import hashlib
import json
import os
import random
import time
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from urllib.parse import quote, quote_plus

import requests

ENDPOINTS = {
    "alpha": "https://api-alpha.sam.gov/contract-awards/v1/search",
    "production": "https://api.sam.gov/contract-awards/v1/search",
}
PAGE_LIMIT = 100
SYNC_RESULT_CAP = 400_000
RETRYABLE_STATUS = {429, 500, 502, 503, 504}
FORBIDDEN_VALUE_CHARACTERS = {"&", "|", "{", "}", "^", chr(92)}
ALLOWED_FILTERS = {
    "awardOrIDV",
    "awardOrIDVTypeCode",
    "contractingDepartmentCode",
    "contractingSubtierCode",
    "modificationNumber",
    "naicsCode",
    "piidSubtierCode",
    "productOrServiceCode",
}


class PartitionRequired(RuntimeError):
    pass


class QuarantinedRun(RuntimeError):
    pass


@dataclass(frozen=True)
class PageEvidence:
    page: int
    retrieved_at: str
    status: int
    attempts: int
    sha256: str
    byte_count: int
    raw_path: str
    requested_limit: int
    requested_offset: int
    inferred_empty: bool
    total_records: int | None
    returned_limit: int | None
    returned_offset: int | None
    row_count: int
    key_hash: str | None


def contains_secret(value: str | bytes, api_key: str) -> bool:
    text = (
        value.decode("utf-8", errors="ignore")
        if isinstance(value, bytes)
        else value
    ).lower()
    variants = {
        api_key,
        quote(api_key, safe=""),
        quote_plus(api_key, safe=""),
    }
    return any(candidate.lower() in text for candidate in variants if candidate)


def parse_source_int(value: Any, field: str) -> int:
    try:
        return int(value)
    except (TypeError, ValueError) as exc:
        raise QuarantinedRun(f"invalid source integer: {field}") from exc


def source_key(row: dict[str, Any]) -> str:
    contract_id = row.get("contractId") or {}
    old_contract_id = row.get("oldContractId") or []
    if not isinstance(old_contract_id, (list, dict)):
        raise QuarantinedRun("oldContractId has an unexpected shape")
    subtier = contract_id.get("subtier") or {}
    referenced_subtier = contract_id.get("referencedIDVSubtier") or {}
    identity = {
        "subtier_code": subtier.get("code"),
        "piid": contract_id.get("piid"),
        "modification_number": contract_id.get("modificationNumber"),
        "transaction_number": contract_id.get("transactionNumber"),
        "referenced_idv_subtier_code": referenced_subtier.get("code"),
        "referenced_idv_piid": contract_id.get("referencedIDVPiid"),
        "referenced_idv_modification": contract_id.get(
            "referencedIDVModificationNumber"
        ),
    }
    if not identity["subtier_code"] or not identity["piid"] or identity["modification_number"] is None:
        raise QuarantinedRun("missing source identity component")
    # GSA does not guarantee this composite is globally unique. Retain the
    # full contractId, oldContractId aliases and raw hash, then quarantine any
    # collision whose source identity or content differs.
    canonical = json.dumps(identity, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()


def write_raw_response(
    root: Path,
    page: int,
    attempt: int,
    response: requests.Response,
    api_key: str,
) -> tuple[Path, str]:
    body = response.content
    if contains_secret(body, api_key):
        raise QuarantinedRun(
            "response body reflected the API key; raw bytes were not persisted"
        )
    digest = hashlib.sha256(body).hexdigest()
    path = root / f"page-{page:06d}-attempt-{attempt:02d}-http-{response.status_code}.bin"
    path.write_bytes(body)
    return path, digest


def fetch_with_evidence(
    session: requests.Session,
    endpoint: str,
    params: dict[str, Any],
    evidence_root: Path,
    page: int,
    max_attempts: int = 5,
) -> tuple[requests.Response, int, Path, str]:
    for attempt in range(1, max_attempts + 1):
        try:
            response = session.get(endpoint, params=params, timeout=(10, 90))
        except requests.RequestException:
            if attempt == max_attempts:
                raise QuarantinedRun("transport failure after retry ceiling")
            time.sleep(min(30.0, (2 ** (attempt - 1)) + random.random()))
            continue

        raw_path, digest = write_raw_response(
            evidence_root, page, attempt, response, str(params["api_key"])
        )
        if response.status_code not in RETRYABLE_STATUS:
            return response, attempt, raw_path, digest

        if attempt == max_attempts:
            raise QuarantinedRun(
                f"retryable HTTP {response.status_code} exceeded retry ceiling"
            )

        retry_after = response.headers.get("Retry-After", "")
        delay = float(retry_after) if retry_after.isdigit() else 2 ** (attempt - 1)
        time.sleep(min(60.0, delay + random.random()))

    raise AssertionError("unreachable")


def collect_partition(
    start_date: str,
    end_date: str,
    output_root: Path,
    *,
    environment: str = "alpha",
    confirmation_manifest: dict[str, Any] | None = None,
    deleted: bool = False,
    extra_filters: dict[str, str] | None = None,
) -> dict[str, Any]:
    for value in (start_date, end_date):
        datetime.strptime(value, "%m/%d/%Y")

    try:
        endpoint = ENDPOINTS[environment]
    except KeyError as exc:
        raise ValueError("environment must be alpha or production") from exc

    api_key = os.environ["SAM_GOV_API_KEY"]
    key_label = os.environ["SAM_GOV_API_KEY_LABEL"].strip()
    account_class = os.environ["SAM_GOV_ACCOUNT_CLASS"].strip()
    access_class = os.environ["SAM_GOV_ACCESS_CLASS"].strip().lower()
    if not api_key:
        raise ValueError("API key must be non-empty")
    if not key_label or not account_class:
        raise ValueError("key label and account class must be non-empty")
    if access_class not in {"revealed", "unrevealed"}:
        raise ValueError("access class must be revealed or unrevealed")
    filters = dict(extra_filters or {})
    unknown = set(filters) - ALLOWED_FILTERS
    if unknown:
        raise ValueError(f"unsupported extra filter names: {sorted(unknown)}")
    invalid_values = [
        name for name, value in filters.items()
        if not isinstance(value, str)
        or any(char in value for char in FORBIDDEN_VALUE_CHARACTERS)
    ]
    if invalid_values:
        raise ValueError(
            f"unsupported extra filter values: {sorted(invalid_values)}"
        )
    durable_inputs = [
        str(output_root),
        key_label,
        account_class,
        json.dumps(filters, sort_keys=True),
    ]
    if any(contains_secret(value, api_key) for value in durable_inputs):
        raise ValueError("durable metadata must not contain the API key")

    base_params: dict[str, Any] = {
        "api_key": api_key,
        "lastModifiedDate": f"[{start_date},{end_date}]",
        "limit": PAGE_LIMIT,
        "includeSections": "contractId,coreData,awardDetails,awardeeData",
        **filters,
    }
    if deleted:
        base_params["deletedStatus"] = "yes"

    request_contract = {
        "contract_version": 1,
        "environment": environment,
        "endpoint": endpoint,
        "key_label": key_label,
        "account_class": account_class,
        "access_class": access_class,
        "lastModifiedDate": base_params["lastModifiedDate"],
        "deletedStatus": base_params.get("deletedStatus"),
        "limit": PAGE_LIMIT,
        "includeSections": base_params["includeSections"],
        "extra_filters": filters,
    }
    request_contract_sha256 = hashlib.sha256(
        json.dumps(
            request_contract, sort_keys=True, separators=(",", ":")
        ).encode()
    ).hexdigest()

    confirmation_snapshot_sha256: str | None = None
    confirmation_run_id: str | None = None
    confirmation_manifest_sha256: str | None = None
    if confirmation_manifest is not None:
        if confirmation_manifest.get("page_walk_reconciled") is not True:
            raise ValueError("confirmation manifest did not reconcile")
        if (
            confirmation_manifest.get("request_contract_sha256")
            != request_contract_sha256
        ):
            raise ValueError("confirmation manifest uses another request contract")
        confirmation_snapshot_sha256 = confirmation_manifest.get(
            "snapshot_sha256"
        )
        confirmation_run_id = confirmation_manifest.get("run_id")
        if not isinstance(confirmation_snapshot_sha256, str):
            raise ValueError("confirmation manifest has no snapshot fingerprint")
        if not isinstance(confirmation_run_id, str) or not confirmation_run_id:
            raise ValueError("confirmation manifest has no run ID")
        try:
            confirmation_text = json.dumps(
                confirmation_manifest, sort_keys=True, separators=(",", ":")
            )
        except (TypeError, ValueError) as exc:
            raise ValueError("confirmation manifest is not canonical JSON") from exc
        if contains_secret(confirmation_text, api_key):
            raise ValueError("confirmation manifest contains the API key")
        confirmation_manifest_sha256 = hashlib.sha256(
            confirmation_text.encode()
        ).hexdigest()

    run_id = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S.%fZ")
    evidence_root = output_root / run_id
    evidence_root.mkdir(parents=True, exist_ok=False)

    rows: list[dict[str, Any]] = []
    unique_keys: set[str] = set()
    record_hashes: dict[str, str] = {}
    page_hashes: set[str] = set()
    page_evidence: list[PageEvidence] = []
    expected_total: int | None = None
    reported_total: int | None = None
    page = 0
    caught: Exception | None = None

    try:
        with requests.Session() as session:
            session.headers.update({
                "Accept": "application/json",
                "User-Agent": "contract-awards-migration/1.0",
            })

            while True:
                params = {**base_params, "offset": page}
                response, attempts, raw_path, digest = fetch_with_evidence(
                    session, endpoint, params, evidence_root, page
                )
                observed_at = datetime.now(timezone.utc).isoformat()
                page_evidence.append(PageEvidence(
                    page=page,
                    retrieved_at=observed_at,
                    status=response.status_code,
                    attempts=attempts,
                    sha256=digest,
                    byte_count=len(response.content),
                    raw_path=str(raw_path),
                    requested_limit=PAGE_LIMIT,
                    requested_offset=page,
                    inferred_empty=False,
                    total_records=None,
                    returned_limit=None,
                    returned_offset=None,
                    row_count=0,
                    key_hash=None,
                ))

                if response.status_code == 204:
                    page_evidence[-1] = PageEvidence(
                        page=page,
                        retrieved_at=observed_at,
                        status=204,
                        attempts=attempts,
                        sha256=digest,
                        byte_count=len(response.content),
                        raw_path=str(raw_path),
                        requested_limit=PAGE_LIMIT,
                        requested_offset=page,
                        inferred_empty=True,
                        total_records=None,
                        returned_limit=None,
                        returned_offset=None,
                        row_count=0,
                        key_hash=None,
                    )
                    if page == 0:
                        expected_total = 0
                        break
                    if expected_total is not None and len(rows) == expected_total:
                        break
                    raise QuarantinedRun(
                        "unexpected 204 before expected rows arrived"
                    )

                if response.status_code != 200:
                    raise QuarantinedRun(
                        f"non-retryable HTTP {response.status_code}"
                    )

                content_type = response.headers.get("Content-Type", "").lower()
                if "json" not in content_type:
                    raise QuarantinedRun("HTTP 200 response is not JSON")

                try:
                    payload = response.json()
                except ValueError as exc:
                    raise QuarantinedRun("invalid JSON response") from exc

                page_rows = payload.get("awardSummary")
                if not isinstance(page_rows, list):
                    raise QuarantinedRun("awardSummary is not an array")

                total = parse_source_int(
                    payload.get("totalRecords"), "totalRecords"
                )
                returned_limit = parse_source_int(payload.get("limit"), "limit")
                returned_offset = parse_source_int(
                    payload.get("offset"), "offset"
                )
                page_evidence[-1] = PageEvidence(
                    page=page,
                    retrieved_at=observed_at,
                    status=200,
                    attempts=attempts,
                    sha256=digest,
                    byte_count=len(response.content),
                    raw_path=str(raw_path),
                    requested_limit=PAGE_LIMIT,
                    requested_offset=page,
                    inferred_empty=False,
                    total_records=total,
                    returned_limit=returned_limit,
                    returned_offset=returned_offset,
                    row_count=len(page_rows),
                    key_hash=None,
                )
                if returned_limit != PAGE_LIMIT or returned_offset != page:
                    raise QuarantinedRun(
                        "returned page metadata differs from request"
                    )

                if expected_total is None:
                    expected_total = total
                    reported_total = total
                    if expected_total > SYNC_RESULT_CAP:
                        raise PartitionRequired(
                            f"reported total {expected_total} exceeds synchronous cap"
                        )
                elif total != expected_total:
                    raise QuarantinedRun("totalRecords changed during page walk")

                keys = [source_key(row) for row in page_rows]
                if len(keys) != len(set(keys)):
                    raise QuarantinedRun(
                        "candidate source key repeated within a page"
                    )
                key_hash = hashlib.sha256("\n".join(keys).encode()).hexdigest()
                page_evidence[-1] = PageEvidence(
                    **{
                        **asdict(page_evidence[-1]),
                        "key_hash": key_hash,
                    }
                )
                if page_rows and key_hash in page_hashes:
                    raise QuarantinedRun("repeated non-empty page detected")
                page_hashes.add(key_hash)

                duplicate_keys = unique_keys.intersection(keys)
                if duplicate_keys:
                    raise QuarantinedRun(
                        "candidate source key repeated across pages"
                    )
                unique_keys.update(keys)
                for key, row in zip(keys, page_rows):
                    canonical_row = json.dumps(
                        row, sort_keys=True, separators=(",", ":")
                    )
                    record_hashes[key] = hashlib.sha256(
                        canonical_row.encode()
                    ).hexdigest()
                rows.extend(page_rows)

                if len(rows) >= expected_total:
                    break
                if not page_rows:
                    raise QuarantinedRun("empty page before expected total")
                page += 1
    except (PartitionRequired, QuarantinedRun) as exc:
        caught = exc
    except Exception as exc:
        caught = QuarantinedRun(
            f"unexpected collector failure: {type(exc).__name__}"
        )

    page_walk_reconciled = (
        caught is None
        and expected_total is not None
        and len(rows) == expected_total
        and len(unique_keys) == expected_total
        and len(record_hashes) == expected_total
    )
    snapshot_sha256: str | None = None
    if page_walk_reconciled:
        snapshot_material = "\n".join(
            [request_contract_sha256] + [
                f"{key}:{record_hashes[key]}" for key in sorted(record_hashes)
            ]
        )
        snapshot_sha256 = hashlib.sha256(
            snapshot_material.encode()
        ).hexdigest()
    if caught is None and not page_walk_reconciled:
        caught = QuarantinedRun("partition did not reconcile")
    if (
        caught is None
        and confirmation_snapshot_sha256 is not None
        and confirmation_snapshot_sha256 != snapshot_sha256
    ):
        caught = QuarantinedRun("confirmation replay did not converge")
    accepted = (
        caught is None
        and page_walk_reconciled
        and confirmation_manifest is not None
    )
    manifest = {
        "run_id": run_id,
        "finished_at": datetime.now(timezone.utc).isoformat(),
        "environment": environment,
        "endpoint": endpoint,
        "credential": {
            "key_label": key_label,
            "account_class": account_class,
            "access_class": access_class,
        },
        "query": {
            "lastModifiedDate": f"[{start_date},{end_date}]",
            "deletedStatus": "yes" if deleted else None,
            "limit": PAGE_LIMIT,
            "includeSections": base_params["includeSections"],
            "extra_filters": filters,
        },
        "reported_total": reported_total,
        "expected_total": expected_total,
        "expected_total_inferred": (
            reported_total is None and expected_total is not None
        ),
        "collected_rows": len(rows),
        "unique_source_keys": len(unique_keys),
        "page_walk_reconciled": page_walk_reconciled,
        "request_contract_sha256": request_contract_sha256,
        "snapshot_sha256": snapshot_sha256,
        "confirmation": None if confirmation_manifest is None else {
            "run_id": confirmation_run_id,
            "manifest_sha256": confirmation_manifest_sha256,
            "snapshot_sha256": confirmation_snapshot_sha256,
        },
        "accepted": accepted,
        "failure": None if caught is None else {
            "type": type(caught).__name__,
            "reason": str(caught),
        },
        "pages": [asdict(item) for item in page_evidence],
    }
    manifest_text = json.dumps(manifest, indent=2)
    if contains_secret(manifest_text, api_key):
        raise QuarantinedRun(
            "manifest serialization contained the API key; it was not persisted"
        )
    (evidence_root / "manifest.json").write_text(
        manifest_text, encoding="utf-8"
    )
    if caught is not None:
        raise caught
    return {"manifest": manifest, "rows": rows}

Save the example as contract_awards_collector.py, install its only third-party dependency with python -m pip install requests, and set SAM_GOV_API_KEY, SAM_GOV_API_KEY_LABEL, SAM_GOV_ACCOUNT_CLASS and SAM_GOV_ACCESS_CLASS in the worker environment. The label and classifications are non-secret manifest values; the API key remains secret. Then run two complete walks of one closed alpha partition:

from pathlib import Path

from contract_awards_collector import collect_partition

baseline = collect_partition(
    "08/04/2026",
    "08/05/2026",
    Path("evidence/contract-awards"),
    environment="alpha",
)
result = collect_partition(
    "08/04/2026",
    "08/05/2026",
    Path("evidence/contract-awards"),
    environment="alpha",
    confirmation_manifest=baseline["manifest"],
)
print(result["manifest"]["accepted"])

A converged pair prints True and creates two evidence/contract-awards/<run_id>/ directories with the final response for every page plus manifest.json. The first manifest can be page_walk_reconciled: true while remaining accepted: false; the second validates and records that baseline run ID and manifest hash, then becomes accepted only when its fingerprint of the versioned non-secret request/access contract, sorted source keys and canonical record content matches. That binding prevents an orphan digest or an empty result from an unrelated query or environment from confirming another partition. Expected collection, reconciliation and convergence failures receive an accepted: false manifest with a sanitized reason. Inspect that evidence before loading result["rows"]; a completed request is not the acceptance condition. Promote with environment="production" only after supplying the corresponding production key and passing the alpha contract tests.

The template deliberately does not print the request exception or prepared URL because common HTTP clients include the full query string in diagnostic text. Before saving a response, it scans the body for raw and URL-encoded forms of the key and fails closed if a service or proxy reflected the credential. It rejects the key in the output path, label, account class, filters or baseline manifest, then scans the fully serialized manifest again before writing it. It also closes the session, preserves safe raw bytes before JSON parsing and refuses to normalize a partition that does not reconcile. The candidate key is not presented as a GSA uniqueness guarantee: persist the complete contractId, oldContractId aliases and raw record hash, then investigate any collision instead of overwriting it.

The in-memory rows list keeps the tutorial readable, not infinitely scalable. A high-volume implementation should stream page records into provisional storage, reconcile the complete partition from its manifest and source keys, and publish the partition only after the acceptance gates pass.

Production additions should include:

• durable object storage with retention and access controls;
• an attempt-level manifest for transport failures and all retry responses;
• automated partition subdivision when the synchronous cap is exceeded;
• a schema contract for every selected response section;
• explicit source-key rules for any documented record class that legitimately lacks a usual key component;
• Decimal parsing for dollar strings rather than binary floating point;
• a two-lane current/deleted schedule;
• asynchronous extract polling and secure download handling;
• downstream idempotency and tombstone tests; and
• metrics and alerts that use accepted partitions, never merely completed HTTP calls.

Run overlapping lastModifiedDate syncs

The replacement API exposes lastModifiedDate, which is the natural incremental boundary for changed records. It is still a date-range query, not a transactional cursor with documented snapshot isolation.

Use a checkpoint strategy like this:

1. choose short calendar partitions;
2. collect and validate the entire partition;
3. advance the accepted checkpoint only after the page walk reconciles and a closed-partition replay fingerprint converges;
4. overlap at least the recent boundary when the next run begins;
5. upsert by source-action key and retain every observed raw version;
6. derive changes only between replay-converged accepted partitions; and
7. periodically replay older partitions to detect late disclosure, correction or operational gaps.

The exact overlap should be measured from source behavior and business latency, not copied from a generic recipe. Start by re-reading at least the previous complete day; a two-day overlap gives a safer initial baseline. Date-only filters make boundary testing particularly important because GSA does not document timezone, boundary inclusivity, result ordering or snapshot isolation. Store the request range and the minimum and maximum returned modification timestamps so a reviewer can see whether the observed records agree with the requested slice. Advance from the last accepted closed day—never use the collector's current clock time as an unproven high-water mark.

Separate a source change from a business event

lastModifiedDate means the source record changed. It does not, by itself, say a new contract was awarded, funding was added or performance ended. A later update can be an administrative correction, closeout action, status change or modification with no new obligation.

Preserve the new source version first. Then classify the change using modification number, reason, action obligation, status, dates and changed fields. This keeps ingestion evidence separate from alert semantics.

Re-read the defense disclosure horizon

A revealed-data collector without eligible DoD unrevealed access cannot retrieve recent contracts both funded and awarded by DoD. Once a record crosses the 90-day rule, it may appear in a collection even though the signed action is older than the current sync window. GSA does not promise that becoming publicly revealed advances lastModifiedDate.

Maintain a scheduled replay by signed-date cohort and agency in addition to the ordinary modified-date sync. Measure late public arrivals and publish that limitation with any near-real-time defense analysis.

Collect deleted contracts separately

The API supports a dedicated deletedStatus=yes query and can return contracts deleted within the previous six months. When that parameter is sent, the API returns deleted records only. (GSA deleted-contract documentation)

Run this as an independent lane:

GET https://api.sam.gov/contract-awards/v1/search
  ?api_key=REDACTED
  &deletedStatus=yes
  &lastModifiedDate=[08/04/2026,08/05/2026]
  &includeSections=contractId
  &limit=100
  &offset=0

For every returned source key:

• retain the raw deletion response;
• link it to the last accepted non-deleted version;
• record deletion observation time and requested window;
• produce a tombstone for downstream consumers;
• prevent a later replay of an older active snapshot from resurrecting the row; and
• keep the evidence even when a public-facing product removes the record.

Do not infer deletion because a row disappears from an ordinary query. Absence can result from a changed filter value, moving page membership, delayed disclosure, access changes, a failed partition or a source correction. Only explicit source evidence should drive a deletion tombstone.

The six-month deleted-record horizon creates a recovery requirement. A collector that is offline for longer can no longer assume the endpoint will replay every missed deletion. Monitor the deletion lane separately and retain your own history.

Use asynchronous extracts for backfills

Adding format=json or format=csv turns a bounded search into an asynchronous extract request. GSA says extracts can return the first 1,000,000 matching records. The initial response provides an export token and a download URL containing REPLACE_WITH_API_KEY; the client supplies its key when requesting the generated file.

Use synchronous JSON when:

• the partition is narrow and comfortably below 400,000 matches;
• low-latency pagination is useful;
• the job needs immediate page-level validation; or
• a small source sample is being reconciled during development.

Use an extract when:

• bootstrapping a bounded historical partition;
• a page walk would consume an unreasonable share of the daily allowance;
• JSON or CSV files fit the downstream batch workflow; or
• a broad, stable slice is easier to validate as one generated artifact.

An extract is not permission to send an unbounded query. One million is also a ceiling. Partition large jobs by date and stable source codes, inventory every requested slice and verify that generated files cover the complete manifest. GSA does not document a token lifetime; download a ready artifact promptly and regenerate the job rather than assuming an old token remains valid.

Secure the extract workflow

The finished download request still needs the API key in its URL. Keep the token and redacted template separate from the credential. Do not write the completed URL to the manifest, object-store metadata, job log or error tracker. Polling behavior must be operation-specific: preserve and inspect a sanitized extract response because a 400 can mean generation is still in progress as well as a real request failure.

For each file, retain:

• the redacted query and export token;
• request and ready timestamps;
• HTTP status and content type;
• byte size and cryptographic hash;
• archive member names and row counts;
• schema fingerprint;
• source-key uniqueness and required-field coverage; and
• reconciliation with the query’s expected population.

Treat “file downloaded” as a transport outcome. Acceptance occurs only after archive, schema, identity and count checks pass.

Map XML responses to JSON without losing meaning

In the FPDS SOAP/XML representation, a coded element can carry the code as its text and the descriptive label as an attribute. GSA’s example represents country of origin with USA as the value and UNITED STATES as the name attribute. The SAM.gov JSON representation uses an object with explicit code and name properties.

That is a semantic improvement, but a naive converter can still lose data.

Store the pair

For every coded field, preserve both the source code and source label. Use the code for stable grouping when appropriate; show the label for human interpretation. Version internal mappings rather than overwriting historical labels.

Preserve null states

Keep these states distinct:

• property absent because the section was not requested;
• property absent inside a requested section;
• JSON null;
• empty string;
• a literal source placeholder;
• code present with label missing;
• label present with code missing; and
• parsing or schema failure.

Collapsing them into one blank value hides whether the source, request contract or parser caused the gap.

Expect structural moves

The replacement response groups fields under contractId, coreData and awardDetails, with further nested structures for organizations, place of performance, product or service, awardee information and transaction data. An XPath-to-flat-key rewrite will not preserve the hierarchy.

Create a versioned transformation map with:

• legacy XPath;
• new JSON path;
• source type and nullable behavior;
• normalization rule;
• destination field;
• fixture record;
• equality or expected-variance assertion; and
• owner for unresolved differences.

Keep the raw XML from the dual run and raw JSON from the replacement. A normalized table alone cannot explain a disputed migration result.

Reconcile the parallel run before cutover

Run both sources against matched populations while the Atom feed remains available. Exact row-count equality is useful, but it is only the first layer.

Reconcile by:

• query partition and retrieval cutoff;
• issuing subtier and contracting department;
• award versus IDV;
• base action versus modification;
• PIID and referenced-IDV relationships;
• action type and reason for modification;
• signed, approved and modified dates;
• positive, zero and negative action obligations;
• total obligations kept separate from action obligations;
• awardee UEI, CAGE and legal name;
• NAICS and PSC code/label pairs;
• null and missing-field rates;
• closed and deleted lifecycle states; and
• public DoD records by signed-date cohort.

Classify every mismatch:

1. expected schema or naming variance;
2. intentional query-scope difference;
3. source timing or disclosure difference;
4. legacy parser defect;
5. replacement parser defect;
6. moving-page or incomplete-partition failure;
7. access or section-selection difference; or
8. unresolved source discrepancy.

Do not hide an unexplained variance inside a percentage tolerance. A tolerance can be an acceptance gate only when the excluded population is measured and its reason is documented.

Use a reader cutover, not a destructive replacement

During the dual run, write legacy and replacement evidence to separate immutable namespaces and normalize both into versioned staging tables. Once the gates pass:

1. freeze the accepted legacy and replacement checkpoints;
2. switch a small downstream reader cohort to the replacement view;
3. compare operational metrics and business outputs;
4. expand the reader cohort;
5. keep the last legacy checkpoint available for rollback; and
6. retire Atom credentials and jobs only after the rollback window closes.

Rollback should change which accepted dataset readers use. It should not delete source evidence or rewrite history.

FPDS API migration checklist

Before declaring the project complete, confirm all of the following.

Scope and access

• [ ] Every FPDS Atom job and consumer is inventoried.
• [ ] Public revealed versus authorized unrevealed scope is written down.
• [ ] The correct SAM.gov personal or system account is approved.
• [ ] Contract Awards read permissions and system-account IPs are tested.
• [ ] The daily request allowance supports the proposed schedule and retry budget.
• [ ] API keys are absent from browser code, logs, manifests, stored URLs and exceptions.
• [ ] Ninety-day key and system-password rotation plus credential-sharing boundaries have owners.

Request contract

• [ ] Each legacy query has a direct, structural, redesigned or unsupported mapping status.
• [ ] Date, code, null, range, OR and NOT semantics have fixtures.
• [ ] Unsupported legacy office-name and description filters have an approved replacement behavior.
• [ ] limit, offset and includeSections are explicit.
• [ ] Large populations are partitioned below the synchronous or extract ceiling.

Source model

• [ ] Base actions, modifications, awards, orders and IDVs stay distinguishable.
• [ ] The source key includes agency namespace, PIID and action context.
• [ ] Referenced-IDV relationships are preserved.
• [ ] Action obligations, total obligations and derived family totals are separate.
• [ ] Codes and names are stored as source pairs.
• [ ] Null, absent, empty and failed-parse states remain distinct.
• [ ] D&B-sourced awardee fields have an approved attribution, use and redistribution decision.

Synchronization and evidence

• [ ] Every page or extract has raw bytes, hash, retrieval time and redacted query evidence.
• [ ] Page totals, rows and unique source keys reconcile, then a closed-partition key-and-content replay converges before acceptance.
• [ ] Recent modified-date windows overlap and checkpoints advance only after acceptance.
• [ ] Deleted records run through a separate deletedStatus=yes lane.
• [ ] Recent DoD signed-date cohorts are replayed for delayed public disclosure.
• [ ] Failed partitions remain visible and cannot publish as zero.

Cutover

• [ ] Dual-run mismatches are classified rather than averaged away.
• [ ] Counts, identities, relationships, obligations, nulls and lifecycle states pass written gates.
• [ ] Downstream reports and alerts have replacement-source acceptance tests.
• [ ] Reader traffic can switch independently of source collection.
• [ ] The last accepted legacy and replacement checkpoints are retained.
• [ ] Rollback, credential revocation and job retirement have been rehearsed.

Frequently asked questions