Our own link checker flagged SEC.gov filings as broken. They weren't.

The false positive

A routine pass with a generic link-checking script came back with three flagged URLs on this site's delisting radar page — all direct links to 8-K filings on sec.gov, all returning HTTP 403. A 403 on a filing that was live and correct a week ago looks exactly like link rot: something moved or got taken down, go fix the link.

Except the filings hadn't moved. Opening the same URLs in an actual browser worked instantly, every time. The 403 wasn't about the link at all — it was about who was asking.

SEC.gov requires you to identify yourself

SEC.gov's fair-access policy asks every automated requester to send a User-Agent header with a real company name and contact email — not a browser impersonation, not a generic tool signature. Requests that don't are throttled or blocked outright, silently, with a plain 403. No error message explains why; there's nothing to distinguish it from a page that's genuinely gone.

$ curl -sI -A "curl/8.4.0" \
    "https://www.sec.gov/Archives/edgar/data/.../form8-k.htm"
HTTP/2 403

$ curl -sI -A "Edge Thirteen hello@edgethirteen.com" \
    "https://www.sec.gov/Archives/edgar/data/.../form8-k.htm"
HTTP/2 200

Same URL, same request, only the header changed. Most link checkers, uptime monitors, and scraping libraries ship a default User-Agent that looks nothing like this — a library name and version, or nothing at all — so they get the 403 by default and report a dead link that was never dead.

The fix, and why it already worked here

Every fetch this site makes to sec.gov or data.sec.gov — for company fundamentals, ticker lookups, filing text — already sends a compliant header, because the data pipeline behind Edge Thirteen's screens was built against SEC's own rate-limit rules from day one:

// lib/edgarLookup.ts -- every fetch to sec.gov / data.sec.gov in this
// codebase goes through the same header, per SEC's published fair-access
// policy (17 CFR 202.5): identify yourself, or get rate-limited or blocked.
const UA = { "User-Agent": "Edge Thirteen hello@edgethirteen.com" };

fetch("https://data.sec.gov/api/xbrl/companyfacts/CIK0000320193.json", {
  headers: UA,
});

The false positive was entirely in the audit tool, not the product — it was checking outbound links with a generic signature instead of identifying itself. The fix for a link checker (or any script hitting EDGAR) is the same one-line header change: send a real, honest identifier instead of a default. Spoofing a browser User-Agent works around the block without answering what SEC is actually asking for, and creates a new problem — you can't tell a real block from a real outage anymore either.

The general lesson

A 403 from a government or regulatory data source is not evidence the resource is gone — it's evidence the request didn't meet that source's access policy. Before trusting any automated "broken link" or "down" finding against sec.gov, data.sec.gov, or similar public-data APIs, re-request with a real identifying header and see if the answer changes. If it does, the bug was in the checker, not the target.

Same instinct that caught a table-of-contents parsing bug and an off-by-one field mapping bug in bank regulatory data: don't trust the first plausible-looking answer a public data source gives you — verify it against a second request before you build on top of it.