Federal agencies now run AI systems that approve disability claims, screen procurement bids, process FOIA requests, and determine who qualifies for public housing. In fiscal year 2024, U.S. federal agencies collectively received over 1.5 million FOIA requests a 25% year-on-year surge while simultaneously deploying machine-learning tools to manage that very backlog. That collision of scale and automation is precisely where AI governance in the public sector becomes a constitutional, not just a technical, question. This article examines the three domains where algorithmic systems are reshaping government accountability in the United States: procurement rules, FOIA transparency obligations, and algorithmic fairness in public benefits.
The Regulatory Foundation: Where Federal AI Policy Stands Today
Two OMB memoranda issued on April 3, 2025 now define how federal agencies acquire and deploy artificial intelligence. M-25-21 “Accelerating Federal Use of AI through Innovation, Governance, and Public Trust” requires agencies to publish an AI Strategy within 180 days, assess enterprise-wide AI maturity, and identify barriers to responsible adoption. M-25-22, which replaced the Biden-era M-24-18, governs how agencies actually buy AI, applying to solicitations issued on or after September 30, 2025.
These memos rescinded and replaced the Biden administration’s Executive Order 14110 framework, shifting the federal posture from one that prioritized civil-rights guardrails toward a model that emphasizes competitive market efficiency and reduced procurement friction. The core obligations for safety and fairness remain, but the enforcement architecture is now meaningfully different.
OMB M-25-21 requires every covered federal agency to develop a written AI Strategy within 180 days of the memo’s April 3, 2025 effective date meaning the first deadlines fell in late September 2025.
Two layers of regulation now sit beneath OMB guidance. The Federal AI Governance and Transparency Act (HR 7532), introduced in March 2024, would codify requirements for procurement policies, data ownership protections, and training-data safeguards into statute. It has not yet passed, but its framework signals congressional intent. At the agency level, the Federal Acquisition Regulation (FAR) governs the mechanics of purchasing and aligning OMB’s AI procurement guidance with FAR’s existing structure remains an active area of legal and administrative debate.
|
Framework |
Core Requirement |
|
OMB M-25-21 (Apr 2025) |
AI Strategy within 180 days; innovation, governance, public trust pillars |
|
OMB M-25-22 (Apr 2025) |
AI acquisition standards; replaces M-24-18; applies to new contracts from Sep 30, 2025 |
|
OMB M-26-04 (Dec 2025) |
Unbiased AI Principles; disclosure requirements for contractors |
|
HR 7532 (pending) |
Statutory procurement, data ownership, and training-data safeguards |
|
Executive Order 14275 (Apr 2025) |
FAR reform directive; streamlining procurement across executive agencies |
AI in Federal Procurement: What Agencies Must Now Do
Federal procurement is the largest lever the U.S. government has over AI standards. When the General Services Administration sets a requirement, it ripples through thousands of vendor contracts. That purchasing power exercised responsibly can pull the private sector toward higher-quality, more inclusive AI systems. Exercised carelessly, it can lock agencies into opaque, unaccountable tools for years.
What M-25-22 Actually Requires
The procurement memo builds three core obligations for agencies buying AI. First, market transparency: the government must benefit from a competitive American AI marketplace, meaning agencies cannot default to single-source contracts without documented justification. Second, taxpayer performance tracking: agencies must monitor AI system performance after deployment, not just at procurement. Third, vendor disclosure: under the companion memo M-26-04, contractors must comply with Unbiased AI Principles and disclose how their systems handle bias and accuracy across demographic groups.
The practical implication is significant. A vendor selling a benefits-eligibility screening tool to the Department of Labor, for instance, will need to demonstrate the system’s error rates by race, gender, and disability status before contract award and maintain that reporting throughout the performance period. This is a meaningful shift from a world where procurement focused almost exclusively on cost and technical capability.
The Unresolved FAR Alignment Problem
The Federal Acquisition Regulation predates the current AI era by decades. Its risk management frameworks, contract terms, and audit mechanisms were designed for tangible goods and traditional software not for probabilistic systems whose outputs shift with retraining. Executive Order 14275 directed the FAR Council to modernize the regulation, but FAR changes take years. In the interim, agencies are managing a gap between what OMB policy requires and what their contract vehicles actually allow them to enforce.
The General Services Administration is developing a digital ‘acquisition journey’ platform to guide agencies through AI procurement an acknowledgment that the current FAR framework alone is insufficient for AI-specific risk management.
Smaller agencies face particular challenges. A mid-size agency without a dedicated AI procurement team has no realistic path to conducting the kind of pre-award bias audits that M-25-22 envisions. The guidance exists. The institutional capacity to act on it does not yet, for many.
Public-sector AI governance is not limited to federal agencies. Public schools, universities, and education departments face the same accountability problem when AI systems process sensitive records and influence access to services. That is why AI governance in education needs to be treated as part of the broader public trust and data rights conversation.
FOIA in the Age of AI: Transparency or Opacity at Scale?
The Freedom of Information Act has been the primary mechanism for public accountability over government operations since 1966. AI changes the FOIA equation in two directions simultaneously: it expands the volume of government records that must be managed, and it introduces the question of whether AI-assisted record processing must itself be disclosed.
The Volume Crisis
Federal agencies received 1.5 million FOIA requests in fiscal year 2024 a 25% increase from the prior year and closed the fiscal year with a backlog of 267,056 unresolved requests. Ninety-two percent of covered agencies received more requests than in FY2023; 43 agencies saw volume more than double. Over $54 million was spent on FOIA litigation-related activities in FY2024 alone. The State Department, responding to these numbers, is now testing two AI models to help search its centralized databases and archives, which hold more than 3 billion records.
MITRE Corporation developed the FOIA Assistant prototype, built on natural language processing, to help analysts identify and categorize exempt material faster. The system was trained on datasets annotated by experienced FOIA analysts a reasonable approach to keeping human expertise in the loop. Multiple agencies have tested it, and initial feedback has been encouraging. But the same tool that speeds up record retrieval can also automate decisions about which records are withheld, and that is where the governance question becomes acute.
The Transparency Paradox
Researchers and advocacy organizations have surfaced a central irony: agencies deploying AI to process FOIA requests have largely not disclosed what systems they are using or how those systems make redaction recommendations. As legal scholar Ronald Capaldi argued in the Journal of the National Association of Administrative Law Judiciary, the U.S. government should implement algorithmic audits to monitor how AI is used in FOIA processing, and agencies should be required to publicly explain how AI systems are deployed when managing records requests.
There must be procedures for challenging decisions where machine algorithms are used including when those systems could be unnecessarily or illegally withholding information. (Transparency advocate quoted by NBC News, 2023)
A specific legal gap compounds the problem. FOIA’s trade secret exemption originally designed to protect proprietary business processes from disclosure can be invoked by vendors to shield the AI systems they’ve sold to the government. Capaldi’s research calls for clarifying that exemption to prevent it from blocking disclosure of how third-party AI tools function within agency FOIA workflows. Without that clarification, the public faces a situation where AI withholds information and the reason it does so is also withheld.
|
AI FOIA Application |
Risk if Unregulated |
|
Automated record search across billions of documents |
Search queries may be systematically biased by training data |
|
Exemption classification (identifying protected material) |
Over-redaction without mechanism for challenge |
|
Intake prioritization and routing |
Requesters from press or advocacy orgs may be de-prioritized |
|
Vendor-built redaction tools (trade secret protected) |
No public visibility into decision logic |
Algorithmic Fairness in Public Benefits: Where the Stakes Are Highest
When a private algorithm produces a biased output, someone gets a less relevant search result or a poorly targeted ad. When a government algorithm does the same, someone loses their disability benefits, their public housing placement, or their eligibility determination for food assistance. The asymmetry in consequence is the defining reason AI governance in the public sector requires stricter standards than the private market typically enforces.
Social Security and Disability Processing
The Social Security Administration has deployed AI to analyze medical evidence and reduce backlogs in disability claim processing. In principle, faster AI-driven analysis means fewer applicants waiting months in financial hardship for determinations that should take weeks. In practice, the same systems introduce new failure modes. Non-diverse training data may lead to bias or errors particularly for medical conditions not well represented in the SSA’s historical listings. Automated systems optimized for throughput may produce incorrect denials that require appeals, shifting the burden back to applicants who may lack the resources to navigate that process.
Nationwide AI-assisted transcription of disability hearings has also been introduced, with the intention of reducing administrative burden on judges. But transcription errors at hearings with poor audio quality can remove critical testimony from the official record a technical failure with legal consequences for the applicant.
Welfare and Benefits Administration
A 2020 survey of 142 federal agencies found that 45% were using or planning to use machine learning algorithms to streamline benefits operations. The promise is speed faster decisions, less financial uncertainty for applicants. The risk is what researchers call ‘false consensus bias’: when AI systems trained on average patterns make systematic errors for populations that diverge from those averages, which is precisely the population most likely to need public assistance.
State-level implementations have produced documented harms. Indiana’s automated benefits system terminated tens of thousands of Medicaid and food-stamp recipients based on algorithmic error in 2009. Arkansas deployed an AI system to calculate hours for home-care workers that cut services to hundreds of people with disabilities a decision courts later found violated due process because the system’s logic was not disclosed to those affected.
Courts have found that when an automated government system makes a consequential determination terminating benefits, reducing services and the system’s logic is not disclosed, this may constitute a violation of procedural due process under the Fifth and Fourteenth Amendments.
Disparate Impact and Algorithmic Discrimination
Federal anti-discrimination law Title VI, the Fair Housing Act, the Equal Credit Opportunity Act prohibits practices that produce disparate impact by race, gender, or disability status, regardless of discriminatory intent. AI systems that produce disparate outcomes are therefore legally actionable under existing law, not just subject to emerging AI-specific regulation.
Research published in Frontiers in Artificial Intelligence (2024) identified five primary types of algorithmic discrimination relevant to government systems: bias by algorithmic agent, feature selection bias, proxy discrimination (using facially neutral variables that correlate with protected characteristics), disparate impact, and targeted exclusion. All five have documented precedents in U.S. public sector deployments.
|
Bias Type |
Government Example |
Legal Exposure |
|
Feature selection bias |
Criminal risk scores using ZIP code |
Equal Protection, Title VI |
|
Proxy discrimination |
Benefits AI using disability-coded language patterns |
ADA, Rehabilitation Act |
|
Disparate impact |
Face recognition misidentification by skin tone |
Title VII, Title VI |
|
Algorithmic agent bias |
Biased training datasets in SSA medical review |
Due Process (5th/14th Amendment) |
What Good AI Governance Looks Like in Practice
The policy gap between aspiration and implementation is substantial. Identifying it is useful. Filling it requires specific institutional mechanisms, not just principles.
Risk Tiering for High-Stakes Decisions
The most defensible governance architecture applies differentiated oversight based on consequence. Systems that make or significantly influence rights-affecting decisions benefits eligibility, criminal risk assessment, housing placement require human review of all adverse outcomes, documented audit trails, and published error-rate data disaggregated by demographic group. Systems used for internal administrative efficiency scheduling, document routing, transcription require lighter oversight proportionate to their lower stakes.
OMB M-25-21 gestures at this tiering concept, but leaves agencies substantial discretion in how they define high-stakes contexts. A more durable framework would codify specific categories of government decision-making where AI use triggers mandatory transparency, independent audit, and appeals rights regardless of which administration is in office.
Mandatory Pre-Deployment Bias Auditing
Before any benefits-related AI system is deployed, agencies should be required to conduct a pre-deployment bias audit using test datasets that reflect the demographic composition of the likely affected population. This is not a novel idea it mirrors the concept of pilot testing that federal programs already apply to new benefit rules. What’s novel is applying it systematically to algorithmic systems rather than human-administered policies.
The Illinois General Assembly’s 2024 proposal to require AI disclosures in grant funding agreements represents one model: attaching governance requirements to funding mechanisms rather than attempting to regulate AI directly. At the federal level, a similar approach through grant conditions, contract clauses, and appropriations riders could extend governance requirements faster than comprehensive AI legislation.
Meaningful Algorithmic Explainability
A system is not accountable if those it affects cannot understand how it reached a decision. Meaningful explainability in the public sector context means the agency must be able to provide a plain-language explanation of why a specific applicant’s claim was denied or flagged one that the applicant can evaluate and challenge. Black-box models that cannot produce this output have no place in rights-affecting government decisions, regardless of their predictive accuracy.
The ISO/IEC 42001 Framework: A Governance Standard Government Agencies Can Use
ISO/IEC 42001:2023 the international standard for AI management systems provides a structured framework that government agencies can adopt to close the gap between policy aspiration and operational reality. Where OMB memos set requirements, ISO 42001 provides the implementation architecture for meeting them.
The standard’s AI Management System (AIMS) framework addresses six areas directly relevant to public sector deployment: organizational AI policy, risk assessment, data governance, human oversight mechanisms, incident response, and continual improvement. Each maps to a specific obligation in the current OMB AI governance framework.
|
ISO 42001 Requirement |
Corresponding OMB Obligation |
|
AI risk assessment process (Clause 6.1) |
M-25-22: pre-award bias analysis for rights-affecting systems |
|
AI policy and objectives (Clause 5.2) |
M-25-21: agency AI Strategy within 180 days |
|
Data governance controls (Clause 8.4) |
M-25-22: training data protection and ownership |
|
Human oversight mechanisms (Annex B.6) |
M-25-21: human review for high-stakes AI decisions |
|
Performance monitoring (Clause 9.1) |
M-25-22: post-deployment performance tracking |
|
Incident response (Clause 8.8) |
M-24-10: AI incident reporting requirements |
Critically, ISO 42001 certification provides an externally verifiable signal of governance maturity something OMB policy guidance alone cannot deliver. An agency or contractor certified against ISO 42001 has had its AI governance controls audited by an independent body. That external verification layer is what transforms internal policy commitments into accountable practice.
What Needs to Change: A Practical Reform Agenda
The architecture of federal AI governance is more developed than it was three years ago. The gaps that remain are not primarily conceptual the principles of fairness, transparency, and accountability are well understood. The gaps are institutional and legal.
- FOIA must explicitly cover AI decision logic. Agencies should be required to disclose the AI systems used in processing FOIA requests, including vendor identity, system architecture, and the basis for automated redaction recommendations. The trade secret exemption should be narrowed to prevent vendors from shielding government-deployed AI from public scrutiny.
- Benefits AI must carry due process obligations. Any AI system used to make or significantly influence eligibility determinations should be subject to: pre-deployment bias audit, plain-language explainability for adverse outcomes, independent appeals process, and annual published accuracy reports disaggregated by demographic group.
- Procurement must close the FAR gap. The FAR Council’s AI modernization process directed by Executive Order 14275 should produce contract clauses that make the M-25-22 bias disclosure obligations enforceable through standard contract remedies, not just policy guidance.
- Congress needs to pass durable legislation. OMB memoranda change with administrations. The Federal AI Governance and Transparency Act (HR 7532) would codify the most critical obligations procurement standards, data ownership, training-data safeguards into statute. Statutory requirements cannot be rescinded by executive memo.
- Agencies need institutional capacity, not just guidance. Policy without capacity is aspiration. The federal government needs designated AI governance officers with actual authority, cross-agency bias audit resources, and procurement teams trained in AI-specific risk assessment. The GSA’s planned acquisition guidance platform is a step in the right direction building the institutional infrastructure the guidance assumes already exists.
Frequently Asked Questions
What is the current legal framework for AI governance in the U.S. public sector?
The primary framework consists of OMB Memoranda M-25-21 and M-25-22 (April 2025), which govern federal AI use and procurement respectively. These replaced Biden-era memos and operate alongside the Federal Acquisition Regulation, relevant civil rights statutes (Title VI, ADA, Equal Protection Clause), and the pending Federal AI Governance and Transparency Act. No comprehensive federal AI statute has yet been enacted, meaning the regulatory landscape is largely executive-branch driven.
Are federal agencies required to disclose when AI is used to make decisions about me?
Not comprehensively. OMB M-25-21 requires agencies to publish AI use case inventories, and M-26-04 introduces Unbiased AI Principles for contractors. However, there is no universal legal requirement that agencies proactively notify individuals when AI systems have influenced specific decisions affecting them. Advocacy organizations have pushed for such a requirement, particularly in benefits administration and FOIA processing.
Can a government benefits decision made by AI be appealed?
Generally, yes existing administrative appeals processes apply regardless of whether the initial determination was made by a human or an algorithm. Courts have found that automated decisions without adequate explanation may violate procedural due process. The practical challenge is that appeals processes were designed for human-made decisions, and claimants may have difficulty challenging an AI output when the system’s logic is not disclosed.
How does FOIA apply to the AI systems that government agencies use?
FOIA requesters can ask for records about an agency’s AI systems, including procurement contracts, performance reports, and internal guidance documents. However, vendors can invoke trade secret exemptions to block disclosure of proprietary system architecture. Researchers argue this exemption is being used too broadly, and that AI decision-making logic used in government functions should not qualify for trade secret protection.
What is ISO/IEC 42001 and why is it relevant to government AI?
ISO/IEC 42001:2023 is the international standard for AI management systems. It provides a structured framework covering risk assessment, data governance, human oversight, and continual improvement that organizations can implement and certify against. For government agencies and their contractors, ISO 42001 certification provides an externally verified signal of governance maturity that OMB policy guidance alone cannot supply. GAICC offers training and certification programs specifically designed to build this expertise.
What is disparate impact in AI, and does it violate federal law?
Disparate impact occurs when an AI system produces outcomes that disproportionately disadvantage a protected group by race, gender, disability status, or national origin even without discriminatory intent. Under Title VI, the Fair Housing Act, the ADA, and the Equal Protection Clause, disparate impact by government systems can constitute illegal discrimination. The 2024 Frontiers in AI research identified five distinct pathways by which algorithmic discrimination occurs, all of which have precedents in U.S. public sector deployments.
Conclusion
The federal government is not a passive observer of the AI governance debate it is one of the largest AI deployers in the world, making consequential decisions about millions of Americans through algorithmic systems every day. The policy infrastructure to govern that deployment has improved substantially since 2023, but the gap between what the memos require and what agencies can actually deliver remains real and documented.
The most important single step any agency can take right now is to conduct a complete inventory of where AI touches rights-affecting decisions benefits, housing, disability, FOIA and apply differentiated governance to those high-stakes systems first. The framework to do that systematically exists. ISO/IEC 42001 provides the implementation architecture. OMB M-25-21 provides the policy mandate. What remains is institutional will and professional expertise.
If you are a government professional or contractor working to build that expertise, GAICC’s ISO/IEC 42001 certification programs are designed for exactly this context giving practitioners the structured knowledge to implement accountable, auditable AI governance in organizations where the stakes are public trust.
