The software development industry is currently navigating a crisis that mirrors other shifts in industrial history. For many years, the primary mechanism of accountability in technology was the peer review process. This is a “ceremony” in which a human engineer manually inspects another’s work. This “craftsmanship” model relies on the assumption that human output was bounded by human temporal and cognitive limits, allowing for 100% manual oversight. 

However, the emergence of AI coding agents has been disrupting this equilibrium. By generating code at a volume and velocity that exceeds the capacity of manual human review, these agents have created a “review bottleneck” that threatens the existing model of professional accountability. This poses a dilemma about whether human responsibility can persist in a world where agents are increasingly becoming creators and reviewers.

This phenomenon is not unprecedented, though. Over the last two centuries, sectors ranging from heavy manufacturing to global finance and the legal profession have faced similar challenges. These were periods where the throughput of production outpaced the existing mechanisms of human oversight. In each instance, the solution was not the abandonment of accountability, but its relocation.

In all cases, accountability shifted from the inspection of individual units of work to the governance of the systems that produce them. To understand how software engineering might resolve its current bottleneck, one can analyze how these previous industrial transformations redefined the relationship between human judgment and automated output.

From Artisan to Governor

A similar situation to the one now being faced by software engineers and software organizations can be traced back to the pre-industrial craftsmanship model. In medieval Europe, production was governed by guilds, unions of craftsmen who established strict rules for product and service quality. During this era, quality was not an external metric: it was an identity. A craftsman’s reputation was tied to the integrity of their work, and guilds ensured accountability through strict skill tests and 100% manual inspection of every item before sale. 

This model persisted into the early 19th century in the United States, where craftsmen sold locally and had a tremendous personal stake in meeting customer needs. If quality failed, the individual craftsman faced immediate and personal economic consequences.

The first disturbance to this model occurred during the Industrial Revolution. The factory system divided artisanal trades into specialized, repetitive tasks, stripping laborers of their autonomy and transforming them into factory workers. As machines accelerated production, the volume of output became too great for the traditional model of oversight based on craftsmanship. 

Factories initially attempted to maintain accountability through “reactive inspection”, by checking finished products at the end of the line and scrapping or reworking those with defects. However, this proved economically unsustainable as production speed increased.

By the early 20th century, Frederick Winslow Taylor’s “scientific management” further intensified this pressure. Taylor used time-and-motion studies to dismantle complex jobs into simple steps that workers performed without deviation. While this maximized efficiency, it stripped workers of power and led to a decline in quality, as workers no longer felt a sense of personal responsibility for the final product. 

This era mirrors the current “throughput-first” pressure in software companies, where engineers are asked to prioritize speed at the expense of deep review.

The Statistical Revolution and the Birth of Systemic Accountability

The manufacturing industry’s response to the failure of manual inspection was the birth of statistical quality control (SQC). In 1924, Walter Shewhart of Bell Telephone Laboratories introduced the control chart, a breakthrough that allowed organizations to predict and prevent defects before they occurred. Shewhart’s innovation marked the first time accountability was shifted from the outcome to the process.

W. Edwards Deming expanded this into a global doctrine in the 1950s, arguing that management, not the front-line worker, was primarily responsible for quality because management controlled the system. Deming’s “14 Points for Management” famously included the directive to “cease mass inspection” as a fundamental step toward quality, while his “Seven Deadly Diseases” identified broader structural barriers to improvement. This historical shift suggests that in the age of AI agents, accountability might move from the “individual PR” to the “generative pipeline.”

Era Primary Quality Mechanism Responsibility Locus Impact of Throughput
Craftsmanship (1000–1800) 100% Manual Inspection Individual Artisan Low (limited by human speed)
Industrial Revolution (1800–1920) End-of-line Audit/Scrap Shop Supervisor Moderate (bottlenecks emerge)
Statistical QC (1920–1960) Control Charts/Sampling Systems Engineers High (process monitoring replaces inspection)
Total Quality Management (1960–1990) Continuous Improvement Every Employee Very High (quality built into the system)
Quality 4.0 (2010–Present) Predictive AI/IoT Algorithm Governance Infinite (real-time automated correction)

In modern times, quality can leverage AI and Big Data to shift from monitoring to prediction. Systems are now continuous, connected, and proactive, allowing for a level of oversight that human reviewers could never achieve. This transformation shows that when speed exceeds human review capacity, the “human” element of accountability needs to shift: from the output/outcome to the design and auditing of the intelligent systems.

Zero Quality Control and the Poka-Yoke Paradigm

One of the most radical answers to the bottleneck problem came from Shigeo Shingo, the architect of the Toyota Production System. Shingo argued that traditional “routine sampling inspection” was inherently flawed because it only discovered defects rather than eliminating them. He proposed “Zero Quality Control” (ZQC), a system designed to achieve zero defects by catching errors at the source.

The foundation of ZQC is poka-yoke or “mistake-proofing”. These are devices or procedural constraints that make it physically or logically impossible to commit an error. For example, a car that won’t start unless the driver’s foot is on the brake is a poka-yoke.

In the context of software agents, this would imply a shift from individual human review towards “automated gates”. These would be formal verification, property-based testing, and security linters, acting as digital poka-yokes.

Shingo’s system also emphasized “successive checks,” where each worker checks the work of the previous step before starting their own task. This created a chain of accountability that achieved a 75% decrease in defects at leading car manufacturers within two years.

For software engineering, this suggests a “tiered review” model where agents check other agents, with humans serving as the final auditors of the checklists rather than the code itself.

Accountability at Microsecond Speeds

The transformation of financial markets also provides a good comparisson to the current software challenges. In the “old days” of the New York Stock Exchange, trades were executed by human market-makers on a physical floor. This manual system relied on handwritten tickets and personal interactions, which limited the number of trades and introduced human errors. Accountability was direct: if a broker misrecorded a trade, they were personally liable for the discrepancy.

The advent of electronic communication networks (ECNs) in the 1990s eliminated human brokers and improved execution speeds, paving the way for high-frequency trading (HFT). Today, HFT firms use supercomputers and sophisticated algorithms to execute orders in microseconds. At this speed, it’s biologically impossible for a human to review in real-time, creating a scenario where “computers make all the decisions”.

Regulatory Evolution and the Registration of Developers

When HFT began to dominate the markets, regulators initially struggled with the same question software companies are asking today: how can a human be held accountable for an automated decision? 

The response was a shift in the legal definition of responsibility. Regulators realized that if they could not hold humans accountable for individual trades, they must hold them accountable for the code that generates those trades.

FINRA Rule 1220(b)(3) was a landmark in this shift. It requires that associated people “primarily responsible for the design, development or significant modification of an algorithmic trading strategy” register as “Securities Traders”. This rule mandates that the engineers themselves must pass qualification exams and adhere to continuing education requirements.

This shift from “reviewing trades” to “auditing system design” resolved the bottleneck. Humans are held accountable not for what the computer did in a specific microsecond, but for the failure to design a system that could prevent that specific microsecond of chaos.

This move transformed the nature of work for software engineers in finance. They were no longer “just programmers”: they were licensed professionals with personal liability for the behavior of their algorithms. If an algorithm caused a market disruption, regulators could trace the “significant modification” back to a registered individual who failed their supervisory duties. 

This provides a potential blueprint for software engineering: if an AI agent produces a catastrophic bug, the accountability lies with the registered “Agent Lead” who designed the prompts and validation gates.

Systemic Integrity via Regulation SCI

The SEC further formalized this systemic accountability through Regulation Systems Compliance and Integrity (Reg SCI). This regulation requires firms that maintain market infrastructure to have comprehensive policies and procedures for their “market-impacting” technologies. Firms must perform annual audits, conduct “capacity stress tests,” and maintain detailed “change management records”.

Under Reg SCI, accountability is ensured by:

  1. Objective Personnel Reviews: Annual audits of system integrity by people independent of the development team.
  2. Incident Reporting: Mandated immediate notification to the SEC within 24 hours of any “SCI event” (system disruption or compliance issue).
  3. Corrective Action: A legal requirement to mitigate harm immediately upon the discovery of an automated failure.
Regulatory Requirement Accountability Mechanism Relevant People
FINRA 1220 Registration Professional Certification/Testing Lead Developers/Supervisors
SEC Regulation SCI Systemic Policy & Procedure Audits CTOs/Compliance Officers
SEC Rule 10b-5 Fraud/Manipulation Liability The Entire Firm
OATS Audit Trails Granular Traceability of Every Order The Firm & Regulators

This shift from “reviewing trades” to “auditing system design” resolved the bottleneck. Humans are held accountable not for what the computer did in a specific microsecond, but for the failure to design a system that could prevent that specific microsecond of chaos.

Information Overload

The legal profession has encountered a similar bottleneck in the field of e-discovery. In modern litigation, the volume of electronically stored information (ESI) is so vast that it is impossible for lawyers to manually review every document. The traditional manual review process was once the standard of accountability, but it became the primary cause of increasing costs and delays in the court system.

To solve this, the legal industry adopted Technology-Assisted Review (TAR). TAR uses machine learning to identify relevant documents based on an initial “seed set” of human-reviewed examples.

This raised a profound ethical question: if an attorney certifies a discovery response as “complete and correct” without having read 90% of the documents, have they violated their ethical duty?

The “Reasonable Inquiry” Standard

The judiciary resolved this by redefining the standard of accountability. Under Federal Rule of Civil Procedure 26(g), an attorney must verify a discovery response after a “reasonable inquiry”. The courts have ruled that “reasonable inquiry” does not mean “perfection” or “manual review of everything”. Instead, it means the implementation of a statistically validated process.

In the landmark case Da Silva Moore v. Publicis Groupe, the court officially endorsed predictive coding as a valid means of meeting discovery obligations. Accountability was maintained by shifting the lawyer’s duty from “reading” to “validating”.

Validation Metric Definition Judicial Benchmark
Recall The fraction of all relevant documents actually retrieved. Generally 70% to 80%+
Precision The fraction of retrieved documents that are actually relevant. Varies by case proportionality
Elusion The percentage of relevant documents missed in the “null set”. Used to confirm the stopping point

The attorney’s accountability now hinges on their ability to defend the methodology of the TAR process. If a lawyer can show they used an iterative training process, applied statistically sound sampling, and achieved a “reasonable and proportional” recall rate, they are held to have fulfilled their professional duty.

This might apply to the software review bottleneck: accountability could shift from “reading every line of code” to “certifying the recall rate of the automated test suite.”

The Ethical Duty of Technological Competence

The legal profession also introduced a new ethical mandate: the “Duty of Technological Competence” (ABA Model Rule 1.1, Comment 8). This requires attorneys to stay abreast of the benefits and risks associated with the technology they use. They cannot “blindly approve” the output of an AI tool.

If a lawyer lacks the expertise to understand the predictive coding algorithm, they are required to “associate with a non-lawyer technical expert” who can provide that guidance.

This transformation suggests that for software engineers, accountability will require a higher level of competence in evaluating AI agents. An engineer who simply clicks “merge” on an agent’s PR without understanding the agent’s potential for hallucination or “model drift” would be considered incompetent, much like a lawyer who blindly relies on a “black-box” discovery too.

Radiology and Medical AI

In medicine, particularly radiology, AI is being deployed to handle a massive increase in imaging volume. An AI algorithm can perform pattern recognition faster than a human, as it “never tires or sleeps”. However, the net number of mistakes can still increase even if the error rate decreases, simply because the volume of scans is so much higher.

The medical industry has maintained a “human-in-the-loop” model to manage this. Under the current malpractice law, the physician remains personally accountable for the diagnosis regardless of whether AI was used. There is no “shared responsibility” with the AI software developer under current federal statutes.

The Liability Costs of Efficiency

The medical industry has discovered that the workflow choice dramatically impacts liability. Research shows that if a radiologist reads a scan after seeing the AI’s suggestions, they are more susceptible to “automation bias,” and jurors are significantly more likely to side with a plaintiff if an error occurs. In contrast, performing an independent read before consulting the AI increases the total read time but decreases legal liability.

Workflow Model Read Time Liability Risk Accuracy
AI Only (Autonomous) Ultra-Low Extreme (Legally Unproven) Varies (Risk of Data Drift)
Human-First (Double Read) High Low Highest
AI-First (Single Read) Low High Moderate (Bias Risk)

This creates a narrative that software engineering must consider. If an engineer merges code after an agent has suggested it, their liability for a bug may be perceived as higher because they are anchored to the agent’s work rather than thinking independently.

Accountability in medicine is maintained by treating AI as a “confirmatory tool” rather than a “primary decision-maker”. The radiologist remains the “learned intermediary” who must apply the standard of care for their field, regardless of the algorithm’s output.

The FDA Predetermined Change Control Plan (PCCP)

The most forward-looking model for software accountability might come from the FDA’s “Predetermined Change Control Plan” (PCCP). Traditionally, any significant update to a medical device’s software required a new regulatory submission. But AI/ML algorithms learn and change constantly. The PCCP allows manufacturers to pre-define planned modifications and the protocols for validating them.

If the FDA authorizes a PCCP, the manufacturer can implement changes without a new marketing submission, provided they stay within the “bounds” of the pre-approved plan. The PCCP requires: Description of Modifications: Defining the scope of future changes. Modification Protocol: Detailing the methodology for development, validation, and implementation, including bias mitigation and failure handling. Impact Assessment: A risk-benefit analysis of the proposed changes.

This lifecycle-based oversight is a candidate for the software industry. Instead of reviewing every PR, a company could define an “Engineering PCCP” for its AI agent. As long as the agent’s code follows the pre-approved protocol and passes the impact assessment, the need for manual human review of every line is removed, while accountability remains with the leaders who signed off on the PCCP.

The Transformation of Work

The transition to high-throughput systems across these industries has fundamentally changed how people work, and a common concern is that automation leads to “deskilling”. If the human role collapses into “fast approval,” junior workers may never develop the foundational competence needed to handle complex edge cases.

However, economic research from MIT suggests that automation can actually increase the value of labor by removing the “easiest tasks”. For instance, even as computers took over tasks for bookkeepers, their real hourly wages rose by 40% because the remaining work demanded more expertise.

Role Task Automated Task Augmented Economic Outcome
Bookkeeper Calculation/Entry Financial Analysis Wages Up 40%
Taxi Driver Local Navigation Passenger Matching Wages Down 13%
Proofreader Spell-Checking Stylistic Coaching Wages Up (Highly Specialized)
Lawyer Doc Identification Strategy & Synthesis High Fees for AI-Governance

The lesson for software engineering is that accountability for syntax and standard libraries is being automated away. The human engineer’s frontier is shifting toward architectural oversight, security modeling, and the validation of intent. The bottleneck is not a lack of reviewers, but a lack of reviewers who have upshifted their expertise to the level required to govern an agentic workforce.

Conclusion

As high-throughput automation reshapes the industrial landscape, the solution hasn’t been to work faster, but to work higher. Across diverse sectors, the resolution to throughput-induced bottlenecks consistently lies in shifting accountability from output inspection to system governance.

Manufacturing solved this via Poka-Yoke pipelines, where automated gates prevent errors at the source rather than catching them later. Similarly, the financial and legal industries managed the flood of automated output by replacing manual, item-by-item review with rigorous validation of the methodology itself - certifying the “recall” of discovery processes or the “system integrity” of trading algorithms. Medicine offers a parallel through “Predetermined Change Control Plans,” which validate the protocols governing AI evolution rather than individual diagnoses.

In all these cases, humans remain accountable by accepting a new role: System Governor. They are no longer responsible for the individual act of production, but for the integrity of the system that produces it.

Maybe these are paths for moving forward in software engineering.