Source-linked AI summary
From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI
Margaret-Anne Storey
TL;DR
Generative AI accelerates software development while potentially shifting risk from technical debt toward cognitive and intent debt. The paper proposes a Triple Debt Model spanning code, people, and externalized knowledge, and discusses practices for preserving understanding and intent. It concludes that healthy AI-assisted software requires deliberate investment in both, while debates remain about documentation and AI’s role in creating it.
Problem
AI can generate code faster than developers understand it, creating a need to account for erosion of shared understanding and missing externalized rationale alongside technical debt.
Method
The paper proposes a Triple Debt Model covering technical debt in code, cognitive debt in shared understanding, and intent debt in externalized goals, constraints, and rationale.
Results
The paper identifies cognitive and intent debt as increasingly important risks in AI-assisted development and discusses practices for diagnosing and mitigating them.
Takeaways & Limitations
Teams should invest deliberately in shared understanding and intent artifacts alongside code quality when developing software with AI.
Takeaways & Limitations
Whether intent should be documented and whether AI should autogenerate such documentation remain unresolved practitioner debates.
Abstract
from arXiv · showhide
Generative AI is accelerating software development, but may quietly shift where the most significant risks lie. As AI generates code faster than teams can understand it, two under appreciated forms of debt accumulate: cognitive debt, the erosion of shared understanding across a team, and intent debt, the absence of externalized rationale that developers and AI agents need to work safely with code. This article proposes a Triple Debt Model for reasoning about software health, built around three interacting debt types: technical debt in code, cognitive debt in people, and intent debt in externalized knowledge. Cognitive debt is a team-level, project-level property reflecting the erosion of shared understanding across a software system over time, leading to increasingly inadequate shared mental models for reasoning about and safely changing the system. Intent debt refers to the absence or erosion of explicit rationale, goals, and constraints that guide how humans and agents evolve the system. We discuss how generative AI changes the relative importance of these debt types, how each can be diagnosed and mitigated, and surface points of debate for practitioners.
What is a Software System, Really?
A software system spans goals and intent, executable code and structure, and shared understanding. System health depends on alignment across all three layers, not code quality alone.
- Goals and intent include the requirements, constraints, and objectives captured in specifications, tests, and documentation.
- Code and structure implement system intent through source code, architecture, dependencies, and deployment infrastructure.
- Shared understanding consists of team members’ mental models of how the system works and can be reasoned about.
- Safe change and coordination require sufficient shared understanding across the team, rather than complete knowledge by one individual.
- System health depends on alignment among all three layers; unclear intent can cause purpose drift, while inadequate understanding makes safe change difficult.
Technical Debt: The Well-Known Layer
Technical debt is the long-term cost of code-quality shortcuts and constrained architectural decisions. Its visibility and established practices make it comparatively manageable, while generative AI increasingly automates remediation.
- Technical debt arises when short-term delivery takes priority over long-term code quality, producing messy code and constrained future evolution.
- Technical debt accrues interest because postponing remediation makes future system changes more costly.
- Technical debt is comparatively visible and manageable through practices such as test-driven development, refactoring, and code review.
- Generative AI increasingly supports technical-debt reduction through automated refactoring, code-smell detection, and test generation.
Cognitive Debt: The Invisible Layer
Cognitive debt is the team-level erosion of shared understanding that makes systems harder to reason about and change safely. AI-assisted development can accelerate this accumulation when developers accept generated code without building comparable mental models.
- Cognitive Debt: The Invisible Layer: Cognitive debt is a team- and project-level accumulation of gaps in shared understanding over time.
- Cognitive Debt: The Invisible Layer: The paper distinguishes its usage from individual neural-engagement and comprehension-debt accounts by emphasizing team-level, longitudinal erosion.
- Cognitive Debt: The Invisible Layer: Cognitive debt is not new, but AI-assisted development may increase the rate of accumulation and make the gap harder to detect.
- Cognitive Debt: The Invisible Layer: AI-generated code can be accepted without developers building the mental models needed to understand system behavior and reason about change.
- Cognitive Debt: The Invisible Layer: Cognitive surrender adopts AI outputs with minimal scrutiny, potentially inflating confidence even when the AI is wrong.
- Cognitive Debt: The Invisible Layer: Teams may deliberately trade understanding for delivery speed, accumulating cognitive debt while pursuing rapid feedback.
- Cognitive Debt: The Invisible Layer: Signals include resistance to change, unexpected results, slow onboarding, weakened transactive memory, and a low bus factor.
- Cognitive Debt: The Invisible Layer: Mitigation combines human review and pair programming with walkthroughs, retrospectives, deliberate communication, and reimplementation.
Intent Debt: The Forgotten Layer
Intent debt is the absence or erosion of explicit goals, constraints, rationale, and specifications needed to guide system evolution. The paper proposes living intent artifacts while noting that AI cannot replace human judgment in deciding what best serves users.
- Intent Debt: The Forgotten Layer: Intent debt accumulates when goals, constraints, specifications, and rationale are unclear, poorly articulated, or absent from consultable artifacts.
- Intent Debt: The Forgotten Layer: Practices such as specifications, tests, domain knowledge, and design rationale may become critical again as AI generates more code.
- Intent Debt: The Forgotten Layer: Requirements documents, decision records, plans, tests, and specifications externalize system intent for both teams and AI agents.
- Intent Debt: The Forgotten Layer: Intent is best captured when key decisions are made because recovering it later can be difficult or impossible.
- Intent Debt: The Forgotten Layer: Recognizable symptoms include behavior drift, AI agents needing extensive clarification, and loss of articulated non-functional constraints.
- Intent Debt: The Forgotten Layer: Intent-first workflows use executable specifications and tests, decision and rationale records, and context artifacts for AI-assisted development.
- Intent Debt: The Forgotten Layer: These approaches cannot replace human decision-making and creative judgment about which software solution best serves users.
Three Layers of Software System Health
Software system health depends on three interacting layers: code, shared understanding, and externalized intent. Their erosion limits changeability, safe reasoning, and continued alignment with what the team meant to build.
- Three Debt Types: Technical debt lives in code and accumulates when implementation decisions compromise future changeability.
- Three Debt Types: Cognitive debt lives in people and grows when shared understanding erodes faster than it is replenished.
- Three Debt Types: Intent debt lives in artifacts and arises when goals and constraints are poorly captured or maintained.
- Interactions: The three debts reinforce one another: unclear intent impairs understanding, weak understanding encourages poor implementation decisions, and messy code further reduces understanding.
- Interactions: Managing software health requires proactive attention to all three layers rather than only the easiest one to measure.
How AI Is Shifting the Balance
Generative AI may reduce technical debt through automation while increasing the relative importance of cognitive and intent debt. It produces code faster than teams can understand it and can weaken the feedback that previously supported understanding and intent capture.
- Changing Debt Dynamics: AI-assisted refactoring, code review, and test generation may reduce technical debt accumulation and ease legacy-system improvement.
- Changing Debt Dynamics: AI can partially mitigate cognitive and intent debt, but becomes a risk multiplier when humans surrender cognition and fail to capture intent.
- Changing Debt Dynamics: AI generates code faster than teams can build the understanding required to change it safely.
- Changing Debt Dynamics: Underspecified prompts, plausible-looking outputs, and reduced implementation friction can produce results that miss system needs or intent.
Implications for Practice
The paper recommends treating understanding and intent as explicit parts of AI-assisted development. Teams should build genuine understanding, capture intent early, and monitor all three debt dimensions together.
- Treat Understanding as a Deliverable: Teams should treat shared understanding as a first-class deliverable supported by walkthroughs, retrospectives, knowledge transfer, and suitable tools.
- Intent-First Workflows: Intent-first workflows capture requirements, constraints, decisions, plans, and acceptance criteria before AI-assisted development proceeds.
- Resist Automation of Understanding: Teams should resist automating understanding because generated explanations can create its appearance without building accurate mental models.
- Monitor the Three Layers in Tandem: Teams should monitor technical, cognitive, and intent debt together using measures such as onboarding time, knowledge concentration, requirements coverage, and intent-behavior audits.
- Monitor the Three Layers in Tandem: Software systems depend on intent in artifacts, behavior in code, and understanding distributed across teams and artifacts.
Points for Debate
The paper identifies unresolved debate over whether intent should be documented and how AI should support that work. It also leaves uncertain how much cognitive debt is acceptable as development becomes more automated.
- Should We Document Intent?: Practitioners disagree about whether capturing intent is worth the effort, and the debate over documenting intent remains unsettled.
- Can AI Help Make Implicit Knowledge Explicit?: Some propose that AI can surface implicit knowledge, while others value documentation primarily for the human understanding created during its production.
- Can AI Help Make Implicit Knowledge Explicit?: Whether AI should autogenerate documentation remains an open question.
- Debt as a Risk or Strategy: The acceptable level of cognitive debt remains uncertain because understanding may be distributed across people and artifacts and may not require implementation-level detail.
Conclusion
Healthy software in the AI-assisted era depends on preserving alignment among intent, code, and shared understanding. Teams should manage understanding with the same care and urgency as code quality.
- Conclusion: AI-assisted development may shift risk from technical debt toward cognitive and intent debt.The paper frames this as a hidden reversal requiring tools and processes to support paying down both forms.
- Conclusion: Software teams that navigate AI-assisted development successfully will invest deliberately in understanding and intent, not only code quality.
- Conclusion: Maintaining software health requires alignment among intent, code, and shared understanding.