"A Second Set of Eyes": The Process and Challenges of Software Documentation Review

1 Introduction

• Prior work defines documentation quality attributes and assessment methods [117], yet insufficient or obsolete content persists even with dedicated documentation resources.The paper identifies a gap between knowing what high-quality documentation looks like and understanding how practitioners achieve it.Source: §1 · p. 2
• Automatic documentation generation, especially with LLMs, adds hallucination and consistency challenges, increasing the urgency of understanding how documentation quality is achieved in practice [28] [60] [71] [4] [57].Without that understanding, the paper argues, adequate support for documentation processes is difficult to provide amid rapidly evolving tools.Source: §1 · p. 2
• Practitioners achieve documentation quality through a collaborative, tool-supported review process with five generally recurring stages, specialized expertise, and goals spanning content, presentation, and user experience.The stages are self review, technical review, editorial review, play testing, and post-publication feedback.Source: §1 · p. 2
• Writers face process challenges including poorly planned cycles, competing stakeholder priorities, and expertise gaps, alongside missing documentation-specific tools, tool maintenance burdens, and LLM unreliability and security risks.These challenges expose gaps between how documentation review operates and current organizational support.Source: §1 · pp. 2–3
• Semi-structured interviews with 31 experienced technical writers from varied organizations and industries examine how documentation quality is achieved and what challenges reviews present.The authors focus on writers’ specialized quality expertise, which prior documentation research has largely overlooked in favor of developers’ perspectives.Source: §1 · p. 2
• The study characterizes review coordination and tools, identifies review challenges and workarounds, and derives a design space based on intervention timing and responsibility for decision making.The design space is intended to help researchers and industry professionals explore alternatives and identify improvement opportunities.Source: §1 · p. 3

2.1 Collaboration in Software Engineering Teams

• Prior software engineering studies show that coordination needs exceed formal processes, expertise is unevenly distributed, and practitioners may disagree on whose standards apply; this paper studies documentation review as a new context for that work.Writers’ coordination is articulation work: it can be substantial and invisible to the colleagues who benefit from it.Source: §2.1 · pp. 3–4

2.2 Reviewing in Software Engineering

• Code review research finds that author–reviewer exchange, supplied rationale, author standing, and reviewer availability shape review; documentation review involves role-based reviewers and has not been examined under these conditions.Review-support tools have largely been developed for code, while writers rely on workarounds for documentation review.Source: §2.2 · p. 4

2.3 Documentation Quality and Practice

• Documentation tools mainly address accuracy, while other quality dimensions and the complete review process remain unsupported; writers are accountable for quality although reviewers have no obligation to help.Prior work also describes low documentation priority and limited credit for documentation work.Source: §2.3 · p. 4

3 Research Design

• Semi-structured interviews with experienced technical writers and qualitative analysis were used to investigate how practitioners establish and maintain documentation quality.The study design was approved by the authors’ university research ethics board.Source: §3 · p. 5

3.1 Participants and Recruitment

• Writers were recruited through professional communities and screened for review familiarity and substantial documentation experience; analysis indicated saturation on major themes by 31 interviews.Writers were selected because they coordinate review work, such as finding reviewers and reconciling conflicting feedback, which may be invisible to developers and managers. The five stages emerged within the first 15 interviews, and challenges stabilized around 20.Source: §3.1 · p. 5

3.2 Interview Protocol and Data Analysis

• Interviews were remote, screen-recorded, and guided by questions that evolved from broad exploration to targeted probes about review stages.Sessions lasted 45–70 minutes; participants could demonstrate practices by sharing screens. Once themes stabilized, targeted questions were used when participants raised relevant aspects, while open-ended questions remained in use.Source: §3.2 · p. 6
• Analysis combined transcript and screen-recording review, open coding, and author peer debriefing; reflexivity and discussion replaced inter-rater reliability calculations.The first author conducted all 31 interviews and led iterative analysis; code disagreements were resolved by returning to transcript examples and refining definitions. The codebook is in the supplementary material.Source: §3.2 · p. 6

3.3 Validation

• Of 13 validation respondents, 12 confirmed the analysis; among nine respondents whose quotes appeared, seven confirmed context and two raised minor phrasing concerns.Participants received a two-page results summary and the full draft, and could respond by questionnaire or email. P5 partially agreed, arguing that writer understanding was primary and collaborative review served as verification; the authors clarified phrasing concerns.Source: §3.3 · p. 7, Figure 2 · p. 7

3.4 Limitations

• Findings reflect experienced writers’ accounts, mainly from organizations with dedicated technical writing roles, and may miss other perspectives, informal workflows, and practices outside defined stages.Writers may not know how others experience the workflow or about workarounds beyond their involvement; focusing on recurring practices may also underrepresent informal review. Research-design trade-offs and author positionality are discussed in the supplementary material.Source: §3.4 · p. 7

4 RQ1: How to Achieve Documentation Quality

• Reviewers assess factual correctness and comprehensiveness, information structure and communication, and users’ discovery and interaction as three related dimensions of documentation quality.The study names these dimensions Content Quality, Presentation Quality, and User Experience Quality, respectively.Source: §4 · p. 8
• Because practitioners cannot directly measure whether users achieve their goals through documentation, they evaluate it using quality attributes they believe contribute to user success.The study notes that reliably assessing these attributes presents challenges.Source: §4 · p. 8

4.1.1 Users Do Not Provide Clear Indicators of Documentation Quality (17 participants).

• User channels—from analytics to votes and comments—yield feedback that requires interpretation, and users may not assess technical accuracy or care about stylistic consistency.Participants noted that frequently viewed content might signal a product problem rather than successful documentation.Source: §4.1.1 · pp. 8–9
• Reviewers’ differing expertise and standards can miss audience needs; a second perspective surfaces blind spots and helps reconcile disagreements about documentation scope and content.Participants described review discussions as a way for product managers and engineers to reach agreement about which features and details belong in customer-facing documentation.Source: §4.1.2 · p. 9

4.2.1 Overview.

• Review spans self, technical, and editorial review, followed by play testing and post-publication feedback; organizations adapt these stages to their needs.Release-focused organizations described shorter reviews, while regulated industries used formal approval workflows; smaller teams may lack editorial oversight, while larger teams may involve specialists.Source: §4.2.1 · pp. 9–10

4.2.2 Self Review (27 participants).

• Writers conduct self review largely alone, testing content against the software and reliable sources while using experience and style guides to check presentation.Testing varies by documentation type, such as verifying GUI steps in the interface or running API calls; writers may seek SME or developer input for complex topics.Source: §4.2.2 · p. 10

4.2.3 Technical Review (28 participants).

• Technical reviewers verify accuracy and completeness, surface implicit knowledge, and negotiate whether documentation should explain system features or prioritize users’ workflows.Developers test procedures and check prerequisites, instructions, screenshots, and code samples; review also guards against editorial changes removing technical details.Source: §4.2.3 · pp. 10–11

4.2.4 Editorial Review (28 participants).

• Editorial reviewers, usually fellow writers, assess language, structure, organizational style, and user experience, with product owners or managers contributing brand and legal considerations.They check links, narrative continuity, visual design, and whether content works for users with different technical backgrounds.Source: §4.2.4 · pp. 11–12

4.2.5 Play Testing (13 participants).

• Play testers use fresh eyes to follow documentation on production software, checking that instructions work and identifying skipped prerequisites or assumed knowledge.This stage is often informal and opportunistic, depends on reviewer bandwidth, and is reported mainly for tutorials.Source: §4.2.5 · p. 12

4.2.6 Post Publication Feedback (18 participants).

• Post-publication feedback, often relayed through support staff or indirect signals, reveals missing implementation steps, unexpected use cases, and user-facing findability problems.Writers use these reports to prioritize practical updates; API documentation generated from code can expose search and presentation limitations through user complaints.Source: §4.2.6 · pp. 12–13

4.3 Tools Used in Review Process

• The tool landscape maps intervention timing and decision-making responsibility, showing how self review spans drafting and feedback incorporation while collaborative decisions bring broader expertise and coordination overhead.The framework also highlights reliance on general software-engineering tools adapted for documentation, and opportunities for tools built specifically for documentation workflows.Source: §4.3 · p. 13, Figure 3 · p. 14

4.3.1 Proactive (25 participants).

• During proactive work, writers use API tests, collaborative diagram feedback, LLMs, editor extensions, and style tools to catch content and presentation problems while drafting.Participants also use documentation-site local builds and staging, plus CI/CD checks for broken links, images, and URLs; linting and style guides made reviews easier and faster by letting teams focus on conceptual clarity.Source: §4.3.1 · pp. 14–15
• During technical and editorial review, automated task tests, CI/CD checks, and pull-request style rules catch functional and presentation errors before publication, sometimes blocking merges until fixes are made.Participants also envisioned semantic comparisons across documentation sets, while LLM-based review agents were described as complementing rule-based checks.Source: §4.3.2 · p. 15

4.3.2 Corrective (14 participants).

• For post-publication feedback, teams combine votes, comments, support-ticket counts, heatmaps, and search queries to identify documentation issues and gaps in user navigation.Feedback arrives through repositories, on-page comments, forums, or Reddit; some teams also use retrieval-augmented search and chatbots to generate query-based summaries or tailored answers.Source: §4.3.3 · p. 16

5 RQ2: Challenges During Review

• Review requires overhead work such as recruiting reviewers, directing document sections, and reconciling feedback.The paper describes this as articulation work imposed by distributed review.Source: §5 · p. 16
• This overhead is not usually recognized as separate from documentation production and surfaces when breakdowns make it nameable.The passage says participants named it only when it broke down enough to become visible.Source: §5 · p. 16
• The reported challenges are mapped to the design space introduced in §4.3, with * marking concerns spanning an entire dimension.The passage specifies the mapping notation as (T, DM).Source: §5 · p. 16

5.1 Process Challenges

• Short, inflexible documentation windows leave writers documenting incomplete, untested features, while added requests further reduce time and content quality.Deadlines remain fixed around product launches, marketing campaigns, or regulatory requirements.Source: §5.1.1 · p. 16

5.1.1 Poor Planning of Documentation Cycles (21 participants).

• Writers have little authority to secure timely reviews because reviewers prioritize core responsibilities, leaving persuasion as their only option.Reviews may occur at the reviewer’s convenience, including at the last minute.Source: §5.1.2 · p. 16

5.1.2 Competing Priorities (17 participants).

• Reviewers’ competing priorities and perceptions of documentation as extra work make collaboration difficult and can lead to last-minute reviews.Writers also described interpersonal friction when colleagues view requests for more attention as creating work rather than improving quality.Source: §5.1.2 · p. 17

5.1.3 Expertise and Communication Gaps (19 participants).

• Expertise and communication gaps can make review a rubber stamp for senior writers, while reviewers may skim, overlook critical issues, or approve unread documentation.Reviewers may lack context about writers’ research, document purpose, or audience needs; language and cultural differences can prompt disagreements about wording, and long documents or frequent requests can contribute to disengagement.Source: §5.1.3 · p. 17

5.2.1 Lack of Dedicated Tooling for Documentation (21 participants).

• Developer-focused tools exclude less technical reviewers, so writers use Google Docs or PDFs and manually transfer feedback back to source files.Participants said Google Docs is common because its collaborative comments resemble GitHub reviews, while the underlying tools remain designed for code.Source: §5.2.1 · pp. 17–18
• Textual diffs and flattened files obscure the published layout, while version control cannot separate meaningful edits from automatic formatting changes.Reviewers wanted to comment directly on a production-like website; one participant tracked proposed edits by saving and renaming separate file copies.Source: §5.2.1 · p. 18

5.2.2 Tool Maintenance Overhead (16 participants).

• Maintaining review tools falls to individuals, competing with documentation work and creating a single point of failure.Some participants favored commercial solutions with support teams over custom implementations; LLM use also requires maintaining prompts and troubleshooting outputs as needs evolve.Source: §5.2.2 · p. 18
• LLM context limits and security policies prevent writers from supplying full documentation and supporting information for review.They split long documents into smaller sections and convert content to plain text because LLMs struggle with tables, XML, and hyperlinks, losing context that can make feedback misleading.Source: §5.2.3 · p. 18

5.2.3 Unreliability of LLM Generated Feedback (25 participants).

• Without confidential internal context, LLMs may rely on outdated or generic information and still affirm that documentation is complete and accurate.Participants distrusted this sycophantic feedback because an LLM cannot identify missing information it has not been given.Source: §5.2.3 · pp. 18–19

5.3 Existing Strategies

• Writers negotiated, persuaded, or coerced reviewers to keep the process running, substituting for a mechanism that required participation and leaving shortcomings unrecorded for organizational improvement.These strategies relied on writers’ skill and standing because they lacked authority to compel review.Source: §5.3 · p. 19

5.3.1 Strategic Communication (17 participants).

• Writers distributed review requests across development, asked specific questions, and used live discussion when complex concepts or misunderstandings required clarification.Early pull requests and incremental review cycles built reviewer familiarity before final approval; precise, actionable questions reduced reviewer burden, while calls, screen sharing, and meetings helped resolve confusion.Source: §5.3.1 · p. 19

5.3.2 Institutional and Social Pressure (12 participants).

• To secure participation, writers escalated to management for explicit timelines or expectations and framed review as mutually beneficial by emphasizing its business impact.Participants described these approaches when colleagues did not readily review documentation.Source: §5.3.2 · p. 20
• Writers made review participation visible by discussing it in public channels, where peers and leaders could see contributions and nonresponse, turning review into a visible professional responsibility.Because the review process did not record reviewer participation, writers created that indicator themselves by naming reviewers publicly.Source: §5.3.2 · p. 20

6.1 Practitioners Lack Shared Quality Standards and Incentives to Review

• Organizations should establish shared criteria for content and user-experience quality so reviewers can assess documentation independently and writers can arbitrate conflicting feedback.Style guides already offer a common standard for presentation, but participants reported no equivalent for content or user experience; superficial reviews and weak connection to reviewers’ primary work further limit useful feedback.Source: §6.1 · pp. 20–21

6.2 Organizations Do Not Recognize Writers’ Coordination Work

• Organizations should formally recognize writers as articulation hubs and track review coordination alongside documentation changes, making its effort visible and measurable.Because review coordination is untracked, writers’ effort remains invisible and its costs accumulate as scalar debt; persistent records should identify each task, owner, and status.Source: §6.2 · p. 21

6.3 Existing Tools Overlook the Collaboration Aspect of Reviews

• As users increasingly access documentation through LLMs, organizations may lose website analytics and comments that indicate how well documentation serves users, risking quality decline.Current LLM implementations also produce unreliable outputs and raise data-security concerns because writers cannot provide the full documentation or internal review context.Source: §6.3 · p. 22
• Documentation tools largely support individual quality checks, while collaborative review remains a major research gap despite involving negotiation, conflicting priorities, and interpersonal dynamics.Docs-as-code tools offer version control and engineering integration but can exclude non-technical reviewers and lack support for rendered-output evaluation or reviewer consensus; code review research offers collaboration tools, but documentation review lacks comparable empirical foundations.Source: §6.3 · p. 22, Figure 4 · p. 22

6.4 Designing Intelligent Tools to Support Collaboration in Reviews

• Collaborative review tools could reduce shallow engagement with role-specific context and questions, while rendered, task-matched workspaces could include non-technical reviewers without developer tooling.These tools should empower writers rather than enable monitoring or pressure to optimize metrics at the expense of useful work.Source: §6.4 · p. 23

• Documentation quality emerges through a coordinated review process involving multiple practitioners with complementary expertise.Source: §7 · p. 23
• Existing workflows and tools inadequately support review because required expertise and coordination are invisible to organizational evaluation systems, forcing writers to develop workarounds.Source: §7 · p. 23
• The authors propose a design space for diagnosing process failures and exploring alternatives, and call for research on purpose-built collaborative tools and technologies such as LLMs to augment human expertise and collaboration.Source: §7 · p. 23