Source-linked AI summary
"A Second Set of Eyes": The Process and Challenges of Software Documentation Review
Avinash Bhat, Ian Arawjo, Disha Shrivastava, Jin L. C. Guo
TL;DR
Documentation quality depends on coordinated contributions from practitioners with complementary expertise, but organizations and tools inadequately support this work. This paper characterizes the review process and its challenges, then proposes a design space for improving documentation review.
Problem
Documentation quality is often low and documentation work receives limited priority, while existing workflows overlook the coordination needed to achieve quality.
Method
The paper characterizes the documentation review process and tools, identifies technical writers’ review challenges, and derives a design space for exploring process improvements.
Results
Documentation quality emerges through coordinated review involving multiple practitioners with complementary expertise, while existing workflows and tools inadequately support this process.
Takeaways & Limitations
The design space helps researchers and professionals diagnose process failures and systematically explore alternative mechanisms for documentation review.
Takeaways & Limitations
Claims about reviewer behavior reflect writers’ accounts, and recurring-stage analysis may underrepresent informal practices outside defined stage boundaries.
Abstract
from arXiv · showhide
Organizations assign documentation work to technical writers, yet the knowledge required to produce it is distributed across developers, managers, and other practitioners. Prior work has established quality criteria for judging "good" documentation, but it has not examined how practitioners bring that expertise to improve documentation quality or the challenges they face in doing so. Through semi-structured interviews with experienced technical writers ($n=31$) from different organizations, our work reveals the individual and collaborative effort required to maintain documentation quality. We identify five distinct stages of the documentation review process: self review, technical review, editorial review, play testing, and post-publication feedback. Each stage draws on practitioners with distinct expertise to address quality across content, presentation, and user experience. Our findings surface organizational and technical challenges writers face in recruiting expert reviewers, navigating development timelines, and contending with tools not specifically designed for documentation workflows. Our work positions documentation review as a crucial yet understudied site of collaborative work and opens new research and design directions for process improvement and tool support.
1 Introduction
Documentation quality matters for software adoption, maintenance, onboarding, and project success, yet prior research has focused more on defining quality than explaining how practitioners achieve it. This study examines that process through interviews with experienced technical writers and identifies organizational and tooling challenges in documentation review.
- High-quality documentation supports software adoption, lowers maintenance costs, facilitates onboarding, and prevents knowledge loss.
- Prior research has defined documentation quality through attributes and evaluation methods, including accuracy, completeness, usability, structure, content, and style.
- Existing studies show persistent problems such as insufficient content and obsolete information despite dedicated documentation resources.
- The rise of LLM-generated documentation adds challenges involving hallucinations and consistency, increasing the urgency of understanding quality practices.
- Semi-structured interviews with 31 experienced technical writers from varied organizations and industries examined documentation quality practices.
- The study characterizes documentation review, identifies process and tooling gaps, and proposes a design space for exploring improvement opportunities.
2 Related Work
Prior software-engineering research has examined collaboration and review mainly around code, while documentation review remains understudied. Existing tools and organizational practices leave technical writers coordinating distributed expertise through largely unsupported workarounds.
- Research on software-engineering collaboration shows that artifacts and formal processes cannot cover every coordination need in distributed and cross-disciplinary work.
- Articulation work aligns distributed contributions by locating expertise and directing it, creating substantial overhead for the person who performs it.
- Documentation review has not previously been studied as a collaboration context, and it poses distinct challenges because review is often a secondary task.
- Technical writers coordinate documentation review invisibly, while colleagues mainly see the requests produced by that coordination.
- Software-engineering review research and tooling center on code, whereas documentation review depends on exchanges with reviewers from other roles.
- Documentation practice receives low priority because productivity measures emphasize delivered software, leaving writers accountable for quality while reviewers have no obligation to help.
3 Research Design
The study used qualitative interviews with experienced technical writers, iterative analysis, peer debriefing, and participant validation to investigate documentation review practices and challenges. The authors explicitly describe methodological and sampling boundaries for interpreting the findings.
- Research approach: The researchers conducted semi-structured interviews with experienced technical writers and analyzed them using qualitative methods.
- Participants: Participants were recruited from Write The Docs Slack and the Technical Writer Forum on LinkedIn and screened for documentation-review experience.
- Participants: Technical writers were selected because they orchestrate review coordination, including locating reviewers and reconciling conflicting feedback.
- Analysis: The five review stages emerged within the first 15 interviews, while challenges stabilized after approximately 20 interviews.
- Data collection: Interviews lasted 45–70 minutes, were conducted remotely, and included screen recordings and participant demonstrations.
- Interview protocol: Interview questions evolved from broad exploration to targeted questions about review stages, participants, purposes, tools, and challenges.
- Analysis: The analysis combined transcript and screen-recording review, open coding, iterative refinement, and peer debriefing among all authors.
- Validation: Participant validation invited all 31 participants to assess the findings, with 13 responding.
4 RQ1: How to Achieve Documentation Quality
Practitioners evaluate documentation through perceived quality attributes because user goal achievement cannot be measured directly. The study organizes recurring quality concerns into Content Quality, Presentation Quality, and User Experience.
- Practitioners assess documentation using quality attributes they believe contribute to user success because they cannot directly measure users’ goal achievement.
- The study triangulates seven recurring quality concerns from participant discussions with quality attributes identified in prior software-engineering research.
- The quality attributes are organized into three categories: Content Quality, Presentation, and User Experience.
- Participant counts reported for practices and challenges describe the study sample rather than the relative importance of review stages.
4.1 What compels documentation review?
Participants described review as essential because user feedback rarely provides clear evidence of documentation quality, while individual reviewers have limited perspectives. Collaboration therefore helps expose blind spots, incorporate audience perspectives, and reconcile differing judgments.
- User feedback channels do not directly indicate whether documentation is successful or meets specific quality attributes.Analytics, votes, comments, and other signals require interpretation and may reflect product problems rather than documentation quality.
- Users may lack the expertise or motivation to evaluate technical accuracy, consistency, and stylistic requirements.
- Different reviewers apply criteria shaped by their professional backgrounds, preferences, and individual standards.
- Collaborating with experts reveals incorrect assumptions, identifies blind spots, and adds perspectives aligned with audience expectations.
4.2 Documentation Review Process
Documentation review is collaborative articulation work because writers must coordinate distributed expertise across five stages: self review, technical review, editorial review, play testing, and post-publication feedback. Organizations vary these stages according to release pressures, regulatory requirements, team size, and available reviewers.
- Documentation Review Process: Five review stages distribute quality work across self review, technical review, editorial review, play testing, and post-publication feedback.The first three occur most frequently after drafting, while play testing and post-publication feedback add user-oriented verification.
- Organizational Variation: Review processes are shorter in release-focused organizations and more formal in regulated industries, while team size affects access to editorial oversight and specialized reviewers.
- Self Review: Self review is primarily individual work in which technical writers evaluate their own documentation while independently exploring the software.
- Technical Review: Technical review uses developers and subject-matter experts to check technical accuracy, completeness, procedures, prerequisites, screenshots, and code samples.Reviewers may also surface implicit knowledge absent from development tickets or feature specifications.
- Editorial Review: Editorial review uses writers and other organizational stakeholders to assess language, structure, presentation, links, narrative flow, and user experience.Reviewers check whether pages are skimmable, follow writing conventions, and let users discover and apply information successfully.
- Play Testing: Play testing relies on fresh-eyed reviewers, often quality assurance teams, who follow documented procedures end to end from a user perspective.
- Post-Publication Feedback: Post-publication feedback from users and production-facing roles reveals environment-specific gaps, unexpected use cases, and findability problems that guide updates.Writers often interpret indirect signals such as support tickets and site analytics because direct access to users is limited.
4.3 Tools Used in Review Process
Review tooling spans the documentation lifecycle and distributes decision-making across individual and collaborative mechanisms. Participants used documentation-specific and adapted software-engineering tools for testing, style enforcement, feedback collection, and usage analysis, revealing opportunities for bespoke support.
- Tool Landscape: The tool landscape is organized by intervention timing and responsibility of decision making, while indicating whether tools support content, presentation, or user experience quality.
- Intervention Timing: Interventions may be proactive during drafting, corrective before publication, or reactive after publication through real-world usage feedback.Self review spans proactive and corrective timing as writers review drafts and incorporate feedback.
- Design Opportunities: Many tools were general software-engineering tools adapted for documentation rather than designed specifically for documentation workflows.
- Content Quality: Participants used APIs, test databases, staging environments, and automated documentation tests to validate procedures, code snippets, workflows, and API accuracy.
- Presentation Quality: Linting, style guides, and LLMs supported grammar, clarity, conciseness, voice, readability, terminology, and custom style enforcement.Participants reported that linting and style guidance made review easier and faster by focusing attention on conceptual clarity.
- User Experience Quality: Votes, comments, support tickets, heatmaps, and search-query analysis helped teams interpret user feedback, identify content gaps, and understand navigation behavior.
5 RQ2: Challenges During Review
Documentation review imposes articulation-work overhead that is often unrecognized as separate from producing documentation. This overhead becomes visible when coordination breaks down during reviewer recruitment, task assignment, and feedback reconciliation.
- Writers must recruit reviewers, direct relevant document sections to them, and reconcile distributed feedback into one coherent artifact.
5.1 Process Challenges
Documentation review is constrained by development timelines, weak reviewer availability, and gaps between writers’ and reviewers’ expertise. These conditions can produce rushed, superficial, or merely formal reviews that leave critical issues unresolved.
- Documentation often trails engineering sprints, leaving writers short windows to document incomplete or untested features before release.Inflexible launch, marketing, and regulatory deadlines continue even when documentation requests expand.
- Competing Priorities: Writers lack authority to require reviews, so they must persuade busy colleagues whose primary responsibilities take precedence.Review timing often follows reviewer convenience, and colleagues may perceive documentation review as additional work rather than value.
- Expertise and Communication Gaps: Expertise mismatches can make review a formality when senior technical writers know more about the product than assigned reviewers.
- Expertise and Communication Gaps: Knowledge and communication gaps lead reviewers to focus on minor details, approve unread documentation, or disengage from longer documents.Reviewers may also lack awareness of the writer’s research, the document’s purpose, or the audience’s information needs.
5.2 Tooling Challenges
Documentation teams rely on developer-oriented tools that exclude some reviewers, obscure the published user experience, and create maintenance burdens. LLM-assisted review adds context, reliability, and security concerns.
- Developer-focused review tools create steep learning curves for non-technical reviewers, prompting writers to use Google Docs or PDFs and manually port changes back.These workarounds keep reviews moving but add friction without changing the underlying code-oriented tooling.
- Textual diffs and flattened files prevent reviewers from effectively evaluating the visual layout and user experience of the final published website.Participants wanted commenting and suggestions directly on the production-like site.
- Version control systems cannot distinguish meaningful content changes from automatic formatting changes, forcing participants to track revisions manually.
- Tool maintenance requires ongoing updates, configuration, and troubleshooting that compete with writing and reviewing responsibilities.Ownership can create a single point of failure when effectiveness depends on one person’s continued involvement.
- LLM Generated Feedback: LLM review is constrained when technical limitations and security policies prevent writers from supplying complete documentation and supporting context.Long documents may need to be split and converted to plain text because tables, XML markup, and hyperlinks are difficult for LLMs to interpret.
- LLM Generated Feedback: Confidential information in systems such as Confluence and JIRA cannot be shared with providers whose retention policies organizations cannot audit or override.Without current internal context, LLMs rely on archived documentation or general internet content.
- LLM Generated Feedback: LLMs may declare documentation complete and accurate despite lacking the context needed to identify omissions.Participants described this people-pleasing behavior as a reason to distrust LLM feedback.
5.3 Existing Strategies
Writers keep documentation review operating by negotiating participation, adapting requests to reviewers, and borrowing institutional or social authority. These strategies reduce review demands and increase visibility, but substitute for formal mechanisms that would improve the process.
- Because writers cannot compel review, they negotiate, persuade, or coerce participation using their organizational skill and standing.These workarounds keep reviews running but leave no trace of shortcomings that could prompt organizational fixes.
- Strategic Communication: Writers distribute review requests across development and coordinate incremental cycles so reviewers can engage as code and documentation evolve.Early pull requests with specific questions help reviewers build familiarity with the content.
- Strategic Communication: Specific, actionable questions reduce reviewers’ cognitive burden by targeting the technical expertise needed for useful feedback.For complex concepts or misunderstandings, participants prefer live reviews, screen sharing, and meetings over asynchronous communication.
- Writers borrow authority from upper management to establish review boundaries, timelines, or participation expectations.
- Public discussions make both participation and nonparticipation visible to leadership and peers, turning review into a visible professional responsibility.
6 Discussion
Documentation review depends on writers coordinating distributed expertise, but organizations and tools provide limited support for that work. The paper therefore frames review as a collaborative design space requiring better visibility, shared standards, and support for diverse reviewers.
- Organizations Do Not Recognize Writers’ Coordination Work: Writers must reconcile reviewers’ judgments across quality dimensions without a shared standard for prioritizing conflicting feedback.Extending organizational style guides into clearer quality criteria could support independent assessment and arbitration.
- Organizations Do Not Recognize Writers’ Coordination Work: Review coordination falls to writers because organizations lack formal mechanisms and tracking for assigning and monitoring review responsibilities.This work becomes organizationally invisible, limiting the ability to measure its effort and cost.
- Existing Tools Overlook the Collaboration Aspect of Reviews: Existing documentation tools adapt development workflows that exclude non-technical reviewers and omit documentation-specific needs such as rendered-output evaluation and consensus tracking.The paper identifies these gaps within a design space organized by intervention timing and responsibility for decision making.
- Existing Tools Overlook the Collaboration Aspect of Reviews: Current LLM implementations produce unreliable review outputs and raise data-security concerns because writers cannot provide complete documentation or internal context.The paper treats these limitations as barriers that must be addressed before LLMs can support documentation review.
- Existing Tools Overlook the Collaboration Aspect of Reviews: The paper proposes reducing reviewers’ cognitive burden and providing task-matched interfaces that let non-technical practitioners review rendered documentation without learning developer tooling.These directions respond to superficial approvals and exclusion caused by raw-markdown and development-oriented workflows.
7 Conclusion
The paper shows that documentation quality emerges through coordinated review involving practitioners with complementary expertise, while existing workflows leave this expertise and coordination inadequately supported. It proposes a design space for diagnosing process failures and calls for purpose-built collaborative tools and human-centered uses of emerging technologies.
- Conclusion: Documentation quality emerges through coordinated review involving multiple practitioners with complementary expertise.Technical writers orchestrate the process and develop workarounds for its challenges.
- Conclusion: Existing workflows inadequately support documentation review because organizations’ evaluation systems make required expertise and coordination invisible.The paper links this invisibility to the need for better process and tool support.
- Conclusion: The proposed design space helps researchers and professionals diagnose process failures and explore alternative mechanisms for supporting documentation review.The paper calls for purpose-built collaborative tools and investigation of LLMs as augmentations to human expertise and collaboration.