Rescribe: Authoring and Automatically Editing Audio Descriptions

INTRODUCTION

• Fitting important content into limited video time requires trial and error; novices may over-describe less comprehension-relevant details, while extended descriptions disrupt source audio and increase playback time.Audio descriptions also require specialized scripting, recording, and editing skills.Source: INTRODUCTION · p. 1
• Rescribe lets authors draft and record desired content, then uses dynamic programming to choose placements and shortening options; authors can visualize and refine the results.It also considers lengthening the source track, such as by extending silence or looping music, to create extended-inline descriptions; one draft can produce extended, inline, and extended-inline versions.Source: INTRODUCTION · pp. 1–2
• 45% to 0%: Rescribe reduced the average share of descriptions overlapping dialogue or other descriptions, and users placed more descriptions within available gaps than with a timeline-based editor.In interviews, all 7 blind and visually impaired users preferred inline and extended-inline descriptions to extended descriptions.Source: INTRODUCTION · p. 2

BACKGROUND

• The work builds on prior research in audio-description creation, writing support, and constraint-based media editing.Source: BACKGROUND · p. 2

Describing videos

• Inline descriptions play alongside source audio, extended descriptions pause it, and extended-inline descriptions extend source material to accommodate more description without disrupting playback.Source: Figure 2 · p. 2
• Existing tools include text-to-speech timeline editors and spoken-description recording tools; authors using the latter still iteratively refine recordings to fit available space.Rescribe also surfaces transcripts to help estimate gaps, find vocabulary, and avoid repeating information already inferable from narration.Source: Describing videos · p. 2

Writing support tools

• Rescribe applies extractive summarization and sentence compression to audio descriptions, conservatively adapting these methods to preserve task-specific meaning.Source: Writing support tools · pp. 2–3
• For independently created videos without reliable scripts, Rescribe uses authors’ drafts, placement timing, and recording timing; it can shorten sentences and optionally lengthen the audio track.Unlike approaches that select script sentences and place them greedily, it favors subtle edits such as removing adjectives or short phrases.Source: Writing support tools · p. 3

Text and optimization-based media editing

• Prior transcript-based media tools support manual text manipulation for editing, but do not explore automatic text manipulation; Rescribe brings that approach to a new editing domain.Source: Text and optimization-based media… · p. 3

DESIGNING FOR AUDIO DESCRIPTIONS

• Rescribe’s design draws on five audio-description guideline sources and covers content, placement, style, and preparation.The interface includes a timeline, transcript, and description pane for navigating and authoring descriptions.Source: DESIGNING FOR AUDIO DESCRIPTIONS · p. 3, Figure 3 · p. 4
• The guidelines call for describing important visuals, avoiding dialogue overlap and spoilers, matching tone, and using clear, concise, objective language.Source: DESIGNING FOR AUDIO DESCRIPTIONS · p. 3
• Rescribe leaves objectivity, avoiding overlap with important non-dialog audio, and vocabulary support for future work; professionals identified limited timing and correct vocabulary as challenges.Professionals also emphasized preserving playback for workflow compatibility.Source: DESIGNING FOR AUDIO DESCRIPTIONS · p. 3

SYSTEM

• The timeline shows source speech, descriptions, and remaining gaps; authors can use speech and gap timelines to locate candidate placements and preview descriptions alongside source audio.Estimated description duration is based on 0.3 seconds per word.Source: SYSTEM · p. 4
• The time-aligned transcript displays source speech and descriptions, with gaps shown as spaces; clicking a word or gap navigates to its video location.Authors can consult source terminology and check what has already been described.Source: SYSTEM · p. 4
• The description pane supports scripting, recording, and time-aligned navigation; authors can lock text, timing, or description presence and revise or delete descriptions.The interface also displays a start-frame thumbnail and retains only the last recording for each description.Source: SYSTEM · p. 4, Figure 4 · p. 4
• After rendering, authors can select included words with a slider or toggles, rewrite text, and re-record; word-based and slider edits do not require rerecording.Included text appears black and omitted original text gray.Source: SYSTEM · pp. 4–5, Figure 5 · p. 5

ALGORITHMIC METHODS

• Rescribe relies on automatic methods to identify candidate locations, generate revisions, align selected revisions to audio, and produce the final audio track.Source: ALGORITHMIC METHODS · p. 5

Identifying description locations

• Description locations are identified from speech, music, and silence labels, with silence and qualifying music marked as extendable.Music qualifies when a segment is at least 30 seconds and beat detection finds at least 60 bpm.Source: Identifying description locations · p. 5

Generating candidate descriptions

• Candidate revisions are generated by parsing descriptions and dropping adjectives, prepositional phrases, or compound phrases that can typically be removed without grammatical revision.Candidates must preserve film-specific phrases, recurring video-specific phrases, quoted phrases, and phrases marked as on-screen text.Source: Generating candidate descriptions · p. 5
• Figure 6 ranks candidate descriptions by coherence, with lower scores indicating greater coherence.Source: Figure 6 · p. 5

• Each description’s composition cost combines candidate text quality, C(di), with placement quality, P(di).Source: Scoring audio description compositions · p. 5
• Candidate cost combines coherence, informativeness, and edit quality using weights 1, 500, and 10, respectively.Coherence uses GPT-2 log loss; informativeness rewards relevant nouns; edit quality penalizes required audio cuts, with an additional penalty when the final word is removed.Source: Scoring audio description compositions · p. 6
• The algorithm can skip a description by assigning cost 10,000 when the proposed duration is shorter than every generated candidate.Source: Scoring audio description compositions · p. 6
• Placement incurs an infinite penalty for overlap with source speech or another description, and a penalty of 10 within 0.3 seconds of an overlapping segment.Placements without a silence or music label also receive the penalty of 10.Source: Scoring audio description compositions · p. 6
• Extended-inline placement is penalized by the seconds added to an extendable music or silence gap; non-extendable gaps receive an infinite score.Source: Scoring audio description compositions · p. 6

• Dynamic programming finds the globally optimal composition by minimizing recursive cost across possible video placements and candidate text lengths.The table considers placements from 0 to the video length and candidate lengths up to the longest candidate.Source: Optimizing audio description… · p. 6
• The search is constrained to placements within 2 minutes and at least 1 shot of the original start time, which can skip otherwise possible placements farther away.The algorithm’s runtime is O(NML), where M is the video length.Source: Optimizing audio description… · pp. 6–7

Rendering audio descriptions

• Extended-inline descriptions lengthen music with loops and extend silence or ambient noise using sampled short segments, up to twice the original segment length.Music loops use identified transition points; ambient noise and silence use randomly sampled segments to avoid salient repetition artifacts.Source: Rendering audio descriptions · p. 7

EVALUATIONS: COMPARING AUDIO DESCRIPTIONS

• Seven blind or visually impaired participants compared extended, inline, and extended-inline versions made from the same draft descriptions for one 9-minute-36-second video.The inline version reduced description length by 52% while preserving 94% of segments; extended-inline reduced length by 76% while keeping 97% of segments.Source: EVALUATIONS: COMPARING AUDIO… · p. 7
• Figure 7 records agreement ratings for understanding, distracting edits, and future use after randomized playback of the three versions.Source: Figure 7 · p. 7, EVALUATIONS: COMPARING AUDIO… · p. 7
• The interview supports only initial reflections: participants heard one description variation for one video without a particular task, so long-term use and task performance remain unknown.Source: EVALUATIONS: COMPARING AUDIO… · p. 7

Findings

• All participants preferred inline and extended-inline descriptions over extended descriptions when placement avoided important audio.Most found extended descriptions distracting and flow-disrupting; some considered them acceptable for unavoidable or necessary content.Source: Findings · pp. 7–8
• Inline descriptions met users’ expectations and preserved music flow, though some heard chopped words or accepted small omissions rather than pausing the video.No users rated inline descriptions as distracting.Source: Findings · p. 8
• All participants said they would use extended-inline descriptions; 6/7 strongly disagreed that looping was distracting, and information-dense opening titles were a potential use case.Two participants wanted a choice between inline and extended-inline depending on setting or content interest.Source: Findings · p. 8

EVALUATIONS: CREATING AUDIO DESCRIPTIONS

• 80 (σ = 59) words with Final Cut Pro versus 96 (σ = 47) with Rescribe; users valued Final Cut’s fine-grained scrubbing and Rescribe’s gap navigation.Rescribe users also valued contextualizing descriptions with narration and budgeting time based on remaining gaps.Source: EVALUATIONS: CREATING AUDIO DESCRIPTIONS · p. 9
• Most users recorded the descriptions they scripted in both tools; Final Cut Pro omissions reflected time limits or decisions to avoid overlap, while Rescribe users mainly edited scripts.No Final Cut Pro users edited their scripts.Source: EVALUATIONS: CREATING AUDIO DESCRIPTIONS · p. 9
• 6 of 8 Final Cut Pro participants failed to fit descriptions into the gaps, and overlaps made source speech or descriptions difficult to understand.Some participants who successfully placed descriptions in Final Cut Pro preferred that composition.Source: EVALUATIONS: CREATING AUDIO DESCRIPTIONS · p. 9
• First-time describers often asked what content to describe; some would publish their preferred clip, while others wanted richer descriptions or more practice.One participant said they would want to make a description longer and richer before publishing.Source: EVALUATIONS: CREATING AUDIO DESCRIPTIONS · pp. 9–10

EVALUATIONS: INFORMAL EXPERT DEMO

• All three professionals expressed enthusiasm about testing Rescribe in their workflows and requested output to professional audio-editing tools for additional mixing.They saw Rescribe as a possible replacement for scripting software currently repurposed from captioning.Source: EVALUATIONS: INFORMAL EXPERT DEMO · p. 10
• Professionals were more excited about placement and shortening optimization than visualization of audio gaps.Two said the tool preserved authoring while automating a tedious iteration step.Source: EVALUATIONS: INFORMAL EXPERT DEMO · p. 10
• Professionals remained skeptical of vision-based scripting assistance but saw value in identifying already-described objects and suggesting relevant descriptions.Source: EVALUATIONS: INFORMAL EXPERT DEMO · p. 10

DISCUSSION AND FUTURE WORK

• Rescribe retargets extended descriptions to inline descriptions, improving ease and efficiency without overlapping content.Source: DISCUSSION AND FUTURE WORK · p. 10
• Manually set cost-function weights may be improved through learning, while existing description corpora largely cover movies and TV rather than independently produced content.Extended-inline descriptions also do not exist to the authors’ knowledge.Source: DISCUSSION AND FUTURE WORK · p. 10
• Text-based editing cannot use paraphrases requiring words absent from the recording, because synthesized speech is low quality when training examples are scarce.Current automated captioning and scene-description tools were too general or inaccurate to be useful.Source: DISCUSSION AND FUTURE WORK · p. 10
• The authors propose integrating audio description into video production, including suggesting edits or flagging visual content unexplained by the script.Source: DISCUSSION AND FUTURE WORK · p. 10
• Audio descriptions are available for around one third of newly produced traditional media but rarely exist for user-generated video platforms.The authors argue that hosting platforms should support audio description and provide education initiatives.Source: DISCUSSION AND FUTURE WORK · p. 10
• Few guidelines address common YouTube formats such as vlogs, how-tos, and reviews, and audience-preference research could inform better guidance for new describers.Source: DISCUSSION AND FUTURE WORK · p. 10

CONCLUSION

• Rescribe enables people to create narrated inline audio descriptions quickly for videos without audio descriptions.Source: CONCLUSION · p. 10