PaperlayerView exact revision on arXiv

Source-linked AI summary

No Pixel Left Behind: Filling Gaps in Anime Colorization

Masahiro Kono, Akinobu Maejima, Yuki Koyama, Yotam Sechayk, Takeo Igarashi

arXiv:2609.00800v1cs.HCcs.CVcs.GR

TL;DR

Professional anime colorization remains largely manual, and small unpainted gaps are difficult and costly to detect and fill. The paper introduces GapFill, which combines workflow-compatible gap detection with contextual deep-learning color suggestions. In a study with 13 professional colorists, GapFill improved performance and usability, while findings highlighted the roles of contextual ambiguity and trust in AI assistance.

  • Problem

    Small unpainted enclosed gaps in professional anime colorization are difficult to detect and fill, yet existing tools provide inadequate specialized support.

  • Method

    GapFill integrates automatic gap detection, interactive inspection and correction, one-click or batch filling, and contextual color prediction based on neighboring flat-color regions.

  • Results

    GapFill significantly improved performance, particularly in gap-detection and filling tasks, in a comparison with conventional tools involving 13 professional colorists.

  • Takeaways & Limitations

    Usability was not determined by prediction accuracy alone; contextual color ambiguity and users’ trust shaped how GapFill complemented existing tools.

Abstract

from arXiv · show

Animation production workflows often involve digital colorization of line art, where small unpainted regions ("gaps") frequently occur and remain an underexplored challenge. We conducted a formative study in Japanese animation (anime) pipelines and found that while the paint bucket tool is widely used for base coloring, tiny enclosed areas are frequently overlooked, resulting in time-consuming manual detection and filling. We introduce GapFill, a tool grounded in professional practices that reduces the effort of gap detection, zooming, and color selection. Our deep-learning method suggests appropriate fill colors by referencing surrounding regions, leveraging the flat-color nature of anime-style images. In a user study with 13 professional colorists, our system improved performance and usability in gap-filling tasks over conventional methods. The study also suggested that prediction accuracy alone is not the primary factor for usability, that appropriate colors can be contextually ambiguous, and that GapFill can complement existing tools depending on users' trust in new AI-powered assistance.

1 Introduction

Anime colorization remains largely manual, and professional workflows expose a costly, underexplored problem: tiny unpainted gaps are difficult to detect and fill. GapFill combines automatic detection, contextual color suggestions, and workflow-compatible interaction, with evaluation showing improved gap-filling performance and usability.

  • Digital anime colorization fills flat colors into hand-drawn line-art regions for every frame and remains largely manual despite digital production tools.The workflow transitioned from cel painting to software such as Clip Studio Paint.
  • Professional colorists use paint-bucket filling on segmented regions while referencing model sheets, but small enclosed gaps from line intersections or minor regions are often overlooked.Gaps commonly occur at hair tips and between fingers.
  • A single unpainted pixel can require a costly retake, making gap detection and filling a time-consuming production burden.
  • GapFill detects gaps with circular highlights, magnifies them on hover, supports color correction by dragging, and enables batch or one-click filling.These interactions target manual detection, zooming, and color-selection effort while preserving user control over AI assistance.
  • Its deep-learning method predicts plausible gap colors by identifying neighboring regions likely to share the target color, exploiting anime’s flat and discrete color structure.
  • A study with 13 professional colorists found significant improvements, particularly for detecting and filling gaps, while indicating that usability depends on more than prediction accuracy and can depend on trust.The evaluation compared GapFill with conventional tools across full colorization and final checking tasks.

2 Related Work

Related work spans creativity-support tools, automatic line-art colorization, AI collaboration, and broader anime-production research. However, existing colorization methods often struggle with small intricate regions, while anime’s flat discrete colors differ from the continuous tones targeted by many inpainting methods.

  • Creativity-support tools include methods for imprecise drawings, vector-sketch region inference, and professional flat coloring.
  • AI-collaboration research emphasizes creator trust, acceptability, explainability, controllability, cognitive load, and creative efficacy.
  • Automatic line-art colorization has used color propagation, cross-frame region matching, graph correspondence, CNN-based colorization, and limited-example learning.
  • Existing methods often struggle with small or intricate regions in production settings, and specialized small-region work such as pupil filling remains limited in scope.
  • Anime-production research combines cultural, industrial, technical, and HCI perspectives, including tools for storyboarding, pre-production, rendering, super-resolution, and in-between generation.
  • Figure 3 depicts a workflow that prepares guided line art, applies model-sheet colors on separated layers, and produces the fully colored image.

A Formative Study

The formative study examined professional anime colorization workflows and identified small unpainted gaps as a frequent, time-consuming quality problem. It also informed GapFill’s workflow-integrated design, including automatic detection, magnification, and context-based color assistance.

  • A Formative Study: Professional anime colorization applies flat colors to clean binary line art within the broader shiage finishing process, under strict specifications because errors can trigger retakes.Colorists work in CSP using tools including Paint Bucket and brushes, with colors guided by model sheets and studio-specific conventions.
  • A Formative Study: The two-stage formative study combined a questionnaire of 20 professional colorists with interviews of four experienced professionals and workflow screen recording.The questionnaire covered experience, tools, and workflow challenges, while interviews deepened interpretation of those findings.
  • 3.1.3 A Practical Challenge in Colorization: Small Unpainted Gaps.: Small unpainted enclosed regions, or gaps, were the most frequently cited coloring issue, mentioned by 11 of 20 participants and often occurring near hair tips, fingers, and line intersections.These regions can be visually difficult to detect and may result from unintended line intersections or isolated minor areas.
  • 3.1.3 A Practical Challenge in Colorization: Small Unpainted Gaps.: Gap handling remained manual: professionals used Black Light, Leftover Pen, Enclose and Fill, or Paint Bucket methods but still had to detect, zoom, select, and fill gaps themselves.Black Light exposes gaps as bright dots, while the other tools fill along strokes or within enclosed areas.
  • 3.1.3 A Practical Challenge in Colorization: Small Unpainted Gaps.: Gap inspection could take several minutes per complex frame, and 60% agreed that addressing unpainted gaps was time-consuming (M= 5.0, SD= 1.8).Professionals differed between batch processing at the end and fixing gaps immediately, motivating workflow-consistent support.
  • 3.1.3 A Practical Challenge in Colorization: Small Unpainted Gaps.: 65% reported often addressing unpainted gaps, while 85% considered selecting their appropriate color important for completing an animation.Participants linked these tiny regions to visible transparency and blemishes in the final output.
  • 3.1.3 A Practical Challenge in Colorization: Small Unpainted Gaps.: Color decisions primarily referenced local context: 90% consulted surrounding colors, alongside model sheets and adjacent frames when available.Artists relied on surrounding elements, tones, subject knowledge, and intuition when the appropriate color was ambiguous.
  • 3.2 Design Principles for GapFill: GapFill was designed around these needs with on-demand gap detection, circular highlights, suggested colors, pop-up magnification, correction, and sweep-based or apply-all filling.The design aimed to reduce repetitive detection, zooming, and color-selection effort while preserving compatibility with existing practices and user control.

4 Design and Implementation of GapFill

GapFill combines automatic gap detection, contextual color prediction, magnified inspection, manual correction, and batch application within a professional painting interface. Its prediction model infers likely fill colors from surrounding flat-color regions using localized masks and region correspondence.

  • 4.1 User Interface: GapFill detects enclosed unpainted regions and highlights them automatically, reducing the need to locate gaps manually.Detection uses enclosed transparent regions below a user-adjustable pixel threshold, identified with grid-based traversal such as BFS.
  • 4.1.3 Hover-Activated Pop-up Magnification for Quick Inspection.: Hovering over a highlighted gap opens a fixed 5× magnified view for inspecting the proposed fill without zooming the canvas.The magnified view is centered on the detected gap and indicates its position with a translucent marker.
  • 4.1.4 In-Circle Color-pick for Correcting AI Suggested Colors.: Users can correct a suggested color by dragging within the magnified highlight to sample a nearby pixel color.The sampled color dynamically replaces the corresponding gap’s temporary fill color.
  • 4.2 Color Prediction Method: The U-Net predicts a spatial likelihood map from a local context patch, line-art mask, and unpainted-gap mask, then selects the most likely matching-color region.The method predicts colors indirectly through region-to-region correspondence rather than directly regressing color values.
  • 4.2.2 Creating Synthetic Training Dataset.: Synthetic training data segments line drawings, identifies gaps below 10 pixels, extracts local context, and learns correspondence to the closest large same-color region.This construction reflects the observation that small regions often share colors with neighboring regions in professional anime images.
  • 4.1 User Interface: Suggested fills can be applied across multiple selected highlights by sweeping over them or using an apply-all button.The interaction marks crossed gaps for confirmation and fills the selected regions together upon release.

5 User Study with Professionals

The user study evaluated GapFill with professional colorists through production-like tasks, performance measures, usability ratings, interviews, and feature feedback. GapFill was faster for gap-filling tasks, left no gaps across the evaluated tasks, and received generally favorable usability evaluations, while preferences varied for initial coloring and AI assistance.

  • 5.1 User Study: Fourteen professional colorists participated, with one excluded from analysis because of technical issues, leaving 13 analyzed participants.Participants had 1–10 years of professional experience and used CSP for daily colorization.
  • 5.1.2 Procedure Overview.: The study used three time-limited tasks: coloring from scratch, detecting and filling gaps, and evaluating automated color predictions.The time limits were 150 seconds for Task A, 90 seconds for Task B, and 30 seconds for Task C.
  • 5.2.1 Task Performance.: 12.54 seconds faster in Task B1 and 14.36 seconds faster in Task B2, GapFill significantly outperformed the Baseline in gap-filling speed.Task B1 means were 45.15 seconds for GapFill versus 57.69 for Baseline; Task B2 means were 51.91 versus 66.27.
  • 5.2.1 Task Performance.: GapFill left zero unpainted gaps across all participants and tasks, whereas the Baseline left gaps for several participants.The Baseline included five remaining gaps for P6 in Task B2 and smaller residual counts for other participants.
  • 5.2.2 Subjective Usability Evaluation: Eight participants preferred GapFill in LQ1 and eleven in LQ2, while ten responded positively to LQ6 regarding its intended function.P3 consistently preferred the Baseline and cited limited trust in AI assistance.
  • 5.2.2 Subjective Usability Evaluation: Ten participants rated intuitiveness positively, and all except P3 responded positively about practical adoption, with eight strongly positive.Participants also described adaptation to the new tool and its usefulness under deadline pressure.
  • 5.2.3 Feature-Level Evaluation: Participants rated automatic highlighting, color suggestion, magnification, in-circle color-pick, and batch application positively, although preferences and use strategies varied.Feedback valued reduced detection, zooming, and color-selection effort, while also identifying adaptation and magnification improvements as concerns.

5.3 Evaluation of Our Color Prediction Method

GapFill’s color predictions were evaluated subjectively and objectively, showing positive production acceptance alongside substantially higher exact-match accuracy than a naive baseline.

  • Subjective Evaluation: Production acceptability received M=4.5, SD=1.6, and Md.=5 across four evaluation images.Responses were aggregated using the median across images for each participant to avoid pseudoreplication.
  • Subjective Evaluation: Participants generally regarded predictions as highly accurate, while identifying minor, consistently localized corrections around pupils.The localized corrections were described as easy to address, but detailed regions with diverse colors remained challenging.
  • Subjective Evaluation: Some minor inaccuracies may go unnoticed after post-production, although professionals still valued neat results and adjusted their tolerance under deadline pressure.
  • Objective Evaluation: 81.68% accuracy (83,345/102,041) was achieved by our method, compared with 37.02% (37,776/102,041) for the Naive baseline.Accuracy was measured as exact agreement with ground-truth colors for unpainted gaps in a professional anime title excluded from training.
  • Objective Evaluation: 74ms average inference time per patch on an NVIDIA RTX 6000 Ada Generation supported real-time interaction.

6 Discussion and Future Work

Discussion findings position prediction accuracy as conditional rather than decisive for usability: controllability, task sensitivity, production context, and workflow integration shape acceptance and trust.

  • Accuracy as a Conditional Factor: Prediction accuracy alone is unlikely to determine usability; manual control, task sensitivity, and production context also matter.Minor errors may be tolerated when easily corrected, whereas critical inaccuracies can erode trust.
  • Accuracy as a Conditional Factor: Visually indistinguishable fills across participants question whether pixel-level accuracy and definitive ground truth adequately measure quality.The appropriate color can be ambiguous for tiny, blended gaps, unlike standardized model-sheet colors.
  • Future Directions: Technical Improvements and Reconsidering: Detailed regions such as pupils remain an accuracy challenge, while sparse local context and the lack of dynamic updates constrain future improvements.
  • Future Directions: Technical Improvements and Reconsidering: Future evaluation should examine gap size, position, and surrounding contrast to identify when approximate colors remain perceptually acceptable.
  • Limits of Trust in New AI-Powered Assistance: Professionals valued gap visualization and controllable assistance, but some continued double-checking or manually reproduced AI suggestions because of reliability concerns.
  • Limits of Trust in New AI-Powered Assistance: Miscoloring visually sensitive regions such as eyes can alter character impression, so GapFill cannot serve as the final product without human quality assurance.
  • Toward Integration with Existing Tools: Combining GapFill with established tools may preserve trusted manual verification while adding automation speed across different production stages.Participants specifically envisioned combining systems and integrating GapFill with familiar practices such as the Black Light Method.

7 Conclusion

GapFill addresses small unpainted regions in professional anime colorization by combining workflow-integrated assistance with local-context color prediction, and its evaluation involved 13 professionals.

  • GapFill targets small unpainted “gaps” in professional anime colorization workflows.
  • The tool reduces effort for gap detection, zooming, and color selection while integrating into existing pipelines.
  • Its deep learning method leverages anime flat-color characteristics and local cues to suggest colors for gaps.
  • A user study with 13 professionals evaluated GapFill’s gap-filling support.

A Analyses of Enclosed Regions in Anime Images

The analysis examined enclosed regions across 51,892 frames from 12 episodes to characterize their quantity, sizes, and proximity to larger same-colored regions.

  • 51,892 frames from 12 episodes of a professionally produced anime series were analyzed using flood-fill-based BFS segmentation.
  • The analysis measured the number of enclosed regions, their size distribution, and distances from small regions to larger regions of the same color.

A.1 Number of Enclosed Regions

Anime frames contain many enclosed regions, creating a substantial space in which small unpainted gaps may occur.

  • 115 enclosed regions appeared per frame on average, with a median of 89 across anime frames.

A.2 Region Size Distribution

Very small enclosed regions are common in anime images, motivating their use as a practical definition of potential gaps.

  • Approximately 54% of enclosed regions were 10 pixels or smaller.This frequency motivated defining small regions as those under 10 pixels when considering potential unpainted gaps.

A.3 Distance to Large Same-Color Regions

Small enclosed regions are usually located close to larger regions of the same color, supporting local contextual color inference.

  • Approximately 90% of small regions had a large same-color region within 10 pixels.Distances used the closest pair of pixels between each small region and its nearest large same-color region.

B Supplementary Statistical Analysis

Supplementary paired t-tests reinforce the efficiency findings: GapFill did not significantly change Task A time but significantly reduced time in both Task B subtasks.

  • B Supplementary Statistical Analysis: Task A showed no significant time difference: mean difference −2.33, 99% CI [−16.91, 12.24], t(11) = −0.497, p = .31440.
  • B Supplementary Statistical Analysis: Task B1 showed a significant time reduction: mean difference −12.54, 99% CI [−22.11, −2.96], t(12) = −3.999, p = .00088.
  • B Supplementary Statistical Analysis: Task B2 also showed a significant time reduction: mean difference −14.36, 99% CI [−26.53, −2.20], t(10) = −3.742, p = .00192.
Loading 2609.00800v1…