PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text to Image Generation

Kuan Xing, Ye Wang, Changyi Gan, Yuheng Li, Thao Nguyen, Yi Chang, Yilin Wang

arXiv:2608.06751v1cs.CVcs.AI

TL;DR

Artist-grounded image generation can substitute canonical artist shortcuts for the requested scene. Atelier plans explicit, evidence-grounded controls and improves style fidelity, source-structure preservation, and shortcut avoidance across evaluated generators.

  • Problem

    Artist-name prompts leave scene preservation, stylistic decisions, and inappropriate motif avoidance underspecified, enabling systematic canonical shortcutting in artist-grounded generation.

  • Method

    Atelier infers a structured control state, binds artist evidence to scene roles, compiles generation plans, and revises outputs using global and local authenticity feedback.

  • Results

    Atelier improves artist-level style fidelity and source-painting resemblance while reducing shortcut substitution across evaluated artist-grounded generation settings.

  • Takeaways & Limitations

    Artist-grounded generation benefits from explicit, role-bound artistic controls that distinguish intended scene structure from stylistic transformation and shortcut avoidance.

  • Takeaways & Limitations

    ArtIntentBench currently covers only Van Gogh and Qi Baishi because reliable artist-grounded evaluation requires curated evidence and human validation.

Abstract

from arXiv · show

Artist-grounded image generation requires more than appending an artist name to a prompt. Image models often respond to artist names through canonical shortcuts, such as recurring motifs, generic palettes, or overrepresented period signatures, rather than preserving the user's intended scene. We introduce Atelier, a shortcut-aware control-state planning framework for artist-grounded image generation. Atelier translates underspecified artistic intent into an explicit control state that separates scene anchors, preserve/transform decisions, style-regime hypotheses, role-bound artist evidence, and shortcut-avoidance constraints. It grounds this state using artist-level knowledge and local patch references, compiles backend-aware generation plans, and iteratively refines candidates through global and local authenticity feedback. We further introduce ArtIntentBench, a benchmark covering Van Gogh and Qi Baishi across artwork re-rendering, period/style-controlled generation, historically unseen subjects, shortcut auditing, and human preference evaluation. Across open-weight and closed-source generators, Atelier improves artist-level style fidelity, preserves source structure more faithfully, and substantially reduces shortcut substitution compared with prompt-engineered, retrieval-augmented, and general-purpose agent baselines. These results suggest that artist-grounded generation is bottlenecked not only by image synthesis, but by the upstream inference of explicit, evidence-grounded artistic controls.

1 Introduction

Artist-grounded generation fails when models substitute canonical artist shortcuts for scene-specific artistic decisions. Atelier addresses this as language-to-control translation by inferring explicit, evidence-grounded controls before synthesis and refining outputs with global and local authenticity feedback.

  • Problem: Canonical shortcutting is a systematic failure in which artist conditioning inserts high-frequency motifs or treatments unsupported by the requested scene.The paper distinguishes such shortcuts from valid canonical associations and frames them as reliance on spurious cues instead of context-sensitive visual language.
  • Problem: The framework targets artistic decisions that synthesis alone does not reliably infer, including period, composition, motif selection, palette, brushwork, material treatment, and local texture.The intended output must preserve the requested scene while making these artist-specific decisions explicitly.
  • Contribution: Atelier translates underspecified artist-grounded requests into explicit control states covering scene anchors, preserve/transform policies, style-regime hypotheses, retrieval targets, and anti-shortcut constraints.It retrieves artist-level knowledge and local patch evidence, binds patches to scene roles, compiles backend-aware plans, and iteratively refines candidates.
  • Evaluation: Evaluation separates intermediate control-state quality from final-image quality and combines global assessment with local authenticity diagnosis.This reflects failures that can remain hidden in plausible overall images, including errors in brushwork, motif treatment, material texture, and period-specific surface language.
  • Evaluation: The introduction proposes frozen, human-validated shortcut taxonomies for Van Gogh and Qi Baishi to measure shortcut substitution across open-weight and closed-source generators.The audit uses neutral prompts, independent review by three annotators, and majority voting, with a pattern counted when at least two annotators select it.

2 Method

Atelier instantiates Shortcut-Aware Artistic Control Planning (SACP) as a closed-loop system that converts underspecified artistic requests into explicit, evidence-grounded controls. It retrieves role-bound global and local artist evidence, compiles backend-aware plans, and iteratively refines candidates using global and patch-level authenticity feedback.

  • Structured control state: The control state separates scene reading, preserve-versus-transform decisions, historical or style-regime intent, retrieval and anti-shortcut constraints, and backend execution preferences.Scene reading captures objects, regions, relations, and mood, while style-regime fields can encode Van Gogh periods or Qi Baishi motif, composition, ink, inscription, and seal cues.
  • SACP and Atelier: Atelier converts underspecified requests into explicit control states, retrieves role-bound artist evidence, compiles backend-aware plans, and iteratively refines candidates through critic feedback.The closed loop includes scene interpretation, evidence retrieval, planning, execution, evaluation, memory updates, and final selection.
  • Closed-loop refinement: Candidate images are evaluated by a global critic and AuthCritic, a trained patch-level authenticity critic, whose feedback updates constraints, preferences, memory, and subsequent refinement rounds.AuthCritic first identifies low-scoring patches, while holistic evaluation supports candidate selection and acceptance decisions.
  • Artist evidence retrieval: The knowledge base combines artist-level style priors, curated historical summaries, and annotated local patches bound to semantic scene roles rather than appended as generic style examples.Global references supply period, motif, composition, and palette priors, while local patches ground roles such as subjects, backgrounds, materials, and textures.
  • Planning and execution: The planner produces a backend-aware generation plan containing scene anchors, style targets, reference bindings, negative shortcut constraints, and backend-specific parameters.The executor maps these controls to reference, adapter, or LoRA-style resources when supported, or to compact textual realizations for general-purpose backends.

3 Experiments

Across artwork re-rendering, unseen-subject generation, and cross-artist transfer, Atelier achieves stronger style fidelity while preserving source or modern-subject structure. Audits show improved control-state grounding and reduced shortcut substitution, while patch-bank coverage remains a key limitation.

  • Artwork Re-rendering and Cross-Artist Transfer: Atelier achieves the lowest IntroStyle W2 for Van Gogh re-rendering and Qi Baishi transfer across both open-weight and closed-source settings, while also leading Qi Baishi DINO-cosine and LPIPS.Van Gogh re-rendering scores are 73.52 open-weight and 64.22 closed-source; Qi Baishi transfer scores are 68.33 and 57.09, respectively.
  • Historically Unseen Subject Generation: For historically unseen modern subjects, Atelier achieves the lowest IntroStyle W2 in both settings and more consistently preserves recognizable subject structure than direct and agent-based baselines.The W2 scores are 86.61 closed-source and 91.28 open-weight; agent baselines often substitute modern elements with period-appropriate equivalents.
  • Control-State Audit: Control-state audits verify period recovery, preserve/transform decomposition, local patch-role binding, identity protection, and anti-shortcut checks across the benchmark’s task settings.When period information is structured, agreement is perfect; caption-based inference reaches top-1 recovery of 40.0% open-weight and 38.0% closed-source, with top-2 recovery of 66.0% and 62.0%.
  • Human Evaluation: Human evaluations rate historically unseen-subject composites highest across panels, while Van Gogh re-rendering emphasizes source retention and Qi Baishi performance is limited mainly by motif and composition.Unseen-subject composite scores range from 8.30–9.34 open-weight and 7.64–8.98 closed-source; Van Gogh source retention is at least 8.4, whereas Qi Baishi composites range from 5.81–6.65 and 4.69–6.10.
  • Shortcut Audit and Limitations: Shortcut audits find high baseline shortcut rates, whereas Atelier’s remaining weakness is knowledge-base coverage rather than binding or perceiver failure.Qi Baishi baseline SSR ranges from 55.00% to 78.75% for closed-source systems and 40.00%–53.75% for open-weight direct baselines; patch-evidence usefulness is the weakest human-rated dimension.

4 Related Work

Prior work addresses style transfer, personalization, agentic generation, evaluation, and shortcut mitigation, but the supplied passages identify unresolved challenges in translating ambiguous artistic intent into structured controls and diagnosing local artistic authenticity. Existing approaches often target fixed style statistics, global evaluation signals, or specific training-stage confounds rather than the full artist-grounded generation problem.

  • Artistic style transfer: Style-transfer methods render content using exemplar feature statistics or artist-work collections, while ChipGAN adds medium-specific constraints for Chinese ink painting.These approaches primarily transfer surface statistics from a fixed style target.
  • Personalization and style-conditioned Image Editing: Personalization and style-conditioned generation adapt models to specified subjects, concepts, or styles [42], but do not translate ambiguous artistic intent into structured artist-grounded controls.Instruction-based editing additionally uses natural-language instructions and structural conditions.
  • Agentic and tool-using generation systems: Language agents use decomposition, tools, memory, and iterative feedback [34] [41], while image-generation and editing agents apply perceive–plan–execute–evaluate loops [48] [51].These frameworks motivate closed-loop assistance for complex image-generation and editing tasks.
  • Evaluation of image generation and artistic authenticity: Global image-generation metrics assess quality, alignment, or preference, but are less suited to local failures involving brushwork, motifs, or material texture.Art-specific judgment remains difficult for strong multimodal models, while patch-level VLM style concepts partially align with art-historian judgment [26].
  • Shortcuts in text-to-image generation: Shortcut learning is a recognized deep-network failure mode, and Goyal et al. mitigate personalization shortcuts during adapter training by exposing confounding-attribute pathways.Their setting targets confounds such as pose, expression, and lighting alongside target identity.

5 Discussion

Artist-grounded evaluation is difficult to scale because reliable validation must assess period fidelity, motif appropriateness, brushwork, material treatment, and shortcut substitutions beyond what automatic metrics capture well.

  • 5 Discussion: ArtIntentBench adopts a depth-first evaluation design because artist-grounded validation requires nuanced judgments that generic automatic metrics cannot reliably capture.The evaluation targets period fidelity, motif appropriateness, brushwork, material treatment, and shortcut substitutions rather than relying on image availability alone.

Limitation

The benchmark covers only two artists, reflecting the substantial curation and expert validation required for reliable artist-grounded evaluation.

  • The benchmark’s two-artist scope is intentional because reliable evaluation requires curated artist knowledge, patch-level evidence, and expert or art-trained human validation.

Ethics and Copyright Statement

The study uses public-domain or museum-held art sources and supports reproducibility by releasing code, metadata, prompts, schemas, and evaluation protocols without redistributing copyrighted images.

  • Ethics and Copyright Statement: The Van Gogh corpus uses public-domain reproductions, while the Qi Baishi corpus uses public museum digital collections of traditional Chinese art.Sources include WikiArt, the National Gallery of Art, and public museum collections.
  • Ethics and Copyright Statement: The authors will release code, metadata, benchmark prompts, the control-state schema, and evaluation protocols, while copyrighted source images will be referenced through links rather than redistributed.The released metadata is intended to support reproducibility.

A Control-State Walkthrough

The appendix walks through Atelier’s control state on two recorded requests: one exposes all five components, while the other traces three refinement rounds and memory-mediated critic feedback. It also explains how the state is distributed across runtime artifacts rather than serialized as one JSON object.

  • Appendix A: The walkthrough exhibits all five components of z = (s, q, h, r, b) on one held-out request and follows three refinement rounds on another.The second trace shows critic feedback being transported through reflective memory into subsequent plans.
  • Runtime representation: At runtime, z is distributed across the scene specification, world model, knowledge_evidence block, and compiled plans rather than serialized as one JSON object.Appendix D, Table 14, gives the field-level mapping across these artifacts.
  • Runtime representation: The excerpts group recorded fields by control-state component while retaining implementation names such as heavy_judge_score for the global-critic score sg.Elisions are marked with ....

A.1 Single-Round Control State

Atelier converts the still-life request into an explicit control state that separates scene roles, preserved structure, style translations, period routing, retrieval evidence, and shortcut constraints. It then compiles that state into three backend-specific generation calls while preserving structural anchors and propagating role-level style guidance.

  • A.1 Single-Round Control State: The scene reader classifies the request as an object still life and assigns foreground mass, field surface, and background halo roles with distinct translation axes.The axes cover surface relief, palette relation, contour pressure, brush rhythm, and directional motion.
  • A.1 Single-Round Control State: The control state preserves object-silhouette integrity and the tabletop support plane while translating role-specific surface, palette, contour, brush, and motion properties.
  • A.1 Single-Round Control State: Because the request explicitly specifies the Paris period, period policy uses a single Paris-period route, while retrieval may select an adjacent-period family through scoring rather than hard filtering.The selected bound family is indexed under the Nuenen period, but the generation route remains Paris.
  • A.1 Single-Round Control State: Retrieval binds the foreground-object role to a patch family but leaves the field-surface and background-halo roles dependent on work-level evidence.The foreground family contains representative patch patch_03958, whereas the other roles use unbound-family fallback.
  • A.1 Single-Round Control State: The planner compiles the state into N = 3 parallel calls with shared structural and style constraints, including negative prompts against canonical Van Gogh shortcuts.The plan uses remote_flux_lora, qwen_t2i, and longcat_t2i, while carrying preserve targets and per-role translation axes into backend text.

A.2 Three-Round Refinement Trace · B Prompt Library

The Qi Baishi refinement trace shows fixed-backend, memory-mediated prompt revision improving critic scores while stopping below the acceptance threshold. The prompt library operationalizes Atelier’s evidence-grounded control state through structured scene extraction, clarification, intent inference, and shortcut-avoidance rules.

  • A.2 Three-Round Refinement Trace: Global-critic scores rise from 47 to 49 to 57 while AuthCritic aggregates reach 42, 67, and 67; FinalSelect chooses Round 3, but the episode remains below τ ⋆= 85 and terminates with recommendation revise.Hunyuan is locked after Round 1, so the trace isolates iterative execution-prompt refinement rather than backend changes.
  • A.2 Three-Round Refinement Trace: The first-round critic identifies weak calligraphic energy, limited ink-density transitions, generic decorative-illustration drift, and overly uniform brushwork, storing canonical failure tags with localized repair targets.Preserved strengths include subject count, left-side inscription and seal, and blank-paper reserve.
  • A.2 Three-Round Refinement Trace: Subsequent prompts translate those findings into material and stroke instructions—dry-brush fraying, graded or translucent washes, ink underpainting, pressure-varying calligraphy, and abbreviated duck forms—while retaining subjects and layout.Round 2 retains the composition while adding concrete repair instructions, and Round 3 carries them forward with ink gradation and translucent petal color.
  • B Prompt Library: The prompt library reproduces deployed templates for Atelier’s perceiver, planner, global critic, local AuthCritic, and control-state audit judges, with two audit listings restricted to fields included in the reported analysis.The listings preserve implementation prompt text while documenting the deployed component interfaces.
  • B.1 Perceiver Prompts: The perceiver prompt converts raw requests into a minimal JSON scene specification containing regime, subject, required and forbidden entities, scene roles, period preference, confidence, and a reasoning note.The clarification prompt additionally decides whether follow-up questions are useful and infers implicit reference intent without replacing the requested subject.
  • B.1 Perceiver Prompts: Perceiver rules require outputs to follow the user request rather than artist-associated motifs, prohibit inferring periods from artist names or canonical resemblance, and prevent forbidden entities from becoming positive cues.Scene regimes and roles must be selected from closed catalogs according to explicit framing and compositional organization.
  • B.1 Perceiver Prompts: The clarification policy asks only about missing factors that improve artistic control, such as mood, setting, expressive goal, life context, or explicit period preference, while treating period signals as artist-specific preferences.It supports low-intrusion personalization and distinguishes implicit work-family references from the user’s actual subject.

C Evaluation Criteria and Selection Policy

This appendix defines Atelier’s closed-loop evaluation policy, including numeric thresholds, stopping and selection rules, and critic input contracts. These rules remain fixed across all experiments reported in §3.3.

  • The appendix specifies the closed-loop stop condition, per-round winner rule, and trajectory-level FinalSelect rule used by Algorithm 1.It also defines the input contracts for Atelier’s two critics.
  • All thresholds and policy rules are held fixed across every experiment reported in §3.3.
  • Together, Algorithm 1 and these rules constitute the full Atelier configuration.

C.1 Runtime Constants · C.2 Global-Score Aggregation

The runtime policy uses uniform thresholds, budgets, and routing cardinalities across Section 3.3 episodes, while backend locking depends on round-one quality. Global scores combine eight critic dimensions and an artifact penalty, then undergo recommendation caps and confidence-aware calibration.

  • C.1 Runtime Constants: Runtime thresholds, budgets, and routing cardinalities are applied uniformly to every episode in Section 3.3.These constants are defined together with the pool-and-lock policy.
  • C.1 Runtime Constants: Round 1 explores a track-specific backend pool before locking its winner for subsequent rounds.The policy therefore narrows backend choice after the initial track-specific exploration.
  • C.1 Runtime Constants: The backend lock is released when the selected outcome is low-quality: sg < τℓ, a reject, or a low-confidence uncertain recommendation.No additional 60-point lock gate is used.
  • C.2 Global-Score Aggregation: The global critic returns eight subscores xj ∈[1, 5] and an artifact penalty a ∈[0, 3].These values form the inputs to the base-score computation.
  • C.2 Global-Score Aggregation: The score weights prioritize style authenticity and period match at 18 each, followed by intent preservation, brushwork directionality, and impasto texture at 12 each.Motif match and composition match receive 10 each, while palette match receives 8.
  • C.2 Global-Score Aggregation: The base score is capped at 59 for reject, 69 for uncertain, or 84 for revise, then calibrated using confidence, blocking tags, actionable gaps, and request-anchor violations.The resulting calibrated value is the score sg used by the downstream policy.

C.3 Stop Condition · C.4 Candidate Selection

The deployed policy terminates episodes through ordered success, plateau, rejection, budget, and error conditions, then selects a final candidate using trajectory-wide evidence and fallback ranking rules. Within-round selection similarly incorporates global scores, AuthCritic evidence, backend preferences, and candidate order when available.

  • C.3 Stop Condition: The policy applies terminal conditions in a fixed order, using score, recommendation, and canonical failure-tag overlap to evaluate episode progress.It defines overlap as |A ∩ B|/|A ∪ B|, with empty-set overlap equal to 1.
  • C.3 Stop Condition: Success termination requires both specified success terms, while a hard near-threshold plateau is detected after three consecutive revise recommendations.The supplied passages state that both success terms are required but do not expose their formulas.
  • C.3 Stop Condition: A generic four-revision plateau guard is unreachable at the reported budget T = 3, whereas a two-round near-plateau is logged as risk rather than treated as terminal.The generic guard requires score span at most 2, mean tag overlap at least 0.6, and maximum score below τ⋆−3; the logged risk uses span at most 1, overlap at least 0.5, and maximum score at least τ⋆−3.
  • C.3 Stop Condition: Episodes also stop after at least two recent low-score rejects without current improvement, at t = T, or after unrecoverable retry failure; FinalSelect then returns y⋆.The reject condition uses scores no greater than 59, and FinalSelect operates on the completed trajectory.
  • C.4 Candidate Selection: With a usable AuthCritic sidecar, within-round selection combines global-critic and AuthCritic candidate evidence, including score, real-patch probability, and candidate index.The passage identifies sg(y), sa(y), pa(y), and i(y), but the selection formula is not included in the supplied text.
  • C.4 Candidate Selection: Without a usable sidecar, candidates are ordered by global score, accept status, confidence, backend preference, and earlier index, with backend-specific tie-breaking.Low-quality exact ties prefer a non-FLUX candidate; otherwise the default backend tiebreak favours FLUX.
  • C.4 Candidate Selection: FinalSelect evaluates the full trajectory with stored second-stage scores and generation rounds, using local evidence when available and lexicographic fallback otherwise.Without usable local evidence, it ranks by (sg(y), s2(y), −t(y)); rounds remain in multi-backend exploration when their winner is rejected, low-scoring, or uncertain and low-confidence.

C.5 AuthCritic Output Contract and Escalation

AuthCritic operates as a grounded reasoning model that emits a strict, ordered JSON record for each patch. Low-scoring patches are escalated to global critique with their crop and stated cause, capped at four patches per candidate.

  • AuthCritic Output Contract: AuthCritic must emit an ordered JSON object containing source labels, grounding fields, visual and style descriptions, a short reason, and the difference from the real master.The required keys are source_type, grounded_title, grounded_period, source_global_summary, patch_visual_description, patch_style_description, reason_short, and difference_from_real_master.
  • AuthCritic Output Contract: The source_type label distinguishes real-master, synthetic-master-style, and other-painter patches, while non-real-master records explain departures through brushwork, palette, contour, surface, or motion.Artist-specific aliases such as real_vangogh_patch are normalized during parsing; real-master predictions are grounded in a work title and period.
  • Escalation: Patches scoring below τp or πp are escalated with their crop and reasoning record as focus signals for global critique, with at most Kp = 4 patches per candidate ordered by ascending score.This connects suspect regions to stated causes during the two-stage evaluation.

C.6 AuthCritic Candidate Aggregation · C.7 Global-Critic Input Projection

AuthCritic aggregates patch-level authenticity evidence into candidate scores, while the global critic receives a deliberately projected request-state view plus selected low-scoring patch evidence. The projection excludes the full state and evidence bundle, leaving other fields to condition retrieval, planning, and generation.

  • C.6 AuthCritic Candidate Aggregation: AuthCritic excludes context-only crops when any local style-evidence patch is available; otherwise, it scores all sampled patches.
  • C.6 AuthCritic Candidate Aggregation: Candidate scoring uses the real-master fraction, mean proximity-to-real, and a validity-related term over scoring patches.The variables include fr, mean synthetic-master proximity, valid-proximity fraction, patch count, and proximity parse-error rate.
  • C.6 AuthCritic Candidate Aggregation: When no synthetic-master patch has valid proximity, mean proximity-to-real falls back to the mean across all valid proximities.
  • C.6 AuthCritic Candidate Aggregation: The associated candidate probability is the real-master fraction, while missing proximity fields and parse status trigger a legacy rounded-mean fallback.The legacy fallback averages per-patch source-type scores.
  • C.7 Global-Critic Input Projection: The global critic prompt projects the raw and clarified requests, working period hypothesis and alternatives, selected motif and composition cues, and low-scoring patch metadata.
  • C.7 Global-Critic Input Projection: The candidate image and flagged patch crops are attached visually, while the remaining state and evidence fields condition retrieval, planning, and generation without entering the critic prompt.

D Control-State and Memory Schema Reference … D.3 Memory Structure

The appendix specifies runtime schemas for the abstract control state z and memory m, enforced through strict JSON contracts, normalization, fallback values, and explicit field mappings. It also defines audit fields and three-component cross-round memory for planning and evaluation.

  • D Control-State and Memory Schema Reference: Runtime realizes z = (s, q, h, r, b) and memory m as records under a prompt-defined strict-JSON contract with normalization and fallback substitution.Perceiver and planner prompts specify required keys; absent or malformed fields receive corresponding fallback values.
  • D.1 Planner Output Contract: The planner outputs six required top-level JSON keys, while the runtime additionally records a derived routing_plan.The six-key contract and normalization behavior are documented in Table 13.
  • D.2 Field Map for z = (s, q, h, r, b): Table 14 maps each component of z to concrete runtime record paths, with abbreviated paths under knowledge_evidence.The mapping connects the abstract state components to the fields consumed by runtime records.
  • D.2 Field Map for z = (s, q, h, r, b): The runtime rebuilds base s and q from fixed interpretation inputs, permits planner revisions within a round, and applies path-specific merge behavior.The Qi Baishi merge restores authoritative scene and anti-shortcut fields, whereas the Van Gogh path retains the planner-emitted world model when present.
  • D.2 Field Map for z = (s, q, h, r, b): Additional world-model audit fields record scene-preservation logic, regime hypotheses, role constraints, uncertainty, and shortcut-related controls without directly feeding generation.Examples include scene_logic_to_preserve, prompt_scene_regime, allowed_scene_roles, role_coverage_audit, and unresolved_uncertainties.
  • D.3 Memory Structure: Persistent memory m contains trajectory, reflective, and selection-related information across rounds.Trajectory memory stores each round’s executed plan, evaluation outcomes, failure and repair information, authenticity scores, and selection record; reflective memory is read at the next Derive step and supplied to the planner.
  • D.3 Memory Structure: Trajectory memory records the executed positive and negative prompts, guidance note, critic outcomes, failure tags, repair targets, AuthCritic score and yes-probability, and selection record for each round.These records support cross-round tracking of plans, evaluations, repairs, authenticity judgments, and candidate selection.
Loading 2608.06751v1…