Source-linked AI summary
When Creators Meet the Metaverse: A Survey on Computational Arts
Lik-Hang Lee, Zijun Lin, Rui Hu, Zhengya Gong, Abhishek Kumar, Tangyao Li, Sijia Li, Pan Hui
TL;DR
Computational arts need to be understood within expanding virtual-physical spaces where users, creators, and digital marketplaces interact. The paper surveys metaverse-relevant artistic components and practices, finding broad creative and trading activity while identifying unresolved ownership, valuation, environmental, and generation challenges.
Problem
The metaverse creates new virtual-physical contexts for computational arts, but artists and technologists need an integrated account of its creative components, practices, and unresolved challenges.
Method
The paper conducts a state-of-the-art survey covering virtual entities and scenes, textual and auditory arts, collaboration, robotics, immersive technologies, and virtual creativity.
Results
The survey documents diverse computational artworks and growing NFT-based digital-art trading, including differentiated tokens representing in-game collectibles with specifications proportional to rarity.
Takeaways & Limitations
The metaverse can function as a broad canvas and marketplace for computational artists, while participation extends beyond professional creators to users and collaborators.
Takeaways & Limitations
NFTs do not eliminate illegal appropriation or scams, and NFT-backed artwork valuation lacks a commonly accepted conclusion while large-scale adoption may consume substantial energy.
Abstract
from arXiv · showhide
The metaverse, enormous virtual-physical cyberspace, has brought unprecedented opportunities for artists to blend every corner of our physical surroundings with digital creativity. This article conducts a comprehensive survey on computational arts, in which seven critical topics are relevant to the metaverse, describing novel artworks in blended virtual-physical realities. The topics first cover the building elements for the metaverse, e.g., virtual scenes and characters, auditory, textual elements. Next, several remarkable types of novel creations in the expanded horizons of metaverse cyberspace have been reflected, such as immersive arts, robotic arts, and other user-centric approaches fuelling contemporary creative outputs. Finally, we propose several research agendas: democratising computational arts, digital privacy, and safety for metaverse artists, ownership recognition for digital artworks, technological challenges, and so on. The survey also serves as introductory material for artists and metaverse technologists to begin creations in the realm of surrealistic cyberspace.
1 INTRODUCTION
The metaverse connects computational arts with virtual-physical spaces, creating opportunities for artists and participants to produce, experience, and trade digital artworks. This survey reviews the enabling components, creative forms, and research challenges of computational arts in that emerging environment.
- Metaverse context: The metaverse is an open 3D virtual space where users can socialise, learn, work, collaborate, create, and play, making content creation central to its sustainability.Participants can become digital creators and computational artists rather than merely consuming professional contributions.
- Computational arts: Computational arts increasingly involve computers and AI as active participants in creation, extending beyond simple digitisation toward interactive and generative artworks.The survey situates these developments within a broader shift toward virtual-physical blended realities and intelligent entities.
- Virtual art trading: NFTs can specify transferred rights and support ownership recognition, but they do not eliminate illegal copying, scams, unresolved valuation questions, or environmental concerns.The survey frames NFTs as licenses outlining ownership rather than complete protection against appropriation or fraud.
- Survey contribution: The survey presents metaverse-driven worlds as a malleable canvas for digital arts and combines reviews of computational arts with reviews of metaverse development.It is positioned as an integrated review intended to introduce artists and technologists to this interdisciplinary area.
- Survey scope: The survey covers virtual scenes and characters, textual and auditory arts, user-centric collaboration, robotics, and augmented or virtual reality as metaverse-relevant creative forms.Its structure spans foundational virtual components alongside embodied, immersive, and collaborative artistic practices.
2 VIRTUAL PHOTOGRAPHY / CINEMATIC SIMULATION
Computational arts use virtual photography, CGI, game engines, and simulation to create dynamic characters, environments, and personalised visual experiences across virtual and physical worlds.
- Virtual Photography: CGI expands photographic and cinematic practice through virtual influencers, digitally scanned portraits, and animated virtual characters embedded in visual narratives.Examples include @lilmiquela, a 3D portrait of Barack Obama, and Ed Atkins’s character-based video artworks.
- Virtual Production: Improved game-engine usability and graphics quality make real-time virtual production more accessible to cultural and artistic creators.Immediate visual feedback reduces waiting during content creation and makes workflows more intuitive.
- Cinematic Simulation: Game engines support real-time rendering, programmed mechanisms, and interactivity, enabling cinematic simulations with unexpected dynamic events.These works can combine computationally generated imagery with cinematic camera movements and editing techniques.
- Cinematic Simulation: Live simulation explores virtual agency and interactivity through autonomous video-game-like systems that generate responses to events.Ian Cheng’s work uses multiple sub-agent systems with distinct personalities and desires.
- Style Transfer: Neural style transfer can process virtual or physical videos in real time, applying user-defined styles at object-level granularity.Different styles can be assigned to different object classes, supporting personalised artistic experiences.
3 CALLIGRAPHY
Computational calligraphy adapts to diverse linguistic and emotional contexts while extending handwritten styles into realistic 3D virtual artworks.
- AI calligraphy systems can mimic expert handwriting across linguistic contexts, including features such as stroke width, skew, ligatures, and slant.
- Few-shot style transfer enables ZiGAN [208] to learn Chinese calligraphy without requiring substantial samples from calligraphy masters.
- Emotion-aware calligraphy adjusts Chinese character structures using detected states such as peaceful, happy, and sad.
- Photometric stereo reconstructs Chinese calligraphy as 3D virtual objects whose heightmaps represent the artwork’s strength and personality.
4 POETRY
AI-generated poetry can connect images, topics, linguistic creativity, and virtual-world interaction, but scene consistency and flexible visual-textual representation remain open challenges.
- AI-generated poetry can support linguistic creativity and two-way communication through human-to-human and human-to-AI gamified interactions.
- Image-conditioned systems generate Chinese five- and seven-character quatrains by combining scene recognition with poetry generation.
- Scene-poetry systems should be evaluated for fluency, meaningfulness, phonological compliance, and coherence.
- Automatic poetry generation faces topic drift, repetitive word usage, and difficulty maintaining semantic consistency between poems and images.
- The metaverse motivates visual poems that spatially blend textual and visual content beyond rigid automatically generated outputs.
5 AUDITORY AND MUSICAL METACREATION
Metaverse musical arts span gesture-controlled virtual instruments, shared concerts, and machine-assisted composition, while audience scale and realistic concert simulation remain limiting factors.
- Musical Instruments for the Metaverse: Virtual reality musical instruments let performers control sound through gesture and motion, with some systems supporting collaboration in shared virtual spaces.
- Music Concerts and Festivals in the Metaverse: VR platforms provide stages for concerts and festivals, offering artists access to potentially large audiences despite pandemic-related disruption to live performance.
- Music Concerts and Festivals in the Metaverse: VRChat limits each room to 40 users, requiring multiple instances for larger audiences and complicating performer presence and concert staging.
- Musical Creativity for the Metaverse: Musical Metacreation treats machine behaviors as creative when unbiased human judges deem them novel and creative.
- Musical Creativity for the Metaverse: Deep learning music systems use note sequences or raw audio, including RNN, LSTM, Markov, and CNN-based approaches for composition and vocal generation.
6 USER-CENTRIC AND COLLABORATIVE APPROACHES FOR ARTISTIC CREATION
User-centric computational arts combine human bodies, biological signals, movement, and AI agents in collaborative creation, including co-creative dance and embodied artistic interaction.
- Privacy and Consent: Such captured data can be converted into artistic representations when privacy is managed appropriately and users provide informed consent.
- Body and Brain Signals: Body sensors and EEG-based brain-computer interfaces allow biological states, social behaviors, and emotions to contribute to artistic creation.
- User-Centric Artistic Representation: User emotions or brain activity can be transformed into artistic effects for virtual characters and avatars.
- Co-creative Dance: Computational agents can interpret user movements and collaborate with human performers through autonomous avatars in virtual dance environments.
- Co-creative Dance: LuminAI uses unsupervised learning to support human dance improvisation and co-creative dancing with virtual agents.
- Open Challenges: Technology-inspired dance still has open challenges involving audience connection, expressive augmentation, choreography, aesthetic harmony, and human–virtual integration.
7 PHYSICAL EMBODIMENT OF ART: ROBOTICS AND DRONES
Robotics and drones expand computational art through musical, theatrical, dance, painting, and collaborative performances, while blending physical embodiments with metaverse-linked virtual agents. These forms also introduce interaction and safety considerations.
- Robotic arts: Human artists can collaborate with artistic robots representing virtual avatars or AI-assisted agents, gaining new embodied environments and experiences.The paper suggests these physical embodiments may inspire unexplored artistic performance landscapes.
- Robotic arts: Robotic artists span musical performance, theatre, dance, painting, and collaboration with human performers.Some musical robots generate rhythms through machine learning, while others perform alongside human musicians.
- Robotic arts: Robots increasingly combine mechanical performance with human-like appearance, communication, cameras, robotic arms, and AI algorithms.Examples include Sophia and Ai-Da, with Ai-Da drawing through her robotic arm, AI algorithms, and eye camera.
- Drones and theatre: Theatre robots provide new performance formats and can support STEAM education, including robot opera performances in schools.Figure 14a depicts five robots used in a Robot Opera.
- Drones: Drones create aerial light shows and can perform with humans as movable, costumed objects in new audience experiences.Drone shows use multiple LED-equipped unmanned aerial vehicles, while interaction can expose audiences to propeller-related injury risks.
8 ARTISTIC CREATIONS IN BLENDED ENVIRONMENTS AND VIRTUAL CREATIVITY
Blended virtual-physical environments provide canvases for AR, VR, and mixed-reality artworks, while virtual environments are studied as tools for nurturing creativity. Research links environmental design and priming to improved creative performance, but identifies sensory limitations.
- Blended environments: Virtual and physical spaces can jointly function as artistic canvases through VR drawing, AR overlays, heritage-site interventions, and projected light artworks.Examples include decorating physical heritage surfaces with AR and placing virtual buildings inside historic buildings.
- Virtual creativity: Virtual creativity research examines domain skills, creativity-relevant processes, task motivation, social environment, and physical environment.The framework is based on componential creativity theory, augmented by evidence that physical environments can influence creativity.
- Virtual creativity: VR supports creativity-relevant processes through brainstorming, avatars, social identification, recording, and reversible operations.These features can address shortcomings of non-VR creative collaboration, including preserving discussions and undoing actions.
- Virtual creativity: Immersive VR environments may increase interest and learning motivation, but missing facial expressions can negatively affect creativity-related user experience.Adding senses such as smell is proposed as another way to influence cognition, positive emotions, and task motivation.
- Virtual creativity: Both topic-relevant and specific virtual environments improved creative performance, with topic-relevant settings supporting priming and design thinking during ideation.The reported mechanism is that priming can influence unconscious cognition and behavior, supporting team performance and personal design thinking.
9 RESEARCH AGENDA
The research agenda centers on making computational arts accessible while resolving infrastructure, interoperability, privacy, safety, ownership, embodiment, and machine-creativity challenges. The survey emphasizes that metaverse art requires both technical development and governance.
- Democratising computational arts and education: Democratising computational arts depends on accessible open-source tools such as Processing, Blender, and Unreal Engine, whose broader impacts remain to be studied.These tools allow many people to begin producing multimedia or 3D content relatively quickly.
- Virtual scenes and characters: Virtual-scene creation remains constrained by weak platform interoperability and software usability, forcing creators to transfer content across cumbersome workflows.The agenda calls for better integration among creation, interaction, and networked presentation platforms.
- Digital privacy and safety: Metaverse artists require stronger privacy and safety protections, especially in online creative communities and for vulnerable creators.The agenda explicitly highlights underage content creators and the social impacts of artists’ works.
- Digital art trade: NFT ownership alone does not establish an effective digital-art market; leasing, rights enforcement, legal understanding, disputes, and copying remain unresolved.The reNFT project proposes NFT rental, but functioning rental markets still require collective enforcement and institutional input.
- Embodiment and machine creativity: Computational arts still need frameworks for judging machine creativity, while current virtual musical interfaces lack the physical feedback musicians experience with real instruments.The survey frames both issues as open challenges for future metaverse creation.
- Technological infrastructure: Metaverse creation requires mature AR/VR devices, networks, and edge-cloud infrastructure because generative art demands computation while mobile devices face resource and battery constraints.AR and VR headsets therefore rely strongly on task offloading.
10 CONCLUDING REMARK
The survey expects metaverse-driven computational arts to reshape virtual and physical environments through diverse technologies, media, embodiment, and robotics. It calls for interdisciplinary collaboration between artists and technologists to develop the artistic metaverse and its infrastructure.
- Conclusion: The survey covers computational artworks spanning virtual photography, cinematic simulation, calligraphy, poetry, musical metacreation, immersive arts, virtual creativity, embodiment, and robotics.These works collectively frame the metaverse as a space for reshaping both virtual and physical environments.
- Conclusion: Constructing an artistic metaverse requires interdisciplinary research by technologists and artists across computational arts and technological infrastructures.The conclusion calls for joint investigation of their integrated facets.