PaperlayerView exact revision on arXiv

Source-linked AI summary

Artificial Intelligence in the Creative Industries: A Review

Nantheera Anantrasirichai, David Bull

arXiv:2007.12391v6cs.CVcs.AIcs.LG

TL;DR

Creative work poses challenges for AI because it often involves unconstrained, one-to-many outputs rather than clearly defined tasks. This paper reviews AI methods and applications across five creative-industry categories, finding that AI is most promising as a human-centered tool or collaborative assistant, while fully autonomous original creation remains constrained.

  • Problem

    Creative work often produces unstructured, one-to-many outputs, while AI systems perform best on constrained problems and may struggle to generate consistently realistic or valuable original results.

  • Method

    The paper reviews AI technologies and organizes creative-industry applications into content creation, information analysis, enhancement and post-production, information extraction and enhancement, and data compression.

  • Results

    The review finds that AI is likely to expand as a creative tool or collaborative assistant, whereas significant challenges remain for AI as the sole generator of original work.

  • Takeaways & Limitations

    Maximum benefit in creative industries is expected when AI is designed to augment rather than replace human creativity.

  • Takeaways & Limitations

    AI performance depends on training data, whose availability, diversity, and labeling are limited or imbalanced in some applications.

Abstract

from arXiv · show

This paper reviews the current state of the art in Artificial Intelligence (AI) technologies and applications in the context of the creative industries. A brief background of AI, and specifically Machine Learning (ML) algorithms, is provided including Convolutional Neural Network (CNNs), Generative Adversarial Networks (GANs), Recurrent Neural Networks (RNNs) and Deep Reinforcement Learning (DRL). We categorise creative applications into five groups related to how AI technologies are used: i) content creation, ii) information analysis, iii) content enhancement and post production workflows, iv) information extraction and enhancement, and v) data compression. We critically examine the successes and limitations of this rapidly advancing technology in each of these areas. We further differentiate between the use of AI as a creative tool and its potential as a creator in its own right. We foresee that, in the near future, machine learning-based AI will be adopted widely as a tool or collaborative assistant for creativity. In contrast, we observe that the successes of machine learning in domains with fewer constraints, where AI is the `creator', remain modest. The potential of AI (or its developers) to win awards for its original creations in competition with human creatives is also limited, based on contemporary technologies. We therefore conclude that, in the context of creative industries, maximum benefit from AI will be derived where its focus is human centric -- where it is designed to augment, rather than replace, human creativity.

1 Introduction

AI is advancing rapidly across applications, but creative work differs from routine tasks because it depends on imagination and experiences that may not follow general rules. This paper reviews AI technologies and creative-industry applications, distinguishing AI as a creative tool from AI as a potential creator.

  • Creative industries: Creative work demands innovation and imagination that may not follow the conformity on which AI accomplishments rely.Creatives draw on lifetime experiences to ask questions that constrained learning systems cannot readily address.
  • AI adoption: Three quarters of surveyed artists would consider AI assistants for image search, editing, and other non-creative tasks.The survey covered artists in the US, UK, Germany, and Japan.
  • AI adoption: Research publications on AI relevant to creative industries grew by more than 500% in many countries, reaching 1,490% in Taiwan.Most publications concerned image-based data, while company usage was especially prominent in games, immersive applications, advertising, and marketing.
  • Paper scope: The paper reviews AI technologies and organizes creative applications into content creation, information analysis, enhancement and post-production, information extraction and enhancement, and data compression.It also discusses challenges and future potential in the creative industries.

2 An introduction to Artificial Intelligence

AI comprises codes, techniques, algorithms, and data that enable computer systems to emulate human-like behavior and make decisions. Current technologies generally operate as narrow AI, restricted to specific tasks within limited domains.

  • Artificial Intelligence: AI combines codes, techniques, algorithms, and data to enable computer systems to emulate human-like behavior and make decisions.The paper notes that such decisions can sometimes match or exceed human performance.
  • AI categories: General AI or strong AI refers to a machine exhibiting full human intelligence, whereas current systems are normally narrow AI or weak AI.Narrow AI operates in limited domains and performs specific tasks.

2.1 Machine Learning, Neurons and Artificial Neural Networks

Machine learning is data-driven and learns from examples rather than predetermined equations or models. The section introduces neural-network foundations, supervised training through backpropagation, and the paper’s overlapping application categorization.

  • Machine learning: Machine learning learns information directly from large amounts of example data without relying on a predetermined equation or model.Its algorithms adaptively converge toward an optimum solution and generally improve as more training samples become available.
  • Application categorization: The paper’s application categories are useful but can overlap because several creative applications span multiple groups.The authors explicitly qualify the categorization as a helpful organizational device rather than an exclusive taxonomy.
  • Neural foundations: An artificial neuron independently weights inputs, sums them, and passes the result through a nonlinear activation function.The model represents a neuron’s action potential and transmission to other neurons.
  • Neural-network training: Supervised multilayer perceptrons use error backpropagation to compute a loss-function gradient and minimize loss across repeated presentations of the input dataset.The loss function maps multiple input event values to one real number representing event cost.

2.2 An Introduction to Deep Neural Networks

Deep learning uses multilayer neural networks, with CNNs especially suited to multidimensional data such as images. The section traces CNN development and explains how deep architectures learn increasingly semantic features, while GANs generate realistic synthetic data through competition between two networks.

  • Deep learning: Deep learning is a subset of machine learning that uses deep neural networks with multiple hidden layers containing learnable weights and biases.CNNs are often used when processed data occupy multiple dimensions, such as images.
  • CNN development: The Neocognitron introduced hierarchical convolutional and pooling layers for visual pattern recognition, while LeNet5 later advanced handwritten-character recognition.LeNet5 used convolutional and average-pooling layers followed by fully connected layers and a softmax classifier.
  • CNN operation: CNNs learn task-specific filters that progress from edges and simple shapes to higher-level semantic features.This layered representation lets CNNs exploit both low-level visual features and higher-level understanding of the data.
  • Generative models: GANs pair a generator that creates images with a discriminator that checks whether they are real or generated.Their competition drives generated pictures toward greater similarity with real images and reduces deceptive results.
  • Paper emphasis: The paper emphasizes deep learning because of its prominence and performance across creative-industry applications.It critically reviews methodologies that achieve or could achieve good performance.

2.3 Current AI technologies

The review emphasizes data, deep-learning architectures, and generative models as foundations for AI applications in creative industries, while noting important practical limitations.

  • AI and the Need for Data: Training databases must match the application domain in size, coverage, and statistics to improve model generalization and avoid overfitting.Reliable labels are also essential for supervised learning, including inputs resembling real use and ground-truth outputs.
  • AI and the Need for Data: Limited labeled datasets and class imbalance can bias models toward majority classes while treating minority-class features as noise.The review gives colorization as an example, where frequent sky-blue pixels can outweigh rarer pink flowers.
  • AI and the Need for Data: Data augmentation increases dataset volume and diversity through transformations such as cropping, flipping, translation, rotation, and scaling.These transformations can improve minority-class representation and help prevent overfitting without collecting new data.
  • Convolutional Neural Networks (CNNs): CNNs learn hierarchical visual features through convolutional layers, pooling, and nonlinear activations, progressing from edges to semantically meaningful representations.Architectures may use skip connections, bridge sections, or modified convolutions to capture local and global features.
  • Convolutional Neural Networks (CNNs): Encoder-decoder CNNs reconstruct outputs from encoded representations and support applications including style transfer and image restoration.U-Net uses contracting and expansive paths, while residual learning helps avoid vanishing gradients in deep architectures.
  • Generative Adversarial Networks (GANs): GAN-based image generation remains constrained by resolution and artefacts, despite near-realistic 1024×1024-pixel portraits.Many methods generate 256×256-pixel images, and transition-area artefacts persist in higher-resolution results.
  • Variational Autoencoders (VAEs): VAEs regularize the encoding distribution to support generation and are more stable during training than GANs, whereas GANs produce more realistic images.The decoder samples from the regularized latent distribution to generate new data.

3 AI for the Creative Industries

AI is being applied across creative production and analysis, but creative work remains distinct from routine tasks because it depends on originality, experience, and audience understanding.

  • AI for the Creative Industries: AI applications in creative industries include audio, image and video analysis, gaming, journalism, script writing, filmmaking, social media analysis, and marketing.Autotune is an early example that automatically fixes vocal intonation errors.
  • AI for the Creative Industries: Creative tasks require original thinking, extensive experience, and audience understanding, whereas repetitive production tasks are generally more amenable to automation.The review defines creativity as producing original and unusual ideas or making something new or imaginative.
  • AI for the Creative Industries: Twitter uses automatic cropping to create thumbnails showing salient image regions, while BBC experiments with automated framing, sequencing, and shot selection for live events.Initial BBC results indicate that the algorithm needs improvement before replacing human operators.
  • AI for the Creative Industries: The review organizes creative AI applications into content creation, information analysis, enhancement and post-production, information extraction and enhancement, and data compression.Applications may combine multiple categories within the same workflow.
  • AI for the Creative Industries: Post-production workflows can combine information extraction with enhancement methods such as super-resolution, colorization, 3D reconstruction, and frame-rate interpolation.These combinations can create new experiences, enhance existing material, or repurpose archives.

3.1 Content Creation

AI supports content creation across stories, games, language, and journalism, but current systems are most effective when working with humans or within constrained tasks.

  • Script and Movie Generation: AI has been used both to generate stories and to organize or search supporting data across creative domains.Applications span art, fiction, journalism, gaming, and entertainment.
  • Script and Movie Generation: RNN-based Benjamin generated the screenplay for Sunspring, but its unnatural storylines led to more fluid results when later used selectively with humans.The sequel It’s No Game used Benjamin in selected areas and in collaboration with humans.
  • Gaming: AI supports gaming through procedural content generation, interactive narratives, personalization, and synthetic environments.GameGAN was trained on 50,000 PAC-MAN episodes to create content for future level-layout generation.
  • Language and Text Generation: NLP systems support translation, image-to-text generation, and rewriting content for different channels or audiences.A conditioned GAN with an RNN generated understandable English from Chinese poems, though grammatical errors remained.
  • Journalism and Text Generation: Automated journalism tools assist human reporters with drafts, templates, story selection, and election coverage, improving production efficiency in constrained cases.Examples include Forbes’s Bertie, The Washington Post’s Heliograf, and Microsoft’s automated MSN selection systems.

3.1.3 Music Generation

AI music-generation systems analyze musical patterns and use neural generative models to transform transcribed ideas or composer-style mixtures into new music.

  • Music Generation: AI-assisted composition systems analyze musical patterns such as chords, tempo, and instrument-specific sequences to support music creation.The process uses machine-learning algorithms trained on musical data.
  • Music Generation: DeepJ composes music conditioned on a mixture of composer styles using a biaxial LSTM architecture.Other recent systems use LSTM-based generative models for music generation.
  • Music Generation: Music-generation research also includes direct waveform-synthesis methods that operate on audio waveforms rather than musical notation.These methods learn from or act directly on the audio signal.

3.1.5 Animation

AI methods automate animation, rendering, and immersive-media production, making some workflows easier, faster, and more realistic while introducing quality, cost, content, and physiological constraints for extended immersion.

  • Animation: AI automates animation workflows, making production easier, faster, and more realistic than traditional frame-by-frame methods.Pose estimation and face tracking support real-time animation.
  • Animation: Adobe Character Animator provides real-time lip synchronization, eye tracking, and gesture control from webcam and microphone inputs.The software has been adopted by Hollywood studios and online content creators.
  • Rendering: AI assists rendering through 3D-view synthesis, shading, dynamic texture synthesis, and realistic lighting for animation and visual effects.Some systems combine geometric computer vision, machine learning, and multiple depth sensors.
  • Immersive Media: VR market growth has not met earlier predictions because of equipment cost, content availability, and physiological effects from conflicting sensory interactions.The cited physiological effects are particularly relevant over extended time periods.
  • Immersive Media: AR, VR, and MR applications use AI to create more realistic content and to track and localize objects and users.Immersive experiences require high-quality animated worlds or 360-degree video, creating compression and visual-quality challenges.

3.1.7 Deepfakes

AI enables realistic manipulation and generation of text, images, and video, but deepfake systems still struggle with natural figures and can be used for harmful purposes.

  • Deepfakes: Text generators can produce coherent paragraphs with basic comprehension, translation, and summarization, but can also create fake news and abusive spam.The passage presents both creative capabilities and misuse risks.
  • Deepfakes: Deepfake technologies replace parts of media with synthetic content while preserving other elements such as hair, body, and action.Examples include face replacement and mouth-movement synthesis for altered speech.
  • Deepfakes: Deepfake generation primarily uses generative architectures including VAEs and GANs.These architectures support the rapid progress of deepfake creation.
  • Deepfakes: Perfectly natural deepfake figures remain challenging because generated faces may not blink naturally.Deepfakes have also been used for pornography, political distress, social unrest, and blackmail.
  • Image Generation: AI can generate image captions from visual content, whereas generating images from text is more complicated and has produced blurred foreground details in early work.Conditional GAN techniques were proposed to stabilize training and improve sample diversity.

3.2 Information Analysis

AI supports information analysis in creative industries by classifying, retrieving, summarizing, recommending, and targeting content using learned patterns from large datasets.

  • AI analyzes large training datasets to classify content and predict outcomes for advertising, film analysis, image retrieval, and video retrieval.These capabilities can help producers acquire information, analysts market products, and journalists retrieve investigation-relevant content.
  • Text categorization: Text categorization indexes documents and supports content analysis through methods including decision trees, support vector machines, CNNs, and RNNs.Modern approaches use embeddings, CNNs for text features, and LSTM-based RNNs for sentence-level prediction.
  • Content Retrieval: AI methods automate media annotation by analyzing audio, objects, and scenes, reducing reliance on time-consuming manual metadata creation.Automatic annotation supports retrieval of images, video, and other media.
  • Content Retrieval: Content-based retrieval matches image features such as points, lines, shapes, colors, and textures rather than relying on metadata.Music retrieval similarly converts sound features into queryable representations through tagging, humming, sound search, and acoustic fingerprinting.
  • Recommendation Services: Recommendation systems suggest content using content-based filtering, collaborative filtering, knowledge-based systems, or hybrids built from user and audience data.Applications include music and movie recommendations, playlist curation, advertising targeting, and film-release strategy.
  • Content Summarization: RNN-based attention models generate summaries ranging from abstracts and short paragraphs to personalized sentences, supporting recommendation services.Summarization extracts important document content into concise representations.

3.3 Content Enhancement and Post Production Workflows

AI-based enhancement and post-production methods restore, upscale, recolor, denoise, deblur, and complete creative media. These methods achieve strong results across several workflows but remain constrained by perceptual ambiguity, missing information, and difficult real-world distortions.

  • AI enhancement tools improve the quality and management of images, video, audio, and documents degraded by acquisition conditions, noise, or ageing.The paper frames these systems as assistive tools, particularly for mass-produced content.
  • Colorization: Colorization restores or adds color to archival, infrared, and aged visual media, using CNN-, encoder-decoder-, U-Net-, and GAN-based methods.Examples include World War One film restoration and infrared-to-natural-color conversion.
  • Colorization: Colorization remains challenging despite deep-learning advances, with video methods addressing temporal color flickering that can arise from frame-by-frame processing.The 2019 CVPRW challenge involved six teams, all using deep learning.
  • Super-resolution: Super-resolution upsamples low-resolution images or video for modern formats, but aliasing, sub-pixel frame shifts, and unavailable high-resolution information restrict reconstruction quality.Video methods can use successive frames to construct a high-resolution frame.
  • Super-resolution: Perceptual-loss SRGAN methods improve quality and sharpness, but high subjective quality and textural detail cannot simultaneously achieve low distortion loss such as MAE or PSNR.More complex networks generally produce better PSNR results, while many state-of-the-art methods use residual learning and ℓ1 losses.
  • Restoration: Video deblurring exploits information across neighboring frames through stacked inputs, synthetic blur training, recurrent networks, and temporal blending.Atmospheric turbulence remains difficult because random spatially varying perturbations combine blur, ripple, and intensity fluctuations.
  • Restoration: Deep-learning denoising methods now represent the state of the art, using architectures such as DnCNN, FFDNet, MWCNN, VNLnet, U-Net, ResNet, and DenseNet.These methods address audio, image, and video noise using residual maps, multiscale transforms, non-local patches, and spatiotemporal processing.
  • Inpainting: Inpainting estimates lost, damaged, or unwanted image and video regions, with GANs, CNNs, recurrent feedback, and semantic features supporting content replacement.DFNet outperformed other evaluated methods over a wide range of missing pixels.

3.4 Information Extraction and Enhancement

AI-based information extraction and enhancement supports recognition, tracking, multimodal interpretation, segmentation, and 2D-to-3D reconstruction across creative applications.

  • Information extraction: Deep learning extracts progressively abstract features from data, enabling recognition and subsequent enhancement or transformation of creative signals.Successive convolutional layers perform statistical analysis from low- to high-level representations.
  • Recognition: Object recognition systems identify scene objects using segmentation, with Mask R-CNN providing bounding boxes, classes, and pixel-level masks.Object recognition is supported by large labeled image datasets and deep-network methods such as Mask R-CNN and FPN.
  • Recognition: YOLO enables real-time object detection and tracking, while VOC benchmarks report mAP values of 80.8% for YOLOv3 on VOC2010 and 86.5% for NAS-Yolo on VOC2012.YOLO processes images frame by frame and predicts regional bounding boxes and probabilities.
  • Multimodal analysis: Deep learning supports speech, music, emotion, action, salient-object, and multi-object recognition tasks across audio, video, security, editing, VR, and compression applications.Examples include audio fingerprinting, speech interaction, emotion analysis, spatio-temporal action recognition, and visual-attention-based salient-object detection.
  • Tracking: Deep learning improves multi-object tracking by combining recognition, recurrent networks, and segmentation; ReMOTS achieves 83.9% mask-based MOT accuracy.ReMOTS fuses Mask R-CNN segmentation with a ResNet-101 backbone extended using FPN.
  • 3D reconstruction and rendering: Deep learning enables 3D reconstruction from images through volumetric, surface-based, and multi-plane representations, including 2D face-to-3D conversion using CNNs, autoencoders, and GANs.Single-image reconstruction is ill-posed, but models learn scene and object semantics from paired 2D RGB images and 3D ground truth.

3.5 Data Compression

Rising visual-media demand intensifies the tension between network capacity and required bitrate. Neural compression improves coding gains and supports perceptual reconstruction, but performance depends strongly on training data.

  • Motivation: Creative-media demand for higher-quality, more immersive video is increasing network pressure, with video projected to comprise 82 percent of internet traffic by 2022.The projected annual traffic was 4.8 zettabytes, creating tension between network capacity and required video bitrate.
  • Conventional compression: Traditional video compression exploits temporal redundancy through inter-frame motion prediction and compensation, unlike frame-independent image encoding.Image coding primarily exploits spatial redundancy through transform-based decorrelation, quantization, and symbol encoding.
  • Neural compression: Deep neural networks can achieve consistently greater coding gain than conventional approaches and optimize prediction, motion estimation, transforms, quantization, entropy coding, and filtering.Neural methods are being considered for components of mainstream standards including VVC and AV2.
  • Perceptual compression: ViSTRA2 achieves average BD-rate savings of 12.6% (PSNR) and 19.5% (VMAF) over HM, and 5.5% and 8.6% over VTM.The method adapts spatial resolution and effective bit depth, then upsamples at decoding with a deep convolutional neural network.
  • Limitation: Compression performance depends on training data, with large and diverse datasets improving coding gain across CNN-based coding tools.The BVI-DVC database supports improvements in tools including loop filtering and post-decoder enhancement.

4 Future Challenges for AI in the Creative Sector

The review identifies ethical, technical, and methodological challenges for AI in creative industries, while emphasizing strong performance in constrained, data-rich applications and the continuing need for human oversight.

  • 4.1 Ethical Issues, Fakes and Bias: AI can appear intelligent while lacking awareness of wider context, emotion, and social relationships, creating cultural and societal concerns.The review also links AI collaboration with human creativity to potential challenges for artists’ rights.
  • 4.1 Ethical Issues, Fakes and Bias: AI systems can generate fake images, videos, conversations, and other content, raising concerns about authorship, piracy, originality, and misuse.The authors argue that creations may be attributed to algorithm developers and require careful definition of ownership and exploitation.
  • 4.1 Ethical Issues, Fakes and Bias: Unrepresentative or uneven training data can produce biased and judgmental results, while changing data distributions may require periodic retraining.The review highlights risks in recommendations, face recognition, autofocus, and algorithmic decision-making.
  • 4.2 The human in the Loop – AI and Creativity: AI performs well for information extraction, analysis, and enhancement in constrained domains with abundant data, especially when deep learning connects low-level features to semantic meaning.These applications are presented as successes of AI supporting and enhancing creative-industry workflows.
  • 4.2 The human in the Loop – AI and Creativity: Current AI tools should be embedded in workflows with human feedback, checking, and critical decision-making rather than used as isolated black boxes.Human-reported faults can be fed back to adjust the model.
  • 4.2 The human in the Loop – AI and Creativity: Creative generation remains difficult because semantics map to many inconsistent outputs, while GANs are not yet robust enough to create consistently realistic or valuable results.Current GAN-based methods are also limited to relatively small images and can produce foreground–background transition artefacts.
  • 4.3 The future of AI technologies: Future creative AI depends on diverse, contextual, high-dimensional, and bias-free training data, although defining, cleaning, organizing, and labeling such data is resource intensive.The review notes that audience preferences and contemporary trends make creative quality difficult to define.
  • 4.3 The future of AI technologies: Deep Reinforcement Learning can learn through penalties and rewards, extract weak annotation information, and transfer semantic features to other tasks without manual labeling.The authors note strong results in games and Go, but say substantial research is needed because the human processes underlying creativity remain poorly understood.

5 Concluding Remarks

The paper reviews AI technologies and creative-industry applications, distinguishing AI as a creative tool from AI as an autonomous creator. It concludes that AI’s near-term value is greatest when it augments human creativity rather than attempts to replace it.

  • 5 Concluding Remarks: The review covers AI technologies and applications across content creation, information analysis, enhancement, extraction, and data compression.Its discussion emphasizes ML–AI methods’ dependence on computational hardware and large training datasets, particularly image and video data.
  • 5 Concluding Remarks: AI is expected to become more widely used as a tool or collaborative assistant supporting acquisition, production, post-production, delivery, and interactivity.The review distinguishes this role from AI’s potential to create original work independently.
  • 5 Concluding Remarks: Significant challenges remain for AI as the sole generator of original work, especially when tasks require external context, long chains of inference, or diverse and unbiased training data.The review therefore considers competition with human creatives for awards to be some way off.
  • 5 Concluding Remarks: Creative applications should remain human-centric and design AI to augment rather than replace human creativity.
Loading 2007.12391v6…