PaperlayerView exact revision on arXiv

Source-linked AI summary

PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion

Yiduo Guo, Zekai Zhang, Yaobo Liang, Dongyan Zhao, Nan Duan

arXiv:2311.01767v2cs.CL

TL;DR

Existing evaluations largely omit LLM completion of complex, multi-turn, multi-modal tasks in PowerPoint environments. PPTC addresses this gap with a benchmark and final-file evaluation system, finding that GPT-4 leads single-turn performance but struggles across complete sessions because of accumulated errors, long templates, and multi-modality.

  • Problem

    Existing LLM evaluations focus on basic language tasks or API translation rather than completing multi-turn, multi-modal instructions in complex PowerPoint environments.

    Source: §1 · p. 1

  • Method

    PPTC evaluates PowerPoint creation and editing across 279 multi-turn sessions, while PPTX-Match compares prediction-file attributes and position relations with label files.

    Source: §1 · pp. 2–3, §2.3 · p. 4

  • Results

    GPT-4 is strongest among the tested models, reaching 75.1% turn-based accuracy for creating new PPT files but only 22.7% session-based accuracy there and 6.0% for editing templates.

    Source: §1 · p. 3, §4.5 · p. 8

  • Takeaways & Limitations

    Error accumulation, long PPT-template processing, and multi-modality perception remain key challenges for LLM-based PowerPoint task completion.

    Source: §7 · p. 11

    The benchmark covers instructions executable through API sequences and excludes subjective visual evaluation and non-API operations such as drawing by mouse.

    Source: §8 · p. 12

Abstract

from arXiv · show

Recent evaluations of Large Language Models (LLMs) have centered around testing their zero-shot/few-shot capabilities for basic natural language tasks and their ability to translate instructions into tool APIs. However, the evaluation of LLMs utilizing complex tools to finish multi-turn, multi-modal instructions in a complex multi-modal environment has not been investigated. To address this gap, we introduce the PowerPoint Task Completion (PPTC) benchmark to assess LLMs' ability to create and edit PPT files based on user instructions. It contains 279 multi-turn sessions covering diverse topics and hundreds of instructions involving multi-modal operations. We also propose the PPTX-Match Evaluation System that evaluates if LLMs finish the instruction based on the prediction file rather than the label API sequence, thus it supports various LLM-generated API sequences. We measure 3 closed LLMs and 6 open-source LLMs. The results show that GPT-4 outperforms other LLMs with 75.1\% accuracy in single-turn dialogue testing but faces challenges in completing entire sessions, achieving just 6\% session accuracy. We find three main error causes in our benchmark: error accumulation in the multi-turn session, long PPT template processing, and multi-modality perception. These pose great challenges for future LLM and agent systems. We release the data, code, and evaluation system of PPTC at \url{https://github.com/gydpku/PPTC}.

1 Introduction

PPTC evaluates LLMs completing multi-turn, multi-modal PowerPoint tasks using final-file matching rather than API-sequence matching. Results show GPT-4 leads but struggles with session-level completion, long templates, and non-textual operations.

Benchmark

  • PPTC contains 279 multi-turn sessions with varying difficulty, hundreds of multi-modal instructions, and tasks requiring single or multiple APIs.Sessions contain 2 to 17 turns, and over half of the instructions require multiple APIs.Source: §1 · pp. 2–3

Evaluation

  • PPTX-Match evaluates the produced PowerPoint file’s attributes and position relations, allowing any API sequence that reaches the correct final status.The system checks prediction files against label files rather than assuming one unique correct API sequence.Source: §1 · p. 2

Results

  • GPT-4 achieves 75.1% turn-based accuracy for creating new PPT files but only 22.7% session-based accuracy, while editing-template session accuracy falls to 6.0%.These results illustrate the impact of accumulated errors across multi-turn sessions.Source: §1 · p. 3
  • The benchmark measures three closed-source and six open-source LLMs, with GPT-4 identified as the strongest model among those tested.The evaluation includes GPT-4, ChatGPT, Davince-003, and open-source models such as LLaMa-2.Source: §1 · p. 3

Challenges

  • GPT-4 reaches 38.1% turn-based accuracy on editing tasks and 24% accuracy on instructions involving non-text modality operations, especially position-related operations.The findings identify long PPT-template processing and multi-modality perception as major challenges.Source: §1 · p. 3

2 PPTC Benchmark

PPTC benchmarks multi-turn PowerPoint creation and editing with diverse multimodal instructions, and evaluates completed files rather than requiring one API sequence. Its design combines curated sessions, validated reference outputs, attribute-level matching, and varied session, API, and modality complexity.

2.1 Benchmark Overview

  • PPTC contains 279 multi-turn sessions spanning new-PPT creation and existing-template editing, with sessions ranging from 2 to 17 turns.The benchmark includes 229 creation sessions and 50 template-editing sessions.Source: §2.1 · p. 3, §2.4 · p. 5
  • Each instruction requires contextual understanding, PPT-content analysis, and selection of appropriate APIs before executing an API sequence into a resulting PPT file.The framework supplies PPT content through a reader and exposes Python implementations of PPT operations because LLMs cannot directly process PPT files or operate the software interface.Source: §2.1 · p. 3

2.2 Benchmark Collection

  • The benchmark is collected from experienced PowerPoint workers, then quality-checked for instruction clarity, API execution, and goal achievement in the label files.Workers create feasible API sequences and label files, while the authors review and revise errors found during validation.Source: §2.2 · p. 4

2.3 PPTX-Match Evaluation System

  • PPTX-Match evaluates prediction files against label files by exact-matching non-position attributes and checking predefined spatial relations for position attributes.Completion requires no incorrect non-position matches and no violated position rules.Source: §2.3 · pp. 4–5

2.4 Benchmark Statistics Analysis

  • Session complexity spans 3–13 turns at the 5th–95th percentiles, while API sequences span 1–5 APIs at those percentiles and reach 29 APIs at maximum.Long sessions can propagate earlier errors, and long API sequences require sub-goal decomposition, API selection, and reliable ordering.Source: §2.4 · p. 5
  • Multimodal coverage includes hundreds of instructions, with 120 involving charts and 292 involving position operations.Position instructions require understanding spatial information and applying modality-specific APIs; one instruction may involve multiple modalities.Source: §2.4 · p. 5, Figure 3 · p. 5

3 Algorithms

The algorithms section addresses complex PPT instructions through planning and reduces prompt redundancy through content and API selection. It considers stepwise reasoning methods alongside filtering based on relevance.

3 Algorithms

  • The paper categorizes performance-enhancement methods into planning algorithms for decomposing instructions and selection algorithms for choosing useful environmental information or APIs.This organization targets complex instructions requiring multiple intermediate steps and prompts containing redundant information.Source: §3 · p. 5, §3.1 · p. 5, §3.2 · p. 6

3.1 Planning Algorithms

  • Zero-shot-CoT and Tree of Thoughts are considered as planning methods for generating intermediate reasoning or evaluating alternative reasoning paths.Zero-shot-CoT prompts step-by-step reasoning, whereas Tree of Thoughts represents reasoning states as a tree and generates evaluations or votes.Source: §3.1 · p. 5

3.2 Selection Algorithms

  • Content selection prompts the LLM to retain selected PPT shapes, while API selection ranks API descriptions by embedding cosine similarity and retains the top k APIs.Both methods replace the full corresponding input with a filtered subset during API-sequence generation.Source: §3.2 · p. 6

4 Experiments

The experiments evaluate nine LLMs on turn- and session-based PowerPoint completion, comparing models, prompting algorithms, and challenges in multi-turn, long-template, and multimodal settings.

4.1 Large Language Models Selected for Evaluation

  • The benchmark evaluates three closed-source and six open-source LLMs using turn-based and session-based accuracy, with token and API counts as cost measures.Experiments use deterministic generation with temperature zero and a 2048-token maximum for the evaluated APIs.Source: §4.1 · p. 6, §4.2.1 · p. 7, §4.3 · p. 8

4.2.1 Turn-Based and Session-Based Evaluations

  • Turn-based evaluation assumes previous turns are correct, whereas session-based evaluation feeds the model’s own prior outputs and resulting PPT files into subsequent turns.The latter therefore measures complete multi-turn sessions and allows earlier errors to influence later turns.Source: §4.2.1 · p. 7, Figure 4 · p. 7

4.4 Main results

  • GPT-4 consistently outperforms other evaluated LLMs, reaching 75.1% turn-based accuracy for creating new PPT files.It also incurs lower API cost than other closed-source LLMs, but the highest token expense when editing PPT templates.Source: §4.4 · p. 8
  • Open-source models trail closed-source models, while Code-LLaMa, WizardLM, and Vicuna improve LLaMa-2 performance through code pretraining or instruction fine-tuning.Code-LLaMa improves LLaMa-2’s turn-based accuracy on creating new PPT files by 20.4%.Source: §4.4 · p. 8
  • Planning and selection methods improve GPT-4’s turn-based performance, but complex ToT costs 5–10 times more tokens than zero-shot CoT without outperforming it.Content and API selection improve turn-based performance by 1–5% and can reduce API-related token cost, while session-based gains are absent or slight.Source: §4.4 · p. 8

4.5 Three challenges in our PPTC benchmark

  • The main challenges are error accumulation across sessions, processing long PPT templates, and multimodal perception, especially spatial operations.GPT-4 reaches only 22.7% session accuracy for creating new PPT files, 6.0% for editing templates, and 24% accuracy on position-related instructions.Source: §4.5 · pp. 8–9

5 Analysis

The analysis examines GPT-4’s error patterns, model-size effects, and dialogue-history ablations in PPTC. Larger models improve turn accuracy, while dialogue history helps, but multi-turn session completion remains difficult.

5.1 Error Analysis of GPT-4 in our benchmark

  • Figure 5 reports average turn-based accuracy for chart, table, picture, position, and pure-text instructions in the creating-new-PPT-file task, while excluding task 2 accuracy.Source: Figure 5 · p. 10
  • GPT-4 errors include position mistakes, unavailable-API hallucinations, PPT-content misunderstanding, and violations of PowerPoint task rules.Session failures also accumulate across turns, with models sometimes repeating earlier infeasible APIs.Source: §5.1 · p. 10

5.2 Does bigger LLM work better on PPTC?

  • Larger LLaMa-2-chat models raise creating-new-PPT turn accuracy from 13.2% at 7B to 30.1% at 70B, without a clear session-accuracy correlation.The authors suggest that completing more intermediate steps does not ensure completion of the entire session.Source: §5.2 · p. 10

5.3 Does dialogue history help LLMs to generate the API sequence?

  • Removing dialogue history lowers GPT-4 accuracy from 75.1% to 73.1% for new-PPT creation and decreases editing-template performance by 6.2%.Dialogue history supplies background and demonstrates correct API usage.Source: §5.3 · pp. 10–11

6 Related Works

Related work evaluates LLMs on language tasks, API generation, embodied task completion, web interaction, and complex tool-assisted environments. This paper distinguishes itself by focusing on task completion in office software.

LLM evaluation and prompting

  • Prior LLM evaluations emphasize zero-shot or few-shot language abilities and translating instructions into tool APIs.Examples include calculator, RapidAPIs, and Hugging Face API use.Source: §6 · p. 11

Task completion benchmarks

  • Task-completion benchmarks test action sequences for robots, websites, Android environments, autonomous agents, and API selection.The cited benchmarks include SayCan, VirtualHome, WebShop, Android in the wild, APIBench, ToolBench, AgentBench, and WebArena.Source: §6 · p. 11

AI assistant systems for complex task completion

  • Complex AI assistants connect LLMs with tools or train end-to-end models using visual, environmental, sensor, or API information.Examples include TaskMatrix, Visual ChatGPT, Hugging-GPT, Voyager, Gorilla, and PaLM-E.Source: §6 · p. 11
  • Unlike these systems, the paper focuses on AI assistant task completion within office software.The target environment is PowerPoint rather than robotics, web navigation, or general API generation.Source: §6 · p. 11

7 Conclusion

The paper introduces PPTC and its evaluation system for measuring LLM completion of PowerPoint instructions. GPT-4 is strongest but still performs poorly on entire sessions, and the analysis identifies three limiting error factors.

7 Conclusion

  • PPTC measures LLM ability to complete user instructions within PowerPoint, using hundreds of multi-turn sessions and varying instruction difficulty.The benchmark targets task completion in the PowerPoint software context.Source: §7 · p. 11
  • The PPTX-evaluation system compares different LLMs, while results identify GPT-4 as strongest but weak at completing entire sessions.The conclusion reports a gap between turn-level strength and full-session completion.Source: §7 · p. 11
  • Three main error factors limit performance, and the benchmark and findings are intended to help design better AI task-completion assistants.The conclusion frames the findings as resources for future system design.Source: §7 · p. 11

8 Limitations

The benchmark excludes subjective instructions and operations that cannot be completed through direct API execution.

8 Limitations

  • PPTC excludes subjective goals such as making a slide more beautiful because generated-file quality is difficult to evaluate automatically.It also excludes non-API operations such as drawing by dragging the mouse.Source: §8 · p. 12

A The API Reference File

The API reference lists the available APIs and provides 49 feasible APIs for task completion.

B The Prompt for Content Selection Algorithm

The content-selection prompt is paired with reference API material covering slide, text, and image-related operations.

Related papers

Loading 2311.01767v2…