Source-linked AI summary
Using Poly-Encoders for Computationally Efficient Automated Creativity Assessment
Sam Grouchnikov, Phillip Gregory, Jiho Noh
TL;DR
Automated creativity assessment remains subjective, costly, and difficult to deploy reliably. This study fine-tunes Poly-Encoders with BERT-family encoders on human-rated SCTT responses and finds comparable performance to large language models with substantially greater efficiency. The findings support Poly-Encoders as a practical option for scalable creativity assessment, especially in educational settings, while retaining important limitations.
Problem
Creativity assessment is subjective, resource-intensive, and difficult to automate while preserving alignment with human creative intent.
Method
The study fine-tunes a Poly-Encoder on approximately 18,000 human-labeled SCTT responses, using BERT-family encoders and a regression-based scoring architecture.
Results
Poly-Encoders achieve comparable performance to large language models while delivering efficient CPU scoring, with reported times of 0.010 and 0.0022 seconds per candidate.
Takeaways & Limitations
Poly-Encoders provide a practical framework for classroom implementation through standard-hardware accessibility and local-run privacy benefits.
Takeaways & Limitations
The study is limited by SCTT label noise, inability to generalize to unseen prompts, potential language and length biases, GPU requirements, and limited interpretability.
Abstract
from arXiv · showhide
Automated creativity assessment has been a long standing challenge, with traditional methods often being resource intensive or lacking practical accuracy. We introduce a novel approach by using Poly-Encoder for computationally efficient and accurate automated creativity assessment. We fine-tuned a Poly-Encoder on a public dataset from the Scientific Creative Thinking Test, comprised of approximately 18,000 human-rated question responses. Our method leverages small pre-trained BERT encoders, achieving performance comparable to fine-tuned Large Language Models while significantly reducing computational demands. Experiments with the BERT-family models and poly-code counts achieved Pearson correlations of up to r = 0.74, 95% CI [0.73, 0.75] with human raters, matching the performance of resource intensive LLMs. This study bridges the gap between high performance and computational efficiency, potentially enabling widespread implementation of automated creativity assessment on accessible consumer-grade hardware. With some limitations, our findings suggest that Poly-Encoders are a promising alternative to LLMs for practical, scalable creativity assessment in various contexts, especially educational.
1 Introduction
Creativity assessment is difficult because human evaluation is subjective, costly, and inconsistent. The study asks whether AI can automate assessment while preserving alignment with human creative intent and practical deployment constraints.
- Human creativity assessment is subjective, time-intensive, and resource-intensive, with inconsistency often emerging between evaluators.
- Effective AI-based assessment must manage test complexity, deployment efficiency, and alignment with human intent.
- Scientific creativity assessment must capture reasoning while aligning with psychometric criteria such as fluency, originality, and elaboration.
- Cloud-based LLM assessment can introduce data-privacy risks and recurring API costs that may be prohibitive for school districts.
- The study investigates how Poly-Encoders can assess creativity rather than similarity and whether they can run efficiently on consumer hardware.
2 Related Work
Earlier semantic-distance models missed sentence-level nuance, while fine-tuned LLMs improved scoring but remained computationally demanding. Poly-Encoders are presented as an efficiency–accuracy middle ground for creativity assessment.
- Semantic-distance models such as SemDis and GloVe fail to capture semantic nuance and sentence-level context needed for creativity assessment.
- Prompt-based GPT-4 methods achieved Pearson correlations between 0.2 and 0.67 with human ratings for novelty and feasibility.
- Fine-tuned GPT-3 and T5 models reached correlations up to r = 0.81 on 27,000 Alternative Uses Task responses.
- Fine-tuned LLaMA-2-7b reached r = 0.74 on approximately 18,000 Scientific Creative Thinking Test responses, but LLM methods remained computationally sub-optimal.
- Poly-Encoders use learnable global attention codes with lightweight BERT encoders to provide semantic representations deployable on consumer-grade hardware.
3 Methods
The study trains Poly-Encoders on human-rated scientific creativity responses using BERT-family encoders and tests architectural and poly-code configurations. A lightweight regression head replaces similarity scoring for deeper creativity scoring.
- Datasets and Training Schematics: The model was trained on approximately 18,000 SCTT responses paired with human creativity labels across 15 prompts.Prompts served as context inputs and responses as candidate inputs.
- BERT for Context/Candidate Embeddings: BERT-family models were explored for context and candidate encodings to examine the trade-off between representation quality and computational efficiency.
- Poly-Code Comparisons: Poly-code counts of 64, 128, 256, and 512 were tested while holding other hyperparameters constant.
- Architectural Tweaks: The modified architecture passes poly-context embeddings, candidate vectors, and their element-wise products through a lightweight regression head.This replaces the original dot-product similarity mechanism and allows nonlinear scoring.
- Architectural Tweaks: Figure 1 compares the original Poly-Encoder architecture with the study’s modified version.
4 Results
BERT-family and poly-code experiments show strong alignment with human creativity ratings, while efficiency is the principal advantage over larger models. The best configurations match resource-intensive LLM performance with rapid CPU scoring.
- BERT Comparisons: DeBERTa-v3-Large achieved the strongest correlation with human creativity scores among tested encoders, while RoBERTa variants produced comparable results.The reported difference between DeBERTa and RoBERTa variants was significant across poly-code counts.
- Poly-Code Count Comparisons: DeBERTa-v3-Large achieved the same correlation as LLaMA-2-7B while using a fraction of its resources.
- Prediction Alignment: The model’s predicted scores showed strong alignment with human ground-truth ratings, correctly predicting most creativity scores within a reasonable threshold.
- Prediction Alignment: DeBERTa-v3-Large produced a mean absolute error of 0.0077 between predicted and ground-truth scores.
- Efficiency, Scoring Times: 0.010 and 0.0022 seconds were the final per-candidate scoring times for DeBERTa-v3-Large and RoBERTa-Model, respectively, on an Intel i5 CPU.
5 Conclusions
Poly-Encoders achieve comparable performance to large language models with substantially greater computational efficiency, supporting practical classroom deployment on standard hardware. However, limitations include dataset label noise, poor generalization to unseen prompts, potential language-related bias, GPU requirements, and limited interpretability.
- Performance and Efficiency: Sub-second scoring on consumer-grade hardware makes Poly-Encoders suitable for real-world classroom deployment with limited resources.The architecture encodes prompts once and uses lightweight BERT-based encoders to reduce memory requirements.
- Performance Analysis: Figure 3 compares predicted with ground-truth scores and reports cumulative accuracy across error thresholds.
- Poly-Code Impact: Variations in poly-code count do not significantly affect performance, while BERT encoder family and size do.Differences between BERT families with the same poly-code counts were significant at p < 0.05.
- Implications and Future Work: Poly-Encoders provide a framework for classroom implementation through standard-hardware accessibility and local-run privacy benefits.Future work includes multidimensional creativity assessment, generalization to the Alternative Uses Task, and textual justifications for scores.
- Limitations: The study is limited by SCTT label noise, inability to generalize to unseen prompts, language-related biases, GPU requirements, and black-box scoring.The paper identifies interpretability as a crucial area for future investigation because scores without transparent reasoning provide incomplete understanding.