Source-linked AI summary
Jigsaw: Large Language Models meet Program Synthesis
Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, Rahul Sharma
TL;DR
PTLMs can generate code from natural-language intent but provide no correctness guarantees because they treat code as text rather than understanding its syntax and semantics. Jigsaw augments black-box PTLMs with multi-modal inputs, program analysis, and synthesis-based pre- and post-processing, and the authors report improved accuracy and learning from user feedback on Pandas tasks.
Problem
PTLM-generated code lacks correctness and quality guarantees because these models do not understand program syntax or semantics.
Method
Jigsaw augments black-box PTLMs with multi-modal specifications, program analysis, and synthesis-based pre-processing and post-processing for code synthesis.
Results
Jigsaw yields significantly higher accuracy than baselines on two Pandas datasets and improves with user feedback over time.
Takeaways & Limitations
The authors conclude that post-processing and multi-modal synthesis can improve PTLM-generated code quality for large APIs such as Pandas.
Takeaways & Limitations
The study used only 25 participants, and its task variations may not represent real-world task variations; large-scale productivity evaluation was beyond scope.
Abstract
from arXiv · showhide
Large pre-trained language models such as GPT-3, Codex, and Google's language model are now capable of generating code from natural language specifications of programmer intent. We view these developments with a mixture of optimism and caution. On the optimistic side, such large language models have the potential to improve productivity by providing an automated AI pair programmer for every programmer in the world. On the cautionary side, since these large language models do not understand program semantics, they offer no guarantees about quality of the suggested code. In this paper, we present an approach to augment these large language models with post-processing steps based on program analysis and synthesis techniques, that understand the syntax and semantics of programs. Further, we show that such techniques can make use of user feedback and improve with usage. We present our experiences from building and evaluating such a tool jigsaw, targeted at synthesizing code for using Python Pandas API using multi-modal inputs. Our experience suggests that as these large language models evolve for synthesizing code from intent, jigsaw has an important role to play in improving the accuracy of the systems.
1 INTRODUCTION
Jigsaw augments black-box pre-trained language models with program analysis, synthesis, and multi-modal specifications to improve Pandas code synthesis. It also incorporates user feedback so the system can improve with usage.
- System design: Jigsaw combines natural-language intent with test cases or input-output examples to produce code snippets.The system is implemented as a Jupyter notebook extension with interactive cards for editing inputs, inspecting results, and copying outputs.
- Evaluation: Jigsaw targets Python Pandas code and is evaluated on the author-created PandasEval1 dataset and a hackathon dataset created by 25 users.The evaluation measures overall accuracy and component accuracy across the two datasets.
- Results: Feedback from the first hackathon session enabled users to solve about 10% more tasks in the second session.The authors attribute this improvement to learning changes in the pre-processing and post-processing modules.
- Results: Using GPT-3 and Codex, the authors report higher accuracy than baselines on both datasets.The paper also reports gains from learning from user feedback over time.
- System design: Its architecture adds pre-processing and post-processing around black-box PTLMs to address syntax, argument, reference, and semantic errors.Post-processing checks and transforms generated code so it passes supplied tests and other quality checks.
2 JIGSAW OVERVIEW
Jigsaw is an interactive, multi-modal code-synthesis system that treats PTLMs as black boxes and uses domain-specific analysis and repair to improve generated Pandas code. Its post-processing relies on I/O examples and learned transformations to correct recurring errors.
- Overview: Jigsaw accepts natural-language descriptions and test cases, while its notebook interface lets users refine ambiguous specifications and provide feedback.The interactive workflow supports both developer interaction and system improvement.
- Design principles: The system treats GPT-3 and Codex as black boxes that are queried rather than fine-tuned.This supports plug-and-play use as models evolve and keeps domain-specific improvements complementary to Jigsaw.
- Pipeline: Jigsaw uses pre-processing to contextualize queries and post-processing to check, transform, and repair PTLM-generated code.The post-processing module uses I/O examples to select appropriate transformations and search API arguments.
- Pandas setting: The system is instantiated for Python Pandas data-transformation tasks and invoked from Jupyter Notebook through a magic command.Example interactions include loading a CSV file and modifying a dataframe column.
- Failure modes: Context selection affects generated-code quality, while Jigsaw identifies recurring reference, argument, and semantic errors for repair.Semantic repair includes enumerative argument search and AST-to-AST transformations learned from feedback.
3 JIGSAW ARCHITECTURE
Jigsaw uses PTLM prompting, correctness checks, enumerative search, and learned AST transformations to synthesize and repair Pandas code. Its context bank and repair rules can be updated from user feedback and are intended to generalize across APIs and languages.
- Pre-trained language models: GPT-3-style prompting supplies question-answer examples before a query, with the examples defining the task the model should perform.For Jigsaw, the examples pair English intent descriptions with desired program snippets.
- Pre-processing: The pre-processing module selects similar intent-code pairs from an offline context bank to form the PTLM context.The bank is built from Pandas documentation, tutorials, and other examples, and user feedback can expand it.
- Correctness checking: Post-processing executes candidate snippets on supplied inputs and checks whether outputs match specified outputs before applying repairs.The checks can also be extended with static analyses for security vulnerabilities and other errors.
- Semantic repair: Variable-name repair searches possible in-scope variable permutations, while argument repair searches inferred argument spaces for the generated API-function sequence.These transformations use information from natural-language descriptions, test cases, and variables in scope.
- Semantic repair: Jigsaw seeds enumerative search with API functions from incorrect snippets and learns AST-to-AST transformations from user feedback.Argument transformations and AST-to-AST transformations are grouped as Semantic Repair in the experiments.
- Learning from usage: The context bank and transformation rules can grow from usage data, while the underlying design is described as extensible across APIs and languages.The paper notes that API-specific transformation rules can be learned from users.
4 DATASETS
The evaluation uses two datasets of concise Python Pandas tasks, including user-generated natural-language variations from a two-session hackathon. These variations preserve some imprecision to study realistic user interaction and learning from feedback.
- PandasEval1: PandasEval1 contains 68 Python Pandas tasks solvable by composing at most 2–3 Pandas functions.The authors created it from queries found in online forums such as StackOverflow.
- PandasEval2: PandasEval2 contains 21 Pandas tasks posed to 25 users across two hackathon sessions.Users translated pictorial task illustrations into natural-language queries.
- Data curation: User queries sometimes misidentify the intended update, such as changing both the country and IATA columns instead of only the country column.The example concerns replacing France and Paris values in a dataframe.
- Data curation: The dataset retains imprecise and partially correct user queries after removing clearly incorrect formulations.This preserves realistic variation in how users express task intent.
- PandasEval2: PandasEval2_S1 and PandasEval2_S2 differ through task variants designed to test whether feedback from the first session improves later usage.Three tasks were unchanged, nine differed in constants, and nine differed in arguments.
5 EXPERIMENTS
Jigsaw is evaluated against black-box PTLMs and other synthesis methods on two Pandas datasets, including offline accuracy, component utility, robustness, and learning from user feedback. The experiments show that contextual preprocessing and program-analysis post-processing improve accuracy, learned updates improve later-session performance, and users complete more tasks.
- Offline evaluation: Jigsaw improves black-box PTLM accuracy by 15%-40% across both datasets, both PTLMs, and context-selection strategies.The comparison is between PTLM and Semantic Repair columns in Table 2.
- Offline evaluation: Useful Transformer context improves PTLM performance over NO-CONTEXT by about 5% without post-processing and up to 15% with post-processing for Codex.For GPT-3, the context advantage is significant on PandasEval2 but statistically insignificant on PandasEval1.
- Offline evaluation: Variable Name transformations improve accuracy by 10%-30% for Codex and 10%-15% for GPT-3, while Semantic Repair adds about 5% for Codex and 6%-11% for GPT-3.Semantic Repair combines Variable Name transformations with Argument and AST-to-AST transformations.
- Temporal evaluation: Updated context and transformation modules improve PandasEval2_S2 accuracy by over 30% for GPT-3 and about 15% for Codex despite the second session being more challenging.Updates were learned from first-session user submissions and feedback.
- Temporal evaluation: Users solved 71% of tasks in session one and 82% in session two, while Jigsaw post-processing remained faster than PTLM inference.Codex inference averaged about 7 seconds versus under 3 seconds for post-processing; GPT-3 took 30-40 seconds versus under 10 seconds.
6 THREATS TO VALIDITY
The evaluation uses manually assembled datasets and a small hackathon study, limiting how broadly the findings generalize. Comparisons with AutoPandas also differ in inputs, runtime constraints, and manual correctness assessment.
- The Pandas datasets cover common programming patterns but are not representative of all Pandas programs in the wild.
- The two-session hackathon may not represent real-world task variation, and its 25 participants cannot establish statistically significant productivity gains at scale.
- Jigsaw receives both natural-language descriptions and I/O examples, whereas AutoPandas receives only I/O examples.
- The AutoPandas comparison uses a three-minute timeout, while Jigsaw takes less than a minute per task; longer AutoPandas timeouts might improve its performance but reduce interactivity.
7 RELATED WORK
Related systems differ from Jigsaw in API scale, model type, input modality, and ability to learn from user feedback. Jigsaw combines large language models with multimodal synthesis and program repair ideas.
- Prior work includes large-API systems without large models, large-model systems without multimodal specifications, and multimodal systems for small APIs.
- TDE relies on rich type information absent from Pandas, while AutoPandas synthesizes Pandas code from I/O examples using graph neural networks and enumerative search.
- AutoPandas and TF-coder use smaller specific models and lack mechanisms for incorporating user feedback.
- Jigsaw is related to program-repair work because it learns code transformations to fix language-model outputs, and its interface draws on notebook visualization tools.
8 CONCLUSION AND FUTURE WORK
Jigsaw combines multimodal synthesis with post-processing to improve code generated for large APIs, but its current notion of quality and specification remains limited. Future work must address broader quality and richer requirements.
- Jigsaw’s post-processing improves code generated by PTLMs, and its multimodal synthesis outperforms baselines using only PTLMs or only I/O examples.
- The current evaluation largely equates code quality with correctness on I/O examples, omitting performance, security vulnerabilities, and licensing attribution.
- Natural-language intent and I/O examples can be weak or ambiguous; richer specifications such as preconditions, postconditions, invariants, and resource bounds may be needed.