Octopus: On-device language model for function calling of software APIs

• External APIs extend LLMs with up-to-date information and specialized functionality, addressing limitations in real-time updates and specialized tasks.Source: §1 · p. 1
• Smaller task-oriented LLMs reduce the computational burden of large models but face greater risks of hallucination and formatting errors.Source: §1 · pp. 1–2
• Octopus trains compact models on more than 30,000 APIs using curriculum learning to improve selection among similar functions.Source: §1 · p. 2
• Across selected base models, the approach outperforms GPT-4 benchmarks and supports practical external-API integration on platforms including mobile devices.Source: §1 · p. 2
• Conditional masking enforces desired output formats, improves accuracy, reduces validation loss, and preserves inference speed.Source: §1 · p. 2

• LLM tool integration has progressed from model-specific fine-tuning toward demonstrations, code interpreters, retrieval frameworks, simulated environments, API frameworks, and advanced reasoning.Source: §2 · p. 2
• Iterative dataset refinement improves prompts, response quality, chain-of-thought processes, and function-calling accuracy.Source: §2 · pp. 2–3
• Software applications require rigid structures such as JSON, but existing parsers do not resolve all function-call formatting failures.Source: §2 · p. 3

3 Methodology

• The methodology combines dataset preparation, model training, and conditional-mask inference to improve function-call model development.Source: §3 · p. 3

3.1 Dataset collection and refinement

• Approximately 30,000 frequently used APIs were collected from RapidAPI Hub and processed through documentation extraction and refinement.Source: §3.1 · p. 3
• API examples were converted into structured Python-style functions containing concise names, arguments, and descriptions to support smaller models’ code-reasoning capabilities.Source: §3.1 · pp. 3–4
• The dataset uses irrelevant and semantically similar functions as hard negatives, with ranks 5–10 selected to make function discrimination more challenging.Source: §3.1 · p. 4
• Positive single-API queries include chain-of-thought responses, while GPT-4 verification removes approximately 1,000 invalid data points from a dataset of about 150,000 examples.Source: §3.1 · p. 4

3.2 Octopus

• Octopus fine-tunes CodeLlama 7B, Gemma 2B and 7B, and Stable Code LM 3B using a shared template, LoRA, and 8-bit quantization.Source: §3.2 · p. 5

3.3 Inference using conditional mask

• Conditional masking sets incompatible token positions to 0 and valid positions to 1, using known function names and type-aware rules to constrain generation.Source: §3.3 · pp. 5–6
• The conditional mask ensures format-valid outputs and significantly improves function-call robustness, while the loss curves track training and validation behavior.Source: §3.3 · p. 6, Figure 2 · p. 6

4 LLM Evaluation for Function Calling

• The evaluation compares Octopus with GPT-4 and GPT-3.5 using zero-shot baselines and examines the effect of retrieval techniques during training.Greedy search was used for pretrained models because precise function-name and parameter selection was required.Source: §4 · p. 6, §4.2 · p. 7

4.1 Evaluation dataset and benchmark

• The benchmark samples queries for four randomly selected APIs, balances solvable and unsolvable cases 1:1, and uses human-annotated ground truth for exact function-name and argument matching.Format errors were ignored for models not trained on the dataset, including GPT-3.5.Source: §4.1 · p. 7

4.2 Performance without the conditional mask

• GPT-4 achieved the highest accuracy without masking, with inaccurate parameters—not function-name identification—as the primary source of errors.Examples include correcting a misspelled function name or generating an unintended country parameter.Source: §4.2 · p. 7

4.3 Performance with the conditional mask

• Conditional masking improves parameter generation by preventing unexpected values for enum-like inputs such as country names.The technique is evaluated through the function-call accuracy reported in Table 1.Source: §4.3 · p. 7, Table 1 · p. 7
• Two 7B Octopus models surpassed GPT-4 after conditional masking, while the mask could not be applied to GPT-3.5 or GPT-4 because their APIs expose no logits.The comparison is reported for function-call accuracy with and without the conditional mask.Source: Table 1 · p. 7, §4.3 · p. 8

5 Conclusion

• The framework trains models on practical software APIs, refines data and prompts with negative sampling and curriculum learning, and evaluates API calling against GPT-4.Conditional masking is introduced to address mismatched output formats.Source: §5 · p. 8

A.1 Impact of conditional masking on inference performance

• The validation-loss analysis compares causal-language-model inference with and without a conditional mask.The appendix introduces the unmasked validation loss before deriving the masked case.Source: §A.1 · p. 11
• Conditional masking partitions the vocabulary into unmasked V1 and masked V2, assuming the true label belongs to V1 during inference.The masked validation-loss expression is derived from this partition.Source: §A.1 · p. 12
• Conditional masking reduces validation loss by restricting the model’s prediction space while preserving the correct label.The appendix explicitly states that validation loss is reduced when masking is applied.Source: §A.1 · p. 12
• Precision without masking is defined by whether the maximum predicted index matches the ground-truth label index.The indicator function returns 1 when the condition holds and 0 otherwise.Source: §A.1 · p. 12
• With masking, argmax prediction is constrained to V1, increasing the likelihood that the selected index matches the ground-truth index when yi ∈ V1.The appendix concludes that this focused search improves precision.Source: §A.1 · p. 12

B.1 Dataset template

• The template instructs the model to identify relevant functions first, then infer their arguments from the user query.It presents available functions for fetching data to answer user questions.Source: §B.1 · p. 12
• The template includes a no_relevant_function fallback for queries that cannot be answered by any provided function.The fallback receives the unresolved user query as its argument.Source: §B.1 · p. 12
• The function definitions document URL arguments as strings representing the web address of the requested resource.This argument pattern appears across the download-function examples.Source: §B.1 · p. 13
• Download examples map content-specific requests to functions such as insta_download_url, pinterest_video_dl_api, youtube_downloader, and facebook_dl_link.The examples use URL-like arguments to retrieve or save online content for offline viewing.Source: §B.1 · pp. 12–13
• A worked example calls insta_download_url with an Instagram post URL to obtain download access for offline viewing.The accompanying reasoning explains that the post URL is passed as the function argument.Source: §B.1 · p. 13

• Insertion creates missing character nodes, advances through the trie, and marks the final node as a complete word.Source: §B.2 · p. 14
• Prefix validation traverses each character and checks whether every required child exists.Source: §B.2 · p. 14
• Prefix enumeration performs depth-first traversal from the root, appending each non-root accumulated prefix before exploring its children.Source: §B.2 · p. 14
• Prefix search traverses to the prefix node, then recursively collects completed words while optionally retaining the queried prefix.Source: §B.2 · p. 14
• The recursive word collector appends strings at end-of-word nodes and extends results through each child subtree.Source: §B.2 · p. 14