BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformer

• GPT-like models use causal objectives with one-directional context, whereas bidirectional models use denoising objectives to exploit context from both directions.Source: §1 · p. 1
• Instruction tuning and reinforcement learning from human feedback improve task alignment by incorporating task-specific instructions and feedback on generated outputs.Source: §1 · p. 2
• Large language models have achieved fluent, coherent, and increasingly general text generation across diverse language tasks, sometimes surpassing human-level performance in evaluations.Source: §1 · pp. 1–2
• Current models face limited memory over long sequences, hallucinations, and difficulty capturing complex dependencies, restricting use in some high-stakes applications.Source: §1 · p. 2
• BATGPT introduces a bidirectional autoregressive architecture for language generation, dialog systems, and question answering, aiming to improve effectiveness across natural language applications.Source: §1 · p. 2

• Autoregressive, autoencoding, and encoder-decoder models represent the main families of pretrained language models, each with distinct design strengths.Source: §2 · p. 2
• Instruction tuning and self-instruct methods formulate tasks as generation problems and expand task-specific training instructions to improve performance across tasks.Source: §2 · p. 2
• Scaling-law research studies trade-offs among model size, data, computation, and performance, while Chinchilla reports better performance from smaller models under equal compute budgets.Source: §2 · p. 2
• RLHF aligns model outputs with human needs through feedback, while AI feedback can reduce the number of required human labels.Source: §2 · p. 2
• BATGPT combines bidirectional autoregression, parameter expansion, and reinforcement learning to address persistent limitations while retaining strengths from earlier models.Source: §2 · p. 3

3.1 Bidirectional Autoregressive Pre-training

• Bidirectional autoregressive pre-training predicts each token from preceding or subsequent tokens, capturing dependencies in both forward and backward contexts.The objective maximizes sequence likelihood over pre-training data and is intended to produce coherent, fluent outputs.Source: §3.1 · p. 3

3.2 Instruct Tuning

• Instruction tuning uses prompt-response pairs, while multi-round dialogue training improves preservation of conversational context over extended chats.The prompted dataset optimizes the likelihood of generating responses given prompts.Source: §3.2 · p. 3

3.3 Reinforcement Learning from Human Feedback

• BATGPT combines human and AI preference feedback, trains a reward model, and updates the policy with PPO while regularizing against earlier model distributions.The preference platform compares outputs for acceptability and helpfulness, and the expanded objective preserves capabilities from pre-training and instruction tuning.Source: §3.3 · p. 4

3.4 Parameter Expansion

• Function Preserving Initialization expands model width by mapping target weights to source parameters while preserving the source model’s outputs.The expansion uses mappings for target columns and rows, uniform sampling, and rescaling to preserve function.Source: §3.4 · pp. 5–6
• Progressive stacking expands depth by repeatedly training and stacking shallower models, reducing training time without sacrificing final-model quality.BATGPT trains a 15B foundation model and initializes the 30B and 70B versions from it using joint width and depth expansion.Source: §3.4 · p. 6

4.1 Evaluation on CMMLU

• BATGPT ranked second among Chinese-oriented language models on CMMLU and achieved its highest average accuracies in STEM and Others.Its reported category accuracies were 33.49% for STEM and 42.14% for Others.Source: §4.1 · p. 6
• CMMLU results show robust performance across humanities, social science, STEM, China-specific topics, and other subjects.The reported accuracies were 35.38% in Humanities, 36.31% in Social Science, 37.00% in China-specific topics, and 42.14% in Others.Source: §4.1 · p. 6, Table 1 · p. 7
• 38.48% overall accuracy was achieved with direct-answer prompting on zero-shot CMMLU, compared with 35.91% using chain-of-thought prompting.On the STEM subset, direct-answer and chain-of-thought accuracies were 33.74% and 34.47%, respectively.Source: §4.1 · p. 6, Table 2 · p. 7

4.2 Evaluation on C-EVAL

• 57.4% average accuracy was achieved on C-EVAL, including 72.1% in Social Science and 36.9% on the C-EVAL HARD subset.Additional category scores were 60.7% in Humanities, 50.5% in STEM, and 53.3% in Others.Source: §4.2 · p. 7, Table 3 · p. 7

5 Conclusion

• BATGPT combines bidirectional autoregressive modeling, parameter expansion from smaller models, and reinforcement learning from AI and human feedback.The paper describes these components as addressing limited memory and hallucinations while improving alignment performance.Source: §5 · p. 7
• The approach is reported to produce highly fluent, coherent, and contextually relevant text for broad natural-language applications.The conclusion connects these capabilities to BATGPT’s intended use across a wide range of applications.Source: §5 · p. 7
• The paper acknowledges remaining challenges involving further capability refinement, bias, fairness, and responsible deployment.These concerns are identified as requiring continued research and development.Source: §5 · p. 7