Source-linked AI summary
Teaching Software Engineering for AI-Enabled Systems
Christian Kästner, Eunsuk Kang
TL;DR
AI courses often focus on modeling rather than the engineering needed for complete AI-enabled systems. The paper presents a software-engineering course with lifecycle lectures and simulation-based assignments; its first offering surfaced constraints in audience, simulator scale, practical depth, and teaching tools.
Problem
AI courses often emphasize algorithms and modeling in fixed-data settings, while engineering complete AI-enabled systems involves production, reliability, scale, and update challenges.
Source: §1 · p. 1Method
The authors designed a software-engineering course for students with ML backgrounds, combining lifecycle-focused lectures with hands-on assignments built around a simulation infrastructure.
Source: §3 · p. 2, §3.1 · p. 2, §3.2 · p. 3Results
The first offering was taught to 12 graduate students and informed recommendations about prerequisites, simulator engineering, practical grounding, and tooling.
Source: §4 · pp. 3–4Takeaways & Limitations
AI-enabled systems often sit toward the complex end of software projects, where software-engineering techniques can address concerns such as safety, risk, scalability, and robustness.
Source: §3 · p. 2The authors struggled to find mature tools or standard techniques for teaching emerging topics such as fairness and explainability.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Software engineers have significant expertise to offer when building intelligent systems, drawing on decades of experience and methods for building systems that are scalable, responsive and robust, even when built on unreliable components. Systems with artificial-intelligence or machine-learning (ML) components raise new challenges and require careful engineering. We designed a new course to teach software-engineering skills to students with a background in ML. We specifically go beyond traditional ML courses that teach modeling techniques under artificial conditions and focus, in lecture and assignments, on realism with large and changing datasets, robust and evolvable infrastructure, and purposeful requirements engineering that considers ethics and fairness as well. We describe the course and our infrastructure and share experience and all material from teaching the course for the first time.
1 INTRODUCTION
AI courses commonly teach algorithms and modeling, but give little attention to engineering complete systems that include AI. Such systems raise practical concerns beyond model accuracy, including deployment, reliability, scale, and updates.
1 INTRODUCTION
- AI courses often give little attention to software engineering for complete systems that include AI.The paper contrasts this gap with the abundance of formal, practical, and online AI courses.Source: §1 · p. 1
- Real-world AI systems raise engineering questions about robust pipelines, production deployment, quality monitoring, mistakes, and risk.Other concerns include tradeoffs among learning cost, inference time, updateability, interpretability, scalability, and data and model versioning.Source: §1 · p. 1
2 RELATED WORK AND SCOPE
Prior work and teaching materials address aspects of engineering AI, but the paper identifies limited academic attention to broader software-engineering concerns in AI-enabled systems. The course focuses on applying software-engineering techniques to build systems with or around AI components.
2 RELATED WORK AND SCOPE
- The course reverses the AI4SE focus: it teaches how software-engineering techniques can build better systems with or around AI components.The paper distinguishes this from using AI techniques to solve software-engineering problems.Source: §2 · p. 1
- Industry accounts describe practical engineering challenges in AI systems, including technical debt, testing, data provenance, A/B testing, and team organization.The paper cites examples from Google, Uber, and Microsoft.Source: §2 · p. 1
- Academic work covered deep-learning project challenges and ML testing, while requirements, architecture, and process appeared rarely in the literature reviewed.The authors also identify Hulten’s book as the closest broad coverage they found for the intended course and assign it as required reading.Source: §2 · pp. 1–2
3 COURSE DESIGN
The course applies software-engineering methods to AI-enabled systems, where unreliable, context-dependent components can create system-level concerns. Its lectures and hands-on assignments cover the lifecycle, using a simulation to make production concerns and feedback effects tangible.
3 COURSE DESIGN
- The course assumes a working relationship with a data scientist and focuses on design, implementation, operation, and quality assurance from a software-engineering perspective.It aims to help engineers turn ML ideas into scalable and reliable products.Source: §3 · p. 2
- AI components can be unreliable, depend on environmental assumptions, and have non-local effects, making system-level design and quality assurance important.The course connects these concerns to established software-engineering techniques for unreliable components, requirements, modularity, and production monitoring.Source: §3 · p. 2
3.1 Scope and Lectures
- Lectures cover requirements, architecture, implementation and operation, quality assurance, and process, including safety, fairness, deployment, monitoring, and interdisciplinary work.The course explicitly transfers software-engineering concepts and techniques while treating AI components largely as black boxes.Source: §3.1 · p. 2
3.2 Assignments
- Core assignments use a simulation where student predictions affect a modeled world, enabling work on scaling, production concerns, and feedback loops.The simulator uses a hidden ground-truth model, event streams, and an API; it can also expose students to drift, corrupt data, bias, and unintended effects of differing goals.Source: §3.2 · p. 3
- Five group assignments progress from offline modeling and tradeoff analysis to infrastructure deployment, automated model updates, and feedback-loop analysis.The infrastructure work includes production evaluation, automated canary releases, deployment without downtime, and A/B testing.Source: §3.2 · p. 3
- Five individual assignments address engineering concerns, safety analysis, architecture tradeoffs, fairness, and security threat modeling..Source: §3.2 · p. 3
4 EXPERIENCE
The first course offering involved 12 graduate students and informed recommendations about audience, simulator design, practical grounding, and teaching tools. The authors report constraints in prerequisites, simulation realism and scale, course depth, and available tools.
4 EXPERIENCE
- The class was taught for the first time in Fall 2019 to a small group of 12 graduate students, and that experience yielded recommendations for future semesters and others wishing to teach such a class.Source: §4 · p. 3
- Requiring both ML and software-engineering experience narrowed the target audience, while students often had gaps in one area.The authors suggest separate versions for students from each background or integrating course modules into existing software-engineering classes.Source: §4 · p. 4
- The simulator’s 160,000 users and event rate were too low to prompt serious operating-cost and performance tradeoffs, and movie recommendation offered limited learning-technique tradeoffs.The authors suggest scaling it up and exploring tasks involving pictures, audio, or video.Source: §4 · p. 4
- The course used varied scenarios to teach system-specific tradeoffs, but the authors suggest exploring fewer scenarios or implementations in greater depth..Source: §4 · p. 4
- Mature tools and standard techniques for emerging topics such as fairness and explainability were difficult to find for teaching.The authors identify a need for mature tools and benchmarks for AI-enabled system developers and educators.Source: §4 · p. 4
5 CONCLUSION
AI-enabled systems are challenging to build and maintain, so the authors designed a course to broaden software-engineering thinking for students interested in AI and made its materials available under a Creative Commons license.
5 CONCLUSION
- The authors designed a course to broaden software-engineering thinking for AI-interested students and made all course material available under a Creative Commons license.The course addresses the challenges of building and maintaining systems with AI components.Source: §5 · p. 4