Source-linked AI summary
A Survey on Online Judge Systems and Their Applications
Szymon Wasik, Maciej Antczak, Jan Badura, Artur Laskowski, Tomasz Sternal
TL;DR
Online judge systems are increasingly used across contests, education, recruitment, data mining, and optimization, but the literature lacked a formal definition and unified survey. The paper classifies these systems, formalizes their definition and common evaluation procedure, and examines Optil.io as a crowdsourcing-based example. The review and competition analysis support online judges as a way to address complex industrial- and science-driven challenges, while noting practical constraints from closed-source systems and evaluation design.
Problem
The literature lacked a formal definition and comprehensive application-based classification of online judge systems, limiting systematic analysis across their uses.
Method
The paper surveys online judge systems by application, formalizes the system definition and common evaluation procedure, and analyzes an Optil.io competition.
Results
The Optil.io competition used crowdsourcing to address a complex optimization problem, with 85 users submitting 1191 solutions during 18 days and correct submissions reaching approximately 50% at the end.
Takeaways & Limitations
Online judge systems can support contest organization, education, data-mining challenges, development platforms, and crowdsourced optimization applications.
Takeaways & Limitations
Most existing online judge systems are closed source, making them difficult to adapt to varied user expectations; configurable self-hosted systems therefore remain important.
Abstract
from arXiv · showhide
Online judges are systems designed for the reliable evaluation of algorithm source code submitted by users, which is next compiled and tested in a homogeneous environment. Online judges are becoming popular in various applications. Thus, we would like to review the state of the art for these systems. We classify them according to their principal objectives into systems supporting organization of competitive programming contests, enhancing education and recruitment processes, facilitating the solving of data mining challenges, online compilers and development platforms integrated as components of other custom systems. Moreover, we introduce a formal definition of an online judge system and summarize the common evaluation methodology supported by such systems. Finally, we briefly discuss an Optil.io platform as an example of an online judge system, which has been proposed for the solving of complex optimization problems. We also analyze the competition results conducted using this platform. The competition proved that online judge systems, strengthened by crowdsourcing concepts, can be successfully applied to accurately and efficiently solve complex industrial- and science-driven challenges.
1 INTRODUCTION
Online judges automatically evaluate submitted solutions in reliable, scalable environments and support contests, education, data-mining challenges, and optimization applications. This survey classifies these systems, formalizes their definition and evaluation methodology, and discusses crowdsourcing through an Optil.io example.
- Online judge systems: Online judges automatically verify submitted solutions by executing them on predefined tests while enforcing resource limits.Their infrastructure also computes participant rankings during contests.
- Online judge systems: The field grew from programming contests such as ICPC and IOI, which use automatic evaluation with different participation and scoring models.ICPC uses team-based binary scoring, whereas IOI uses individual participation and partial scores.
- System requirements: Reliable online judging requires precise, deterministic execution-time measurement and scalable cloud infrastructure using concurrency and parallel processing.These requirements become especially important when submissions increase rapidly near contest deadlines.
- Problem types: Online judge problems are generally combinatorial, including decision, search, and optimization problems constrained by discrete variables and objective functions.The systems commonly adjust processing-time limits to ensure feasible evaluation.
- Applications: Crowdsourcing connects optimization challenges with large networks of programmers and scientists through open calls over the Internet.The survey presents Optil.io as an online judge designed to apply this approach to complex optimization problems.
- Survey scope: The survey fills the lack of a formal online-judge definition, classifies systems by application, summarizes common evaluation methodology, and examines an Optil.io contest.Its classification targets practitioners, researchers, and teachers choosing or applying online judge platforms.
2 ONLINE JUDGE SYSTEMS
The article surveys online judge systems, defining their common evaluation procedure and classifying systems by principal objective and application. It reviews systems for contests, education, recruitment, data mining, online compilation, and customizable development platforms.
- Scope and objectives: Online judges aim to provide safe, reliable, continuous, cloud-based evaluation of user-submitted algorithms.Their infrastructure must support coherent evaluation and scalability as submissions increase.
- Evaluation procedure: The common evaluation procedure comprises submission, assessment, and scoring.Submitted code is compiled and verified, tested on problem-specific cases, checked for errors, resource limits, and output compliance, then scored in aggregate.
- Classification: The survey classifies systems into six classes according to their principal objectives, including competitive programming, education, recruitment, data mining, online compilation, and development platforms.Systems that fit multiple groups are assigned to the class representing their principal objective.
- Review method: The review combines searches of Web of Science, Google Scholar, Scopus, the Olympiads in Informatics journal, and the Internet.Included systems had to support at least part of the evaluation procedure, address combinatorial problems, be publicly available, operate properly, and be usable in English or described in English.
- Review method: For each identified usage scenario, the authors compute a coverage coefficient equal to the number of online judge systems implementing that scenario.They selected two to five intuitive usage scenarios for each system before computing coverage.
- Development platforms: Development platforms can be installed locally, administered fully, adapted to user needs, and integrated with external services.Examples include educational systems such as CloudCoder and Tsinghua University Online Judger, while BOSS was last updated in 2009.
3 EVALUATION METHODOLOGY
The evaluation methodology models online judging as a staged process that compiles submissions, executes them on test instances, collects statuses and scores, and aggregates results. It also formalizes online judge systems as cloud services performing evaluation steps.
- Submission phase: The submission phase compiles user code and checks compilation errors, compilation time, and binary-size limits.Static binaries additionally require verification of linked libraries and compatibility with the evaluation architecture.
- Assessment phase: The assessment phase executes the compiled solution on each test instance and can be the most computationally expensive stage.For complex industry-inspired problems, a single execution may take several dozen minutes.
- Test instances: A test instance contains input data, reference output data, and parameters passed to the evaluation engine.Parameters can encode resource limits, random seeds, or output-size limits, while execution parameters may be hidden from the submitted solution.
- Evaluation engine: The evaluation engine maps a binary and test instance to an execution status, evaluation score, and execution statistics.Statuses include Accepted, Time Limit Exceeded, and Memory Limit Exceeded; statistics commonly include peak resource usage.
- Online judge system: An online judge system is defined as an online service that performs evaluation-procedure steps in a cloud.The paper introduces this definition after formalizing solutions, test instances, and evaluation-engine outputs.
4 EXAMPLE APPLICATION BASED ON OPTIL.IO PLATFORM
The Optil.io contest evaluated an NP-hard facility-location challenge through iterative submissions and online ranking. Results showed that user participation and crowdsourcing improved optimization quality, while many early submissions were incorrect.
- The contest used an optimization-based variation of the multiple facilities location problem, whose computational complexity is NP-hard.
- 85 users submitted 1191 solutions during the 18-day contest, followed by another 400 submissions after it ended.
- At the contest’s end, potentially correct submissions comprised approximately 50%, after incorrect submissions initially predominated.
- After five days, improvement slowed substantially unless a significantly larger number of participants joined the competition.
- The Optil.io contest demonstrated that many programming enthusiasts could interactively solve optimization problems through the platform.
5 CONCLUSION
Online judge systems now support competitive programming, education, recruitment, data-mining challenges, online compilation, and customizable deployments. The survey classifies these systems, formalizes their definition and evaluation procedure, and presents Optil.io as an optimization-focused use case.
- Online judge systems have expanded from competitive-programming support to education, recruitment, data-mining challenges, and browser-based development.
- Cloud-based evaluation and Evaluation-as-a-Service have become established terms for the approach used by online judge platforms.
- Most online judge systems are closed source, motivating attention to downloadable, highly configurable systems for competitions and educational evaluation.
- The survey’s classification helps practitioners find platforms suited to different needs and distinguishes this work from narrower earlier reviews.
- The paper formally defines online judge systems and describes their commonly used evaluation procedure.
- Optil.io provides continuous evaluation of algorithms for complex optimization challenges in a safe, homogeneous cloud-based infrastructure.
ELECTRONIC SUPPLEMENT
The electronic supplement provides URLs for online judge systems listed in the article’s tables.
- Table 7 lists online judge systems from Tables 1 to 6 together with their URL addresses.
- The supplement includes online compiler and development-platform URLs such as Codeanywhere, Codepad, CodeSkulptor, Coding Ground, Codio, and Ideone.