Source-linked AI summary
The jsonlite Package: A Practical and Consistent Mapping Between JSON Data and R Objects
Jeroen Ooms
TL;DR
The paper addresses inconsistent and ambiguous conventions for representing R data structures in JSON, especially when practical R types do not map directly to JSON structures. It specifies generalized mappings and consistency principles, using jsonlite as a reference implementation. The result is a practical framework for common R classes, while retaining limitations from JSON and approximate reversibility.
Problem
R packages lack formal consensus on mapping JSON structures to R classes, and edge cases can produce inconsistent behavior and unexpected output.
Method
The paper defines explicit conventions for mapping common R classes to JSON, emphasizes structural and type consistency, and presents jsonlite as a reference implementation.
Results
The proposed conventions cover practical representations for vectors, matrices, data frames, and other common R classes while supporting consistent JSON exchange.
Takeaways & Limitations
Reliable JSON interfaces with R require explicit, unambiguous mappings and structures that remain compatible with clients expecting stable types and keys.
Takeaways & Limitations
The mapping is limited to common R data classes, JSON cannot represent binary data directly, and class-based conversion is not perfectly reversible.
Abstract
from arXiv · showhide
A naive realization of JSON data in R maps JSON arrays to an unnamed list, and JSON objects to a named list. However, in practice a list is an awkward, inefficient type to store and manipulate data. Most statistical applications work with (homogeneous) vectors, matrices or data frames. Therefore JSON packages in R typically define certain special cases of JSON structures which map to simpler R types. Currently there exist no formal guidelines, or even consensus between implementations on how R data should be represented in JSON. Furthermore, upon closer inspection, even the most basic data structures in R actually do not perfectly map to their JSON counterparts and leave some ambiguity for edge cases. These problems have resulted in different behavior between implementations and can lead to unexpected output. This paper explicitly describes a mapping between R classes and JSON data, highlights potential problems, and proposes conventions that generalize the mapping to cover all common structures. We emphasize the importance of type consistency when using JSON to exchange dynamic data, and illustrate using examples and anecdotes. The jsonlite R package is used throughout the paper as a reference implementation.
1 Introduction
The paper addresses the lack of consistent conventions for mapping JSON structures to practical R data types. It proposes explicit mappings and uses jsonlite as a reference implementation while emphasizing type safety and supported scope.
- Motivation: Different R packages lack formal consensus on JSON-to-R mappings, creating ambiguous edge-case behavior and unexpected output.The ambiguity includes basic R structures that do not perfectly correspond to JSON counterparts.
- Motivation: JSON objects and arrays naturally map to named and unnamed R lists, but statistical applications usually require vectors, matrices, or data frames.Homogeneous arrays of primitives are commonly simplified into atomic vectors.
- Type safety: A null inside a homogeneous JSON array can cause parsers to return a list instead of a vector, producing type errors when client code assumes vector output.The ambiguity arises because the array is no longer homogeneous.
- Design principles: The paper argues that JSON-to-R mappings should be explicit, consistent, unambiguous, and aligned with natural R representations for complex structures.It discusses interpreting null as a missing value, such as NA, rather than automatically falling back to a list.
- Reference implementation: jsonlite provides class-based toJSON and fromJSON methods, extensive unit tests, and separate type-based serialization functions for preserving R storage details.The class-based mappings prioritize concise, practical encoding, while serializeJSON and unserializeJSON target reversible type-based serialization.
- Scope and limitations: The approach is scoped to common R data classes and excludes language constructs, custom compound classes, and temporary in-memory properties that cannot be sensibly serialized.The paper recommends standard data structures for R interface inputs and outputs.
- Scope and limitations: JSON remains human-readable and interoperable but lacks binary support and can lose real-number precision through decimal formatting.Binary data can be represented indirectly, for example by encoding it as a base64 string.
2 Converting between JSON and R classes
jsonlite maps common R classes to JSON structures through class-based conventions, addressing ambiguity in vectors, missing values, special types, data frames, and nested data. The package favors consistent representations that preserve practical interoperability while acknowledging cases where JSON cannot fully retain R semantics.
- Vectors: Atomic R vectors map to JSON arrays, although JSON arrays do not express the vectors’ homogeneous type distinction.
- Missing values: Null encoding cannot distinguish numeric NA, NaN, Inf, and -Inf, whereas string encoding conflicts with character values such as the literal "NA".
- Missing values: Missing values in logical and character vectors are encoded as null, while numeric missing and non-finite values may be encoded as strings.
- Special vector types: Dates, timestamps, factors, and complex vectors are coerced to character strings and require manual coercion after parsing to recover their intended R types.
- Vectors: jsonlite encodes vectors of length 0 or 1 as arrays, avoiding length-dependent primitive values that can break clients expecting arrays.
- Data frames: For data frames, jsonlite uses a more natural, interoperable representation, including row-based conventions where primitives inside JSON objects indicate data-frame rows.
- Nested data: Nested JSON records map to nested data frames, but R lacks native relational support, so nested tables are stored as list columns without guaranteed matching fields or flattenability.
3 Structural consistency and type safety in dynamic data
Reliable JSON exchange requires fixed, predictable keys and type-consistent structures, especially when dynamically typed R data are converted for interoperable use. The paper recommends organizing heterogeneous data so consumers can process conceptually similar elements uniformly.
- 3.1 Rule 1: Fixed keys: JSON interfaces should limit objects to a finite set of keys known in advance by all parties.Fixed keys make outputs easier to process, validate, document, and optimize.
- 3.1 Rule 1: Fixed keys: Encoding changing data values as JSON keys makes outputs difficult for software to process and document.Representing timestamps as keys produces different keys across queries; explicit time and price fields provide a more stable structure.
- 3.1 Rule 1: Fixed keys: Fixed-key structures can add properties to each observation without breaking backward compatibility.The structure remains compatible with software designed around known fields.
- 3.3 Rule 2: Consistent types: Consistency concerns conceptual meaning rather than identical structure, so records may omit fields while remaining processable by the same method.The data-frame example contains records with different present fields but the same conceptual kind of observation.
- 3.3 Rule 2: Consistent types: A collection containing different object classes should be separated and organized using a named list.The supplied example separates humans and horses into distinct named groups.
- 3.3 Rule 2: Consistent types: JSON properties and arrays should use consistent types because mixed-type collections are difficult to describe, interpret, and process.The paper recommends avoiding heterogeneous unnamed lists when designing interoperable structures.
A Public JSON APIs
The paper introduces public HTTP APIs as examples of complex real-world JSON structures. These services are free, although some require registration or authentication.
- Public HTTP APIs provide examples of complex structures encountered in real-world JSON data.The examples return substantial amounts of data for inspection in R.
A.1 No authentication required
Some listed APIs allow limited use without registration. This subsection identifies that access condition without describing a specific service.
- Some of the following APIs allow limited use without any registration.
Github
Github is presented as a public code-hosting service with APIs exposing live activity data. The paper gives R examples for retrieving organizations, repositories, issues, and commits.
- Github APIs provide live data on almost all activity in an online code repository.
- The examples use fromJSON to retrieve organizations, repositories, issues, and commits from Github endpoints.
CitiBike NYC
The CitiBike NYC example uses jsonlite to retrieve a public API containing location, status, and current availability for all bike-sharing stations.
- The public API reports location, status, and current availability for all New York City bike-sharing stations.
- fromJSON retrieves the station data from the CitiBike NYC stations endpoint.
AngelList
The AngelList example uses jsonlite to query a startup job-listing directory and extract its startup records.
- AngelList is a job-listing directory for startups.
- The example retrieves AngelList startup data with fromJSON and selects the startups field.
Ergast
The examples show jsonlite retrieving motor-racing, media, company, and government data through web APIs, including APIs that require registration keys or tokens.
- Ergast: The Ergast Developer API provides historical motor-racing data for non-commercial purposes.
- Ergast: fromJSON retrieves Formula One race results and extracts the driver from the first result.
- Registered APIs: Several APIs require free registration for a key or token, which must be appended to requests.
- New York Times: The New York Times provides free APIs for news articles, book reviews, real estate, and other departmental data.
- New York Times: The examples query New York Times article-search, bestseller, and movie-review endpoints using API keys.
- CrunchBase: CrunchBase is an editable free database covering technology companies, people, and investors, queried with fromJSON.
Sunlight Foundation
The Sunlight Foundation example introduces a nonprofit focused on government transparency and accountability, then demonstrates registered API queries for civic data.
- The Sunlight Foundation promotes government transparency and accountability through data, tools, policy, and journalism.
- The example registers an API key for access to Sunlight Foundation services.
- Example queries retrieve bills, legislative speech dates, and legislators through Sunlight-related APIs.
Twitter API access uses OAuth2 authentication before requests can retrieve user timeline data. The example shows obtaining a bearer token and supplying it to an authenticated API call.
- Authentication: OAuth2 authentication requires an application consumer key and consumer secret.The example creates application credentials before requesting an access token.
- Authentication: The client exchanges basic-authenticated application credentials for an OAuth2 access token using the client-credentials grant.The token request posts grant_type=client_credentials with an Authorization header.
- API request: The resulting bearer token authenticates a request for ten statuses from the UCLA user timeline.The API call adds the token in the Authorization header and requests count=10.
B Simple JSON RPC with OpenCPU
OpenCPU uses JSON to pass data-frame inputs to R functions and return their results. The examples show equivalent command-line requests for selecting, reshaping, and analyzing data.
- JSON RPC: jsonlite enables OpenCPU clients to retrieve R objects and remotely call R functions through JSON arguments and return values.Both function inputs and outputs are represented as JSON objects.
- Statistical function call: A variance calculation can be requested through curl by posting the numbers 1:9 as a JSON array.The endpoint receives {"x":[1,2,3,4,5,6,7,8,9]} with Content-Type: application/json.
- Data-frame example: The melt example sends example airquality rows to the reshape2 endpoint with application/json, with both input and output represented as data frames.The request contains rows with Ozone, Solar.R, Wind, Temp, Month, and Day fields.
- Data-frame example: The airquality example selects two rows, identifies Month and Day, melts the remaining variables, and serializes the result with toJSON.The corresponding R code uses airquality[1:2, ], reshape2::melt, and cat(toJSON(y)).