Source-linked AI summary
A Nutritional Label for Rankings
Ke Yang, Julia Stoyanovich, Abolfazl Asudeh, Bill Howe, HV Jagadish, Gerome Miklau
TL;DR
Ranking algorithms can discriminate, exhibit low diversity, and be unstable, while complex processes and hidden assumptions make their methodology difficult to assess. Ranking Facts uses visual widgets to present summarized information about ranking methods and outputs. The demonstration presents the tool across real datasets and notes that its capabilities reflect limitations of the state of the art.
Problem
Rankings can discriminate against individuals and protected groups, exhibit low diversity, and change drastically after small input or methodology changes, while complex processes and hidden assumptions complicate assessment.
Source: §1 · p. 1Method
Ranking Facts uses a nutritional-label approach and visual widgets to summarize ranking methodology and output information, including attributes material to ranked outcomes.
Source: §1 · p. 1, §2.1 · p. 2Results
The paper presents Ranking Facts as a system for producing nutritional labels that explain rankings.
Source: §4 · p. 4Takeaways & Limitations
Users can inspect summarized information about the ranking process through Ranking Facts’ label for ranked outputs.
Source: §1 · p. 1Fairness measures beyond binary categories and richer scoring-function design remain in development; planned additions include suggestions to mitigate fairness and diversity issues.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Algorithmic decisions often result in scoring and ranking individuals to determine credit worthiness, qualifications for college admissions and employment, and compatibility as dating partners. While automatic and seemingly objective, ranking algorithms can discriminate against individuals and protected groups, and exhibit low diversity. Furthermore, ranked results are often unstable --- small changes in the input data or in the ranking methodology may lead to drastic changes in the output, making the result uninformative and easy to manipulate. Similar concerns apply in cases where items other than individuals are ranked, including colleges, academic departments, or products. In this demonstration we present Ranking Facts, a Web-based application that generates a "nutritional label" for rankings. Ranking Facts is made up of a collection of visual widgets that implement our latest research results on fairness, stability, and transparency for rankings, and that communicate details of the ranking methodology, or of the output, to the end user. We will showcase Ranking Facts on real datasets from different domains, including college rankings, criminal risk assessment, and financial services.
1 INTRODUCTION
Rankings can discriminate, exhibit low diversity, and change sharply after small input or methodology changes. Ranking Facts presents ranked outputs with summarized information about the ranking process.
1 INTRODUCTION
- Rankings can discriminate against individuals and protected groups, exhibit low diversity, and change drastically after small input or methodology shifts.The authors note that such instability can make results uninformative and easy to manipulate, and that these concerns also apply to ranked items such as colleges, departments, and products.Source: §1 · p. 1
- Complex processes and hidden assumptions motivate tools that help users assess a model or dataset’s fitness for use and inspect its methodology.The authors argue that retrospective checks place a burden on users, while vendors are better positioned to explain the results.Source: §1 · p. 1
- Ranking Facts uses a nutritional-label approach to explain ranked outputs with appropriately summarized information about the ranking process.The example label covers CS department rankings, with Ingredients and Fairness widgets expanded to show influential attributes and the computation behind fair/unfair labels.Source: §1 · p. 1, Figure 1 · p. 2
- Ranking Facts comprises visual widgets for transparency and interpretability, drawing on the authors’ work on fairness, diversity, transparency, and stability in rankers.The widgets communicate aspects of the ranking methodology or output to users.Source: §1 · p. 1
2 EXPLAINING RANKINGS
Ranking Facts explains ranked outputs through six widgets, each with overview and detailed views. The widgets describe ranking methodology and report stability, fairness, and diversity information.
2.1 Recipe and Ingredients
- Recipe describes the ranking algorithm and attribute weights, while Ingredients lists attributes most material to the ranked outcome, including associations beyond the designer’s stated intentions.Detailed views report minimum, maximum, and median attribute values at the top-10 and overall.Source: §2.1 · p. 2
2.2 Stability
- Stability indicates how much the data or ranking methodology must change to alter the ranking; a detailed view uses score-distribution slope at top-10 and overall.Close scores at adjacent ranks make a distribution unstable; in the example, slope at or below 0.25 is considered unstable, though attribute-specific or uncertainty-based assessments are alternatives.Source: §2.2 · pp. 2–3
2.3 Fairness
- Fairness tests statistical parity for sensitive attributes; the example reports FA*IR [14], proportion, and a pairwise measure, with fairness determined by computed p-values.A ranking measure can compare the protected-group proportion in a selected top-k set with its population proportion.Source: §2.3 · p. 3
2.4 Diversity
- Diversity reports category proportions in the top-10 and overall, and updates as users change ranking methods or weights; the example’s top-10 contains only large departments.The widget can measure demographic categories or categorical attributes of other items, including department size and university region in the example.Source: §2.4 · p. 3
3 DEMONSTRATION SCENARIOS
The demonstration applies Ranking Facts to three real datasets and lets users choose datasets and ranking methods or design scoring functions. A guided example compares stated scoring attributes with attributes associated with the ranked outcome.
3 DEMONSTRATION SCENARIOS
- The demonstrations use CS department data augmented with NRC attributes, COMPAS data for 6,889 individuals, and German Credit data for 1,000 individuals.The CS dataset includes publication count, faculty, average GRE scores, and university region.Source: §3 · p. 3
- Users can select a dataset and ranking method or upload a CSV, then configure scoring attributes and weights with data previews and attribute histograms.The design view supports raw or normalized and standardized data; it requires a categorical sensitive attribute, currently limited to binary attributes, and at least one numerical scoring attribute.Source: §3 · pp. 3–4
- In the CS example, GRE is a scoring attribute but does not correlate with the ranked outcome; similar top-10 and overall GRE ranges and medians support that finding.The presenter explores the nutritional label with the user, including its detailed Recipe widget.Source: §3 · p. 4
4 TAKE-AWAY MESSAGES
The paper presents Ranking Facts as a system for producing nutritional labels that explain rankings. Its capabilities reflect state-of-the-art limitations, with broader fairness measures and richer scoring-function support still in development.
4 TAKE-AWAY MESSAGES
- Ranking Facts produces nutritional labels that explain rankings and is described as modular and easy to extend.The authors present it as, to their knowledge, the first tool to consider interpretability for ranked outputs.Source: §4 · p. 4
- Fairness measures beyond binary categories and richer scoring-function design remain in development; planned additions include suggestions to mitigate fairness and diversity issues.The authors say the tool reflects known limitations of the state of the art.Source: §4 · p. 4