A Century of Portraits: A Visual Historical Record of American High School Yearbooks

Motivation

• Historical photographs can preserve non-verbal details that written descriptions may fail to convey, while large-scale visual analysis remains limited by manual curation.The paper motivates image analysis as complementary to historical texts.Source: §I · p. 1

Visual trends

• Data-driven mining examines changes in smiling, men’s hairstyles, and eyewear popularity, with results consistent with historical records on fashion.The eyewear trend is described as correlated with advances in contact lens technology.Source: §I · pp. 1–2

Model visualization

• The proposed visualization method identifies image parts useful for dating by disabling network units without altering the dating decision.The goal is to expose time-specific visual elements used by the CNN.Source: §I · p. 2

Contributions

• The paper introduces a yearbook-portrait dataset and data-driven analyses of visual patterns, portrait dates, and the image elements used by the dating model.The dataset spans 120 years in the United States.Source: §I · pp. 1–2

Portrait dating

• A CNN dates portraits with median errors of 4 years for women and 6 for men.The authors also report some generalization to historical celebrity portraits despite differences in appearance.Source: §I · p. 2

1) Historical Data Analysis:

• Digital humanities work has used large text corpora to study word usage and historical events, while automated historical-image analysis has been more limited.The paper contrasts its approach with image studies of automobile design, architecture, and dating.Source: 1) Historical Data Analysis · p. 2

2) Modeling Style:

• Earlier fashion studies used supervised annotations or weakly supervised clustering, whereas this paper examines style changes across a much longer period.The yearbook scans include grayscale and lower-resolution images, making color- and texture-based approaches unsuitable.Source: 2) Modeling Style · p. 2

3) Deep Neural Networks:

• VGG is used for portrait dating and visualization; prior CNN visualization methods described here do not explain which input-image locations contribute to classification.The cited approaches include feature- or activation-maximizing images and synthetic face images.Source: 3) Deep Neural Networks · p. 2

4) Deep Neural Network Visualization:

• Unlike activation and region visualization methods,,,, it follows each input’s CNN path to locate image-specific discriminative pixels, including structures beyond objects.This distinguishes its image-specific locations from generic class locations and visual elements constrained to image regions.Source: 4) Deep Neural Network Visualization · pp. 2–3

III. THE YEARBOOK DATASET

• The collection contains 168,055 senior portraits from 949 yearbooks (1905–2013), with 37,921 frontal-facing portraits retained across 814 yearbooks, 115 schools, and 26 states.The final dataset also includes year, state, city, school, and gender annotations.Source: §III · p. 3
• The collection spans year and region (Fig. 2); 46.4% of photos come from the 100 largest US cities, averaging 28.8 faces per yearbook and 329 per school.Figure 2 depicts portrait distribution by year and region.Source: §III · p. 3, Figure 2 · p. 3
• Historical coverage reflects unequal high-school access: under 10% of 18-year-olds graduated in the early 1900s, and Black students, especially in the South, were largely excluded until mid-century.High-school graduation rates had risen to almost 50% by the end of the 1960s.Source: §III · p. 3
• Women make up 53.4% of photos and men 46.6%, but the authors cannot determine whether this imbalance reflects a population bias.They suggest historically greater high-school graduation among girls may contribute to the difference.Source: §III · p. 3
• Processing identified portrait pages, converted images to grayscale, detected and cropped faces, filtered pose, aligned faces, and assigned gender with an SVM plus crowdsourcing for cases below 90% confidence.Facial landmarks supported pose estimation and alignment; gender classification used whitened HOG features.Source: §III · p. 3

A. Getting a Sense of Each Decade

• Average images of male and female students for each decade summarize the main modes of popular fashions in those periods.The images are pixel-means of portraits organized chronologically.Source: A. Getting a Sense of Each Decade · p. 3

B. Capturing Trends Over Time

• The analysis extends portrait trends from attributes present in every image, such as smile degree, to accessories and styles worn by only some people.Kotchemidova’s historical account describes how Kodak advertising helped establish smiling as an expected snapshot expression.Source: B. Capturing Trends Over Time · p. 3, 1) Smiling in Portraiture · p. 3

B. Capturing Trends Over Time — 1) Smiling in Portraiture:

• Lip curvature, computed as the average of two angles from facial keypoints, provides a measure of smile intensity that correlates with AU12 activation on BP4D images.The BP4D validation used six subjects and approximately 2,500–3,000 samples for each AU12 value.Source: 1) Smiling in Portraiture · p. 4
• In the study’s portrait data, average smile intensity rose rapidly from the 1900s to the 1950s and continued increasing thereafter; women consistently smiled more than men on average.Source: 1) Smiling in Portraiture · p. 4
• Expression-recognition software could also detect smiles, but it was not publicly available when the experiments were conducted.Source: 1) Smiling in Portraiture · p. 4

• Smile degree is easy to apply to each portrait in the collection because every subject exhibits some degree of mouth curvature, although it may sometimes be negative.Source: 2) Glasses · p. 4
• The authors fine-tuned VGG on PubFig celebrity portraits labeled for glasses, then applied the classifier to the yearbook collection.Source: 2) Glasses · p. 5
• Glasses were more popular among male students, and changes in their popularity correlated with the introduction of contact lenses.Source: 2) Glasses · p. 5

3) Men’s Hairstyles post 1960:

• Low resolution made fully automatic detection of men’s hairstyles inaccurate enough to require additional manual filtering.Source: 3) Men’s Hairstyles post 1960 · p. 5
• Afros were predominantly popular from the late 1960s through the late 1970s, after which many people switched to a more styled natural hairdo.The hairstyle classifications were manually filtered to remove false positives and add some false negatives.Source: 3) Men’s Hairstyles post 1960 · p. 5

C. Mining for Date-Specific Patterns

• Discriminative mode seeking groups decade-specific women’s hair and facial-accessory styles into visual clusters using portraits cropped to the face and hair.Clusters are ranked by how many top detections come from the target decade, with overlap reduced to improve visual coverage.Source: C. Mining for Date-Specific Patterns · p. 5
• The decade clusters display distinctive styles, including 1940s curly bangs and the 1960s “winged” flip hairstyle, discovered using year labels alone.Only one woman per graduating class was shown to reduce possible photographic or scanning-artifact bias within clusters.Source: C. Mining for Date-Specific Patterns · p. 5

V. DATING HISTORICAL IMAGES

• Decade-specific hair and accessory clusters emerge through discriminative clustering of the portraits.The detected patterns include finger waves in the 1930s, pin curls in the 1940s and 1950s, and straight long hair in the 2000s.Source: Figure 9 · p. 6
• The authors choose classification rather than standard regression because regression’s Gaussian formulation excludes multiple modes.The task extends prior dating of color photographs to black-and-white portraiture, where color-profile changes are unavailable.Source: §V · p. 6

1) Dating Yearbook Portraits:

• Dating is an 83-way classification task for years 1928–2010, with separate women’s and men’s classifiers.The models are separated to discourage reliance on low-level image artifacts; the experiments also include opposite-gender, celebrity, and background-crop tests.Source: §V · pp. 6–7
• On same-gender Yearbook test sets, full-network fine-tuning boosts performance over partial fine-tuning; 65.3% of women’s and 46.4% of men’s test images are classified within five years of the true year.Source: 1) Dating Yearbook Portraits · p. 7
• Most women’s-model confusion occurs between neighboring years, rather than far from the true year.The section relates this pattern to visual trends such as hairstyles that can span single-year boundaries.Source: 1) Dating Yearbook Portraits · p. 7

2) Generalization:

• Background-only crops achieve 2.8% accuracy and 24.1% accuracy within five years, which the authors say is insufficient to date the portraits.The crops are 32 by 32 pixels and taken from portrait corners.Source: 2) Generalization · p. 7
• Both gender-specific models perform substantially worse on opposite-gender yearbook photos, while the fully fine-tuned women’s model improves celebrity-test performance over the baselines.The authors conclude that low-level statistics alone cannot account for dating success and suggest that some Yearbook-learned features generalize.Source: 2) Generalization · p. 7
• Celebrity performance may be limited by the shift from student portraits to glamour shots and by a test set that may be too small.The authors note that celebrity hairstyles can differ from those of the general public.Source: 2) Generalization · pp. 7–8
• Approximately 40% of female-celebrity L1 errors are under a decade, and most predictions fall within two decades of the ground-truth year.The passage says dating does not generalize well for all celebrities.Source: 2) Generalization · p. 8

3) Implementation Details:

• Dating portraits are face-and-hair crops resized to 96 by 96 pixels, with school-separated train and test images to reduce photographic and scanning biases.The test set contains approximately 30% of portraits from 1982–2010; images are noise-reduced before training.Source: 3) Implementation Details · p. 8

VI. WHAT TIME SPECIFIC PATTERNS IS THE CLASSIFIER USING FOR DATING?

• The visualization method disables spatial locations outside an input’s path through the network while preserving the same output.Its goal is to identify which pixels contribute to a particular dating decision.Source: §VI · p. 8

1) Top-Down Selection of Spatial Units:

• The top-down selection objective minimizes KL divergence between the original output distribution and the distribution after spatial-unit removal.The output vector assigns probabilities to years, and c denotes one class entry.Source: 1) Top-Down Selection of Spatial Units · p. 8
• A binary mask disables spatial units, with the same fixed sparsity percentage applied at every layer.The mask is constrained to retain the desired number of active spatial units.Source: 1) Top-Down Selection of Spatial Units · p. 8
• A greedy top-down algorithm removes spatial locations whose removal increases the objective least while meeting the sparsity constraint.It starts with a full mask and jointly disables all feature channels at each selected location.Source: 1) Top-Down Selection of Spatial Units · pp. 8–9

2) Gradient Approximation:

• The original greedy search is too slow for lower layers because it evaluates every spatial location for each unit removed.Each step seeks the location whose removal increases the loss least.Source: 2) Gradient Approximation · p. 9
• A finite-difference estimate of loss change is approximated by the KL-objective gradient times the unit’s activation, requiring one backward pass.The authors describe the linear approximation as crude but effective in practice.Source: 2) Gradient Approximation · p. 9
• Because the KL gradient is zero when the distributions match, the implementation starts with all units disabled and adds those needed to preserve the output.This reverses the direction of the optimization.Source: 2) Gradient Approximation · p. 9

3) Experimental Setup:

• The dating classifier is a fine-tuned VGG network whose fully connected layers are replaced with 1 × 1 convolutions for spatial-unit selection.The fully convolutional model treats each image-pixel as an independent predictor, averages spatial predictions for the image-level date, and uses fixed sparsity sl = 20% across layers.Source: 3) Experimental Setup · p. 9

4) Quantitative Evaluation:

• Our method achieves higher classification accuracy than Simonyan et al. on images containing discovered regions, which retain more discriminative elements.Table II compares classification accuracy and errors on visual elements.Source: Table II · p. 9, 4) Quantitative Evaluation · p. 9
• For the region test, discovered elements are added to the training-set mean image before the fine-tuned network predicts the year label.Figure 12 shows discriminative regions for a 1940 portrait overlaid on the mean training image.Source: 4) Quantitative Evaluation · p. 9, Figure 12 · p. 9

5) Qualitative Evaluation:

• Selected regions capture era-specific styles, including dark lipstick in the 1940s and 1950s, 1960s flat bangs, 1980s curls, and 1990s hair partings.The comparison method tends to select the forehead and nose, which the authors describe as less relevant for predicting the photograph’s era.Source: 5) Qualitative Evaluation · p. 9, Figure 13 · p. 10
• Figure 13 compares celebrity portraits from different eras, with selected spatial units shown for pool5 and conv1; its images are correctly predicted unseen female celebrity portraits.Rows show pool5 units above and conv1 units below; the pool5 receptive fields are shaded with a tent filter for display.Source: Figure 13 · p. 10, 5) Qualitative Evaluation · p. 10

VII. CONCLUSION

• The authors present the public yearbook dataset and mining methods as tools for observing changing fashions and habits within a fixed visual framework.They identify future uses including studying style spread and celebrity influence, fashion-fad cycles, and data-driven style transfer.Source: §VII · p. 10
• Historical-image work pushes computer-vision methods to their limits, and CNNs must be checked for memorizing portrait backgrounds and noise.The authors cite room for improvement in figure-ground and hair segmentation, facial keypoints, face alignment, pose estimation, and attribute and accessory detection.Source: §VII · p. 10