PaperlayerView exact revision on arXiv

Source-linked AI summary

Are You Thinking What I am Thinking? : Examining Conceptual Separation in Neural Architectures

Jaee Ponde, Roshni Agarwal, Subhashis Banerjee

arXiv:2609.00764v1cs.LGcs.AI

TL;DR

The paper asks whether neural networks organize well-defined and ambiguous concepts coherently and relationally in their internal representations, beyond output-level evaluation. It analyzes CNN and LLM activation spaces geometrically and distributionally, finding that conceptual separation broadly follows human-perceived distinctness but weakens for unseen, shifted, and ambiguous concepts. The authors propose activation-space analysis as a diagnostic complement to output-level metrics.

  • Problem

    Output-level metrics reveal little about whether neural networks preserve conceptual structure when classifying well-defined and ambiguous concepts.

  • Method

    The study uses geometric and distributional analyses of internal activations in CNNs and LLMs across familiar, unseen, shifted, related, and ambiguous concepts.

  • Results

    Conceptual separation broadly tracks human-perceived distinctness: familiar CNN classes are coherent, while unseen or shifted concepts and ambiguous LLM topics show weaker separation, including at the covariance level.

  • Takeaways & Limitations

    Activation-space analysis can complement output-level evaluation by diagnosing whether concepts are represented coherently and distinctly before deployment.

  • Takeaways & Limitations

    The findings are indicative rather than conclusive because the study uses 100 samples per class, four architectures, one hate-speech dataset, mainly two distributional moments, and only LLM final-layer activations.

Abstract

from arXiv · show

Neural networks are increasingly employed to identify both well-defined and ambiguous concepts, yet output-level metrics reveal little about how those concepts are represented internally. Our study asks if these networks exhibit \textit{conceptual separation}: if examples of the same concept form coherent representations, and whether related concepts lie closer together in the representation space. We examine this conceptual organisation in Convolutional Neural Networks (CNNs) and Large Language Models (LLMs) through geometric and distributional analysis of their internal activations. In CNNs, familiar ImageNet concepts form coherent and semantically ordered representations, while this coherence weakens for unseen concepts and suffers within-class domain shift. In LLMs, clearly distinct domains remain well separated, related subdomains move closer together, and the distinction between ambiguous topics collapses at both the mean and covariance level. These results suggest that conceptual separation can reveal structure that output accuracy alone cannot, and may serve as a useful diagnostic of how robustly a model represents the concepts it is asked to identify. Code and data available on \href{https://github.com/JaeeRoshniCapstoneProject/Are-You-Thinking-What-I-m-Thinking-Examining-Conceptual-Separation-in-Neural-Architectures}{GitHub}.

1 Introduction

The paper investigates whether neural networks preserve human-like conceptual separation in their internal activation spaces, beyond what output metrics reveal. It studies this question across CNNs and LLMs, including familiar, shifted, unseen, related, and ambiguous concepts.

  • Output-level metrics can coexist with shortcut learning, underspecification, and poor behavior under distribution shift, motivating analysis of activation-space geometry.
  • The study examines conceptual separation in CNNs and LLMs using geometric and distributional analyses of internal activations.CNNs provide a controlled setting through explicit ImageNet class labels, while LLMs extend the analysis to less clearly bounded language concepts.
  • The experiments progress from familiar ImageNet classes to unseen and shifted visual concepts, then to distinct, related, and ambiguous language domains.
  • The paper introduces an architecture-aware empirical framework for studying conceptual separation directly in neural activation spaces.
  • CNN representations are coherent for familiar classes but weaken for unseen concepts and within-class domain shifts, while LLM separation degrades as domains become more ambiguous.
  • The LLM analysis extends beyond mean-level geometry to covariance structure, testing whether distinctions absent from centroids remain visible in distributional shape.

2 Related Work

The paper builds on concept-based interpretability, representation-similarity analysis, probing, and distribution-shift research. It differs by examining how multiple concepts are organized relative to one another in activation space and how that organization changes under shifted inputs.

  • Concept-based interpretability explains networks using human-meaningful concepts, whereas this paper studies the organization of multiple concepts relative to one another.
  • Representational Similarity Analysis, SVCCA, and CKA compare or quantify similarity among learned representation spaces across layers or models.
  • Probing classifiers test whether linguistic or semantic properties can be decoded from internal representations, while this work focuses on conceptual separation itself.
  • Related out-of-distribution and domain-generalization work studies changed inputs, while this paper examines activation-distribution shifts as evidence of stable conceptual structure.

3 Experiments on Convolutional Neural Networks

The CNN experiments test whether activation spaces preserve coherent within-class representations and expected semantic relations, then examine unseen concepts and within-class variation. Familiar ImageNet classes are coherent and semantically ordered, while coherence weakens for unseen concepts and shifts with geographic context but not cat pose.

  • Setup: KL divergence compares relative neuron-activation distributions, providing a holistic measure of conceptual coherence and inter-class proximity.Concept blueprints summarize typical normalized activation patterns; lower divergence indicates closer agreement with a blueprint, while symmetric inter-class divergence averages both directions.
  • In-Distribution Concept Separation: ResNet-50 and MobileNetV2 represent same-class images more similarly than different-class images across Euclidean, PCA, cosine, and Mahalanobis analyses.For familiar cats, dogs, and cars, both architectures preserve the expected separation structure.
  • In-Distribution Concept Separation: Both architectures place cats and dogs closer together than either is to cars in activation space.KL divergence is lowest when images are compared with their own class blueprint and preserves the same semantic ordering across models.
  • Out-of-Distribution Experiments: Unseen Rangoli and microscopy concepts show substantially higher intra-class KL divergence than familiar ImageNet classes.Their images are also mapped across many unrelated ImageNet labels, providing a behavioral counterpart to weaker activation-level coherence.
  • Within-Class Distribution Shifts: Turkish road images consistently diverge more from an Indian-road blueprint than held-out Indian images, indicating sensitivity to within-class distribution shift.The blueprint comparison is proposed as a simple pre-deployment check for changed internal representations under new environments.
  • Within-Class Distribution Shifts: Sleeping and standing cat activation distributions nearly overlap, whereas geographic and contextual road changes produce a clear representational drift.The authors suggest pose may be absorbed into the broader cat representation, while road scenes vary across several feature-relevant visual dimensions.

4 Experiments on Large Language Models

Across two language models, representational separation tracks the human-perceived distinction between concept pairs: clear domains separate, related domains overlap more, and hate versus no-hate speech collapses at both centroid and covariance levels.

  • Experimental setup: The experiments compare three concept pairs spanning clear domain differences, related technical subdomains, and an ambiguous social boundary across GPT-OSS-20B and BERT-large-uncased.Sentence embeddings are mean-pooled from final-layer hidden states; separability is assessed using distance ratios, PCA projections, and distributional metrics.
  • Academic pairs: CS versus Shakespeare forms clearly separated clusters in both models, with cosine separability ratios exceeding 1.4.The observed covariance differences are also significant under permutation testing: 1.238 for GPT and 1.267 for BERT, both p<0.001.
  • Academic pairs: InfoSec versus Theory of Computation remains significantly separated but shows lower cosine ratios and more overlap than the maximally distinct academic pair.Its covariance differences remain significant, with ˜F = 1.268 for GPT and 1.195 for BERT, both p<0.001.
  • Hate versus no-hate speech: Hate versus no-hate speech has effectively no first-moment separation in either model, with complete geometric entanglement and a cosine ratio near unity.The per-neuron KL divergence is the smallest off-diagonal value in the six-category comparison, even below InfoSec versus Theory of Computation.
  • Second-moment analysis: Second-moment analysis preserves strong covariance distinctions for academic pairs but finds weak, non-robust differences for hate versus no-hate speech.For hate speech, ˜F = 0.416 (p=0.025) for GPT and 0.492 (p=0.130) for BERT; the Gaussian KL decomposition is 93–94% covariance-dominated.
  • Prompting as intervention: Prompting does not recover hate-speech separation and instead further reduces covariance differences, while CS versus Shakespeare remains significant after compression.GPT’s ˜F changes from 0.416 to 0.389 for hate speech, versus 1.238 to 1.157 for CS versus Shakespeare, which remains p<0.001.

5 Discussion

Across CNNs and LLMs, representational separability broadly follows human-perceived conceptual distinctness, weakening for shifted, unfamiliar, or ambiguous concepts. Activation-space analysis therefore provides a diagnostic complement to output-level evaluation.

  • Representational separability broadly tracks how distinct concepts are to humans across CNNs and LLMs.
  • Familiar ImageNet classes are coherent and well separated, but unfamiliar concepts show substantially lower intra-class coherence.
  • Geographic domain shifts alter road-image activation structure, whereas cat-pose variation does not produce the same observed sensitivity.
  • LLM representations progress from separated distinct domains to closer related subdomains and nearly indistinguishable hate-versus-no-hate concepts.
  • For ambiguous tasks, downstream classifiers may need to learn decision boundaries over representations lacking a clear conceptual distinction.
  • Conceptual separation is diagnostically useful before deploying models on new domains or ambiguous tasks, complementing output-level evaluation.

6 Limitations and Future Work

The findings are indicative rather than conclusive because the empirical scope, activation analysis, and prompting intervention are limited. Future work calls for broader validation and layer-wise analysis.

  • 6 Limitations and Future Work: The study uses 100 samples per class and four representative architectures, limiting the breadth of its empirical scope.
  • 6 Limitations and Future Work: Hate-speech results come from one platform and time, leaving generalizability across taxonomies and linguistic contexts uncertain.
  • 6 Limitations and Future Work: The analysis focuses mainly on the first two distributional moments and the final LLM transformer layer, potentially missing intermediate-layer structure.
  • 6 Limitations and Future Work: Prompting uses one classification-style template per concept pair, so richer prompting strategies may yield different insights.
  • 6 Limitations and Future Work: Future work proposes broader validation, a unified diagnostic metric, and layer-wise extensions across transformer layers.

A Dataset Examples

The appendix provides representative visual examples for the CNN experiments, covering familiar ImageNet classes, unfamiliar concepts, geographic shifts, and cat-pose variation.

  • In-distribution experiments: Representative cat, dog, and car images illustrate the in-distribution ImageNet-class experiments.
  • Unfamiliar-concept experiments: Rangoli and microscopy images illustrate unfamiliar concepts that form human-defined categories without explicit ImageNet classes.
  • Within-class distribution shift: Road images from India and Turkey illustrate held-out test sets for within-class geographic distribution shift.
  • Pose-variation experiment: Sleeping and standing cat images illustrate the manually selected held-out test sets for pose variation.

B PCA and Distance-Based Analysis

PCA projections and distance-based analyses show that cats, dogs, and cars occupy distinct representation regions, with stronger within-class than across-class similarity and cats closer to dogs than cars.

  • PCA projections show cats, dogs, and cars forming distinct regions across ResNet-50 and MobileNetV2 representations.
  • Within-class cosine and Mahalanobis distances are smaller than across-class distances for cats, dogs, and cars.
  • Cats and dogs are closer to each other than either is to cars in the representation geometry.
  • The distance-based pattern is consistent with KL analysis, indicating expected semantic structure across multiple geometric measures.

C Prediction Consistency for Out-of-Distribution Concepts

The study uses ImageNet predictions as a behavioural view of representational consistency, asking whether images from the same concept map to similar known classes. Familiar concepts produce concentrated predictions, whereas unseen concepts produce dispersed mappings.

  • Prediction consistency: ImageNet predictions are used to test whether different images from one concept map to similar known classes, without requiring correct labels for out-of-distribution concepts.Rangoli and microscopy are not ImageNet classes, so consistency—not classification accuracy—is the target.
  • Representation geometry: Euclidean, cosine, and Mahalanobis intra-class distances are consistently smaller than inter-class distances for cats, dogs, and cars.This geometric pattern indicates more compact representations within each familiar concept than between concepts.
  • Familiar concepts: Over 75% of cat images are predicted as cats, compared with around 90% for dogs and cars.Fine-grained labels are collapsed into broader semantic groups for readability.
  • Out-of-distribution concepts: Microscopy’s most common prediction covers 15% of ResNet-50 images and 12% of MobileNetV2 images, while rangoli’s covers 22% and 14%, respectively.These distributions are substantially less concentrated than those for familiar ImageNet concepts.
  • Out-of-distribution concepts: The top five predictions capture only 37–63% of microscopy and rangoli images, versus 89–99% for cats, dogs, and cars.Out-of-distribution predictions therefore spread across many unrelated ImageNet classes rather than one semantic group.
  • Prediction consistency: Top ImageNet predictions show relatively low variation within each of the cat, dog, and car concepts, unlike the diverse mappings observed for rangoli and microscopy.The comparison links behavioural prediction consistency with concept-specific representation structure.

D Representative examples from the LLM datasets

The LLM experiments use examples from academic and socially sensitive concept classes, with the appendix documenting their examples and analyses. The accompanying figure decomposes separation under unprompted and prompted conditions across two architectures.

  • Dataset examples: The LLM dataset includes Computer Science, Shakespeare, Information Security, Theory of Computation, Hate Speech, and No-Hate Speech examples.Academic pairs comprise short definitional or canonical sentences, while hate and no-hate examples come from social-media posts.
  • Dataset examples: Table 2 presents three examples from each LLM concept class.The table provides representative examples rather than the full text collections.
  • Analyses: The appendix contains distance matrices, six-category KL matrices, Frobenius null distributions, and moment decompositions for GPT-OSS-20B and BERT-large-uncased.These analyses cover both language-model architectures used in the study.
  • Analyses: For the hate-speech distinction, separability ratios collapse to unity and the covariance term contributes more than 90% of Gaussian KL across both architectures.Figure 16 compares unprompted and prompted conditions for GPT-OSS-20B and BERT-large-uncased.

F Hate Speech Data and Labelling Framework

The hate-speech analysis uses a domain-controlled corpus to isolate conceptual differences from writing style, topic, and platform effects. It combines dataset details, geometric comparisons, distributional tests, and annotation context.

  • Data and labels: The hate/no-hate partition contains 100 sentences per class from a 2020 U.S. presidential-election Twitter corpus.The binary partition is derived from the corpus’s hate/offensive-speech annotations.
  • Data and labels: Restricting the data to one event window and platform is intended to prevent domain shift from being conflated with the conceptual distinction.The design holds writing style, topic distribution, and platform conventions more constant.
  • Geometric analysis: Distance matrices compare CS with Shakespeare, InfoSec with Theory of Computation, and Hate Speech with No-Hate Speech using Euclidean and cosine distances for GPT and BERT.The figures organize within-block comparisons by architecture.
  • Data and labels: Hate-speech labels are socially contested, and annotators may disagree on boundary cases because of cultural context, personal sensitivity, and operational definitions.This contrasts with academic domain labels, which can have high inter-annotator agreement.
  • Distributional analysis: Hate Speech and No-Hate Speech have the smallest off-diagonal per-neuron KL divergence of any category pair.Figure 20 reports this comparison for both GPT and BERT.
  • Statistical testing: Permutation-test null distributions are provided for CS versus Shakespeare, Hate Speech, and InfoSec versus Theory of Computation.The tests include GPT and BERT conditions where specified, with prompted and unprompted comparisons for the latter pair.
Loading 2609.00764v1…