Probing BERT in Hyperbolic Spaces

1 INTRODUCTION

• Euclidean geometry is not established for syntactic subspaces, despite structural probes using squared Euclidean distance to recover dependency trees.The paper notes that Euclidean space has intrinsic difficulties modeling trees.Source: §1 · p. 1
• Hyperbolic space is proposed for syntax because its exponentially growing volume better matches tree growth than Euclidean space’s polynomially growing volume.The authors motivate it as providing enough space for tree nodes.Source: §1 · p. 1
• The Poincaré probe recovers higher parsing scores than restricted-capacity Euclidean probes, especially for deeper trees, longer edges, and longer sentences.It does not score baseline embeddings without syntax higher than Euclidean probes, supporting the authors’ distinction between probing and parsing.Source: §1 · p. 2
• The sentiment probe models positive and negative polarity with two meta-representations and finds fine-grained sentiment encoding in BERT.Words with stronger polarity are closer to the corresponding meta-representation; contextualization can change their geometric locations.Source: §1 · p. 2
• The authors present Poincaré probes as tools for revealing hierarchical linguistic structure in BERT and suggest its representations may occupy non-Euclidean metric spaces.The work is exploratory and demonstrates its approach with experiments and visualization.Source: §1 · pp. 2–3

Probing BERT

• Structural probes study how syntax trees are geometrically encoded in BERT, while the Poincaré probe follows this line with hyperbolic geometry and restricted capacity.The paper also frames its sentiment extension as a semantic probe of BERT’s geometric encoding of word sentiment.Source: §2 · p. 3
• The paper distinguishes its aim from other linguistic probes by focusing on geometric representations of syntax and sentiment in BERT.The cited related work also covers morphology, word sense, phrases, semantic fragments, and other syntax and semantics.Source: §2 · p. 3

Hyperbolic Deep Learning

• Prior Poincaré embedding methods learn hierarchies, while this work uses hyperbolic spaces to probe contextualized embeddings.The paper uses Möbius operations and Riemannian optimization techniques from prior hyperbolic-learning work as training infrastructure.Source: §2 · p. 3

3 POINCARÉ PROBE

• The probe uses the standard Poincaré ball with curvature parameter c = 1 and hyperbolic distance based on Möbius addition.The exponential map sends vectors from the tangent space into the hyperbolic space; the Euclidean space is recovered as c approaches zero.Source: §3 · p. 3
• The probe maps BERT embeddings into a tangent space and then into a Poincaré ball, where another linear map constructs the syntactic or sentiment subspace.The projection uses P from the embedding space to the tangent space and Q within the hyperbolic space.Source: §3 · pp. 3–4
• Compared with Euclidean probes, Poincaré probes do not score ELMO0 and LINEAR embeddings higher, but recover higher scores for contextualized embeddings with smaller probe capacity.The authors interpret the latter comparison as greater sensitivity to syntax in contextualized embeddings.Source: Table 1 · p. 4

4 PROBING SYNTAX

• The probe uses squared Poincaré distances between word pairs to recreate tree distances and squared Poincaré distance to the origin to recreate word depth, with two linear transforms (P and Q) restricting modeling power.Optimization uses Adam initialized at learning rate 0.001 and trains for up to 40 epochs. The learning rate decays, and model selection is based on dev loss.Source: §4 · p. 4

4.1 EXPERIMENTAL SETTINGS

• Experiments use Penn Treebank dependencies, restrict probe rank to 64, and assess tree distances and depths with attachment, correlation, and root-identification metrics.The baselines include ELMO0 and LINEAR, expected to lack syntactic information, and ELMO1 and BERT*, expected to encode rich syntax.Source: §4.1 · p. 5

4.2 RESULTS

• Poincaré probes match Euclidean probes on ELMO0 and LINEAR but score higher on deeply contextualized embeddings, consistent with sensitivity to encoded syntax rather than parser capacity.The authors suggest Euclidean probes may underestimate syntax in contextualized embeddings because they cannot model trees.Source: §4 · p. 4, §4.2 · pp. 5–6, §4.1 · p. 5
• Syntax is strongest in middle BERT layers; scores do not increase beyond rank 64, and Poincaré probes are especially better at low ranks.The authors posit an intrinsic syntactic subspace near 64 dimensions but leave this for future investigation.Source: Figure 2 · p. 5, §4.2 · p. 6
• Poincaré probes score better on longer sentences, align more closely with gold edge-length distributions, and recover long-average-length edge types better than Euclidean probes.A PCA visualization also shows a more organized word hierarchy in the Poincaré ball, with the root at the origin and words arranged by depth.Source: §4.2 · pp. 6–7, Figure 3 · p. 6, Figure 4 · p. 6

4.3 SUMMARY ON SYNTACTIC PROBE

• The optimal curvature is about -1; as curvature approaches 0, Poincaré scores converge to Euclidean scores, and curvature 0 recovers the Euclidean probe.The authors describe the geometric interpretation as exploratory: better reconstruction does not establish that BERT’s syntactic subspaces are hyperbolic.Source: §4.2 · pp. 6–7, §4.3 · p. 7

5 PROBING SENTIMENT

• The Poincaré probe recovers fine-grained positive, negative, and neutral word sentiment despite training on sentence-level binary labels.It uses two meta-embeddings as positive and negative roots and locates words by their distances to them.Source: §5 · p. 7

5.2 VISUALIZATION AND LEXICALLY-CONTROLLED CONTEXTUALIZATION

• PCA visualization in hyperbolic space is empirically effective despite distortion, and the authors leave hyperbolic dimension reduction for future work.They note that hyperbolic spaces have no perfect analogy of PCA.Source: §5 · p. 7

5.1 WORD POLARITIES BASED ON GEOMETRIC DISTANCES

• Figure 5 shows sentiment emerging around layer 9, deeper than syntax around layer 7.Its panels A and B project a sentence into the two spaces; dashed word connections indicate no significant distance difference between the two meta-embeddings.Source: Figure 5 · p. 8
• Both probes outperform a BiLSTM classifier, and fixing class meta-representations substantially reduces Euclidean performance while the Poincaré probe can perform better.The authors interpret this pattern as suggesting sentiment may be encoded geometrically; they also report that sentiment emerges around layer 9, later than syntax around layer 7.Source: §5.1 · p. 8
• The recovered sentiment words align better with human intuition for the Poincaré probe than for the Euclidean probe.Table 3 presents top sentiment words recovered by both probes, with colors marking words that align better with intuition.Source: §5.1 · p. 8, Table 3 · p. 8

5.2 VISUALIZATION AND LEXICALLY-CONTROLLED CONTEXTUALIZATION

• Both probes distinguish fine-grained word sentiment in sentence visualizations, while the Poincaré probe separates the two meta-embeddings more clearly and gives more compact embeddings.The authors describe this as a general pattern rather than a special case.Source: §5.2 · p. 9
• Stronger sentiment words locate closer to their corresponding meta-embeddings, while a negative sentence context draws all words closer to the negative embedding, including under simple negation.The comparisons use lexically controlled sentence variants.Source: §5.2 · p. 9, Figure 6 · p. 9
• BERT fails on double negation, gives reasonable localization for an ambiguous sentence, and gives correct localization for a sentence with satire.For the ambiguous sentence, most words are not significantly closer to either meta-embedding.Source: §5.2 · p. 9, Figure 7 · p. 9

6 CONCLUSION

• Poincaré probes recover hyperbolic subspaces for hierarchical information encoded in BERT and provide analytical tools and modeling possibilities concerning tree structures, localizations, and contextualization.The conclusion frames the work as an exploration of BERT embedding geometry in hyperbolic spaces.Source: §6 · p. 9

A ADDITIONAL RESULTS ON PROBING SYNTAX

• Table 4 reports scores for probes trained on distance and depth tasks simultaneously.The accompanying syntax-subspace plots include longer sentences and deeper dev-set trees.Source: A ADDITIONAL RESULTS ON PROBING SYNTAX · p. 12, Table 4 · p. 12
• The syntax-subspace visualizations include a sentence about the Dutch company Burmah and its stake increase.The sentence is identified as Figure 8.Source: Figure 8 · p. 12

A.1 A GRU LAYER TRANSFORMS A PROBE TO BE A PARSER

• The parser variant adds a GRU layer, using each word's GRU hidden state to encode syntax for the Euclidean probe.The Poincaré probe's corresponding equations follow this description.Source: §A.1 · pp. 12, 14
• GRU-based probes generally score higher than probes based on linear maps, while both Euclidean and Poincaré probes score much higher on ELMo0.The authors interpret the ELMo0 result as indicating that these probes can learn syntax trees, rather than probe from deep models.Source: §A.1 · p. 14
• Table 5 reports scores for probes using local information on distance and depth tasks trained simultaneously.The supplied passage describes the table's scope but does not give its individual scores.Source: Table 5 · p. 14

B ADDITIONAL RESULTS ON PROBING SENTIMENT

• Movie Review 5 is balanced, and 10% of its sentences are randomly assigned separately to the development and test sets.The dataset has no official split.Source: B ADDITIONAL RESULTS ON PROBING… · p. 14
• Top sentiment words are ranked by the distance gap between the two meta-embeddings, addressing neutral words that can be equally close to both; subwords and numbers are excluded.Table 8 extends Table 3.Source: B ADDITIONAL RESULTS ON PROBING… · p. 14, Table 8 · p. 15

C CLOSED-FORM FORMULAS OF M ¨OBIUS OPERATIONS

• Appendix C restates definitions of fundamental mathematical operations for the generalized Poincaré ball and refers readers to Ganea et al. (2018) for details.Source: C CLOSED-FORM FORMULAS OF M ¨OBIUS… · p. 14
• The appendix introduces definitions for Möbius matrix-vector multiplication and the exponential and logarithmic maps.The supplied passages identify these operations but do not give their full definitions.Source: C CLOSED-FORM FORMULAS OF M ¨OBIUS… · pp. 15–16

• To make the comparison with the Poincaré probe fairer, the authors extend Euclidean probes to two linear transforms with a nonlinearity between them.Source: D ADDITIONAL RESULTS ON EUCLIDEAN… · p. 16
• Table 9 reports results for the Euclidean probe variations on the distance task; the supplied passage does not state a comparison outcome.Source: Table 9 · p. 16

• The curvature experiments train two-dimensional probes simultaneously on distance and depth tasks using BERTBASE7.Their scores are reported in Table 10.Source: E ADDITIONAL RESULTS ON CURVATURE · p. 16, Table 10 · p. 16
• In Figures 17–19, the circles mark the boundary of the Poincaré models.Source: E ADDITIONAL RESULTS ON CURVATURE · p. 16