Source-linked AI summary
A Transformer-Based Delta Expression Encoder for Psilocybin Transcriptional Response: Architecture, Representations, and Biological Validation
Sai Jayakumar
TL;DR
To understand heterogeneous psilocybin responses, the paper models transcriptional perturbations at cell-type resolution. It trains a Transformer delta expression encoder on pseudobulk profiles and finds separable biological representations and attention-derived drug modules, alongside an HTR2A result inconsistent with simple dose dependence.
Problem
Psilocybin response is heterogeneous and its biological basis remains poorly understood, motivating cell-type-resolved models of transcriptional perturbation.
Source: Introduction · p. 2Method
A 4-layer Transformer classifies DEG status per gene per timepoint from pseudobulk expression, using sentinel values and a two-tier curriculum for unmeasured timepoints.
Source: Introduction · p. 2Results
The model learns drug- and cell-type-separable representations and attention recovers pharmacologically correct drug-specific modules, while the HTR2A-gating test runs counter to simple dose dependence.
Source: Discussion · p. 20Takeaways & Limitations
Attention-derived co-regulation modules and separable embeddings provide evidence of pharmacological structure learned without pathway supervision.
Source: Discussion · p. 20Between-subjects sampling provides no real multi-timepoint animal trajectories, limiting recovery of the biphasic response; independent-dataset validation and multimodal integration remain future work.
Source: Limitations and future directions · p. 21
Abstract
from arXiv · showhide
Understanding why individuals respond differently to psilocybin requires modeling the drug's transcriptional perturbation signature at the cell-type level. I present a Transformer-based delta expression encoder that learns to classify differential gene expression status - upregulated, downregulated, or neutral - from single-nucleus RNA-sequencing data, without supervision from pathway annotations or prior biological knowledge. The model is trained on pseudobulk profiles from 623 examples spanning 18 cell types, 2 drug conditions, and 6 timepoints derived from the Liao et al. 2025 dataset, and achieves 69.4% weighted classification accuracy. Three principal findings are reported, alongside one direct test of a published hypothesis that returned a result inconsistent with that hypothesis. First, per-cell-type classification accuracy ranges from 28.3% (L2/3 IT, a primary HTR2A-expressing psilocybin target) to 99.6% (endothelial cells), consistent with known psilocybin response biology. Second, psilocybin-induced transcriptional downregulation is significantly more stereotyped across individuals than upregulation (Mann-Whitney U=18615.0, p<0.0001), a novel finding with a cortical depth gradient across excitatory subtypes. Third, attention-guided gene co-regulation analysis recovers drug-specific modules without pathway supervision. Separately, a direct test of whether baseline HTR2A expression predicts drug-response separability across cell types found a significant negative correlation (Spearman r = -0.7088, p = 0.0021), the opposite of what a simple HTR2A-gating account would predict.
Introduction
The paper motivates cell-type-resolved modeling of psilocybin’s transcriptional perturbation and introduces a Transformer encoder for classifying gene-expression change. It tests biological structure without pathway supervision while reporting response patterns and architectural limitations.
Introduction
- Clinical response to psilocybin is heterogeneous, while its biological basis remains poorly understood, motivating models of transcriptional perturbation at cell-type resolution.The motivation contrasts cell-type-resolved modeling with bulk tissue averages.Source: Introduction · p. 2
- A 4-layer Transformer classifies DEG status per gene per timepoint from pseudobulk profiles, using sentinel values and a two-tier curriculum for unmeasured timepoints.The design addresses destructive sampling, which gives each animal a real measurement at exactly one timepoint.Source: Introduction · p. 2
- Cell-type accuracy follows a qualitative gradient consistent with known response hierarchy, while a direct HTR2A-gating test contradicts simple HTR2A-dose dependence.The paper presents the HTR2A relationship as a hypothesis to test, not an assumption to validate.Source: Introduction · p. 2
- The paper reports more stereotyped psilocybin-induced downregulation than upregulation, with a cortical depth gradient across excitatory subtypes.This is presented as a novel finding.Source: Introduction · p. 3
- Attention analysis recovers drug-specific co-regulation modules, including serotonin- and NMDA-receptor pathway segregation, without pathway supervision.The contribution list also describes an attention-priority versus classical DEG significance test.Source: Introduction · p. 3
- The paper documents temporal-structure failure and attention–DEG divergence, and leaves integration with other biological layers to future work.The encoder addresses the transcriptional layer in isolation.Source: Introduction · p. 3
1. The delta expression encoder learns biologically coherent cell-type representations
The encoder learns embeddings that separate drug and cell-type information and reflect biological cell-class structure. Its final checkpoint outperforms simple accuracy baselines, while additional probes and clustering characterize the learned representation.
1. The delta expression encoder learns biologically coherent cell-type representations
- 69.4% weighted accuracy (loss 0.7655) at 300 epochs exceeded majority-class (32.7%) and stratified-random (32.8%) baselines by 2.1×.The earlier 150-epoch checkpoint used for UMAP achieved 70.3% accuracy (loss 0.7804); PCA silhouette scores used an epoch-50 checkpoint.Source: §1 · p. 3
- In the top 10 principal components, drug and cell-type silhouettes were 0.1796 and 0.2081, respectively; in 2D UMAP they were 0.0618 and −0.0437.The paper reports UMAP as a visualization rather than the primary separability metric.Source: §1 · p. 3
- Drug identity and cell type were perfectly linearly decodable (1.000 ± 0.000 each), while timepoint was near-perfectly decodable (0.974 ± 0.016) and animal identity was at chance (0.100 ± 0.038).These balanced-accuracy probes used frozen 128-dimensional embeddings and 5-fold cross-validation.Source: §1 · p. 3
- All eight excitatory cortical subtypes formed one cluster, while inhibitory interneurons and glia intermingled; Sncg interneurons and endothelial cells were isolated at maximum distance.Figure 1B describes Ward-linkage clustering of cell-type mean embeddings in PCA space, with branch lengths reflecting PCA distances.Source: §1 · p. 4, Figure 1B · p. 5
- Swapping drug labels shifted embeddings by 14.16 ± 3.61 L2 units, below the natural cross-drug distance of 21.65 and near within-drug variation of 16.19.The perturbation check supports drug identity being encoded, but not as the representation’s sole organizing principle.Source: §1 · p. 4
- Ten principal components explain 90% of embedding variance, and a probe for neuronal class achieved 1.000 ± 0.000 balanced accuracy despite that grouping never being a training label.The reported effective rank is approximately 10/128.Source: §1 · p. 4
2. Psilocybin-induced downregulation is more stereotyped than upregulation across individuals
Psilocybin-induced DOWN labels are more consistent across individuals than UP labels, and the model recalls DOWN tokens more reliably. The variance difference appears across all eight examined excitatory subtypes, while a proposed regulatory explanation remains untested.
2. Psilocybin-induced downregulation is more stereotyped than upregulation across individuals
- DOWN-labeled genes had lower inter-individual variance than UP-labeled genes (mean std 0.1690 vs 0.2403; Mann-Whitney U=18615.0, p<0.0001).Median standard deviations were 0.1443 for DOWN and 0.2164 for UP.Source: §2 · p. 5, Figure 2B · p. 6
- Across eight excitatory cortical subtypes, the UP:DOWN variance ratio ranged from approximately 1.07× to 1.61×; six subtype comparisons were significant and two trended in the same direction.The subtype set included L2/3 IT, L4/5 IT, two L5 subtypes, two L6 subtypes, L6b, and L5 NP.Source: §2 · p. 5
- Differential stereotypy of repressive versus activating transcriptional responses is reported as a novel observation, while the proposed repressor-complex explanation remains untested here.The proposed account invokes uniform repressor-complex responses versus variation in chromatin accessibility, co-activator availability, or prior activity state.Source: §2 · p. 5
- DOWN-token recall was 52.1% versus 23.7% for UP tokens, and the model more often confused DOWN with NEUTRAL than with UP.For UP tokens, DOWN predictions occurred almost as often as NEUTRAL predictions, consistent with less directionally conservative errors.Source: §2 · pp. 5–6, Figure 2A · p. 6
3. Cell-type accuracy gradient is qualitatively consistent with known biology; a direct quantitative test of HTR2A-gating is not
Cell-type accuracy spans from low values in excitatory intratelencephalic neurons to near-ceiling values in endothelial cells, while accuracy is negatively associated with independent DEG counts. Baseline HTR2A expression does not show the positive association with drug-response separability predicted by a simple gating account.
3. Cell-type accuracy gradient is qualitatively consistent with known biology; a direct quantitative test of HTR2A-gating is not
- Accuracy ranged from 28.3% in L2/3 IT neurons to 99.6% in endothelial cells, with the three lowest values in excitatory intratelencephalic neurons.Non-neuronal near-ceiling accuracy may reflect near-universal NEUTRAL prediction rather than gene-selective inference; excitatory examples also had high attention entropy.Source: §3 · p. 7, Figure 3B · p. 9
- Model accuracy had a significant negative relationship with significant psilocybin DEG counts across cell types (Spearman ρ = −0.885, p < 0.001).Accuracy was positively correlated with mean absolute log2FC among significant DEGs (Spearman ρ = +0.885, p < 0.001).Source: §3 · p. 7, Figure 3A · p. 7
- Baseline HTR2A expression was negatively correlated with drug-response silhouette score across 16 cell types (Spearman ρ = −0.7088, p = 0.0021), opposite to a simple HTR2A-dose gating prediction.The silhouette score operationalizes separation between psilocybin and ketamine embeddings in PCA space, pooled across animals.Source: §3 · pp. 7–8, Figure 3C · p. 10
- Baseline HTR2A expression showed no significant relationship with DEG count (ρ = 0.3706, p = 0.1577) and a significant negative relationship with mean absolute log fold-change (ρ = −0.6559, p = 0.0058).At the individual-animal level, no consistent positive drug-axis projection signal appeared across excitatory subtypes (n_pos = 2/11; n_neg = 8/11).Source: §3 · p. 8
4. Attention analysis recovers drug-specific gene co-regulation modules
The model's attention weights were clustered to identify gene co-regulation modules, many of which mapped to known pathways. Drug-selective clusters included pharmacologically coherent content, although the global serotonin-receptor module was not specific to L2/3 IT neurons.
4. Attention analysis recovers drug-specific gene co-regulation modules
- Clustering the gene-gene attention matrix produced 15 modules, 14 annotatable to known pathways; drug-selective modules included serotonin receptor and alkaloid-response clusters for psilocybin and kainate-receptor and postsynaptic-structure clusters for ketamine.The underlying 510-gene panel was preselected for differential-expression significance, so the informative test was correspondence of recovered pathways to drug-specific pharmacology.Source: §4 · p. 10, Figure 4A · p. 12
- Psilocybin-dominant clusters included C2 (1.67×) and C8 (1.52×), while ketamine-dominant clusters included C12 (0.57×), with drug-selective structure emerging without receptor-mechanism supervision.C12 was anchored by Grin2b; C8 included Fos and Bdnf, and the separation of Grin2b from Grin2a was recovered from expression co-variation alone.Source: §4 · pp. 10–11
- Within L2/3 IT neurons, the most psilocybin-dominant cluster contained cytoskeletal and glial-transporter genes rather than canonical serotonin receptors, indicating the global receptor module was not L2/3 IT-specific.The L2/3 IT analysis used 17 psilocybin and 17 ketamine examples; its most ketamine-dominant cluster contained synaptic-vesicle and ion-channel genes.Source: §4 · p. 11
- Incoming attention to Htr2a was diffuse in layers 1–2, became sharply sparse with an approximately threefold magnitude increase in layer 3, and remained sparse after refinement in layer 4.This pattern suggests drug-specific gene prioritization emerges primarily at layer 3 rather than accumulating gradually.Source: §4 · p. 12, Figure 4B · p. 13
5. Attention priorities show no detectable enrichment for classical DEG significance
Temporal attention priorities did not overlap with statistically significant L2/3 IT DEGs at 1h or 72h. This null result limits the temporal-attention analysis specifically: prioritized genes were distinct from classical DEG significance at the resolution tested.
5. Attention priorities show no detectable enrichment for classical DEG significance
- The top 30 genes by temporal attention variability overlapped zero significant L2/3 IT DEGs at both 1h and 72h (hypergeometric p = 1.0 at each timepoint).Expected overlaps were 1.3 genes at 1h and 0.7 at 72h; testing the top 50 and top 100 attention genes did not change the conclusion.Source: §5 · p. 13, Figure 5 · p. 14
- Temporal attention priorities tracked a signal distinct from classical differential-expression significance in the tested L2/3 IT comparisons.The result does not establish whether that distinct signal is pharmacologically meaningful; the passage frames this as an unresolved alternative interpretation.Source: §5 · p. 13
6. Attention flags a candidate sustained suppression pattern in Htr1f
Htr1f stood out in temporal attention analysis and showed an early psilocybin suppression followed by lower expression at 72 hours. The later dip is not statistically significant in L2/3 IT, and platform-dependent baseline-expression disagreement makes the finding hypothesis-generating.
Attention and expression trajectory
- CV = 1.668 placed Htr1f fourth among 510 genes for temporal attention variability; its 72h attention weight was about 9× its 24h value.No other serotonin receptor in the panel showed a comparable temporal spike.Source: §6 · p. 14
- In L2/3 IT neurons, Htr1f suppression at 1h was significant (p<0.001), but the 72h dip was not (p=0.22); expression was lower than ketamine in 14 of 16 cell types.Across cell types, psilocybin expression fell at 1h, partly recovered at 2–4h, and declined again at 72h, unlike ketamine’s relatively narrow range.Source: §6 · p. 14, Figure 6 · p. 15
Convergent analyses
- Temporal attention flagged Htr1f, and co-attention clustering independently placed it in the most psilocybin-dominant serotonin-receptor cluster (P/K ratio 1.665×).That cluster also contained Tek, which is expressed in vascular endothelium.Source: §6 · pp. 14–15
Interpretive caveat
- Baseline Htr1f expression differs between SMART-Seq and MERFISH references, so the apparent 72h suppression could partly reflect a sequencing-method artifact.The source also notes that Htr1f’s function in this circuit is not well characterized.Source: §6 · p. 14
7. Temporal structure is not recovered, and the mechanistic reason is the training data design
The model did not recover psilocybin’s published biphasic temporal response pattern. The authors attribute this limitation to training examples that rarely contain real measurements across multiple timepoints.
Published response pattern
- The reference response is biphasic, with an early 1–2h wave, near-silence at 4–24h, and a return at 72h, which the model failed to recover.The pattern was identified by Liao et al. 2025.Source: §7 · p. 15
Training-data explanation
- Because animals were measured at only one post-administration timepoint, individual examples contain no real temporal trajectory; only 35 of 623 examples provide complete temporal coverage.The authors identify this between-subjects data structure, rather than model capacity, as the reason the biphasic pattern was not learned.Source: §7 · p. 15
Temporal structure test
- Figure 7 shows uniformly high psilocybin–ketamine score correlations across timepoints (r = 0.916, 0.909, 0.901, 0.897, 0.908), with only a 0.019 peak-to-trough difference.The difference was directionally consistent with the expected pattern but negligible in magnitude.Source: §7 · p. 15, Figure 7 · p. 16, Figure 6c
- Figure 7’s confident-score fraction rose monotonically from 1h to 24h before dropping, opposite the two psilocybin peaks at 1–2h and 72h reported by Liao et al. 2025.The tested fraction comprised genes with |score| > 0.1.Source: §7 · p. 15, Figure 6c
Methods
The methods build a Transformer encoder from pseudobulk single-nucleus RNA-sequencing profiles, with sentinel values marking unmeasured timepoints and separate analyses probing learned representations, attention, and response patterns.
Dataset / Gene panel / Pseudobulk construction
- The study used Liao et al. 2025 mouse medial frontal cortex data, selecting 510 genes and averaging nuclei by animal, cell type, drug, and timepoint.The panel combined statistical differential-expression criteria with preselected pharmacologically relevant genes; pseudobulk combinations with fewer than 10 nuclei were excluded.Source: Dataset · p. 16, Gene panel · p. 16, Pseudobulk construction · p. 16
Training example construction
- Each individual example was a 510 × 6 animal-by-cell-type matrix with a shared control mean at 0h and real drug expression only at that animal’s measured timepoint.Other drug-timepoint columns were −1.0 sentinels; 35 population-mean examples supplied complete temporal coverage, yielding 623 examples total.Source: Training example construction · pp. 16–17
Sentinel value rationale / Per-example normalization
- Sentinel values marked unmeasured timepoints, and inputs were normalized across non-sentinel values to reduce shortcuts from imputation or overall expression magnitude.Earlier population-mean imputation could let the model infer cell type from imputed values; −1.0 was never produced by real data.Source: Sentinel value rationale · p. 17, Per-example normalization · p. 17
Model architecture
- The DeltaExpressionEncoder processes 3,060 tokens from 510 genes across six timepoints using a four-layer Transformer, with expression, gene, time, cell-type, and drug information in token representations.Sentinel positions use a learned mask token; each token is classified into DOWN, NEUTRAL, UP, or BASELINE, and mean-pooled representations support embedding analyses.Source: Model architecture · p. 17, Figure 8 · p. 18
Training
- Training randomly masked 25% of real-value tokens and applied class-weighted cross-entropy only to masked tokens, excluding the 0h/BASELINE column from loss.Because 94.0% of evaluable tokens were NEUTRAL, weighted accuracy was the primary metric; the final 300-epoch checkpoint achieved 69.4% weighted accuracy.Source: Training · p. 19
Embedding probing
- Frozen-embedding probes tested drug, cell type, timepoint, animal identity, and biological class; PCA and UMAP separability and drug-swap embedding shifts were also assessed.Linear probes used five-fold stratified cross-validation, while the biological-class probe tested structure beyond the explicit conditioning inputs.Source: Embedding probing · p. 19
Attention analysis / Attention-DEG enrichment test
- Attention analyses ranked genes by temporal attention variability and clustered gene co-attention, with pathway enrichment and DEG-overlap tests evaluating the resulting patterns.The co-attention matrix was clustered after z-scoring and sparsifying; the top 30/50/100 temporally variable genes were tested against Liao et al. DEGs.Source: Attention analysis · p. 19, Attention-DEG enrichment test · p. 19
HTR2A-gating tests / Variance asymmetry analysis
- Baseline HTR2A expression was correlated with response separability and DEG measures, while DOWN- and UP-label variance across individual profiles was compared using Mann–Whitney tests.Variance comparisons were performed both pooled and by excitatory subtype.Source: HTR2A-gating tests · p. 19, Variance asymmetry analysis · p. 20
Discussion
The discussion presents attention-derived gene modules as evidence of pharmacological structure, while treating HTR2A-gating and attention–DEG enrichment results as limits on interpretation. It also identifies boundaries in temporal recovery, non-neuronal attention, and translation from mouse data to human response prediction.
Discussion
- Without pathway supervision, the encoder learns drug- and cell-type-separable representations and recovers pharmacologically correct drug-specific gene modules through attention.The discussion calls these modules its strongest evidence of pharmacological structure, while noting that attention priorities do not detectably overlap with classical DEG significance.Source: Discussion · p. 20
- Three tests found null or negative relationships between baseline HTR2A expression and drug-response separability or DEG magnitude, opposing a simple linear gating account.The authors distinguish this result from evidence that HTR2A and pyramidal cell types are necessary for behavioral effects; ceiling effects or a permissive receptor role are proposed possibilities, not established mechanisms.Source: Discussion · p. 20
- Downregulation's greater stereotypy across individuals is presented as a robust observation across excitatory subtypes, while the proposed repressor-complex explanation remains untested.The mechanistic account is framed as a hypothesis motivated by the variance asymmetry, not a finding established by this study.Source: Discussion · p. 20
- The discussion treats mouse-to-human translation as a major challenge because required-scale human psilocybin transcriptomics data are unavailable.It proposes a cross-species module using orthologous expression to align mouse and human cell-type representations, and notes future work on additional modalities and independent-dataset validation.Source: Discussion · pp. 20–21, Limitations and future directions · p. 21
Limitations and future directions
- The between-subjects training design prevents recovery of the biphasic temporal response because no animal contributes a real multi-timepoint trajectory.The authors state that addressing this limitation would require within-subject longitudinal psilocybin data in mouse, which does not currently exist.Source: Limitations and future directions · p. 21
- Near-uniform attention in non-neuronal cell types suggests heuristic rather than gene-level inference, qualifying interpretation of their high classification accuracy.The discussion also characterizes the HTR2A-gating and attention–DEG enrichment findings as negative or null tests that constrain claims about what the model learned.Source: Limitations and future directions · p. 21
Figure List
The figure list covers embedding structure, classification and response variability, attention patterns, and temporal-response validation. It also includes an architecture schematic and a gene-expression trajectory.
Embedding structure
- Figures 1A and 1B show PCA embeddings colored by drug and cell type, and hierarchical clustering of cell-type mean embeddings.Source: Figure List · p. 22
Classification and response
- Figures 2A–3B present the classification confusion matrix, DOWN-versus-UP variance distributions, accuracy–response-magnitude relationship, and per-cell-type accuracy.Source: Figure List · p. 22
Attention and HTR2A
- Figures 3C–4B cover HTR2A expression versus drug-response silhouette, gene-gene co-attention with cluster annotations, and incoming attention to Htr2a across layers.Source: Figure List · p. 22
Additional figures
- Figures 5–8 depict attention-priority versus DEG overlap, Htr1f expression trajectories, temporal-grammar validation, and the model architecture.Source: Figure List · pp. 22–23