Let the Heads Talk: Beyond Diagonal Graph Attention
TL;DR We connect matrix-valued transport in sheaf networks to communication between attention heads, motivating a graph attention mechanism that learns how heads interact along each edge.
Publications
My research explores how mathematical structure shapes what neural networks learn, from symmetry and geometry to information processing in language models.
Google Scholar ↗2026
Remember to Forget: Gated Adaptive Positional Encoding
TL;DR We introduce GAPE, a mask that augments positional encodings with two mechanisms: one suppresses irrelevant distant context, while the other protects important tokens from that suppression.
Lyapunov Spectral Analysis of Loop Transformer Dynamics
TL;DR Lyapunov spectra distinguish mildly chaotic dynamics in Ouro from contracting dynamics in Huginn, tracing the difference to opposing layer contributions.
Algebraic Priors for Approximately Equivariant Networks
TL;DR We prove that, for finite groups, equivariant latent spaces almost surely contain copies of the regular representation, then use this structure as a parameter-free algebraic prior for approximate equivariance.
2025
Entropy-Lens: Uncovering Decision Strategies in LLMs
TL;DR Layer-by-layer entropy of token predictions reveals how LLMs expand and prune candidate continuations, with patterns specific to model families and tasks.
2024
Metric Learning for Clifford Group Equivariant Neural Networks
TL;DR We extend Clifford group equivariant networks to learn their latent metric from data, allowing the geometry of their representations to adapt to the task.
Bias/Variance is not the same as Approximation/Estimation
TL;DR Two classical explanations of prediction error are often conflated. We establish precisely how bias–variance and approximation–estimation differ and how they connect.