{"success":true,"partial":false,"results":[{"paperId":"8319239866974784291","primaryId":"arxiv:1706.03762","ids":{"arxiv":["1706.03762"]},"title":"Attention Is All You Need","abstract":"The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.","authors":["Ashish Vaswani","Noam Shazeer","Niki Parmar","Jakob Uszkoreit","Llion Jones","Aidan N. Gomez","Lukasz Kaiser","Illia Polosukhin"],"createdDate":"2017-06-12","updateDate":"2023-08-03","score":0.9852259683067269},{"paperId":"5767108596850106300","primaryId":"arxiv:2510.19861","ids":{"arxiv":["2510.19861"]},"title":"Some Attention is All You Need for Retrieval","abstract":"We demonstrate complete functional segregation in hybrid SSM-Transformer architectures: retrieval depends exclusively on self-attention layers. Across RecurrentGemma-2B/9B and Jamba-Mini-1.6, attention ablation causes catastrophic retrieval failure (0% accuracy), while SSM layers show no compensatory mechanisms even with improved prompting. Conversely, sparsifying attention to just 15% of heads maintains near-perfect retrieval while preserving 84% MMLU performance, suggesting self-attention specializes primarily for retrieval tasks. We identify precise mechanistic requirements for retrieval: needle tokens must be exposed during generation and sufficient context must be available during prefill or generation. This strict functional specialization challenges assumptions about redundancy in hybrid architectures and suggests these models operate as specialized modules rather than integrated systems, with immediate implications for architecture optimization and interpretability.","authors":["Felix Michalak","Steven Abreu"],"createdDate":"2025-10-21","updateDate":"2025-10-24","score":0.9514318456497438},{"paperId":"1231078796007356867","primaryId":"arxiv:2606.02332","ids":{"arxiv":["2606.02332"]},"title":"Forget Attention: Importance-Aware Attention Is All You Need","abstract":"Attention retrieves content precisely, but its scores depend only on query--key products, leaving order and recency to positional encodings. Conversely, state-space models (SSMs) such as Mamba-3 track order and recency through input-dependent state transitions that decay and rotate, yet compressing context into a fixed-size state hampers precise retrieval. Hybrids like Samba and Hymba exploit this complementarity only across layers or heads, so SSM transitions never shape attention scores directly. The closest prior work, the Forgetting Transformer (FoX), adds an input-dependent forget gate to the scores but reweights attention by recency alone. We propose SSM-Informed Softmax Attention (SISA), which builds Mamba-3-style state transitions, rotation included, directly into the scores alongside FoX's gate. Each logit receives the weight such an SSM would give that token pair: a decayed match between rotated SSM projections. This rotation already encodes relative position, so SISA drops RoPE; restoring it raises loss. SISA outperforms parameter-matched Transformer and FoX in language modeling and length extrapolation. Among the attention models we compare, hybrids included, only SISA benefits from context beyond the training window. Building SSM transitions into the scores yields stronger attention, and our kernels make it faster and leaner than baselines on their official or standard kernels.","authors":["Suhyeong Shin","Yeongwook Yang"],"createdDate":"2026-10-28","updateDate":"2026-10-30","score":0.9377602233752947}]}