QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm

Qirui Zhou1,3, Shaohui Peng2, Weiqiang Xiong2,3, Haixin Chen1,3, Yuanbo Wen1, Haochen Li2,3, Ling Li2,3, Qi Guo1, Yongwei Zhao1 Ke Gao2 Ruizhi Chen2 Yanjun Wu2 Chen Zhao2 Yunji Chen1,3
1SKL of Processors, Institute of Computing Technology, CAS, Beijing China
1Intelligent Software Research Center, Institute of Software, CAS, Beijing, China
3University of Chinese Academy of Sciences, Beijing, China
Paper Code

Abstract

The attention operator remains a critical performance bottleneck in large language models (LLMs), particularly for long-context scenarios. While FlashAttention is the most widely used and effective GPU-aware acceleration algorithm, it must require time-consuming and hardware-specific manual implementation, limiting adaptability across GPU architectures. Existing LLMs have shown a lot of promise in code generation tasks, but struggle to generate high-performance attention code. The key challenge is it cannot comprehend the complex data flow and computation process of the attention operator and utilize low-level primitive to exploit GPU performance.

To address the above challenge, we propose an LLM-friendly Thinking Language (LLM- TL) to help LLMs decouple the generation of high-level optimization logic and low-level implementation on GPU, and enhance LLMs' understanding of attention operator. Along with a 2-stage reasoning workflow, TL-Code generation and translation, the LLMs can automatically generate FlashAttention implementation on diverse GPUs, establishing a self-optimizing paradigm for generating high-performance attention operators in attention-centric algorithms. Verified on A100, RTX8000, and T4 GPUs, the performance of our methods significantly outshines that of vanilla LLMs, achieving a speed-up of up to 35.16×. Besides, our method not only surpasses human-optimized libraries (cuDNN and official library) in most scenarios but also extends support to unsupported hardware and data types, reducing development time from months to minutes compared with human experts.

Overview

We define Thinking Languge to help LLMs describe attention execution workflows and parameters on GPUs.

Main Results

We conduct our QiMeng-Attention on Deepseek-V3 to evaluate the effectiveness. We evaluate some attention variety such as MHA(multi-head attention) and GQA(group-query attention). We test the performace on different length of input sequence.

Here shows the results on A100 and head dimension of 128. Implemented on Deepseek-V3, QiMeng-Attention achieves a performance enhancement of up to 17.73x compared to vanilla LLMs. Besides QiMeng-Attention can achieve better performances compared to handcraft high performance library such as flash-attn and cuDNN.