[Roadmap] FlashInfer v0.2 to v0.3

[yzh119]

Milestones

Our tentative roadmap includes the following milestones:

• SageAttention-2 in FlashAttention3: Implement SageAttention-2 in FlashAttention3 template
• Flex-Attention Compatible Interface: standarize JIT interface @shadowpa0327
• SM89 Kernel Optimization: Leverage Ada FP8 Tensor Cores for better performance on Ada6000 & 4090.
• Template Refactoring: Refactor FA-2 and MLA templates using CuTE.
• MLA Acceleration: Optimize Multi-Level Attention (MLA) with Tensor Core support, follow up of feat: support MLA decode #551 .
• Triton Porting: Migrate elementwise, normalization, and other kernels (that are not on critical path) to Triton.
• API Standardization: Simplify and standardize the attention APIs for better usability.
• POD-Attention Integration: Implement POD-Attention for improved efficiency of chunked-prefill.
• Nanoflow Parallelism: Expose python-level APIs for performing GEMM and Attention on a subset of SMs, which is required for nanoflow style parallelism, see #591.
• Fused Tree Speculative Sampling: follow up of sampling: fused speculative sampling kernels #259 , we should support tree-speculative sampling as well, we will port the implementation of fused tree-speculative sampling written by @spectrometerHBH from https://github.com/mlc-ai/mlc-llm to accelerate eagle and medusa etc.
• Improvements on Existing Top-P/K Sampling Operators: change the algorithm to guarantee all samples are successful after 32 rounds.
• PyPI wheels: upload wheels to PyPI (pending issue: PEP 541 Request: flashinfer pypi/support#5355)
• RoPE positions in batch attention interface: Fused-RoPE Attention with q_offset and k_offset #701
• FP32 output for bf16 input: Different sequence numbers calculate inconsistent results #696
• Per Head Scale Quantized KV-Cache: How to use low bit KV Cache  #721

---

We welcome your feedback and suggestions!
Let us know what features you'd like to see in FlashInfer.

[johnnynunez]

Initial support blackwell:
#747
10.0 blackwell b100/b200
12.0 blackwell rtx50