HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
Open paper
Benchmark
Model
Rank
Results
common-sense-reasoning-on-winogrande
HySparse (80B MoE)
–
Accuracy: 72.1
math-word-problem-solving-on-math
HySparse (80B MoE)
–
Accuracy: 30.8
sentence-completion-on-hellaswag
HySparse (80B MoE)
–
Accuracy: 79.0
Rank counts only results with a code link.