A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)

Open paper
Benchmark Model Rank Results
long-context-understanding-on-l-evalGALI(Llama3-8b-ins-4k-to-16k)#1Average Score: 59.21
long-context-understanding-on-l-evalGALI(Llama3-8b-ins-4k-to-32k)#2Average Score: 59.10
long-context-understanding-on-l-evalGALI(Llama3-8b-ins-8k-to-32k)#3Average Score: 42.79
long-context-understanding-on-l-evalGALI(Llama3-8b-ins-8k-to-16k)#4Average Score: 42.32
long-context-understanding-on-longbenchGALI(Llama3-8b-ins-4k-to-16k)#1Average Score: 46.22
long-context-understanding-on-longbenchGALI(Llama3-8b-ins-8k-to-32k)#2Average Score: 45.38
long-context-understanding-on-longbenchGALI(Llama3-8b-ins-8k-to-16k)#3Average Score: 45.17

Rank counts only results with a code link.