https://allenai.org/blog/olmpool
OlmPool: How small architectural choices compound to undermine long context extension | Ai2
OlmPool is a controlled suite of 26 models showing how small architecture choices can compound to make long-context extension much harder, even when training...
architectural choices