Densing Law of LLMs

This paper talks about how to measure the quality of large language models (LLMs) in a way that considers both how well they perform and how efficient they are. The authors introduce a new idea called "capacity density," which is like figuring out how much "brainpower" an LLM has compared to its size. Imagine two students who get the same grade on a test – the student who studied less has a higher "study density." Similarly, an LLM that can perform as well as a larger model but with fewer parameters has a higher capacity density. The researchers looked at many popular LLMs and found something interesting: the capacity density of LLMs is doubling every three months! This means that we're getting much better at creating powerful LLMs without needing to make them ridiculously huge. They call this trend the " ...