>>16594562
>These results have never been peer reviewed
Incorrect; the scaling hypothesis is remarkably consistent.
>>16594480
>Grok 3 had 15x the compute
Scaling pre-training still works, just not nearly as well as it did. But scaling pre-training is not the only scaling dimension. Scaling data quality/amount, scaling test-time compute, scaling compute length, all these still work. It's also an open question if pre-training scaling is dead dead, or if we haven't figured out the most efficient way for our current models to wring information from data. We almost certainly haven't, transformers aren't even a decade old.
>>16594406
>But I think we're soon going to run into a situation where every gain will have to come from a clever trick.
>The Dataset size is something we're already running into. It's not enough to have a lot of data. It has to be good data.
Yes. You understand. The frontier AI labs understand this too. That's why they're working on synthetic data generation; billions of dollars and tens of thousands of man-hours concentrated on it. They are not far away from cracking this problem.

Back | Go to board

Board:

/sci/

Thread:

These three graphs are the most important graphs in the world right now. If you understand what they imply, you know everything is about to…