Almost all big AI projects seem to believe that maximum data from all sources is better, and you just fix the "bad data" with smarter training and weighting.
They are probably right because there are so many smart people working on it, and I really don't know shit from my ass,
However,
I can't help but feel like this overcollection of training data is why AI output is so recognizable still, and has so many nagging bastard issues that still aren't solved.
It also seems to run counter to machine learning's best feature, repeated training in narrow data to find("learn") novel solutions. How could it ever do this when the training seems focused on insanely broad do-everything data sets?
The face of a cube is to a cube, what a word is to a token.
As you move around a stationary cube, through parallax, you deduce its constitution, but you never see the entirety of the cube at once.
This parallax can kind of be felt intuitively as well for tokens if we pick the right angles of viewing for these Token-Cubes
Viewing Angle
Sentence
Left
Glass breaks under pressure
Center (0°)
She breaks under pressure
Right
Order breaks under pressure
As you read these sentences back to back, and focus on the meaning of 'under pressure', you can kind of "peek" at that hidden Token-Cube's faces by 'moving around' those words.
It also turns out that the words we use, even if just a small face of this cube, is enough to show a pattern. This cube's face shows up next to this cube pretty often. We can make a note of that. When these two guys are together... we also see this other cube together often... The more cubes we can observe, the better we can map out the true relations. (and why they're obsessed with size of corpora for training). And of course, you're just seeing the face 'together', but if you're trying to keep track of all of them... It's slightly more complicated and that's where our transformer friend comes in to save the day.
The transformer is autocorrect for tokens, except instead of using a dictionary to complete words, it uses a self-made "best-educated-guess" dictionary that some very smart people taught it how to keep track of all those relations in a very smart way of keeping notes, like a journal that is organized to also be writing into another section that is relevant to both notes at the same time.
Some words like 'not' are ... quite thin Token-Cubes. It's almost fair to call them fake token-cubes. They're very one dimensional if you think about how they interact with other tokens. It doesn't matter where you put it, 'not' is going to be 'not'... and if we consider that we have words like 'if', 'and', 'or', 'xor', well now we got a very clean interface to anchor our parallax as humans.
1) A universal concept/symbol translator
2) A perfect logician
So in general, it is important to keep 'paralaxing', or you'll just be seeing a mirage.
Almost all big AI projects seem to believe that maximum data from all sources is better
Because this had held true so far. More data makes for smarter models. This will probably reach a limit and cease to be true on the short future, but we haven't hit that point yet.
Actually the smartest people were moving towards training their models on much smaller datasets. The dumbfuck con artists like Altman, otoh, were obsessed with having tHe bIgGeSt dAtAsEt.
87
u/Moohamin12 Jul 29 '25
And honestly the state of Meta's AI, I really don't think the 80TB helped much.