even though it is worthless whenever next model version is released.
For most version releases I actually doubt that, I’m pretty sure they just keep training the model and release checkpoints every few months. Architectural changes, however, will indeed invalidate the previously trained models, and occasionally they have to do that to keep up with the competition.
But if you do architectural changes even just once a year, that means you’d have to depreciate those training costs over just that one year to be truthful. Meaning they might as well be losses rather than depreciating capital investments.
There is a genuine “training corpus” that is an accumulated asset and grows and feeds new models. The vast majority of training cost, that should be expensed, is the training time compute. Neither of these are expensed. There was about 13 major US model releases in September. At least 7 of these replaced a 1 month old previous model family. Open AI gpt sol went from 5.6 to 6 to 6.1 in about 2 months.
Fast model releases not only lock in structural losses, they juice revenue growth illusions. Model use rate generally drops by 66% 4 weeks after release, because of competition, and a surge of use on release for evaluation.
For most version releases I actually doubt that, I’m pretty sure they just keep training the model and release checkpoints every few months. Architectural changes, however, will indeed invalidate the previously trained models, and occasionally they have to do that to keep up with the competition.
But if you do architectural changes even just once a year, that means you’d have to depreciate those training costs over just that one year to be truthful. Meaning they might as well be losses rather than depreciating capital investments.
There is a genuine “training corpus” that is an accumulated asset and grows and feeds new models. The vast majority of training cost, that should be expensed, is the training time compute. Neither of these are expensed. There was about 13 major US model releases in September. At least 7 of these replaced a 1 month old previous model family. Open AI gpt sol went from 5.6 to 6 to 6.1 in about 2 months.
Fast model releases not only lock in structural losses, they juice revenue growth illusions. Model use rate generally drops by 66% 4 weeks after release, because of competition, and a surge of use on release for evaluation.
New models are trained on older models, so even if the older model is not longer at par, without it the new model couldn’t exists