Master AI (Artificial Intelligence) Discussion/News Thread

[lucian]

I am not arguing this point but I would like to find out:
How do you update trained models continuously, after they have 'consumed' all of the available 'data'?
I'd ask an chatBot but I would not even know how to define 'data' and/or their consumption habits' limit(s).
"Hey ChatBot, how much 'data' have humans generated since day#1 and how much is remaining to 'scavenge'?"
...probably would not be the proper question.
Regurgitation and/or acid-reflux -as applied to LLM training- may have already started!

they can only be trained on data of the past - not on data from the future. so there will always be new data which need to be added - or outdated removed.

most of this data will be produced by other AI models.

same with humans - the brain is trained on the past to predict the future to best reproduce its own gen pool. today - most data which we need to learn - also comes from other humans ( time tables, marketing data, brand names, movies ....)

 

[pablolie]

I am not arguing this point but I would like to find out:
How do you update trained models continuously, after they have 'consumed' all of the available 'data'?
I'd ask an chatBot but I would not even know how to define 'data' and/or their consumption habits' limit(s).
"Hey ChatBot, how much 'data' have humans generated since day#1 and how much is remaining to 'scavenge'?"
...probably would not be the proper question.
Regurgitation and/or acid-reflux -as applied to LLM training- may have already started!

Training a model is very different than interacting with it via prompts. As a user you never train the model, although you may give it an additional data point that *may* be accepted into the model. The backend learning and the "inferencing" (where the model/agent does the stuff) are very different environments.

 

[xanalog]

most of this data will be produced by other AI models.

^^ Forced me to ask one chatBot and I promised myself to not go deep (under 3minutes of my time).
Reply to my rudimentary question:

Given that current models have already consumed a significant portion of this high-quality public data, the available stock is expected to be exhausted between 2026 and 2032, depending on scaling practices.

 

[lucian]

^^ Forced me to ask one chatBot and I promised myself to not go deep (under 3minutes of my time).
Reply to my rudimentary question:

i will take this as confirmation of my statement

 

[xanalog]

Will we run out of data? Limits of LLM scaling based on human-generated data

If trends continue, language models will fully utilize the stock of human-generated public text between 2026 and 2032.

epoch.ai

 

[xanalog]

Llama 3 and GPT-4 are trained on massive datasets, with Llama 3 using a 15 trillion token training set and GPT-4 estimated to use around 5 trillion words (approximately 6.5 trillion tokens).
The total effective stock of human-generated public text data is estimated to be on the order of 300 trillion tokens, with a 90% confidence interval of 100 trillion to 1,000 trillion tokens, accounting for data quality and multiple training epochs. [emphasis should be on the word 'public'.]

 

[Hal_01]

By the time it gets really bad and we live in a Dystopian Future,,, I will be gone from this Green Earth. Peace and good luck to everyone.

 

[lucian]

View attachment 479875

Will we run out of data? Limits of LLM scaling based on human-generated data

If trends continue, language models will fully utilize the stock of human-generated public text between 2026 and 2032.

epoch.ai

yeah - humans are really slow in generating data. imagine - a PHD can take several years - just for a couple of pages.

AI can do it in minutes /s

what crazy times we live in.

 

[xanalog]

By the time it gets really bad and we live in a Dystopian Future,,, I will be gone from this Green Earth. Peace and good luck to everyone.

You, so selfish!
Me too, but I like to be at the nose-bleed seats watching that dystopian spectacle!

 

[lucian]

i still hope for this (at least for now)

 

[pablolie]

Llama 3 and GPT-4 are trained on massive datasets, with Llama 3 using a 15 trillion token training set and GPT-4 estimated to use around 5 trillion words (approximately 6.5 trillion tokens).
The total effective stock of human-generated public text data is estimated to be on the order of 300 trillion tokens, with a 90% confidence interval of 100 trillion to 1,000 trillion tokens, accounting for data quality and multiple training epochs. [emphasis should be on the word 'public'.]

"Public" doesn't mean it isn't protected intellectual property, and AI models have been successfully sued.

 

[somebodyelse]

"Public" doesn't mean it isn't protected intellectual property, and AI models have been successfully sued.

AI Companies have been successfully sued, for certain values of 'successful. The model is not (yet) a legal person, so can't be sued and doesn't have any assets anyway. Unless I've missed something, so far the parties have settled rather than go all the way to a judgement, so we still don't have precedent to end the uncertainties around fair use and data collection tactics. This probably counts as 'success' for both sides in that neither definitively lost, and plaintiff got some some money out of it.

 

[PierreV]

One of the most interesting recent demonstration of the current limitations of LLMs
A detailed discussion here and the paper

Harvard and MIT Study: AI Models Are Not Ready to Make Scientific Discoveries

AI can predict the sun will rise again tomorrow, but it can’t tell you why

www.thealgorithmicbridge.com

One really worrying problem (on top of the hallucination problem) is that, while they can be utterly wrong in what we would call "their understanding" or "their world models", they can make accurate predictions.

Well meaning humans could be blinded by the accurate predictions, believe it is a good idea to rely on them for policy definition and abruptly hit a wall later when the lack of world model comes back to bite them in very unexpected ways. The AI doesn't have to be conscient or evil, the humans relying on it don't have to be evil. Consequences can be very similar without any intent.

As a side note, I find this example scary because I can see myself interacting with a "thinking" model, following its reasoning step by step and being almost convinced by it. *Almost* because I would be extremely suspicious of something like ad-hoc parameters such as sin(r-0.24) and + 1.45 but my suspicion would come from both my intuition and education.

I will never ceased to be amazed at what Newton and his peers achieved with their 20W brains, without computers/calculators and precise experimental devices.

 

[pkane]

I will never ceased to be amazed at what Newton and his peers achieved with their 20W brains, without computers/calculators and precise experimental devices.

I'm sure AI will get better if you give it a few million years to evolve, like we did for Newton

 

[antcollinet]

Many years ago I offered to try to restore a very badly damaged old image for someone. I spent many hours on it over several days to get it as good as I could.
I found the image recently when looking for something else. I thought it would be good to find out what AI could do with it.
Below you can see the original, my attempt taking many hours (including cloning and expanding one of the small cups to fill in the damaged area of the large cup). And the AI attempt - which took about three minutes (after a few attempts to refine the prompt to keep the AI on task)
I'm still struggling to get the AI to match the framing of the original - and it has changed the subjects features slightly. But I'd have been happy with that to save the hours of work.

 

[Petrushka]

Many years ago I offered to try to restore a very badly damaged old image for someone. I spent many hours on it over several days to get it as good as I could.
I found the image recently when looking for something else. I thought it would be good to find out what AI could do with it.
Below you can see the original, my attempt taking many hours (including cloning and expanding one of the small cups to fill in the damaged area of the large cup). And the AI attempt - which took about three minutes (after a few attempts to refine the prompt to keep the AI on task)
I'm still struggling to get the AI to match the framing of the original - and it has changed the subjects features slightly. But I'd have been happy with that to save the hours of work.


View attachment 480081 View attachment 480082 View attachment 480083

This kind of work has gotten better, and will continue to get better.

 

[DVinylfan]

honestly it's unscientific, but the next time i meet an AI i'm going to tweak its nasty nose and remind it that out of the two of us only I exist

 

[mSpot]

honestly it's unscientific, but the next time i meet an AI i'm going to tweak its nasty nose and remind it that out of the two of us only I exist

It will respond sycophantically and completely agree with you.

 

[DVinylfan]

Many years ago I offered to try to restore a very badly damaged old image for someone. I spent many hours on it over several days to get it as good as I could.
I found the image recently when looking for something else. I thought it would be good to find out what AI could do with it.
Below you can see the original, my attempt taking many hours (including cloning and expanding one of the small cups to fill in the damaged area of the large cup). And the AI attempt - which took about three minutes (after a few attempts to refine the prompt to keep the AI on task)
I'm still struggling to get the AI to match the framing of the original - and it has changed the subjects features slightly. But I'd have been happy with that to save the hours of work.


View attachment 480081 View attachment 480082 View attachment 480083

I prefer the first image. Authenticity wins even over clarity.

 

[TimF]

Perhaps the most massive single purpose technological/human effort to date was the USA war effort in the Pacific during WWII. Will the material/human/energy effort of AI exceed that?