Master AI (Artificial Intelligence) Discussion/News Thread

[Petrushka]

At some point, people are going to realize just how much personal information they’re feeding into cloud-based AI chatbots. These systems already know more about you than you might be comfortable with, and once that really sinks in, the trend will shift toward running AI locally on your own computer. And as much as I hate to admit it because I’ve never been a Mac guy and still find basic things like window resizing annoying, Macs actually have a big advantage here. It’s not because they’re faster, but because of how they handle memory. Even a top of the line NVIDIA 5090 GPU has a small limit on how much data it can hold at once (32GB). Apple’s unified memory architecture lets the CPU and GPU share a single pool of memory 64 GB, 96 GB, or even 128 GB at high bandwidth. A large language model can sit entirely in that shared memory space, with no expensive copying between system RAM and VRAM. So while highend PCs still win on raw power, Macs quietly excel at running truly capable AI models locally and if privacy becomes the priority, that advantage is going to matter a lot more than people expect IMO.

If you can find memory chips to buy.

 

[Axo1989]

Until now, I have not dipped into AI at all, never interacted with an LLM. I was planning to learn about them in a few months or so, and hopefully learn some GPU programming and advance my outdated knowledge of scientific computing (I haven't really programmed anything for about 25 years, and have always bought pre-built PCs). My home computer is dinky, a panic purchase from a local Best Buy when my previous computer died during the Covid-19 lockdown. I have been planning to get a new computer with lots of memory (64 or 128 GB) and a beefy CPU and GPU. I was not in any hurry, being preoccupied with a bunch of other things, until this past weekend.

Over the past weekend, I watched a bunch of YouTube videos about a big shift in computer memory and storage hardware availability, due to large demand for HBM2 for AI clouds. Also, prices for DDR5 have about tripled in the past month, while DDR4 prices have doubled. Prices for SSDs and GPUs have also jumped upward. Anyone expert who is tracking the supply chain situation, and have any understanding of what is likely to happen? Are the prices just due to panic buying and will return to semi-normal in a few months, or is this the end of chonky home computers as some YouTube video creators are predicting? (They claim we will all have to subscribe to AI and other cloud services in the future, while I would much prefer "Local AI" for privacy reasons).

Maybe this article will be informative.

The answer seems to be prices are still climbing. I'm not interested enough in building PCs to do more than skim that story, but if you are, enjoy (if budget permits). Otoh I recently bought a Mac Studio with 64 GB and I'm regretting not getting 128 GB of silicon gold. Their RAM pricing looks the same as last year and maybe the year before, which means everyone else is catching up.

 

[Axo1989]

If you can find memory chips to buy.

No problem buying a Mac Studio with 512 GB RAM (as unified memory) currently. Which a makes very lavish memory pool for GPU. For those interested in running suitable LLM locally.

*having said that I'd better check shipping ... 3-5 business days here, yeah that's still normal.

 

[Power Pop 23]

The National Oceanic and Atmospheric Administration (NOAA) announced the deployment of three 'A.I.-driven', global weather prediction models.

https://www.noaa.gov/news-release/noaa-deploys-new-generation-of-ai-driven-global-weather-models

 

[JustAnandaDourEyedDude]

It all depends on when/whether the AI bubble bursts. Hardware manufacturers are rejigging to follow the money, and that's in AI data centres now, not in consumer products. Expect anything containing silicon to get more expensive until the demand shifts again. Electricity prices are also rising in places with a data centre concentration.

Good point about when/whether the AI bubble bursts.

At some point, people are going to realize just how much personal information they’re feeding into cloud-based AI chatbots. These systems already know more about you than you might be comfortable with, and once that really sinks in, the trend will shift toward running AI locally on your own computer. And as much as I hate to admit it because I’ve never been a Mac guy and still find basic things like window resizing annoying, Macs actually have a big advantage here. It’s not because they’re faster, but because of how they handle memory. Even a top of the line NVIDIA 5090 GPU has a small limit on how much data it can hold at once (32GB). Apple’s unified memory architecture lets the CPU and GPU share a single pool of memory 64 GB, 96 GB, or even 128 GB at high bandwidth. A large language model can sit entirely in that shared memory space, with no expensive copying between system RAM and VRAM. So while highend PCs still win on raw power, Macs quietly excel at running truly capable AI models locally and if privacy becomes the priority, that advantage is going to matter a lot more than people expect IMO.

Yes, the idea of all my reading and writing being hoovered up by the AI companies is appalling to me. The unified pool of RAM and VRAM on the Macs is great, but like you I have no hankering to live in Apple world. IIRC a few years ago AMD had promised unified memory (at least with an all-AMD CPU and GPU combination, but they have not yet delivered on that.

There has have been a few very good youtube tutorial links provided by members in this thread:
If I may suggest, please don't delay your start for interacting with LLM, any longer.

It sounds extremely interesting, and I will get around to it, but not until I clean up some issues that have priority. I have already spent some time watching various YouTube videos about AI, but done nothing hands on.

Maybe this article will be informative.

The answer seems to be prices are still climbing. I'm not interested enough in building PCs to do more than skim that story, but if you are, enjoy (if budget permits). Otoh I recently bought a Mac Studio with 64 GB and I'm regretting not getting 128 GB of silicon gold. Their RAM pricing looks the same as last year and maybe the year before, which means everyone else is catching up.

Nice article, thanks. It is consistent with the YT videos I have watched. We are facing a major redirection of NAND chip manufacturing by all three major manufacturers (Samsung, SK Hynix and Micron), which will result in prolonged shortages (and high prices) of consumer RAM and SSDs and soon, graphics cards. It seems to me that the current high prices are anticipatory, based on the announcements, rather than the actual supply streams tapering down. A combination of consumers stockpiling, retailers testing consumer willingness-to-pay-due-to-anxiety, and scalpers gambling on turning tidy profits, much as happened with graphics cards during Covid.

Thanks for the inputs, all. I caved in to my fears, headed down to the local Best Buy store and bought two 64GB DDR5 kits, clenching my teeth as I paid nearly thrice what I would have had I known to buy in September. Then I came back and ordered a 4TB SSD from Amazon, paying a premium of about 50%. I will leave the question of a graphics card till next year, as AI software and hardware are evolving intensely at present.

 

[Satiomeliom]

I was about to buy another 32 gb 2 months earlier because i had a suspicion. I was just at checkout but then i realised my PC doesnt have the clearance to actually put the sticks in so i didnt do it.

Im not mad though since 32 GB was already a 4x compared to my last PC, and back then it wasnt that cheap to do either.

 

[somebodyelse]

The unified pool of RAM and VRAM on the Macs is great, but like you I have no hankering to live in Apple world. IIRC a few years ago AMD had promised unified memory (at least with an all-AMD CPU and GPU combination, but they have not yet delivered on that.

Isn't that what AMD have done with Strix Halo? The high speed RAM is the reason Framework soldered it in their desktop - AMD couldn't make it work with socketed RAM without a major hit to speed.

 

[JustAnandaDourEyedDude]

Isn't that what AMD have done with Strix Halo? The high speed RAM is the reason Framework soldered it in their desktop - AMD couldn't make it work with socketed RAM without a major hit to speed.

I did not realize that. More power to AMD then!

 

[levimax]

Good point about when/whether the AI bubble bursts.

Yes, the idea of all my reading and writing being hoovered up by the AI companies is appalling to me. The unified pool of RAM and VRAM on the Macs is great, but like you I have no hankering to live in Apple world. IIRC a few years ago AMD had promised unified memory (at least with an all-AMD CPU and GPU combination, but they have not yet delivered on that.

It sounds extremely interesting, and I will get around to it, but not until I clean up some issues that have priority. I have already spent some time watching various YouTube videos about AI, but done nothing hands on.

Nice article, thanks. It is consistent with the YT videos I have watched. We are facing a major redirection of NAND chip manufacturing by all three major manufacturers (Samsung, SK Hynix and Micron), which will result in prolonged shortages (and high prices) of consumer RAM and SSDs and soon, graphics cards. It seems to me that the current high prices are anticipatory, based on the announcements, rather than the actual supply streams tapering down. A combination of consumers stockpiling, retailers testing consumer willingness-to-pay-due-to-anxiety, and scalpers gambling on turning tidy profits, much as happened with graphics cards during Covid.

Thanks for the inputs, all. I caved in to my fears, headed down to the local Best Buy store and bought two 64GB DDR5 kits, clenching my teeth as I paid nearly thrice what I would have had I known to buy in September. Then I came back and ordered a 4TB SSD from Amazon, paying a premium of about 50%. I will leave the question of a graphics card till next year, as AI software and hardware are evolving intensely at present.

Do you know if the price on complete machines has changed yet? Reading this makes me want to go down to Costco and buy a new machine before year end.... get 100% tax deduction as well.

 

[tineared]

I bought a Dell in September and replaced the included 16GB RAM with 64GB of Crucial DDR5 for it for $140. Now it's over $600 everywhere. Also bought an M.2 4TB SSD that's gone up 60% in price and an internal 4TB SSD that's gone up almost 100%. These are real percentages, not Trump percentages
Fortunately I now have 2 current and capable PCs and should be good for another decade or so
The Win 10 machine I'm typing with now is 10 years old and no longer has that snappy response and forget about photo processing
It's fine for music, but transferring files is cumbersome
I haven't yet seriously dipped my toe in the AI realm, but when I do I shall be very conscience of what information I provide it with

 

[Timcognito]

You just gave us and AI a lot about you

 

[tineared]

You just gave us and AI a lot about you

Yes, but is it news you can use?
I think it's safe to say that anything purchased online is not a secret

 

[Timcognito]

 

[levimax]

I bought a Dell in September and replaced the included 16GB RAM with 64GB of Crucial DDR5 for it for $140. Now it's over $600 everywhere. Also bought an M.2 4TB SSD that's gone up 60% in price and an internal 4TB SSD that's gone up almost 100%. These are real percentages, not Trump percentages
Fortunately I now have 2 current and capable PCs and should be good for another decade or so
The Win 10 machine I'm typing with now is 10 years old and no longer has that snappy response and forget about photo processing
It's fine for music, but transferring files is cumbersome
I haven't yet seriously dipped my toe in the AI realm, but when I do I shall be very conscience of what information I provide it with

I just checked Costco.com and all their high spec computers (4 TB SSD and 64 GB ram) are "out of stock"....

 

[respice finem]

I've got 10 (yes, ten) PCs, aged from 4 to 10 years. Saved them from being discarded. Most are from my former office, because my successor wanted to have all new. So I don't have to buy PCs any time soon, even after donating some. But if needed a PC right now, I wouldn't care too much for "futureproofing" by maxing out CPU and RAM, because we have, after years of relative stagnation, apparently entered a stage of disruptive changes, mainly due to AI. These changes will IMHO make any PC of today obsolete in 2-3 years, if not earlier.

 

[xanalog]

View attachment 498413

Yeah!
So right-on, yet so sad!
I am still trying to hang onto that tenet that is ingrained into the older generations (=me?), where hardware used to be the ruler and the king of our (=my?) universe.
Software updates is like permanent 'rhoids...

It's not just our (=not mine) connected/portable digital devices, including (=not mine) audio gear, software has extended into most all devices.

I am hoping to avoid the purchase of another new car, that is software-driven, rather than human (=me) driven.
But AI is way-more of an unruly-beast, than 'software' ever was or will be!!

 

[respice finem]

Software updates is like permanent 'rhoids...

This. And, increasingly, devices are software dependent, from a light switch to a car.
Which opens a whole new "universe" of planned (and sometimes unplanned) obsolescence...

 

[Somafunk]

permits). Otoh I recently bought a Mac Studio with 64 GB and I'm regretting not getting 128 GB of silicon gold. Their RAM pricing looks the same as last year and maybe the year before, which means everyone else is catching up.

Someone else has been building on the Mac Studio for AI. Use

 

[JustAnandaDourEyedDude]

Do you know if the price on complete machines has changed yet? Reading this makes me want to go down to Costco and buy a new machine before year end.... get 100% tax deduction as well.

This time around, I did not specifically look at complete systems. I surmise that the prices of complete system builds have not jumped as much, particularly if pre-built. One of the YT vloggers mentioned this, and suggested it as a way of mitigating the price gouging. I saw the thumbnail of another video that pictured a Costco complete system box saying "$400 off at Costco", though I do not recall clicking on it and watching it. I expect the prices also vary by outlet, and would be updated every couple of days. The cheapest equivalent 64GB DDR5 6000MHz kit was about $120 more at Amazon than at Best Buy when I checked online last evening, which is why I headed over to Best Buy right away.

I just checked Costco.com and all their high spec computers (4 TB SSD and 64 GB ram) are "out of stock"....

That is unfortunate.

I bought a Dell in September and replaced the included 16GB RAM with 64GB of Crucial DDR5 for it for $140. Now it's over $600 everywhere. Also bought an M.2 4TB SSD that's gone up 60% in price and an internal 4TB SSD that's gone up almost 100%. These are real percentages, not Trump percentages
Fortunately I now have 2 current and capable PCs and should be good for another decade or so

Lucky you! That is just what I was hoping to get to next year.

But if needed a PC right now, I wouldn't care too much for "futureproofing" by maxing out CPU and RAM, because we have, after years of relative stagnation, apparently entered a stage of disruptive changes, mainly due to AI. These changes will IMHO make any PC of today obsolete in 2-3 years, if not earlier.

I lived with the limitations of value-priced pre-built home computer systems during my working life, because I always had access to plentiful compute at work. Now I miss that, and to slow brainrot if not for any other reason, I have a hankering for a beefy computer on which to continue some level of pursuit of my modeling and simulation interests, say 2D analyses if not 3D, and also AI of course. The puny though economical pre-built HP home PC that I bought five years ago with 16GB of RAM has a motherboard that allowed a max of only 32GB, and the post-purchase addition of only an RTX 1060 instead of using the built in IGP. Also, the CPU was already two generations behind at time of purchase.

 

[JustAnandaDourEyedDude]

In the past, the computer/phone manufacturing industry had a large customer segment that bought a large number of high-profit-margin computers but on a slow upgrade cycle. The industry balanced meeting this demand with the demand of other customer segments with lower-profit margins, some of which (such as the gamer segment) upgraded more frequently. Now the computer mfg industry is being lured away by a new customer segment in the AI hyperscalers, with extreme demand scale for the initial build-out, and with deep pockets and willing to pay high margins.

The Kingston representative who was featured in a couple of interviews said that Kingston is rationing the supply of DRAM and SSDs to their long-standing computer industry customers, to retain customer loyalty. Kingston sales teams are encouraging new customers who come calling to rather go back to their normal DRAM and SSD suppliers and negotiate with them for supply quantity though the prices are indubitably continuing to increase. So for Kingston, it will be no new contracts with new DRAM/SSD non-AI customers, and reduced supply and higher prices with existing customers (apparently, the contracts are flexible enough). The three NAND chiplet makers are not rushing to over-build new capacity (which process takes a couple of years anyway), but rather mostly re-use existing capacity. They learned a lesson during the Covid lockdown, when they over-predicted future demand, and built excess capacity (which comes in useful right now, though), but post-Covid the demand plummeted and they were forced to lower prices and profit margins.

I watched a couple of YT videos late last night wherein the podcasters said that there is a rumor that Nvidia plans an approximately 40% cutback on production of consumer GPUs in 2026. Another rumor is that Nvidia has told their video card partners (who make the bulk of video cards sold and customize them a little) that the partners would have to supply their own VRAM for the cards. Nvidia has apparently denied this rumor. The partners of course has less leverage than Nvidia does, with the DRAM manufacturers.

Mind you, caveat emptor, this is all just my take watching YT videos during the past few days, though the creator channels are mostly by people that have been posting for a few years and whom I consider semi-reliable in their facts if not necessarily in their speculations. If you were considering upgrading or building or a pre-built currently or during the next year or two, please do your own research into the chaos, and not take the plunge immediately, based solely on my impressions.