AI makes mistakes on simple questions

[alvigg]

Paid AI seems to be better. I use it for complicated science. It isn't always right but leads me to new avenues to explore. I treat it as a collaborator to go back and forth with and check whatever we conclude if I can. Humans also make errors and our group checks each other and AI. The collaboration works well. Gemini, Carnegie Mellon and MIT just had a study that indicates spending 10-25% of the time saved checking is what most scientists do. That is an 75-90% time savings.

 

[Petrushka]

I find AI browser searches much more useful than any previous searches. I don’t expect them to be perfect.

I have spent hours trying to track down simple facts on Google, and AI has given better results quickly.

I use it also when shopping for obscure parts. It correctly parses my request, whereas older searches just use a shotgun.

The magical thing about AI is not that it is infallible; the magic is in “understanding” natural language. This is not trivial.

 

[ta240]

If that contractor worked for free, they would be in very high demand despite that shortcoming

Sad but true.

 

[dtaylo1066]

Yes, AI makes mistakes. Just wait until it F's up with drones, traffic lights, power grids, dam releases or missles. Fun times coming.

For those that want the Wild West, I am sure you can't wait for the self-policing by drug companies, food companies and chemical companies.

On the bright side, I think AI will design some great DACs, amp circuits and speaker crossovers. And in three years AI will replace Amir. And he will be bored, as his robot will be tending to the garden and later play cello in his family room.

 

[Brian Hall]

There are two things to consider before asking an LLM model questions like these. Is "thinking" mode on? Gemini defaults to fast answers (flash-lite) which is fine for very simple questions. The other two modes (flash and pro) are more likely to give a correct answer on harder questions. Also a prompt to direct it to the right knowledge can help. "You are an expert on audio engineering" or something similar.

 

[DonH56]

AI makes mistakes on simple questions​

Complex questions, too.

 

[Chrispy]

It's doing what it is designed to do?

 

[alvigg]

There are two things to consider before asking an LLM model questions like these. Is "thinking" mode on? Gemini defaults to fast answers (flash-lite) which is fine for very simple questions. The other two modes (flash and pro) are more likely to give a correct answer on harder questions. Also a prompt to direct it to the right knowledge can help. "You are an expert on audio engineering" or something similar.

Fast is certainly problematic at times. Patience is worth it.

 

[bachatero]

DO NOT USE THE GOOGLE AI MODELS except for Gemini 4. They're all (except for 4) behind the competition by nearly a year, which is an eternity, so use something like Grok 4.7 or GPT-6 or 5.5 Sonnet (this will surely change soon).

 

[gzost]

If that contractor worked for free, they would be in very high demand despite that shortcoming

The current LLMs by the big US tech companies have required the largest capital invest in history, and this is projected to increase further. Current pricing for the use of these models does not remotely reflect that invest. Warnings about the lack of any realistic path to profitability have now come from major investment and central banks. This entire "industry" is running on circular investments and the hope that it achieves "AGI"/“super intelligence" which makes huge classes of labour obsolete before the bill comes due. Personally, I think that there is no way that LLMs can improve all that much further and are clearly not going reach that required level of utility.

This is not to say that the LLMs don't have any utility. ChatGPT did a good job of fleshing out my list of all recorded Mahler symphony cycles with more information, but none of the models I tried were able to remotely assemble the list itself. I put it together solely with information from the public Internet, though with a background of years of browsing around this topic. So useful as an assistant when you have enough knowledge of a topic to play around with the prompt and can verify the result. Useless to dangerous where you can't do that.

 

[Timcognito]

DO NOT USE THE GOOGLE AI MODELS except for Gemini 4. They're all (except for 4) behind the competition by nearly a year, which is an eternity, so use something like Grok 4.7 or GPT-6 or 5.5 Sonnet (this will surely change soon).

There is group at UC Berkeley that continuously tests and ranks the models. As an alumni I heard about and got kind of interested about four years ago but got bored with all the frantic ever changing news so I haven't looked in a while. FYI

Arena Leaderboard | Compare & Benchmark the Best Frontier AI Models

See how leading AI models stack up across text, image, vision, and more. This page provides a high-level snapshot of each Arena. Explore dedicated tabs for deeper insights.

arena.ai

https://arena.ai/company/about

 

[amirm]

Gemini defaults to fast answers (flash-lite) which is fine for very simple questions. The other two modes (flash and pro) are more likely to give a correct answer on harder questions.

I have had really bad results doing that. I just tried it this morning and as it often does, it said something went wrong after a couple of minutes and that was that. Other times it is slower but doesn't produce anything better. I do like the default Flash as many questions I ask have easy answer (really smart search results as suggested by member above) and hate to wait for an answer.

For serious work, I have ChatGPT subscription. That substantially improves its results/experience as otherwise, you can easily run out of tokens. The problem is that even for easy questions per above, it takes a while. The latest model has made this worse as even in its fastest mode, is slower than before.

For anything complicated, I am using both engines and then adding my own independent analysis/verification.

 

[Petrushka]

I’m a bit more positive because I always expected AI to have the same flaws as humans.

This is not a fixable flaw. This is the nature of reality. Knowledge is and always will be incomplete and probabilistic.

AI will always have a superficial grasp of more things than a human, and human experts will always be able to find defects in the detailed knowledge of AI.

I had a drainage ditch dug. The backhoe operator was a genius at his job, but he had a guy with a shovel probing for buried pipes and cables. Machines can amplify our muscles, and relieve us of drudgery, but there will always be niches they do not fill.

 

[nonsynchro]

it's developer updated it's underlying model and training data. Wow, what an incredible waste of energy

The waste of energy is the apostrophes being inserted into the "its" possessive pronoun. This is a science forum of which good grammar is a fundamental part. Senior Members are held to a higher standard.

 

[Petrushka]

The waste of energy is the apostrophes being inserted into the "its" possessive pronoun. This is a science forum of which good grammar is a fundamental part. Senior Members are held to a higher standard.

My browser has developed such an aversion to apostrophes that it no longer even offers them as an option when they’re required.

Its true. Im not exaggerating.

 

[RexrothPigeon]

The waste of energy is the apostrophes being inserted into the "its" possessive pronoun. This is a science forum of which good grammar is a fundamental part. Senior Members are held to a higher standard.

lol wut