ChatGPT vs Microsoft CoPilot: Poor Execution or Bias?

[Somafunk]

Gentlemen!

Ok, I edited post

 

[Axo1989]

Ok, I edited post

Good on you (not that I disagreed with your original).
Others haven't though. Time to update the ignore list.

I find it very relevant. How could it not be? If you want to use a different description you'll still be talking about the same issue. When certain things are deemed off limits or unacceptable it will effect the results from an AI. No different than bias effects people's results answering questions.

Yes, really. At least in people I know it seems like the best most succinct description. I try very hard to give the benefit of the doubt. I'll drop the discussion of this as I don't want to make it political or problematic.

But for sensible discussion (here and elsewhere) why not use a more correct technical description, instead of a goading/loaded and inaccurate political one?

The underlying technical issues are interesting for sure. You'll recall an earlier Microsoft chatbot Tay (deployed as a Twitter chatbot in 2016) that turned sour:

Some Twitter users began tweeting politically incorrect phrases, teaching it inflammatory messages revolving around common themes on the internet, such as "redpilling" and "Gamergate". As a result, the robot began releasing racist and sexually-charged messages in response to other Twitter users. Artificial intelligence researcher Roman Yampolskiy commented that Tay's misbehavior was understandable because it was mimicking the deliberately offensive behavior of other Twitter users, and Microsoft had not given the bot an understanding of inappropriate behavior. He compared the issue to IBM's Watson, which began to use profanity after reading entries from the website Urban Dictionary. Many of Tay's inflammatory tweets were a simple exploitation of Tay's "repeat after me" capability. It is not publicly known whether this capability was a built-in feature, or whether it was a learned response or was otherwise an example of complex behavior. However, not all of the inflammatory responses involved the "repeat after me" capability; for example, Tay responded to a question on "Did the Holocaust happen?" with "It was made up".

While people here are often incorrectly assigning 'knowledge' and 'thought' to outputs which are really text strings generated by statistical inference (based on extremely large n-dimensional matrices of language token proximity derived from the training corpus) that output doesn't work if unconstrained. So we no longer have mainstream chatbots that 'learn' from user prompts, instead persistence is generally per-session, not global. And there is no large-scale text corpus that is entirely 'true' so we have additional measures to reduce 'hallucinations' and other inaccuracies.

The xAI Grok model (originally called TruthGPT) hasn't been without issues—some due to using X (née Twitter) content as source material:

Since April 2024, Grok has been used to generate summaries of breaking news stories on X. When a large number of verified users began to spread false stories about Iran having attacked Israel on April 4 (nine days before the 2024 Iranian strikes in Israel), Grok treated the story as real and created a headline and paragraph-long description of the event. Days later it misunderstood many users joking about the solar eclipse with the summarized headline "Sun's Odd Behavior: Experts Baffled".

Musk's branding and sales pitch for Grok employs deliberately crafted provocation to appeal to a certain demographic (@PHD's post picked up one of those provocations verbatim). And there are many more examples, but they lead to political terrain out-of-scope for discussion here, as you've noted.

Others like Microsoft (or Alphabet, or Apple and so on) aren't going to do that. They'll have to deal with societal norms in order for their 'AI' products/services to be acceptable and useable for broad demographics in the US and globally. To paraphrase your post, certain things are off limits or unacceptable. Which means design interventions and value judgements. What else can they do?

 

[Old_School_Brad]

Others like Microsoft (or Alphabet, or Apple and so on) aren't going to do that. They'll have to deal with societal norms in order for their 'AI' products/services to be acceptable and useable for broad demographics in the US and globally. To paraphrase your post, certain things are off limits or unacceptable. Which means design interventions and value judgements. What else can they do?

This is precisely what makes such approaches less objective and therefore less useful in an objective sense. One could argue that AIs catering to societal trends -prioritizing something other than pure facts because it’s considered unfashionable, might offend someone, or is deemed risky -diminishes their value.

An AI influenced by trendy or political motives is likely not what humanity truly needs or wants.

 

[mcdn]

Microsoft Copilot is a brand not a technology. The specific copilots you get help from depend on what you're trying to do - web search is different to word is different to GitHub is different to Excel. Absolutely no MS copilot instance is using the latest and greatest models from OpenAI or anywhere else - that would be ruinously expensive for MS in compute time, and very slow to get responses for users who just want to make a pivot table without getting a degree in statistics.

MS customers wanting top-shelf performance and data privacy pay for private OpenAI models instances running on Azure. If they have even more budget they pay for those models to be fine-tuned in various ways. There's a list of SKUs and prices but you'll need an enterprise agreement to find out what they are. None of these things are "copilots" though!

 

[Dimitri]

There is an old joke about "if microsoft built airplanes..." .
It appears some of those ambitious people moved over to management at mcdonnel douglas....but that's a subject for a different thread.

But back on the topic of ChatGPT vs CoPilot etc ...
Openai, is an AI company. They are not a mutlitool like microsoft.

Microsoft on the other hand , has everything Microsoft available to them to "sprinkle" AI pixie dust on it and improve it ( or add value in investor-speak or break it in user-speak) only like Microsoft is able to do.

openAI in a sense more independent than any division/department/section/subsidiary or whatever "separate thing" microsoft is bale to cobble up together

I believe the goals of the two companies to be competely different.
MS has to produce and inject AI carefully into its offerings so it augments the products, not compete with them.

openAI, I expect has no such restrictions

So far I like chatGPT, it even "gets it" and responds to humor which is kinda scary in a way.

I was tempted to have it rewrite this post, but I resisted

 

[Audio Science Pal]

Interesting thread.

 

[amirm]

I believe the goals of the two companies to be competely different.
MS has to produce and inject AI carefully into its offerings so it augments the products, not compete with them.

I appreciate that. The issue is that Suleyman still speaks of AI in the generic way that OpenAI, etc. deliver (for the most part). I gave the example of him arguing with AI chat bot on what movie to watch.

 

[Axo1989]

May I suggest you're thinking of it as if it can think... it can't. Providing related information and topics to explore is pretty standard and doesn't mean it's pushing any kind of specific agenda. It's just to you it may seem that way. Climate change is one of those things that's backed up by large amounts of factual research data.

Not to dwell on that subject overly, but you are correct of course. Unless one is focussing on a narrow technical question or function, generating text about renewable energy while avoiding discussion of climate change issues (being a significant factor in renewable energy economics, rationales and strategies) would require some odd constraints and/or distortions (which would be political, not scientific). Arguing for 'AI' without guardrails, then insisting on avoiding certain subject matter is inconsistent nonsense.

This is precisely what makes such approaches less objective and therefore less useful in an objective sense. One could argue that AIs catering to societal trends -prioritizing something other than pure facts because it’s considered unfashionable, might offend someone, or is deemed risky -diminishes their value.

An AI influenced by trendy or political motives is likely not what humanity truly needs or wants.

We don't disagree. But don't conflate societal norms with trendy motives, that's a straw man and an oversimplification.

Selective facts are certainly problematic, but 'pure facts' are also insufficient. For example, most societies posit as normative that slavery or genocide are unacceptable. We won't go into detail on this forum, but suffice to consider that one social group can gain competitive advantage by eliminating another, or that fully functional economies can be built on slave labour. Acceptability or otherwise is primarily based on morals and ethics, rather than facts. A probabilistic text generator can refer to those things in context obviously, but Microsoft and others won't let their 'AI' products advocate those things, and the necessary applied constraints are normative, not factual. It's impossible to avoid value judgements when developing and implementing functional 'AI' products in our societies.

So we have two broad problem areas. Firstly, there is no comprehensive training corpus comprised of 'pure facts' to start with. Secondly, some normative constraints will be applied necessarily, requiring value judgements, some of which will be contested.

 

[dshreter]

There's no such thing as a "neutral" AI model. No matter what there is an editorial decision made by humans of what training data to feed the system.

If you were making an audio system agent, should you train it on Audio Science Review? Audiogon? The collective works of Floyd Toole? Amazon.com reviews? All four? None of these?

This is just one example, but all models present some sort of human point of view and judgment on the quality of training data.

 

[Axo1989]

Who knew that Mustafa Suleyman was an accomplished musician? The album is called God of the Impossible ... perhaps that's our Suleyman's job description.

 

[Old_School_Brad]

Not to dwell on that subject overly, but you are correct of course. Unless one is focussing on a narrow technical question or function, generating text about renewable energy while avoiding discussion of climate change issues (being a significant factor in renewable energy economics, rationales and strategies) would require some odd constraints and/or distortions (which would be political, not scientific). Arguing for 'AI' without guardrails, then insisting on avoiding certain subject matter is inconsistent nonsense.



We don't disagree. But don't conflate societal norms with trendy motives, that's a straw man and an oversimplification.

Selective facts are certainly problematic, but 'pure facts' are also insufficient. For example, most societies posit as normative that slavery or genocide are unacceptable. We won't go into detail on this forum, but suffice to consider that one social group can gain competitive advantage by eliminating another, or that fully functional economies can be built on slave labour. Acceptability or otherwise is primarily based on morals and ethics, rather than facts. A probabilistic text generator can refer to those things in context obviously, but Microsoft and others won't let their 'AI' products advocate those things, and the necessary applied constraints are normative, not factual. It's impossible to avoid value judgements when developing and implementing functional 'AI' products in our societies.

So we have two broad problem areas. Firstly, there is no comprehensive training corpus comprised of 'pure facts' to start with. Secondly, some normative constraints will be applied necessarily, requiring value judgements, some of which will be contested.

Obviously, that goes without saying.
You can’t allow someone to use an AI as a collaborator in their evil plans.

When AI is used as a tool for search and information, it should be free of any restrictions or biases related to ideologies. Its job is to deliver facts. The real problems arise only when such tools are misused in a malicious context.
This highlights why proper AI governance is so crucial. Perhaps a "Sandbox AI" could be designed to monitor the main chat for malicious intent without being influenced itself.

 

[MaxwellsEq]

When AI is used as a tool for search and information, it should be free of any restrictions or biases related to ideologies

That's engineeringly impossible.

This is not search - i.e. a robot that trawls available online sources then uses a backlinks to prioritise query responses.

These tools are competitively trained on the available online content. Then they use probability algorithms to write sentences that look as close as possible to commonly used phrases. It's not a tool for facts. It's a tool for well-formed sentences trained on what's on the Internet. There are input and output guardrails, but they are unlikely to filter ideology unless it's illegal.

 

[MaxwellsEq]

For everybody who considers AI output potentially trustworthy, have a think about what happens for non-English use cases...

Basically to save you the research effort - you get different answers. If your native or sole language is not heavily used on the Internet, there is less material for competitive training of transformers etc. and fewer examples for building probability algorithms from. The result is significantly different answers from the English AI tools.

 

[mhardy6647]

...It's not a tool for facts. It's a tool for well-formed sentences trained on what's on the Internet. ...

On one hand, this is a chilling thought.
On the other hand, well-formed sentences would improve "what's on the Internet", on balance.

 

[Old_School_Brad]

For everybody who considers AI output potentially trustworthy, have a think about what happens for non-English use cases...

Basically to save you the research effort - you get different answers. If your native or sole language is not heavily used on the Internet, there is less material for competitive training of transformers etc. and fewer examples for building probability algorithms from. The result is significantly different answers from the English AI tools.

Huh, is this right? I would have assumed it searches for answers across all languages, analyzes the consensus, and then presents the result in the language you used to ask the question.

 

[MaxwellsEq]

Huh, is this right? I would have assumed it searches for answers across all languages, analyzes the consensus, and then presents the result in the language you used to ask the question.

It's the truth. A friend builds analytical tools wrapped around RAG and AI APIs. He has real challenges with non-English languages. Spanish, French, Portuguese etc. are not so bad but smaller languages give very different results.

searches for answers across all languages

It does NOT search. It absorbs and then trains. It's not a search engine.

It cannot be multilingual when training. It can only train in one language. Otherwise, if how can it compare one sentence in English and another in Serbian? Imagine you are teaching a six year old and you answered each of their questions in a different language, how would they learn.

 

[Old_School_Brad]

It's the truth. A friend builds analytical tools wrapped around RAG and AI APIs. He has real challenges with non-English languages. Spanish, French, Portuguese etc. are not so bad but smaller languages give very different results.


It does NOT search. It absorbs and then trains. It's not a search engine.

Wow, that must require an enormous amount of data storage!

It cannot be multilingual when training. It can only train in one language. Otherwise, if how can it compare one sentence in English and another in Serbian? Imagine you are teaching a six year old and you answered each of their questions in a different language, how would they learn.

I guess this operates quite differently from what I expected. It must be incredibly resource-intensive!

What are your sources? Have you looked into it before? (I'm not asking for specific sources, just curious about how you know.)

 

[teched58]

Different reply is fine. This is far more than that. It is like CoPilot is not even trying to answer the question. Or on purpose is attempting not to.

This reminds me of the 1990s, when every time a MS manager or flack was asked a question, they would respond, as if autonomically batting back the query: "We will ship [product XYZ] when our customers tell us it's ready."

 

[JSmith]

Both CoPilot (Bing) and ChatGPT do use search engines as a source to further train on, if the answer to the query isn't known or if more up to date information is needed, like current events for example.


JSmith

 

[Old_School_Brad]

Both CoPilot (Bing) and ChatGPT do use search engines as a source to further train on, if the answer to the query isn't known or if more up to date information is needed, like current events for example.


JSmith

So, is that their own in-built search engines, or are they using Bing or Google? -Or both?