Does what we hear correspond to what we measure?

[Vintage02]

A

Android phones have a poor reputation for sound out of the box. There are lots of things you can do to fix this.

You were testing your android phone AND the Bluetooth codecs at the same time. Firstly you need to ensure your android setup is bit-perfect where possible.

To test a Bluetooth receiver, you necessarily need a Bluetooth source. It seems to me that the data coming from the streaming flow is digital, then transmitted by the phone's Bluetooth chip, still in digital form. The AptX-HD or LDAC codecs will indeed modify this data since they are not lossless, but we remain in the digital domain. I'm not certain that switching from one phone to another will have a significant effect as long as we stay in the digital domain.

 

[tmtomh]

To test a Bluetooth receiver, you necessarily need a Bluetooth source. It seems to me that the data coming from the streaming flow is digital, then transmitted by the phone's Bluetooth chip, still in digital form. The AptX-HD or LDAC codecs will indeed modify this data since they are not lossless, but we remain in the digital domain. I'm not certain that switching from one phone to another will have a significant effect as long as we stay in the digital domain.

You're right - you're not sure, and no one else is either. That's why @MaxwellsEq says that to isolate the issue of BT transmission/codec to be sure that's what is causing any potentially audible issues, one should make sure that the signal path right up to the BT transmission is bit-perfect.

I understand this can seem tiresome or pedantic, but sorry - if you want to try to "correlate measurements with what we hear" then that requires some rigor and due diligence in setting up listening tests so we're sure about which measurements we are testing with our listening.

Too many folks in this thread are shooting the messengers when it comes to test design.

 

[MaxwellsEq]

But isn't that impossible with any BT codec?

You miss the point. The OP was requesting we run a comparison test where we could see if the Bluetooth codecs was audible. But the OP used two different paths one with Android and one without.

Analysis shows quite a lot of mess in their results, including potentially all the Android out-of-the-box problems.

There's something wrong with levels, something wrong with balance, something wrong with clipping, something wrong with dynamic range compression. Most of this has nothing to do the Bluetooth! Some of us are trying to pin down what the mistake are...

 

[pablolie]

You miss the point. The OP...

I clearly wasn't replying to the OP. I was replying to the comment that one must "ensure your android setup is bit-perfect where possible", which seems impossible to begin with if BT is used. But if other elements other than BT are also lossy, then of course the comparison between the files is even more silly.

 

[Duke]

I would have offered these four options:
- I prefer 1
- I prefer 2
- I hear a difference but don't have a preference
- I hear no difference

I like this idea! I've been wondering how many people listened but did not participate for one of the latter two reasons.

I would also have clarified that the test samples were from the same source, with one of them processed through an additional device.

I think this could have introduced subconscious bias. Let me explain: If I believe I'm really good at detecting processing, or that processing will result in audible artifacts, I might think that I hear a difference when I really do not. And if I believe the opposite, then I might not think I'm hearing a difference when I actually do.

(I remember taking one of the tests that Earl Geddes and Lydia Lee designed for their research into distortion perception. Because Earl had told me what the test was testing for beforehand, my data could not be included in their analysis since my knowledge of what they were looking for could have biased my answers.)

Two additional thoughts: Letting participants see how the voting is going can also introduce bias. I might want to follow the crowd, or I might think that crowds are often duped and therefore be inclined to vote the opposite.

Also, obviously once someone has used software to analyze the tracks, the data they are seeing with their eyes will tend to bias their perceptions. And others reading their posts about their analysis will likewise have their perceptions biased.

Of course I understand that not revealing anything about the test, and concealing the results of the poll until after the poll has closed, and forbidding analysis of the tracks, would be very frustrating for the participants. So imo there's a trade-off between making the poll and thread interesting enough for people participate, and useful from the standpoint of conducting research. I'm not sure that adequately controlled research can even be conducted in this setting. When I have participated in administering controlled blind tests, even the order in which the choices were presented to listeners was randomized to prevent that from being a source of possible bias.

 

[Vintage02]

You're right - you're not sure, and no one else is either. That's why @MaxwellsEq says that to isolate the issue of BT transmission/codec to be sure that's what is causing any potentially audible issues, one should make sure that the signal path right up to the BT transmission is bit-perfect.

According to the information gathered, on Android the streams necessarily pass through AudioFlinger, which is not bit-perfect. We'll have to deal with it...

 

[Timcognito]

We'll have to deal with it...

If we can hear it at all. Nothing like that has even come close to being identified as reason one picked 1 over 2, as there has been no controlled experiment. Keep on trolling.

 

[Vintage02]

Keep on trolling.

Sorry, when I say "We'll have to deal with it..." it's because I have no other way to establish a Bluetooth connection to the receiver. I don't see how that could be considered "keep on trolling". But I may be misunderstanding the meaning of your comment related to the translation.

 

[Timcognito]

Sorry, when I say "We'll have to deal with it..." it's because I have no other way to establish a Bluetooth connection to the receiver. I don't see how that could be considered "keep on trolling". But I may be misunderstanding the meaning of your comment related to the translation.

It's related to unscientific approach and unwillingness to acknowledge any posts that point to that. If asked to pick one of two things and one participates one is forced to pick one of two. I says nothing about the difference. That is why I never chose and have been thinking this an AI generated or trolling adventure.

 

[antcollinet]

I clearly wasn't replying to the OP. I was replying to the comment that one must "ensure your android setup is bit-perfect where possible", which seems impossible to begin with if BT is used. But if other elements other than BT are also lossy, then of course the comparison between the files is even more silly.

I think the intent is bit perfect up until bluetooth. Rather than the usual android mangling of audio and *then* the additional bluetooth compression.

 

[antcollinet]

it's because I have no other way to establish a Bluetooth connection

But what you could do - is the necessary work to ensure android isn't making any additional changes. Or you should just accept you don't have the necessary gear to do the test properly.

BTW an analogue input to bluetooth transmitter can be had relatively inexpensively - though then of course you have the problem of the A to D inside that device.

 

[Thomas_A]

If there are synced and as good as it gets level matched and channel matched files, I can do the ABX. Level and channel differences is no real point to ABX.

 

[krabapple]

I do not think it has anything to do with the topic. It probably has to do with the tendency of inquisition we see on this forum.

Goodness gracious, a tendency to doggedly question, on an audio science forum?

 

[MAB]

Sorry, when I say "We'll have to deal with it..." it's because I have no other way to establish a Bluetooth connection to the receiver. I don't see how that could be considered "keep on trolling". But I may be misunderstanding the meaning of your comment related to the translation.

The Bluetooth device is the easy part. Most people who seriously want to investigate the impact of BT would use an inexpensive eval board instead of an entire phone (for example). And of course would have created level-matched files with no other unintended artifacts or 'tells'.

Most importantly, the test needed to be done blind.

Like in medical studies with a placebo... the investigators don't tell the subjects if they got the placebo or not. Or the taste of Coke vs. Pepsi, a strong preference exists in sighted tests, the majority of that preference goes away when the subjects don't know which beverage they are drinking.

Of course, you never even established that there was an audible difference in your samples, which would also need to be done blind with good test files.

There is a science to experimental design.

 

[krabapple]

It seems to me that immediately saying that recording No. 1 came from a Bluetooth receiver would have irreparably biased the listening. For me, the best tests are when you don’t know you are taking a test... then the responses have a chance of not being biased.

Now there's an idea. Why didn't we think of that?

 

[Vintage02]

It's related to unscientific approach and unwillingness to acknowledge any posts that point to that. If asked to pick one of two things and one participates one is forced to pick one of two. I says nothing about the difference. That is why I never chose and have been thinking this an AI generated or trolling adventure.

It is necessary to put things into context, and if you reread my statements, I did not initiate the topic by talking about a scientific approach but about a listening test. You will also see that I admitted without problem some clumsiness in the test approach. I find that the use of the word "troll" and the verb "to troll" is not very respectful, especially when repeated several times. I believe I have exercised all possible courtesy so that no one feels singled out.

 

[Vintage02]

The Bluetooth device is the easy part. Most people who seriously want to investigate the impact of BT would use an inexpensive eval board instead of an entire phone (for example). And of course would have created level-matched files with no other unintended artifacts or 'tells'.

Most importantly, the test needed to be done blind.

Like in medical studies with a placebo... the investigators don't tell the subjects if they got the placebo or not. Or the taste of Coke vs. Pepsi, a strong preference exists in sighted tests, the majority of that preference goes away when the subjects don't know which beverage they are drinking.

Of course, you never even established that there was an audible difference in your samples, which would also need to be done blind with good test files.

There is a science to experimental design.

One learns from one's mistakes...

 

[tmtomh]

It is necessary to put things into context, and if you reread my statements, I did not initiate the topic by talking about a scientific approach but about a listening test. You will also see that I admitted without problem some clumsiness in the test approach. I find that the use of the word "troll" and the verb "to troll" is not very respectful, especially when repeated several times. I believe I have exercised all possible courtesy so that no one feels singled out.

For what it's worth, I agree that you are not trolling.

 

[Timcognito]

For what it's worth, I agree that you are not trolling.

Maybe not, but is in an advanced training session where the the supposition is that BT augments the signal in audible manner. No matter what the response from ASR that post must be continued in perfect english in a unemotional and dignified manner, pleading ignorance for any mistakes, yet continuing to assert the initial premise.

 

[pablolie]

Of course not. The ratio I was talking about here was 7 votes to 2, which is 3.5 : 1.

Several of the votes were from people that clearly analyzed the files using a variety of tools. The resulting stats are utterly and completely useless.