Master Thread: Are Measurements Everything or Nothing?

[rdenney]

Yes, Dan, but nobody is claiming that the multigenerational loop-back test measures the specific performance of the converters. They are constructing a demonstration to show that the faults of converters are inaudible, if the first conversion measures well (flat response, negligible distortions and noise). It puts the lie to the golden ears who claim they can hear the unmeasurable.

Some of us have measured instrument spectrums pretty carefully. I’m a tuba player and my tuba sound is stronger in the third, fourth and sixth harmonics than it is in the fundamental, else the sound wouldn’t have any carrying power at all. Added distortion would be hard to hear. But it’s easy to hear when it’s a single sine wave, which makes that the more rigorous test if hearing is the designated judge. Without gain-riding the quiet bits like the tails of reverberation to high levels, I can’t hear harmonic distortion at lower than about -36 dB when playing acoustic music, but I can hear it at -60 dB easily with just a sine wave, even though it might not be undesirable. That’s not the point, though, except to emphasize further that ears make a poor judge of electronics in the modern era.

If I was testing a product like a converter for sale, it would have to measure superbly and also fulfill all its operational requirements reliably and robustly.

We focus a lot on harmonic distortion, but noise and clipping are worse problems in amps (and in intermediate gain stages if the gain structure is wrong).

Rick “whose ADC is a Benchmark that has precision gain controls” Denney

 

[antcollinet]

I would expect to see the noise to come up mostly ?

In the plot above there are 6 significant harmonics of the 1kHz tone, plus one or two smaller spikes. It seems likely that with a 32 tone multitone then that there will be at least 192 individual harmonic tones, plus a whole mass of intermodulation tones as every tone within the test signal inter modulates with all the other tones.

This is what makes up the grass at the bottom of the normal multitone test plot - pretty much indistinguishable from the noise floor. I would expect that all these tones will stack up on top of them selves with multiple passes. It will look like the noise floor rising - but will (IMO) be largely distortion.


(look at all the distortion tones created by the 19/20 IMD test and imagine the number multiplied by about 500, - the number of ways 32 tones can pair with each other - plus the approx 200 harmonic tones - why it just looks like noise, and why Amir always uses multitone to state the "distortion free range" of a device.

 

[solderdude]

In the plot above there are 6 significant harmonics of the 1kHz tone, plus one or two smaller spikes. It seems likely that with a 32 tone multitone then that there will be at least 192 individual harmonic tones, plus a whole mass of intermodulation tones as every tone within the test signal inter modulates with all the other tones.

This is what makes up the grass at the bottom of the normal multitone test plot - pretty much indistinguishable from the noise floor. I would expect that all these tones will stack up on top of them selves with multiple passes. It will look like the noise floor rising - but will (IMO) be largely distortion.


(look at all the distortion tones created by the 19/20 IMD test and imagine the number multiplied by about 500, - the number of ways 32 tones can pair with each other - plus the approx 200 harmonic tones - why it just looks like noise, and why Amir always uses multitone to state the "distortion free range" of a device.

Yep it will be a 'music content modulated' increased noise floor which is not audible ass 'noise' but there will also be an actual noise increase.
It is all about audibility thresholds though.
As long as that stays away 60-80dB below the signal it means inaudible ... to me.

 

[antcollinet]

Yes, Dan, but nobody is claiming that the multigenerational loop-back test measures the specific performance of the converters. They are constructing a demonstration to show that the faults of converters are inaudible,

This - It has been stated multiple times now.

 

[antcollinet]

to me.

To pretty much everyone in real world listening.

 

[danadam]

preferably something with say 20 to 32 tones and maybe pink-noise shaped to mimic a music signal (in spectrum).

I would expect to see the noise to come up mostly ?

Here's:

• FiiO K3 (ESS) low-gain -- SNR: 109.5 dB, THD: -91.2 dB
• JCally JM20 -- SNR: 112.2 dB, THD: -124.5 dB

,

3 multitones:

• 32 from AudioPrecision that amirm posted (I resampled it to 44.1k)
• 20 from pkane's Multitone Analyzer
• 16 from REW (No Interharmonic Distortion version)

(The 32-tone is rather dense, so I also included a graph with zoom to 1k-10k range)

FiiO K3:
, , ,

JCally JM20:
, , ,

With -80 dBFS RMS noise:


added to the 20-tone, it covers the distortion in its range:
,

Note: I used line width 1.2 in the graphs to make the "layers" more visible. Here's comparison with line width 1.0 on FiiO K3 and 16-tone:
,

 

[solderdude]

Kind of as expected but with a surprising difference at the same time.

In the K3 the doubling of attempts always leads to 6dB (a doubling) of distortion signal levels.
The JM20, however, seems to do much better (as well as lower in level) and a doubling of attempts only results in 4dB increase in distortion.
What part of this is caused by the ADC ?

I suppose this has something to do with the DS method being used ?

 

[keosoft]

This. To be pedantic: tube/solidstate alone is not too useful a distinction. For example Naim amps are solid state, but carefully designed for audible 2nd order harmonic distortion because their fans want it. OTOH I've heard tube amps (e.g. Manley) that were much less "tubey" in sound than, say, Cary. Circuit design matters!

But. If you look at the tube amps that are actually produced commercially, they are mainly made for customers who *like* 2nd order distortion etc. Tubes seem to lend themselves to designing circuits that do that. The difference shows up plainly in measurements, and it's also dead easy to hear.

Genuine question: are there commercially-available tube amps that don't test with audible distortion?

Some people really *like* colored sound and by gum, they should have it and enjoy it. This site caters to folks who don't.

Genuine question: are there commercially-available tube amps that don't test with audible distortion?

Here is a test of the 4000$ tube Audio Research SP-6C-1 preamp from the Hungarian Hi-fi Magazin from 1990. (you don't speak Hungarian, but you can read the Measurements

Review of Audio Research SP-6C-1 preamp

This amp measured very good, almost perfect (ruler-flat freq. resp., 0.003% thd, - 80 db noise,...), certainly deemed inaudibly imperfect by objectivist view.

And still, when listened in level-matched blind test against a Denon PMA-320A (a mid-level 300$ transistor amp) and an AGI 511A (a ~ 2000$ IC preamp), they were ABSOLUTELY blown-away by the music of the Audio Research! They described it as "enchanting", "a totally exhilarating experience", "a night and day difference in bass and dinamics".

They should've heard no difference at all, all 3 amps measured quite the same (the transistor amps were perhaps a LITTLE bit better in noise, but the AR killed them all in phono-preamp overload headroom).

And this type type of praise was very unusual for them, they did not accept ads, and they were down-to-earth, brutally honest guys. This sincerity cost them some lawsuits from vendors, but they win them all (they always could back up their findings with measurements

So maybe our knowledge of what is audible, what not, human perception, questioning accepted psychoacoustic truths may be worth some Research.

(I know that hi-fi/pro-audio is an annoyingly little business compared to almost anything else tech-related (computers, cars, mobile phones, Military industry,...) so the motivation is essentially non-existent. Its simply not sexy nowdays :l

Ég és Föld: Audio Research SP-6C-1 előerősítő

 

[RickS]

Hearing this thread needs a break and possibly some content relocated.

Locked for now.

P.S.

Upon review, it appears a reasonable technical discussion got personal and more off topic. To address, I have moved some of the posts here: https://www.audiosciencereview.com/...d-savitr-new-a-d-standalone.19645/post-646366

Thread is open again.

 

[Newman]

Who's next?

 

[solderdude]

Genuine question: are there commercially-available tube amps that don't test with audible distortion?

Here is a test of the 4000$ tube Audio Research SP-6C-1 preamp from the Hungarian Hi-fi Magazin from 1990. (you don't speak Hungarian, but you can read the Measurements

Review of Audio Research SP-6C-1 preamp

This amp measured very good, almost perfect (ruler-flat freq. resp., 0.003% thd, - 80 db noise,...), certainly deemed inaudibly imperfect by objectivist view.

And still, when listened in level-matched blind test against a Denon PMA-320A (a mid-level 300$ transistor amp) and an AGI 511A (a ~ 2000$ IC preamp), they were ABSOLUTELY blown-away by the music of the Audio Research! They described it as "enchanting", "a totally exhilarating experience", "a night and day difference in bass and dinamics".

They should've heard no difference at all, all 3 amps measured quite the same (the transistor amps were perhaps a LITTLE bit better in noise, but the AR killed them all in phono-preamp overload headroom).

And this type type of praise was very unusual for them, they did not accept ads, and they were down-to-earth, brutally honest guys. This sincerity cost them some lawsuits from vendors, but they win them all (they always could back up their findings with measurements

So maybe our knowledge of what is audible, what not, human perception, questioning accepted psychoacoustic truths may be worth some Research.

(I know that hi-fi/pro-audio is an annoyingly little business compared to almost anything else tech-related (computers, cars, mobile phones, Military industry,...) so the motivation is essentially non-existent. Its simply not sexy nowdays :l

Ég és Föld: Audio Research SP-6C-1 előerősítő

How 'well' was this test done ?
Was everything measured correctly in situ (not stand alone on a resistor) ?
If the test were repeated would the results/conclusions still be the same ?

 

[keosoft]

Hm, I don't know what you mean by testing 'in situ'.

The test was done by the top State laboratory by Antal Sólymos, the leading measurements/standards expert in Central Europe.
The AR was terminated by 47 kohm standard load when measured.

"If the test were repeated would the results/conclusions still be the same ?"

Of course not! It was 36 years ago! The amp should be refurbished with new caps/passive parts/new tubes (as any amp from 36 years back). But these were built for longevity. Their unit was 8 years old when tested and exceeded all their factory secs without even biasing the tubes. Audio Research was a transparent company which is quite difficult to find nowadays in high-end. Measurements, schematics, part numbers, everything. Nothing was just hype and myths as nowadays.

These are quite hard to find, they sell for 1500-3000 $ as vintage now.

The new units sell for astronomical prices of course.

The factory specs are here:

https://www.hifiengine.com/manual_library/audio-research/sp-6.shtml

 

[MAB]

Genuine question: are there commercially-available tube amps that don't test with audible distortion?

Here is a test of the 4000$ tube Audio Research SP-6C-1 preamp from the Hungarian Hi-fi Magazin from 1990. (you don't speak Hungarian, but you can read the Measurements

Review of Audio Research SP-6C-1 preamp

This amp measured very good, almost perfect (ruler-flat freq. resp., 0.003% thd, - 80 db noise,...), certainly deemed inaudibly imperfect by objectivist view.

And still, when listened in level-matched blind test against a Denon PMA-320A (a mid-level 300$ transistor amp) and an AGI 511A (a ~ 2000$ IC preamp), they were ABSOLUTELY blown-away by the music of the Audio Research! They described it as "enchanting", "a totally exhilarating experience", "a night and day difference in bass and dinamics".

They should've heard no difference at all, all 3 amps measured quite the same (the transistor amps were perhaps a LITTLE bit better in noise, but the AR killed them all in phono-preamp overload headroom).

And this type type of praise was very unusual for them, they did not accept ads, and they were down-to-earth, brutally honest guys. This sincerity cost them some lawsuits from vendors, but they win them all (they always could back up their findings with measurements

So maybe our knowledge of what is audible, what not, human perception, questioning accepted psychoacoustic truths may be worth some Research.

(I know that hi-fi/pro-audio is an annoyingly little business compared to almost anything else tech-related (computers, cars, mobile phones, Military industry,...) so the motivation is essentially non-existent. Its simply not sexy nowdays :l

Ég és Föld: Audio Research SP-6C-1 előerősítő

Sorry, I see no reference to blind, level matched tests.
Did I miss something?
They did a good technical review of the device verifying it measured as it should.
They appear to have performed the standard sighted listening evaluation, and said some nice things.8 Which isn't useful in any way. Again, perhaps I missed something.

That being said, many tests of good performing tube gear have been made, Audio Research is certainly one of them. They are audibly indistinguishable from many other good amps, unless someone tries something funky like drive a pair of Kappa 9 or Scintillas, which is not an appropriate speaker to attempt with such an amp.

Many amps were tested back in the '80s by David Clark (see page 78):

https://americanradiohistory.com/Archive-HiFI-Stereo/80s/HiFi-Stereo-Review-1987-01.pdf

Amps operated within their power envelope, and not hooked up to speakers with inappropriate impedance for the amp, are indistinguishable when level matched. Clark did proper controlled blind tests, even with extreme sceptics. They were not able to hear the differences.
The examples where people can hear differences are often when a tube amp or amp with high output impedance is hooked up to a low-impedance pair of speakers. This case does sound different from an amp with low output impedance, not necessarily better, often not so good.

I'm open to new info, if it is done correctly. Unfortunately, most tests are not done with proper controls.

 

[Rob_Gordon]

Hello there,

Since I've discovered ASR, I became very skeptical of any individual who claims to hear differences between gear.

I understand that what we do defend here is not that we can't ever hear differences, of course, but mostly that we can only hear differences that are measurable.

And I get that within the hi-fi world, lots of folks claim to hear stuff that isn't measured.

I was wondering if any of you knew anything about musicans/gear heads' earring abilities.

I've been down the rabbit hole for months, watching videos and listening to different guitar amps/tubes/pedals. Though it's pretty much never ABX, I do appreciate the fact that the differences between, say, a Marshall Plexi and a Fender Tweed are, indeed, measurable.

What I'm wondering is: do we know how reliable are people's ears/claims in that specific context? Cause, sometimes, their claims do sound like BS as well.

Is there, like, a mixture of things? The two amps do sound different, but what they perceive still is "fictitious" because they see the amp and their perception is altered by the history of the gear, the albums they know have been recorded with it, etc. ?

Isn't it a weird situation where there's empirical data to support the difference, but, at the same time, reasons to believe the difference perceived is still very much unreliable? Like, you could do well in a double blind setting and yet still imaging things when you see the amps. You perceive both "real" and "false" things?

Hopefully this message isn't completely stupid.

 

[solderdude]

Hello there,

Since I've discovered ASR, I became very skeptical of any individual who claims to hear differences between gear.

I understand that what we do defend here is not that we can't ever hear differences, of course, but mostly that we can only hear differences that are measurable.

And they have to reach audible limits which is a very important aspect which has many, many factors that are in play.

And I get that within the hi-fi world, lots of folks claim to hear stuff that isn't measured.

I was wondering if any of you knew anything about musicans/gear heads' earring abilities.

Yes, they don't seem to be performing any better in blind tests (according to research done by Harman etc.)
What they generally are better at is hearing small differences in instruments they are very familiar with.
The reason for that is ... training. Lots and lots of practice and experience with specific instruments.
If you would train your average hifi buff the same trick (tell them what to notice and train them for months) they will be able to do the same.

It is similar to a person getting really good at say ... Sudoku puzzles but when that person is very good at it that doesn't mean he can also do other puzzles just as well.

I've been down the rabbit hole for months, watching videos and listening to different guitar amps/tubes/pedals. Though it's pretty much never ABX, I do appreciate the fact that the differences between, say, a Marshall Plexi and a Fender Tweed are, indeed, measurable.

Yes very measurable and obviously within audibility. Some of those comparisons might be obvious also because of level differences by the way.

What I'm wondering is: do we know how reliable are people's ears/claims in that specific context? Cause, sometimes, their claims do sound like BS as well.

This has been researched a lot. Books by Floyd Toole and Harman research is just the best known but there have been many studies in this field.

Is there, like, a mixture of things? The two amps do sound different, but what they perceive still is "fictitious" because they see the amp and their perception is altered by the history of the gear, the albums they know have been recorded with it, etc. ?

When 2 amps are perceived to sound different either they are (in combination with a speaker) or they don't play equally loud or one is clipping and the other is not.
Guitar amps usually sound different. This is by design.

Isn't it a weird situation where there's empirical data to support the difference, but, at the same time, reasons to believe the difference perceived is still very much unreliable? Like, you could do well in a double blind setting and yet still imaging things when you see the amps. You perceive both "real" and "false" things?

Yep, that's how it works. When 2 amps (or DACs) that are 'identical' within audible thresholds are AB'ed level matched and with statistical relevance can not be distinguished you can still perceive (not the word sound or heard) substantial differences. That's because the brain uses more than just the hearing and level differences, operational differences, price/looks etc all make a difference in how 'sound' is perceived.

Hopefully this message isn't completely stupid.

No it isn't, why would it be ?

 

[Rob_Gordon]

Thank you very much for your thorough answer!

And they have to reach audible limits which is a very important aspect which has many, many factors that are in play.

Yes, of course, I should have specified it, thanks for reminding me.

Yes, they don't seem to be performing any better in blind tests (according to research done by Harman etc.)
What they generally are better at is hearing small differences in instruments they are very familiar with.
The reason for that is ... training. Lots and lots of practice and experience with specific instruments.
If you would train your average hifi buff the same trick (tell them what to notice and train them for months) they will be able to do the same.

It is similar to a person getting really good at say ... Sudoku puzzles but when that person is very good at it that doesn't mean he can also do other puzzles just as well.

Yes very measurable and obviously within audibility. Some of those comparisons might be obvious also because of level differences by the way.

This has been researched a lot. Books by Floyd Toole and Harman research is just the best known but there have been many studies in this field.

Oh ok. I knew they wrote about hi-fi, I didn't know they also researched musicians.

When 2 amps are perceived to sound different either they are (in combination with a speaker) or they don't play equally loud or one is clipping and the other is not.
Guitar amps usually sound different. This is by design.

Yes, of course. I guess I was especially intrigued by the "sound profile" they say they hear. Not only perceiving a difference (cause, yeah, obviously, I know that amp designers aim for a "sound identity") but the type of comment they make "this has more bite ; this is more open ; this is so much bigger".

I'm also skeptical of the "scales" their comments imply. "Night and day"... I don't buy it, most of the time.

Yep, that's how it works. When 2 amps (or DACs) that are 'identical' within audible thresholds are AB'ed level matched and with statistical relevance can not be distinguished you can still perceive (not the word sound or heard) substantial differences. That's because the brain uses more than just the hearing and level differences, operational differences, price/looks etc all make a difference in how 'sound' is perceived.

Yeah, and in that specific world (rock music), gear is mythology.

No it isn't, why would it be ?

Thanks, ahah, well, I guess, I'm always a little intimidated here, I've posted things that I though weren't stupid but were received as such!
thanks again!

 

[solderdude]

Oh ok. I knew they wrote about hi-fi, I didn't know they also researched musicians.

The research was about sound reproduction and they used various 'groups' of listeners including a 'musician' group.
They did not perform 'better than the sound engineer and hifi salesman group.
A funny thing is that most musicians don't really care for highest sound quality where sound engineers do.

Yes, of course. I guess I was especially intrigued by the "sound profile" they say they hear. Not only perceiving a difference (cause, yeah, obviously, I know that amp designers aim for a "sound identity") but the type of comment they make "this has more bite ; this is more open ; this is so much bigger".

I'm also skeptical of the "scales" their comments imply. "Night and day"... I don't buy it, most of the time.

Yep, someone's 'subtle' seems to be someone else's 'night and day'.
A small volume difference only can mean 'night and day' but a small difference in level might just be 'subtle'.
Hard to say what happened during the comparison.

Yeah, and in that specific world (rock music), gear is mythology.

It is mythology in both the recording and 'music reproduction' world

Thanks, ahah, well, I guess, I'm always a little intimidated here, I've posted things that I though weren't stupid but were received as such!
thanks again!

Not all reactions are equally well thought out nor received.

 

[NTK]

Thanks, ahah, well, I guess, I'm always a little intimidated here, I've posted things that I though weren't stupid but were received as such!
thanks again!

Another benefit of participating in ASR. It is a good training ground to develop a thicker skin

 

[Hapo]

...you guys don't have thick skin...just sayin'... .!..!.

 

[Mort]

...you guys don't have thick skin...just sayin'... .!..!.

Funny but still bait.