Master Thread: Are Measurements Everything or Nothing?

[John_Siau]

I have tried to replicate your test, with 2 class AB amplifiers, A250W and OPA549. The later has quite horrible crossover distortion, the former is free of low power crossover distortion. The THD vs. power plots, scaled in dBW as you did, look like this:

View attachment 384210

On the basis of the above graph, it looks like the 0.01W THD of the OPA594 is about 9 to 10 dB lower than our RA-500 test amplifiers. Given this 10 dB reduction, it may be much more difficult to hear the distortion. At this level, the THD would only be 3 to 4 dB SPL (assuming 87 dB speakers) and this would make it very hard to hear (perhaps impossible).

Any files provided for ABX testing should be acquired directly from the speaker terminals and not using a microphone.

Obviously, the files will allow playback at higher levels. With the recorded levels boosted on playback, it may be possible to hear the effect with the OPA594 recording (assuming the noise is low).

 

[JP]

Yes, here is the description from our blog:

At a 1 watt output, this class-AB amplifier produces a distortion waveform that measures 70 dB below the output level of the 1 watt test tone (see graph at the bottom of this application note). This means that the power produced by the THD+N is 70 dB below 1 watt. If we drop the level of the test tone by 20 dB, the output power is 0.01 W. At this output level the amplifier was still producing distortion at a level of 73 dB below 1 watt. At 0.01 watt the distortion waveforms look virtually identical to the 1 watt waveforms. This distortion was clearly audible when we used this amplifier to drive a speaker at 0.01 watt. For our tests we used a stereo pair of Benchmark SMS1 speakers with a sensitivity of 87 dB 1 watt, 1 meter. The tone was reproduced at a sound pressure level of about 67 dB (measured 67 to 68 dB SPL) at the listening position while the amplifier distortion was reproduced at a calculated sound pressure level of about 14 dB (87 dB - 73 dB = 14 dB). With a 0.01 watt 1 kHz test tone, the amplifier distortion was clearly audible through the loudspeaker.

And any attempts with music?

 

[John_Siau]

And any attempts with music?

No. At Benchmark we are not interested in determining how bad a defect needs to be before it can be heard in music. Instead, we aim to reduce noise, THD, phase errors, and frequency response errors to levels that are inaudible (even when using test tones).

Music can hide many defects, but these defects may be revealed by the right piece of music. Running these tests on music would be a long and arduous task. Inaudibility on a few music test tracks does not prove inaudibility. Instead, you need to find a music track that proves audibility.

You can never prove inaudibility using music tracks. You can only prove audibility.

The fact that audibility with a pure tone was confirmed is enough to suggest that we should avoid this particular distortion problem. If it can be heard with a pure tone, there is probably some piece of music somewhere that will reveal the same defect. After all, a musical recording could be a series of pure tones.

 

[DonH56]

^^^ +1

Using single, or perhaps two-three (for IMD), sine waves for testing is much more revealing than musical signals that tend to mask distortion. The problem is compounded because music generally has much less dynamic range than modern electronics, and it is much more difficult to establish a reference when using music compared to a pure sine wave. There are a myriad of studies showing we tolerate much higher distortion in music than when tested with a pure sine wave. The level of distortion detectable (JND, just noticeable difference) in music is thus much higher than when using a pure tone. As @John_Siau said, sine wave testing will more readily expose distortion levels well below what is detectable with (most, likely the vast majority of) music. It is much easier to hear distortion on a single isolated tone than buried in a random sample of many other tones.

 

[krabapple]

Now, for those interested in possible sound difference with the 1kHz sine signal recorded from both amplifiers at 10mW/4ohm, below you will find the zip file with 2 flac files. Please feel free to make a foobar ABX test and tell me if you can hear the difference, and possibly post a test report.

Note: test files (1kHz) are reduced to 16bits/44.1kHz, thus lower "ENOB"
View attachment 384218 View attachment 384219

From playing A and B in foo_abx I knew very quickly that I wouldn't be able to tell them apart. And 16 trials of this tone would trigger my tinnitus to an unhealthy degree so I stopped there.

(fwiw, replaygain scan shows a 0.06 dB difference in required gain to level match them)

 

[Titurel]

No. At Benchmark we are not interested in determining how bad a defect needs to be before it can be heard in music. Instead, we aim to reduce noise, THD, phase errors, and frequency response errors to levels that are inaudible (even when using test tones).

Music can hide many defects, but these defects may be revealed by the right piece of music. Running these tests on music would be a long and arduous task. Inaudibility on a few music test tracks does not prove inaudibility. Instead, you need to find a music track that proves audibility.

You can never prove inaudibility using music tracks. You can only prove audibility.

The fact that audibility with a pure tone was confirmed is enough to suggest that we should avoid this particular distortion problem. If it can be heard with a pure tone, there is probably some piece of music somewhere that will reveal the same defect. After all, a musical recording could be a series of pure tones.

I agree. The problems using natural sound as test signal are manifold, not the least of them being the qualification of a test instrument, i.e., a listener. For example, if listeners aren't familiar enough with violin sound to readily tell one fiddle (or one player) from another, what chance would they have recognizing the addition of small amounts of distortion. Real world instruments (non-electronic) produce a mixture of well correlated sounds (e.g., from the vibrating string) and poorly correlated sounds (the scraping of the bow). The presence of the latter could easily defeat reliable detection of amplifier generated distortion. Add one more fiddle, a viola, and a cello to your test signal and your listeners are pretty much SOL A possible alternative might be a synthesized instrument (not sampled!), one that produces the overtone structure of real world instruments, but omits the poorly correlated components, like bowing, reed noise, etc.

 

[pma]

Now, for those interested in possible sound difference with the 1kHz sine signal recorded from both amplifiers at 10mW/4ohm, below you will find the zip file with 2 flac files. Please feel free to make a foobar ABX test and tell me if you can hear the difference, and possibly post a test report.

In case someone thinks it should be easy to test against pure 1kHz generated tone, I attaching the sine file level matched with the amplifier tones.

 

[pma]

(fwiw, replaygain scan shows a 0.06 dB difference in required gain to level match them)

Yeah, and there are also distortion components that are also computed into final rms value. Take it (that 0.06 dB) as a part of the file difference . I wonder if the original test from 2016 was better matched. Still I would be interested if someone could hear a difference with the posted files "as is", without cheating by means of additional analyser tools. I am serious .

And, regarding the test described by @John_Siau , I wonder if the amp with xover distortion was not oscillating with the speaker used. Was it really a pure xover distortion that was indicated as a sound difference, or was it something else that might have resulted in a xover distortion as a by-product??

 

[Newman]

No. At Benchmark we are not interested in determining how bad a defect needs to be before it can be heard in music.

Then your article needs extensive re-writing, because it emphasises music as if you are interested in it.

Instead, we aim to reduce noise, THD, phase errors, and frequency response errors to levels that are inaudible (even when using test tones).

Err, doesn't everyone?

Music can hide many defects, but these defects may be revealed by the right piece of music. Running these tests on music would be a long and arduous task. Inaudibility on a few music test tracks does not prove inaudibility. Instead, you need to find a music track that proves audibility.

You can never prove inaudibility using music tracks. You can only prove audibility.

The fact that audibility with a pure tone was confirmed is enough to suggest that we should avoid this particular distortion problem.

We should avoid all distortion problems that exceed audibility thresholds. And frequency response problems that exceed audibility thresholds.

If it can be heard with a pure tone, there is probably some piece of music somewhere that will reveal the same defect. After all, a musical recording could be a series of pure tones.

Honestly, and I mean this: big deal. "Probably"? "Some piece"? "Somewhere"? "Could be"? I am reminded of the mp3 at 320 kbps audibility debate: yes, there will be the odd moment of music, probably, somewhere, could be, that will be audibly different to PCM in an ABX...but too many people claim that it sounds audibly inferior all the time, and cite some tone test as their evidence.

John, I am not sure that you entered the discussion with the correct understanding of what the core misunderstandings are that need repeated correction in threads like this. For example:-

- Nobody here thinks that all amplifiers sound the same. So, there is no need to provide evidence that some sound detectably different. It is accepted that some sound different, because their response or distortion levels exceed audibility thresholds.

- Some people here think that amplifiers with excellent measurements sound significantly different. You entered this discussion in reply to a new member making his third post. His first post was largely a claim that class A amplifiers sound audibly superior in the bass and the treble to all other amplifier classes, and his evidence was his sighted listening to music. His second post was that one watt does not equal one watt, citing Pass amps, clearly meaning class A amps sound better 'at one watt' than class AB and class D, and do so in sighted listening to music. His third post said that (his) sighted listening perceptions are important because there are things that are immeasurable (and sighted listening will detect these). He added that Class D gives him a headache after 30 minutes of use because of the harshness of the treble, and that Class AB has the same issue of harshness in the treble but to a lesser degree. When he was challenged on this third post with a request for evidence, that's when you chipped in and said you have the evidence. If you read what you were responding to, it was a request for evidence that immeasurable things are audible in sighted listening to music.

 

[krabapple]

Indeed, John Siau quoted this upon entering the thread:

Amplifier sound quality, and perceptions of are quite valid.

Good. Let's see your evidence. Not assertions and handwaves. Evidence.

But this is the full quote plus a few boldings on my part:

kawzx7:
Amplifier sound quality, and perceptions of are quite valid. I have worked in Quality Control for many years and sometimes you don’t always go by the spec sheet, because sometimes there are other contributors that affect the final output and are sometimes immeasurable. I can tell you this – I do not actively listen for faults. I have absolutely an open mind. I just listen to the music. Class D gave me a headache after 30 minutes of use because of the harshness of the treble. Class AB was more pleasant, but again still gave me the harshness over that I did not desire in my music. Pass labs, who I do not work for or have any relationship with besides being a repeat customer, allow me to listen to my music at the volume levels I desire with no headache or other discomfort - that’s all I can say. Numbers and amplifiers are just numbers. I would need 1000 W class D for subwoofers in my Mustang. But I would never use it for the mains. I’ve heard about class D and all of these different innovations to make it sound better but class D is what it is and that’s why these innovations are necessary.

John Siau, are you saying your data supports kawzx7's claims?
Because I don't see that.
He's condemning an entire class of amps based on listening to music for ~30 min, and claiming the reason is 'immeasurable'.

 

[krabapple]

Audio is always subjective. None of our systems are the same so we have no control group. I know of no one that listens to their music, staring at an oscilloscope.

Our reponse to audio is always subjective -- that is, personal. But conclusions we draw from it aren't necessarily correct. They don't necessarily reflect the objective truth of the matter. It's easy to 'hear' differences that aren't real.

A headache doesn't demonstrate that class D and AB amps intrinsically sound different from Pass amps. You'd have to do a lot more work than you did, to prove that in any scientifically convincing way. The first steps are careful level matching + blind comparison, making sure that the only thing being compared, is the sound of the amps. Or you could start from comprehensive measurements, whose differences might show something predictably audible.

There's simply no good evidence that there's anything 'immeasurable' about amps.

And cars are a lame and tired analogy. We are talking about hearing an amplified signal. Not the multisensorial experience of driving a car.

 

[restorer-john]

Benchmark is in business to sell product and by all accounts, their amplifier is an excellent piece of equipment.

And marketing "papers" in and around HiFi have always been to find or create a "serious" problem with a class of products, blow it out of all reasonable proportion, and then present a unique and "patented" solution along with showing a bunch of oscillograms, men in white lab coats looking serious, plots and maybe some proof in the form of blind/abx tests done with "experts" etc. Job done!

Benchmark followed this playbook precisely. Thing is, way back when this was reviewed here, those of us who look at amplified distortion residuals in class AB amplifiers knew it was very fishy, bordering on deceptive. The standard output of my ancient THD analyzer is a notched and amplified 60dB (1000X) 1V FSD output, specifically for that same purpose. The residual display on my newer analyzer is in sotware, not unlike the AP's residual display.

I've never seen residual plots as bad, and I've looked at a lot. Even when practically shutting off the bias (deliberately misadjusting) you can't get the type of gross crossover distortion Benchmark showed. You'd need a purposely badly designed amplifier or a seriously faulty amplifier with some odd characteristics to produce that type of residual.

John said this:

It is hard to know if this is unusual, especially in lower cost class AB power amplifiers. The RA-500 was a low-cost "studio reference" class AB power amplifier.

I would have expected considerable further investigation would have been warranted before using just one "edge case" or "faulty" design amplifier (the RA500) as the basis for Benchmark's miraculous new solution to a problem (crossover distortion) which was solved by every other competent amplifier designer in the early 1970s.

The article/paper has probably sold a bunch of amplifiers and that's good for Benchmark. But it wasn't good for their hard earned reputation as being technically sound, somewhat unbiased (pun intended) and honest.

 

[pma]

Well said, John.

 

[Jimi Floyd]

I have a DIY DAC built around the TDA1543, just for having the "privilege" of hearing what the fuzz is all about without spending what they ask for it and... It doesn't only measure poorly, It sounds veiled and rolled off. Maybe somebody would call that sound warm or analog-like but I don't agree at all.

 

[antcollinet]

They may measure poorly on paper but good measurements don't necessarily mean better sound,

That is one of the many audiophile myths pushed on you by the audio industry in the interest of selling you poorly performing kit at ridiculous prices.

The problem with the sort of reviews you are listening/reading to is twofold.

First - most of them have a financial interest. They either rely on manufacturers to send them gear they can review - which dry up if the reviews are bad. Or worse, they are reliant on selling advertising - which drys up when the reviews are bad. There is a reason why (with this type of reviewer) high cost manufacturers with high ad spend rarely get a bad review.

Second - even if they are as honest as the day is long, they are using sighted listening to evaluate the gear. This is known to be unreliable (at best). Our auditory system is subject to perceptive biases (we are all subject to this and it is not possible to turn it off - even when you know it is happening) that will alter the sound we perceive even when there is nothing in the sound waves reaching our ears to account for this. So any particular person's perception of the sound quality is relevant only to them : it is likely that anyone else will not perceive the sound in the same way. This is particularly true with well designed accurate/transparent gear - where the imperfections are so small, they fall below the level of human auditory capability - they literally cannot be heard by anyone and the device is audibly perfect. This applies to nearly all modern DACS, including your SMSL.

Of course - it is possible for people to "prefer" badly distorted sound - or at least some people claim this. In which case they might not like accurate, well measuring gear. But this has nothing to do with "better sound", or with HIFI (High fidelity)

 

[antcollinet]

That suggests to my tiny brain that it's not all about the chip,

No, it is not alll about the chip. The analogue electronics designed well to reduce noise and distortion, and good PCB layout to prevent noise bypassing the electronics are more important than the chip (Especially when we are talking SOTA Delta SIgma type chips - which I suspect we are not, given the description you posted above from AN)

The problem is much of the stuff they have left out (noise shaping, post chip analogue filter) are a vital part of the digital to analogue process. Without them you get nothing like a "master tape" analogue waveform, and instead get a mess of noise and distortion.

It is an indication of how poor our auditory system is that even though they are doing this, we can still think

None of the AN gear I have sounds warm, woolly, or harmonically distorted to give a romantic sound - to me it sounds natural and faithful to the original recordings.

This is not a reflection on you. We vastly over estimate our ability to hear imperfections in audio (noise/distortion). Most people would not be able to hear the result of these design choices. This is why they get away with it.

 

[Jimbob54]

More measurements vs ears merged in.

 

[Purité Audio]

It is perfectly fine to enjoy lo-fi, as long as you realise it is just distortion you are enjoying, expensive distortion at that.
Keith

 

[antcollinet]

I think you’re making incorrect assumptions about me in a rather patronising and arrogant way, or at least that’s how I perceive it.

I'm sorry - It is not intended that way.

Most of us here have fallen for the industry propaganda in the past. It is difficult to avoid doing when everything you see and read says the same stuff. It takes quite a lot of "re-programming" to break out of that mindset, and there are not many places you can get that. This is one of them.

Pointing out that this can happen is not a putdown. We have all previously fallen for it ourselves. However it is not so much an assumption - your statement I responded to is one of the results of that propaganda.

 

[Mart68]

"all products in the Audio Note DAC range have no oversampling, no jitter reduction, no noise shaping and no re-clocking. Having removed all the digital filtering that is part of the oversampling, we have also dispensed with all filtering in the analogue domain to further retain good wide band phase-frequency and dynamically coherent behaviour. The end result being a reproduction more reminiscent of master tape in quality, with greater differentiation and contrast between different recordings."

My issue with Audionote is that - IME - their systems do precisely the opposite of this, and suppress the differences in recordings. How they can have the front to make this claim given their wilfully, grossly inaccurate loudspeakers is quite astonishing.

if you like Audionote buy it - I can see/hear the appeal even if it's not for me - but if you want accuracy to the master tape, it's the diametric opposite.