Master Thread: Are measurements Everything or Nothing?

[mctron]

If the engineer gets the product to perform in a way which really pleases him or her, they have the choice of reducing the perofrmance for the sake of pleasing the spec heads, or leaving the mearurements where they are to please those who want the product to perform like a champ.

You must be trolling

 

[GXAlan]

Experiment
Go ahead and try a UMIK-1 or other calibrated microphone. Record your own voice and play it back on your best setup.

Unless you're a voice actor, the way your recorded voice sounds to you probably doesn't sound the way it's being played back. This is the mirage that @Jim Taylor talks about -- we want to sound better/cooler than we really do. You can add a little bit of bass to your bass or even a bit of DRC and then it sounds better.

That doesn't happen all the time. Despite that, some people want it to happen all the time. Some people want to be impressed all the time. They want a magic box that turns garbage into gold, that makes what is naturally mundane into an exciting and visceral experience... they understand it for what it is: a mirage, an attractive lie. And they don't evangelize their personal choice to other people. They don't claim it's "better". They simply enjoy it for what it is.
...
Or they could get a neutral system and face the facts: some recordings are garbage.

Why not have both? No one ever made the rule that you only get one speaker/sound system.

 

[Waxx]

Measurements matter to engineers, and for good reason. Consumers are not so good at explaining their fixation with measurements, when what matters most is sonic performance - if indeed how something sounds means anything to the consumer.
When I started to reaize a product which hasn't measured well could perform impressively, I became skeptical about what the measurements were really telling us about performance.
If the engineer gets the product to perform in a way which really pleases him or her, they have the choice of reducing the perofrmance for the sake of pleasing the spec heads, or leaving the mearurements where they are to please those who want the product to perform like a champ.
If my system - which measures really well - doesn't impress me, I would be a fool to think it's a good system because I am safe in the knowledge it measures well.

Measurements should tell what the gear does. That not all consumers understand it is normal, not everyboyd is technical minded. But that gear is measured and tested and compared to the claims of the company is good because to much of those specs are made up by the marketing department to sell their gear. And the ranking of the gear in one or another way can help consumers to choose the right gear for their preference.

And yes, i understand the taste for coloured gear. I like coloured gear myself, and use it. But even that can be measured (THD, what harmonics, and how loud they go, the custom curves of speakers, ...). Measurements can show you that also. It's not the focus of this site, as most here prefer very neutral sounding gear, but science and measurements are not contrary to that taste, it's just the same as with neutral gear, it can be measured and bogus claims can be unmasked. That is what happens here all the time and is why i'm here.

When Amir measured the ACA amp, i knew it would be high harmonic distortion. That is why i have it in my house and why i like it. It measures bad on the SINAD scale, but the right kind of bad to give me the sound i want. And i could see it in the graphs and measurements that Amir presented.

 

[scruffy1]

addit : this was moved from the thread about denafrip ares ii, but that was actually the point of the post, not to fan a flame war about measurable vs. subjective splendour

still, now it's over here, and i guess i can either read 200 pages of conflicting opinions, or look somewhere else

perhaps this is not the forum i am seeking? anyway, back to the original post :

been reading a lot about this technology, and watching youtube reviews by people who generally seem impressed by it

yeah, i know that readings are some indication of technology's "measure", but ears are an individual's unique portal to the brain

so when the purists are outraged that people couldn't possibly "hear" the differences, i would like them to explain how they can be so assured that their experience is the real one?

i am a would-be audiophile, and i can quite clearly hear the differences between the components that led me to start my upgrade journey

the chaintech av710 sound card that sounded way better tricked to use the wolfson dacs from the rear channels to do the main stereo out (the bass was suddenly magically tighter and with more punch)
the asus xonar upgrade to opa627's - again, mostly the bass, but also the general width of the sound stage
then i tried the burson v5i in their place, and you know what - more expensive didn't make it sound better... synergy ? artefact ? maybe- but it diminished the experience
and the onboard dac in the nad d3020 was painful in comparison to either, with the upper registers being way too sharp and fatiguing
discovered the khadas tone board, and it shat on all the previous dac solutions, for crazy cheap - i don't know about hearing the "hump" in their implementation of the ess es9038q2m, but i do know about hearing detail and separation

in the interim, i have also played with a (wolfson based) alo pan am and an akm ak4993 based stoner acoustics ud130

what i realise is that despite the clarity of the ess chip, and the perhaps for my ears "too clinical" output of the akm, that the warmth of the wolfson implementations touch my heartstrings in a way the others don't, despite their very agreeable but less appealing analogue out

tldr - if a trained musician can tell me that the tonal qualities of the denafrips are the closest to "real life" that they have heard, why isn't that so?

all the engineering analysis in the world is rather pointless when the personal experience of professional musicians' appreciation compared to actual performance consistently report that is more "life-like" regardless of measuring minuscule increased distortion - the explanation that r2r doesn't influence higher order harmonics the same as delta sigma is surely of some consequence in analogue out

real life is imperfect; the nuanced harmonics of a real performance are on their own - any reproduction by its very nature is "lossy" surely, especially if you are filtering high order harmonics to "fix" distortion?

if what you listen to is mostly recordings, your appreciation will be disproportionately tuned to hearing what you are used to, whereas for a performer, the obvious deficiencies in reproductions will be quite audible, or more likely, missing something that is reality in live performance

i'm not a sound engineer, but my ears work okay, and i know what i like

i know that the focal elears and the alo pan am with the french tubes sound the best of all; i know that the russian tubes sound better with the akg q701's on the same amp, but those tubes are not as good as the elears
i understand tubes "colour" the output - but they and the wolfson are a marriage made in heaven for my happiness

i would be really keen for someone to do a double blind test on me with a denafrips, to see if it hits the spot

personally, if it sounds like angels singing, i don't give a rat's arse about the "lesser thd %" that is below my threshold to appreciate - i just want to hear beautiful music that melts my heart


so, feel free to flame me as if i have suggested silver cables make electrons spin better, but please explain why all the best electronic analysis in the world can possibly define what others' cochlears interpret as if that extinguishes the experience

 

[BDWoody]

i would be really keen for someone to do a double blind test on me with a denafrips, to see if it hits the spot

Why not arrange it? You may learn why there is so much skepticism to claims such as yours.

so when the purists are outraged that people couldn't possibly "hear" the differences, i would like them to explain how they can be so assured that their experience is the real one?

No one is outraged by the claims made, as they are too common to be outraged by. It seems more that those making the claims are the ones who become outraged when their 'golden ear' status is challenged. It isn't that the experience being claimed isn't real, as much as it is more likely a result of not really knowing how to compare devices and drawing conclusions that aren't related to sound waves or the ear.

tldr - if a trained musician can tell me that the tonal qualities of the denafrips are the closest to "real life" that they have heard, why isn't that so?

Musicians tend to do quite poorly when their hearing is actually put to the test, funny enough. However, if your trained musician friend can tell the difference between the denafrips and another competent DAC during a properly controlled and conducted test (enough trials to be worthwhile, levels matched, double blind), that would be all the evidence needed to prove all these naysayers wrong. Thing is, nobody has actually demonstrated it yet. Claimed it, yes...over and over. This thread is testament to that, but no actual evidence.

real life is imperfect; the nuanced harmonics of a real performance are on their own - any reproduction by its very nature is "lossy" surely, especially if you are filtering high order harmonics to "fix" distortion?

I'm not sure the point you are making... No one is claiming a recording can exactly duplicate the live performance experience. But to claim that one DAC or another gets you closer to it, due to added noise or distortion is a bridge too far without evidence, when the differences are already vanishingly small.

A DAC has a simple job. To take the digital source material and turn it into analog electrical signals. It really doesn't have the ability to impart so many of these attributes readily and commonly ascribed to them. How well do you understand the measurements typically done? Often, people with the biggest issues with what measurements can't tell us are those who least understand them.

Here is a video done by our host that goes into why the typical comparison isn't worth much of anything, and how to do one properly.

 

[scruffy1]

ignoring all your other stuff (and no. i'm not going to travel to melbourne to hear one, because it's impractical), what can you tell me about harmonic reproduction in an r2r vx sigma delta ? i didn't comment on added noise, i asked about subtracted harmonics - the absence of one is not equal to the presence of another

i'm keen to learn, but not to be told that an analyser will suggest what sounds best to me personally - that's what my ears do

imperfect sound that i like is way more attractive than "0.00000000x measurements of single parameters" that is less appealing

you are dancing about architecture, i think

ps. my ears are by no means "golden", but they are better to me than yours, because they are attached to my brain, so your numbers are numbers, and my hearing is what i experience

edit : by all means set up a double blind, host my visit, and laugh when i fail miserably, and then i will eat humble pie for you
what i would possibly notice is what sounds better for my enjoyment, and from prior experiences i know that will depend on the song and the chain as an amalgam - some things sound better on a different combo, and you can't make it all sound good on one set of gear (otherwise all those people with lots of headphones are pretentious)

 

[Blumlein 88]

ignoring all your other stuff (and no. i'm not going to travel to melbourne to hear one, because it's impractical), what can you tell me about harmonic reproduction in an r2r vx sigma delta ? i didn't comment on added noise, i asked about subtracted harmonics - the absence of one is not equal to the presence of another

i'm keen to learn, but not to be told that an analyser will suggest what sounds best to me personally - that's what my ears do

imperfect sound that i like is way more attractive than "0.00000000x measurements of single parameters" that is less appealing

you are dancing about architecture, i think

ps. my ears are by no means "golden", but they are better to me than yours, because they are attached to my brain, so your numbers are numbers, and my hearing is what i experience

edit : by all means set up a double blind, host my visit, and laugh when i fail miserably, and then i will eat humble pie for you
what i would possibly notice is what sounds better for my enjoyment, and from prior experiences i know that will depend on the song and the chain as an amalgam - some things sound better on a different combo, and you can't make it all sound good on one set of gear (otherwise all those people with lots of headphones are pretentious)

I cannot travel to your location. You can try some old files where I ran a digital file thru an ADC/DAC loop 8 times to increase the difference and see how audible it was.
Did this twice with different pieces of gear. You can try them yourself. Each song snippet has two that are 8th generation copies and one unmolested original. So see if you can pick the odd man out.

8th generation ADC/DAC loop vs the original.....can you hear it?

Download this zip file. It will unzip into a folder with three 30 second snippets of a Fiona Apple song. Two are the original digital file. One is an 8th generation loop thru DAC/ADC. Pick the odd file out on the poll...

www.audiosciencereview.com

Listen and choose the 8th generation digital copy.

Each of these zip files will open into three wav files. One is labeled Reference, one is A and one is B. These are 30 second snippets of full songs. The originals were from CD so all 44.1 khz and 16 bit. Either A or B has been thru 8 generations of DA to AD copying. The other is identical...

www.audiosciencereview.com

 

[Mart68]

ignoring all your other stuff (and no. i'm not going to travel to melbourne to hear one, because it's impractical), what can you tell me about harmonic reproduction in an r2r vx sigma delta ? i didn't comment on added noise, i asked about subtracted harmonics - the absence of one is not equal to the presence of another

There are no subtracted harmonics, but just having that idea in your head can affect your perception of the sound. That's why no-one takes these claims seriously unless you have done a blind comparison. That's really trusting your ears. Even a deaf man can identify which DAC is playing when he can see it's the one that's connected up.

 

[BDWoody]

ignoring all your other stuff

Well, that's convenient isn't it? For there to be good faith exchange, ignoring all the other stuff that you can't easily answer and just making new arguments about how none of this really matters to you isn't going to be very helpful.

i'm keen to learn, but not to be told that an analyser will suggest what sounds best to me personally - that's what my ears do

I don't believe I said that. What are you keen to learn exactly? Seems you are more keen to argue.

imperfect sound that i like is way more attractive than "0.00000000x measurements of single parameters" that is less appealing

Based on what? Sighted, completely uncontrolled listening? I'm guessing you didn't watch the video...?

ps. my ears are by no means "golden", but they are better to me than yours, because they are attached to my brain, so your numbers are numbers, and my hearing is what i experience

Sure, all you need to do now is demonstrate that you can hear what no one else on the planet has ever demonstrated they can hear.

by all means set up a double blind, host my visit, and laugh when i fail miserably, and then i will eat humble pie for you

No one here cares about humble pie, most care about people wasting money on well crafted stories. You don't need to go to Melbourne, and you don't need anyone to do it for you. You can do it yourself with two of the components you already have. I am hoping you do more than just throw out arguments without actually trying to understand what is being said.

what can you tell me about harmonic reproduction in an r2r vx sigma delta ? i didn't comment on added noise, i asked about subtracted harmonics - the absence of one is not equal to the presence of another

Why would that not have more to do with the microphone and recording process, than the reproduction process? If the harmonics are in the signal, the DAC will reproduce them. If the DAC is adding them, that's distortion. I'd prefer to keep my signal as close to what is encoded in my source.

I asked before how well you understand these measurements, and you didn't answer. I'll ask again. How well do you understand these measurements? That would be a great first step, before trashing them as if they don't apply to you because your ears are different...?

perhaps this is not the forum i am seeking?

If you want clarity and honesty, this is the place. If you want to get validation as to how all these things sound so different, there are lots of places to go that will certainly reinforce that belief. It depends on whether you want to understand it or not. Understanding is the cure for the nonsense. Up to you.

 

[Robbo99999]

I cannot travel to your location. You can try some old files where I ran a digital file thru an ADC/DAC loop 8 times to increase the difference and see how audible it was.
Did this twice with different pieces of gear. You can try them yourself. Each song snippet has two that are 8th generation copies and one unmolested original. So see if you can pick the odd man out.

8th generation ADC/DAC loop vs the original.....can you hear it?

Download this zip file. It will unzip into a folder with three 30 second snippets of a Fiona Apple song. Two are the original digital file. One is an 8th generation loop thru DAC/ADC. Pick the odd file out on the poll...

www.audiosciencereview.com

Listen and choose the 8th generation digital copy.

Each of these zip files will open into three wav files. One is labeled Reference, one is A and one is B. These are 30 second snippets of full songs. The originals were from CD so all 44.1 khz and 16 bit. Either A or B has been thru 8 generations of DA to AD copying. The other is identical...

www.audiosciencereview.com

Good work! I tested myself on there, only got 2 out of 5 right, so can't say I could reliably notice a difference - I did leave with the impression that the 8x converted files sounded a bit more crispy in the highs, which was the only thing I was really latching onto when trying to compare the different files, but no real idea if this would be the discerning factor.

 

[Yoda896]

Seems like so many review threads get challenged with:

1. Measurements are not everything.

2. You all never listen.

3. I trust my ears, not graphs.

4. I don't listen to graphs. I listen to music.

5. You all must not listen to music at all.

6. Why don't you all buy the best SINAD gear?

7. I have heard your best SINAD gear and they sound terrible. I don't like any of this Chinese stuff.

8. You don't trust your ears. I/we do.

9. All these reviewers/youtubers/audophiles say these amps, DACs, etc. sound different and you say they don't. They can't all be wrong.

10. Surely designers have created certain house sound for each equipment which your measurements don't show.

11. Your measurements are only at one frequency. You need to also measure X, Y and Z like impulse response, slew rate, etc., etc.

12. You guys run a cult here where you only go by measurements and no one is allowed to disagree.

On and on...

I have had to answer these so many times that I thought it is time to stop having them go into every review as they are not product specific. From here on, any such questions should be posted here. Answers will be given in this thread and simply referenced in future challenges in other threads.

@AdamG247 and @BDWoody, please direct any future posts in review threads to here and not allow discussions there.

Thanks. You all are free to discuss this topic, provide answers, argue, whatever, in this thread.

@amirm first of all I think that your work is invaluable for the community and I want to thank you very much as it inspires me on a nearly daily basis

Being a scientist myself I strongly believe in objective measurements and limits of perception, i.e., differences that can or cannot be heard anymore. Nevertheless being a scientist I am also always worried that there might be important orthogonal directions that are projected out in the measurements that we study, i.e., are we working on understanding shadows of the reality in perfect details (cf. Plato's allegory of the cave https://en.wikipedia.org/wiki/Allegory_of_the_cave). E.g., soundstage might be an example of such a dimension that we miss with our current measurements. Even though nearly all of us will agree, that Sennheiser HD800s sound dramatically different to Focal Clear still both are excellent headphones and both measure very well. I am also not aware that it is possible to capture the specific qualities of a discrete amp or an R2R DAC with measurements and I am not even talking about tube amps.

My question @amirm is thus: does it make sense to further improve, e.g., on SINAD of an DAC/amp into a territory that we anyway will not be able to distinguish and hope that everything else is correlated and thus something that we indeed can hear is improved in parallel to SINAD. Or should we rather try to quantify other orthogonal directions of our hearing that are not correlated with SINAD (or frequency response of headphones or speakers)?

 

[Newman]

@Yoda896 The focus on SINAD that you mention is only for amplifiers and DACs, which are transparent components. They don’t have different sound stages. If you think they do, you are imagining it. This is pretty well-established.

Your last sentence gives the misleading impression that reviews here focus on SINAD for speakers and headphones too. That is not true.

 

[Killingbeans]

the explanation that r2r doesn't influence higher order harmonics the same as delta sigma is surely of some consequence in analogue out

By "explanation", do you mean that this effect has been demonstrated to be real and substantial enough to audible? Or is it just conjecture?

It honestly makes no sense. A DAC has no way to pick and choose between different parts of a signal like that. It would require DSP to do that. The D/A conversion itself is indiscriminate.

 

[Yoda896]

Sorry, I did not want to give the impression that reviews of speakers and headphones focus on SINAD. I am well aware that this is not the case. I also did not want to imply that different amps or DACs have a different soundstage (headphones clearly have different soundstage, though). AMPs and DACs should be transparent, but very often they aren't. My question is rather, if we project onto the relevant subspace with the measurements we currently look at.

Within visual perception it is very well established, that such projections can lead to optical illusions. There is no doubt, that there are similar effects in acoustics, it is just much harder to identify and quantify these effects. E.g., think of "soundstage" as a phenomenon similar to "aerial perspective". Our brain infers a 3D-effect from a 2D-projection. It took artists centuries to understand and refine this "aerial perspective". Similarly, we might not yet have understood "soundstage" well enough to measure it. And "Soundstage" is just one example of these subjective audiophile terms, which might or might not be measured given the right approach.

 

[Blumlein 88]

Sorry, I did not want to give the impression that reviews of speakers and headphones focus on SINAD. I am well aware that this is not the case. I also did not want to imply that different amps or DACs have a different soundstage (headphones clearly have different soundstage, though). AMPs and DACs should be transparent, but very often they aren't. My question is rather, if we project onto the relevant subspace with the measurements we currently look at.

Within visual perception it is very well established, that such projections can lead to optical illusions. There is no doubt, that there are similar effects in acoustics, it is just much harder to identify and quantify these effects. E.g., think of "soundstage" as a phenomenon similar to "aerial perspective". Our brain infers a 3D-effect from a 2D-projection. It took artists centuries to understand and refine this "aerial perspective". Similarly, we might not yet have understood "soundstage" well enough to measure it. And "Soundstage" is just one example of these subjective audiophile terms, which might or might not be measured given the right approach.

Even been in an empty room with no furniture or anything? Remember that echoey, reverb it has when you talk or especially clap your hands. Now add artificial reverb to some recorded sound, and voila, you have your aural illusion of soundstage. There are other effects too.

 

[Yoda896]

Even been in an empty room with no furniture or anything? Remember that echoey, reverb it has when you talk or especially clap your hands. Now add artificial reverb to some recorded sound, and voila, you have your aural illusion of soundstage. There are other effects too.

Great example!
Obviously you could measure reverb? So this would allow to change/optimise soundstage on purpose?
I agree that this is not the only one effect, but as for perspective in visual perception there seem to be more effects involved (cf. https://en.wikipedia.org/wiki/Depth_perception#Aerial_perspective).

 

[Killingbeans]

 

[Waxx]

And a dac should be neutral, not colouring anything in the sound. If i want to colour the sound, the preamp or power amp is a way better place to do that.

Chip based dacs are, to a level where even the mediocore in the rankings are so good it does not matter anymore. A ladder dac is no advantage, mostly reverse as the distortion of the discrete components is often higher than a dac in chip format (that is roughly the same in technical layout). I love discrete circuits, but when it's worse it's worse, and a ladder dac is in my opinion. The only discrete components that could be reasonable in a dac is the output and input buffer. But even there a good opamp based circuit is mostly better.

 

[VintageFlanker]

 

[fpitas]

Just because his system isn't resolving enough, and he blindly believes in measurements, he dismisses all the evidence that so many audio experts have heard.

/How am I doing?