In DAC, Anything audible but unable to be measured(so far)?

[Blumlein 88]

You are correct. However, once again your defense is to indicate correlation to other reviews and your own testing results. That is the exact opposite of the circumstance I described - i.e. one in which there is no other data available. In each specified example you indicate multiple samples are involved.

I'll just give up already. Single unit sample measurements are obviously without reproach here... even in cases where Amir retests and subsequently updates his measurements or methodologies after some feedback. I believe Amir is both competent and highly intelligent... but I don't believe that necessarily translates to infallible or omniscient.

I'm starting to see some of the cult here after all I think.


Right back at ya.

There is no cult of Amir. There is no declaration that all examples tested are perfect or that there can be no defective units. To then take the leap of logic and say these tests by anyone are no better than anecdotal reports is where the problem comes in.

Have you ever worked in a testing lab? I have. You have tests and proper procedures based upon the physics involved. Your testing gear is periodically calibrated with known samples or via physical constants. As long as it is calibrated, working and you follow procedure your results are good. Even for testing of one thing. So Amir's methods are good.

Now given modern manufacturing and QC, only rarely is one sample far off from the normal were you to test 100 of them. Given that knowledge base only if results are strange or the maker knows they are far off and submit other samples do you have much reason to worry. The testing is good and reliable. The products usually are very consistent. If you don't get any red flags, these kinds of test results are far more solid than anecdotal would describe.

Anecdotal: not necessarily true or reliable, because based on personal accounts rather than facts or research.

These tests done here are based upon facts and research. And some of the facts are that modern manufactured electronic goods have very, very, very small variances. So this isn't about Amir.

Do you honestly feel you can put very little faith in the results here because only one unit is tested? Would you need 5 or 30 or 50?

 

[digicidal]

We are not claiming nor is it our mission to invent new science , amir just tests hifi gear.

It's all well documented, reliable and repeatable if one has the equipment.

If one thinks its value is so limited for whatever reason they best seek other indicators of what might suit them for a bit of hifi. In a ideal world all manufacturers would provide this data and then it could be cross checked by individuals in the field.

As a aside , my brother in law was recently convicted of driving while unfit. Convicted on a pee sample take in non sterile surroundings and tested by just one lab... The horror!

That was good enough for the court, as is VADA testing in sport.

I guess my Bother in laws legal council missed a trick by not arguing the illegitimacy of a single sample size.

Yes, it's quite possible. In either case.

Obviously I am horrible at communicating what I mean - and for that I apologize for all the ruffled feathers. I never stated (or at least didn't intend to) that there was no value in measurements. Even where there is only a single sample, it is still meaningful - just less so in cases where there is no similar test data to confirm the results.

It could be a failure in QC, or it could simply be damage in shipping, or it could be a failure due to misuse by the 'donor' of the sample (end user loan). In the cases where a manufacturer provides measurements, or where others have replicated the results - the point is moot because there is another sample.

Of course measurements are meaningful - but when they don't agree with the "known" in a given case, a determination must be made as to where in the process the failure exists. Naturally, it's much more rational to suspect the sample than the methodology, lab equipment, or the operator. However, even in that there is the "faith" that no mistakes were made accidentally.

 

[solderdude]

Can measurements tell you of bass is more weighty, soundstage is wider or imaging is sharper?
I believe different circuit topology, proprietary technology, fabrication processes, composition, chips use can make for different flavors of sound.

Is it measurable? Not sure but I think the human ear can discern much more than a measurement rig.

The properties you speak of are all transducer related. This means also room related.

You may believe it has something to do circuit topology, proprietary technology, fabrication processes, composition, chips used for different flavors of sound but this sadly can not be proven for electronics in blind tests unless there is some purposely applied alteration of the signal.
This is very measurable. These magical properties (incl PRaT) magically disappear when tested properly (not knowing what is playing and level matched)

 

[solderdude]

Of course measurements are meaningful - but when they don't agree with the "known" in a given case, a determination must be made as to where in the process the failure exists. Naturally, it's much more rational to suspect the sample than the methodology, lab equipment, or the operator. However, even in that there is the "faith" that no mistakes were made accidentally.

Agreed, what I have noticed in Amirs measurements is that sometimes when measurements are way off from what is expected performance, Amir always checks his data and contacts the manufacturer or person who sent it in and mentions it.
When it measures well one could expect a cherry picked model but even those will still be close to the others.
It is possible that some equipment works well, or horrible during tests but have issues for some users or work perfectly in their cases.

For electronics it either works well or something is quite wrong.
For transducers (speakers, headphones, microphones, cartridges.. whatever) it certainly is another story and there could well be products that measure quite differently between serial numbers.

 

[digicidal]

There is no cult of Amir. There is no declaration that all examples tested are perfect or that there can be no defective units. To then take the leap of logic and say these tests by anyone are no better than anecdotal reports is where the problem comes in.

Have you ever worked in a testing lab? I have. You have tests and proper procedures based upon the physics involved. Your testing gear is periodically calibrated with known samples or via physical constants. As long as it is calibrated, working and you follow procedure your results are good. Even for testing of one thing. So Amir's methods are good.

Now given modern manufacturing and QC, only rarely is one sample far off from the normal were you to test 100 of them. Given that knowledge base only if results are strange or the maker knows they are far off and submit other samples do you have much reason to worry. The testing is good and reliable. The products usually are very consistent. If you don't get any red flags, these kinds of test results are far more solid than anecdotal would describe.

Anecdotal: not necessarily true or reliable, because based on personal accounts rather than facts or research.

These tests done here are based upon facts and research. And some of the facts are that modern manufactured electronic goods have very, very, very small variances. So this isn't about Amir.

Do you honestly feel you can put very little faith in the results here because only one unit is tested? Would you need 5 or 30 or 50?

Again, not contesting his methodologies at all, nor his equipment, nor his competence at operating it. The only point I was trying to elucidate (admittedly horribly) was what I highlighted in your post. That inconsistencies require broader samples in order to obtain the most meaningful results.

That was exactly what I said... when a well engineered product measures well, there is no reason to doubt or suspect. However, when something aberrant occurs - a larger sample size is necessary in order to determine exactly where the failure is. Just as the example you provide with QC as well as @SIY stating anything less than 100% QC was essentially "poor manufacturing" - the sample size is the most important aspect.

My snide remark was simply that. I merely used the terminology of the linked post, in reference to the presumption of infallibility that was seemingly conferred IMO. Admittedly that was a poor choice.

 

[Thomas savage]

Yes, it's quite possible. In either case.

Obviously I am horrible at communicating what I mean - and for that I apologize for all the ruffled feathers. I never stated (or at least didn't intend to) that there was no value in measurements. Even where there is only a single sample, it is still meaningful - just less so in cases where there is no similar test data to confirm the results.

It could be a failure in QC, or it could simply be damage in shipping, or it could be a failure due to misuse by the 'donor' of the sample (end user loan). In the cases where a manufacturer provides measurements, or where others have replicated the results - the point is moot because there is another sample.

Of course measurements are meaningful - but when they don't agree with the "known" in a given case, a determination must be made as to where in the process the failure exists. Naturally, it's much more rational to suspect the sample than the methodology, lab equipment, or the operator. However, even in that there is the "faith" that no mistakes were made accidentally.

This is why ' results' get scrutinised, here that's done in the threads via the membership and on occasion input from the manufacturers who I'm sure amir also often liaises with behind the scenes to make sure the results are accurate as possible.

It's all transparent, it is as bulletproof is it can be . Sure you can pick holes in it , you can pick holes in anything.

Equipment testing that gets done here is a indicator of performance. It's a point of information, of data, for folks to consider. No one here thinks it's more than that.

 

[Blumlein 88]

Agreed, what I have noticed in Amirs measurements is that sometimes when measurements are way off from what is expected performance, Amir always checks his data and contacts the manufacturer or person who sent it in and mentions it.
When it measures well one could expect a cherry picked model but even those will still be close to the others.
It is possible that some equipment works well, or horrible during tests but have issues for some users or work perfectly in their cases.

For electronics it either works well or something is quite wrong.
For transducers (speakers, headphones, microphones, cartridges.. whatever) it certainly is another story and there could well be products that measure quite differently between serial numbers.

Even microphones are pretty good. Here are three Shure KSM 32's made over more than a decade (I think it is 12 years). Most of the difference is likely me positioning them relative to the speaker source. 1/12 th smoothing I think it was. I've similar tests of half a dozen other condenser mics. Pretty much the same kind of results.

 

[digicidal]

This is why ' results' get scrutinised, here that's done in the threads via the membership and on occasion input from the manufacturers who I'm sure amir also often liaises with behind the scenes to make sure the results are accurate as possible.

It's all transparent, it is as bulletproof is it can be . Sure you can pick holes in it , you can pick holes in anything.

Equipment testing that gets done here is a indicator of performance. It's a point of information, of data, for folks to consider. No one here thinks it's more than that.

I am extremely appreciative of that fact. I know Amir does a ton of work (most poorly compensated at that) on the testing, guides, etc. That's what makes this forum different from most, if not all, of the others. However, there have already been a couple cases in the relatively small number of products reviewed here where a secondary sample generated different results, or where the manufacturer data disagreed by some margin - hence the Benchmark example. Not saying that reflects on Amir or the testing methodology - other than positively, in that they were conducted professionally for our benefit. The manufacturer did not (thus far at least) provide a second sample - and in the absence of additional data - it's impossible to determine exactly why there was a variance between their measurements and Amir's.

 

[amirm]

It could be a failure in QC, or it could simply be damage in shipping, or it could be a failure due to misuse by the 'donor' of the sample (end user loan).

We have a process for this. It involves the manufacturer providing alternative measurements that show different results. When I tested the Benchmark DAC3, I found a small issue regarding jitter. Benchmark stepped in and provided measurements that didn't show this problem. They however used a different analyzer. So they sent me another unit and I tested that. The problem was not there. So there was something peculiar about the original sample.

More commonly we face consumers objecting to data but manufacturer not provide any back up for them. In that case, the objections are without merit. The data we have is the data we have. If the manufacturer is unwilling to send me another unit or run their own test, then that is that.

We are not here after perfection. We are here to go from zero reliable data to a good volume that can be the basis for a judgement on a product. We are improving the situation immensely that way without achieving perfection.

Indeed, perfection is something I avoid. The fastest way to confuse people is to pile on graph after graph. Measurement after measurement. Think of me as a mechanic coming with you to look over a used car you want to buy. His expert opinion will be of a lot of help for you, even if it doesn't achieve perfection.

 

[March Audio]

I don't worry "too much" about any of it - however your statement makes my point. Sample size is too small to be meaningful.

I'm not saying that I believe anything deceptive or incompetent has been published (and certainly not deliberately as many of the SBAF users implied). Conversely, I feel a great effort has been spent on providing transparency where possible. However, statistically it's only slightly more meaningful than no confirmation at all, surely you admit that?

Actually no.

I think you need to come up with some reason why/evidence that a DUT is going to vary in a significant way from sample to sample. Its not my experience of modern electronics that it does.

However if something suspicious/unusual is shown in measurements Amir will seek the input of the manufacturer.

 

[amirm]

Of course measurements are meaningful - but when they don't agree with the "known" in a given case, a determination must be made as to where in the process the failure exists.

This "failure" thing seems a lot more probable to people than reality. I have repaired tons of audio electronics. I can smell a broken device from a mile away. Seemingly people think if something doesn't measure, then there is good likelihood of it being broken. I keep correcting this misconception. After testing near 200 products, we have yet to have a single bad measurement being explainable by a device being broken.

You don't avoid driving your car because you assume you are going to have an accident that day. The probability exists that you might but it is small enough to not be a consideration. By the same token, you need to trust my experience that when I don't say a device is broken, that it is working as designed.

Let's remember that these devices are stereo so the odds of something to break the same way in both channels is very, very low.

I have also repeated measurements of a number of devices. I tested the Schiit Yggdrasil three times for example, using three different samples. I have measured two samples of RME ADI-2 DAC. I have a repeat sample of a Chord DAC coming out shortly. None of these repeat tests have shown differences of note.

I have also analyzed tests run by others and shown them to correlate with mine even though the same and test gear uses are different.

Also, where it exists, I show the manufacturer spec/measurement on my graph. Often these are close, or I explain why they are not (because of different test conditions).

So really, there are a lot of checks and balances here and a strong history of testing to back the practice.

 

[Thomas savage]

We are not here after perfection. We are here to go from zero reliable data to a good volume that can be the basis for a judgement on a product. We are improving the situation immensely that way without achieving perfection.

This is how it is, we all accept this.

 

[Thomas savage]

Nothing like clearing up a argument with a car analogy

These must of become ' un-banned' while I was convalescing.

 

[onion]

Can measurements tell you of bass is more weighty, soundstage is wider or imaging is sharper?
I believe different circuit topology, proprietary technology, fabrication processes, composition, chips use can make for different flavors of sound.

Is it measurable? Not sure but I think the human ear can discern much more than a measurement rig.

There are measurements. And there are solutions. It's just it's not common.
It's all inside the impulse response. It's just we don't know how to process the impulse response the way our brain does but the impulse response is there. And we can use it.
About the recording, tho Paul in psaudio is not well regarded here and he does believe something that isn't necessarily true but according to him all the information prior to the microphone are already recorded including the spatial information. Then rotate the microphones and you get speakers to reproduce what's recorded. In good systems there are depth and good imaging.
And a different solution is to use dummy head and earphones. The results are really good. Just no one wants to do it or it's not beneficial in general.
There is difference between interpretation and measurement. We can already measure everything. Issue is do we really do that or how to interpret. There are many many different tests. Do we do them all? Where is the point that no more measurements are necessary. That's the question.

This goes back to the original question. My thinking is that it must be measurable at the ears if humans can discern a consistent difference. I'm guessing a lot of people on this thread may believe that the measurements as they are (THD, phase and amplitude response etc) already give us all the information we need. I'm guessing they are correct, but it is those measurements taken at the left ear and right ear and their difference; and the differences between those measurements for multiple sound emitters playing live and a stereo playback of those multiple sound emitters; that determine the limit of how well one can hear sound imaging in 3d-space from stereo playback. This does depend on everything in the signal chain including DAC, amp, speakers and room; but each of those factors can be standardised to allow comparative measurements of any individual component.

 

[digicidal]

We have a process for this. It involves the manufacturer providing alternative measurements that show different results. When I tested the Benchmark DAC3, I found a small issue regarding jitter. Benchmark stepped in and provided measurements that didn't show this problem. They however used a different analyzer. So they sent me another unit and I tested that. The problem was not there. So there was something peculiar about the original sample.

More commonly we face consumers objecting to data but manufacturer not provide any back up for them. In that case, the objections are without merit. The data we have is the data we have. If the manufacturer is unwilling to send me another unit or run their own test, then that is that.

We are not here after perfection. We are here to go from zero reliable data to a good volume that can be the basis for a judgement on a product. We are improving the situation immensely that way without achieving perfection.

Indeed, perfection is something I avoid. The fastest way to confuse people is to pile on graph after graph. Measurement after measurement. Think of me as a mechanic coming with you to look over a used car you want to buy. His expert opinion will be of a lot of help for you, even if it doesn't achieve perfection.

Exactly. And to both your credit (and Benchmark's engineering) when those measurements are so amazing that they expose a previously inconsequential variable in another area - referring to the amp test in this case - you proactively search for the most optimal configuration and run the tests again. Again, I apologize for the confusion my choice of words caused - I was not critical of the testing nor reviewing proceedures.

@amirm You are the last to simply accept a result and move on if something doesn't agree with expectations or published specifications - that is apparent in all the reviews I've read here.

@onion - I'd say the current measurements (in almost every case) exceed what we can hear by a fairly significant margin. With most of them we can even reasonably predict what most people perceive as a "good" difference vs. a "bad" one. Whether or not that makes us any closer to truly being able to say what a specific person's response will be is much more debatable. There are many people that find accuracy "sterile and grainy" and some noise levels "warm and textural".

 

[LaLaLard]

The issue is that live, the vocals and instruments are individual sound emitters in 3d-space. When they are recorded and played back through an audio device, the sound emitters are the pair of loudspeakers. This is not the same; it is a transformation of the rich spatial information of the live performance into a stereo representation of that. While a DAC can 100% replicate the recording, it cannot replicate the spatial resolution of the live performance

When you write of phase and amplitude response over the audible range, human ears (lt and rt) process that using the brain. The relative differences in phase and amplitude for the same sound emitter being heard by each ear at the same moment - allows one to perceive the 3d-space within which the sound generates. The result is qualitatively different for a live performance with multiple sound emitters, compared to a stereo representation of that live performance via a recording.

Many high-end DACs market themselves as being able to replicate this spatial resolution with high fidelity. I'm not convinced - I think loudspeakers and room treatments have a role to play, but they are not the full picture.

This is some argument that I can see where you're coming from. However don't forget human being only have one pair of ears --- not several dozens of them floating in the 3d dimension.

How we are able to identify the source of the sound is because of our brain. Our brain picks up the distortion of source of sound and picks up the difference of sound signal from two ears and tells us: you are hearing the sound from coordinate x,y,z.

This does not mean we received a 3d-signal. We merely heard two sound signals, one in our left ear and one in the right ear. No matter how complicated the sound was made, it all came down to, WHAT YOUR LEFT EARDRUM AND RIGHT EARDRUM HAS BEEN VIBRATED RESPECTIVELY.

Understanding this, you won't have concern about loudspeakers trying to reproduce 3d-image. Because they don't have to. THEORETICALLY a pair of perfect loudspeakers are able to 100% reproduce the sound you hear when you are sitting at exactly the right place, and if technology allows eventually.

Before that, we can get to as close to that as possible.

 

[Wombat]

Ho-hum. 4 pages on an inconsequential subjective topic. Seriously, folks.

We need some 'stickies' to refer to, or referring to existing ones to avoid these repetitive topics.

 

[solderdude]

if humans can discern a consistent difference.

When humans can discern a consistent difference under properly conducted 'blind' conditions then yes... In that case they usually show differences that are very measurable yet.

The problem is that most 'reviewers' that hear a consistent difference do so under sighted and often not properly level matched conditions (when possible to do so).
Even here we can see measurements that differ yet are below the borders of audibility.

I have not seen reliable tests that show no measurable differences yet give clearly audible (under proper test conditions) results.
In fact I have 'tested' a few persons by just pretending to swap something or flip a switch (which did nothing) and merely telling people what to listen for. That works like a charm.. nothing has changed. This only works on other people.

This is VERY easy to do yourself... try it and see what people think they can hear but nothing changed in reality. It's how the brain works and the best demonstration of falibility of 'sighted' testing.

You really need to differentiate between sighted listening tests and properly conducted 'blind' tests.
Do note that it is easy to f-up blind tests as well and can give false positives.

 

[Blumlein 88]

This goes back to the original question. My thinking is that it must be measurable at the ears if humans can discern a consistent difference. I'm guessing a lot of people on this thread may believe that the measurements as they are (THD, phase and amplitude response etc) already give us all the information we need. I'm guessing they are correct, but it is those measurements taken at the left ear and right ear and their difference; and the differences between those measurements for multiple sound emitters playing live and a stereo playback of those multiple sound emitters; that determine the limit of how well one can hear sound imaging in 3d-space from stereo playback. This does depend on everything in the signal chain including DAC, amp, speakers and room; but each of those factors can be standardised to allow comparative measurements of any individual component.

The thing is for something to measure differently at the ears, if all you changed was the DAC, it had to measure differently at the output of the DAC. It had to measure differently at the output of the amp. It had to measure differently at the speaker as it went to your ears. We don't have to measure past the DAC.

So if we measure the DAC, and find very nearly identical performance, and yet someone says, "this one has a wider soundstage than the other one", you first need to see if that person is really hearing an identifiable difference. Or just thinking so for other reasons. Most of the time the answer is there is no such difference.

Sound stage width is channel difference in intensity and phase. That is all it can be. We can measure those at the DAC very, very precisely.

 

[Sergei]

This is some argument that I can see where you're coming from. However don't forget human being only have one pair of ears --- not several dozens of them floating in the 3d dimension.

How we are able to identify the source of the sound is because of our brain. Our brain picks up the distortion of source of sound and picks up the difference of sound signal from two ears and tells us: you are hearing the sound from coordinate x,y,z.

This does not mean we received a 3d-signal. We merely heard two sound signals, one in our left ear and one in the right ear. No matter how complicated the sound was made, it all came down to, WHAT YOUR LEFT EARDRUM AND RIGHT EARDRUM HAS BEEN VIBRATED RESPECTIVELY.

Understanding this, you won't have concern about loudspeakers trying to reproduce 3d-image. Because they don't have to. THEORETICALLY a pair of perfect loudspeakers are able to 100% reproduce the sound you hear when you are sitting at exactly the right place, and if technology allows eventually.

Before that, we can get to as close to that as possible.

Hmm, it isn't that simple. A human does have a pair of ears floating in 3d. As I understood the original point: why are we so worried about 0.0001% THD vs 0.0002% THD of a DAC, when we can easily tell, without any double blind testing, that no matter the DAC, the music played through the audio delivery system used in the DAC testing doesn't sound like real music performance at all?

Consider the following classes of audio delivery systems:
(A) Stereo
(B) Multi-channel (e.g 5.1, 7.2, 22.2)
(C) Binaural with generic Head-Related Transfer Function
(D) Binaural with personalized Head-Related Transfer Function
(E) Binaural with personalized Head-Related Transfer Function and Head Tracking

Maybe I'm extending the original point here, or maybe that's what the OP meant: Wouldn't it make even more sense to test the DACs with the (B) through (E) systems, in addition to (A)?

Another way to ask the question: Perhaps the relative crudeness of (A) overrides the fine effects caused by DAC differences that could be otherwise noticed on (B) through (E)? For instance, hypothetically: the listening room's reflections, which in (A) are layered over, let's say, concert hall reflections, may dwarf the subtle differences in echoes reproduction that could - hypothetically - be heard even with (C), more definitively with (D), and beyond doubt with (E).