firedog understands it quite well. That's exactly my point.
"Sampling" proves nothing, and in fact can result in incorrect/skewed conclusions. What is needed is true "random sampling" and a number of samples in the random pool that is statistically significant relative to the entire number of models of DAC in existence. That's "science". Anything else is "science like". Actual scientific testing is difficult and expensive and is one of the reasons it almost is never actually done in the audio world.
It also would be difficult to get a true random sample of the DACs on the market, as "DAC" isn't a scientifically clear concept for these purposes and it would have to be precisely defined beforehand; i.e. what qualifies as a DAC for the test.
I'm not saying anything against Amir, but his method fulfills neither requirement. So his testing is useful as an indication, but it isn't "proof" in the scientific sense. You are welcome to 100% take it as proof, if you'd like. But that's just an assumption on your part, not proof in the sense of scientific proof. You would also need to have his testing repeated and affirmed by others for it to be "proof".
Amir tested a few DACs that happen to be available to him, not a random sample. It's useful fo laymen and in practical terms is probably the only testing we are going to get - but it doesn't tell us anything conclusive about other DACs he didn't test. We actually have no way of knowing that if he tested more DACs that he would get the same results.
If you want to participate in "science" and claim scientifically accurate conclusions, then stick to scientific standards and not some armchair version. The word "science" gets thrown around here and in other audio forums in situations where it doesn't apply and by people who have no concept of what scientific testing is.
It might be helpful to look at some definitions:
Science is a
systematic discipline that builds and organises
knowledge in the form of testable
hypotheses and
predictions about the universe.
Engineering is the practice of using
natural science,
mathematics, and the
engineering design process to solve technical problems, increase efficiency and productivity, and improve systems.
Technology is the application of
conceptual knowledge to achieve practical
goals, especially in a
reproducible way.
A
review is an evaluation of a publication, product, service, or company or a
critical takeon current affairs in literature, politics or culture.
Quality (business), the
non-inferiority or
superiority of something
Quality control (
QC) is a process by which entities review the quality of all factors involved in
production.
Based on these definitions taken from Wikipedia and the point put forward by
@firedog (many years ago), this site might be more aptly named “Audio Engineering Quality Review”. The products reviewed on ASR are the result of engineers applying existing engineering principles and new technologies with new arrangements components to achieve incremental improvements in audio performance at a certain price point, as determined by well established set of measurements.
Other websites and forums are various versions of subjective listening tests done in uncontrolled environments and using an infinitesimally small set of all possible applications and combinations of components. The “measurements” in these forums are always the result of adding or replacing a new element to a “system” (power/source/amplification/transducers/room/ears/brain/physiology), and making judgements about the experienced sound quality.
I submit that neither “ASR” nor subjective reviews have that much to do with “science”. If we were focused on the “science” of audio reproduction, we would be spending more time trying to understand why there is a disconnect between the perceptions of a number of regular listeners around the effects of, say, cabling changes and the failure of current measurement techniques to discern a “measurable” difference. Or why FPGA and Delta sigma DACs that measure identically and well can sound different when placed in a “system”. Listening tests carried out with the level of control, sample size and statistical accuracy necessary to be considered valid are so complicated and expensive as to be impractical. And even if they can be carried out to a level of rigor needed as a baseline or demonstration of concept, they are hardly practical to test every possible product that might offer value but elude differentiation by SINAD.
I suggest a site called “audio science” would be focusing on areas of audio reproduction where there might be a disconnect between readily available measurement tools and some of the more esoteric designs and why those designs elicit different experiences in end users. Is the answer always “it is all in your head based on good marketing” satisfying to everyone? Lots of research was done to determine whether 16/44.1 was adequate bit depth and sampling frequency for mass distribution of digital media, and Amir has attested to doing research around compression algorithms to distribute audio online. These efforts were focused on “good enough”. Where are the research efforts looking at the bleeding edge of what’s possible to make something truly outstanding? Is there more to be measured than jitter and SINAD?
kn
