• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

The Audio Science Review Manifesto???

nintendoeats

Member
Joined
Nov 19, 2018
Messages
86
Likes
143
Location
Ottawa, ON, Canada
In the comments for the first Ciúnas Audio ISO DAC semi-review, @AndrovichIV and @LTig suggested having a FAQ or introduction to address common anti-measurement sentiments that people bring up. Since I am a professional technical writer and quite a fan of ASR, I thought I would try my hand at it. I came up with the following draft, which can of course be expanded or revised ad nauseum.

Note that I have included this in spoiler tags to make it entirely clear that it does not officially represent Audio Science Review in any way.

Hello! Welcome to Audio Science Review. This site is intended as a place to discuss audio under the umbrella of scientific rigor through measurement of equipment. Our friendly community is learning-driven; regardless of your current knowledge-level about audio science, we hope that you will find other users that are excited to teach you what they know and explore new ideas about what they don't.

As an introduction for new users, we have created this outline to help you understand the philosophy at the heart of the ASR community. If it does not align perfectly with your views, that is entirely fair; you are free to challenge it. Just be prepared to present evidence. Arguments about judging audio equipment by measurement have been going on for at least a century, and more aggressively in the last few decades. Many users come here because they have grown weary of this debate.

The basic philosophy of ASR is as follows:
  • High quality consumer audio equipment must have high fidelity. Once a recording is published, the transmission medium (radio waves, record grooves, digital files) and devices it passes through (DACs, amplifiers, cables, speakers) should change that recording as little as possible. If a device does not change the sound passing through it in a way that humans can perceive, the device is called audibly transparent.
  • The mechanics and limits of human hearing have been understood fully for decades. We therefore have a complete definition of the characteristics a device must have to be audibly transparent.
  • The only way to determine if audio equipment is audibly transparent is to take measurements. You must do this using equipment with better characteristics than the equipment you are trying to measure. With modern devices of passable quality, the human auditory system rarely qualifies.
  • Because your perception of sound changes across listening sessions, and even moment to moment, it is unreliable in identifying subtle differences in audio equipment.
  • If you know what equipment is being used in a listening test, your brain will fill in what you expect to hear. If you don’t know what is being used, your brain will just guess. If somebody tells you that they think an expensive device sounds better than a cheap one, the person probably really does hear an improvement in quality. Put both devices behind a curtain however, and the story might be very different. Because of this, repeated double-blinded tests are the only human experiments that matter (unless the difference reported is truly obvious, such as blatant distortion).
The above does not take into account personal taste. You buy audio equipment so you can enjoy music, movies, etcetera; not to get pretty graphs on an audio analyzer. In your personal audio chain, some coloration of sound might be desirable. For example, you might prefer to boost the bass, or the mids. You might even enjoy introducing distortion not present in the original recording. If you want a sound system that is not audibly transparent, it is far better to color your audio using EQ or effects that you control than to buy components with coloration permanently baked into their design. Then you can adjust or defeat the coloration based on the needs of a specific recording, or changes in your tastes.

Thank you for taking the time to read this document, and welcome to the community!

I welcome any and all feedback (including "this is a bad idea, why did you bring this up").

After some discussion, there was general agreement that some kind of FAQ that addressed common arguments would be useful. Below is my working version of something like this.

1. I hear a difference between two things, therefore they must really be different somehow.
◦ Every person who has an interest in audio has said this about some device, file, or setting at some point. Every audio engineer has adjusted the EQ on a channel until they thought it was perfect, only to realize that the channel was disabled the whole time.​
Do an ABX test with proper blinding and a good number of iterations (say 20 for a low-importance test). If you can choose correctly 75% of the time, then there is a very good chance that you really are hearing a difference. If not, remember that you are not alone and the human perceptual system likes to lie to you.​

2. Double-blind testing is unfair, because...
◦ Unblinded subjective testing of audio equipment is inherently unscientific. Here are some common endings to the sentence above, and the standard rebuttal.​
a) ...the way a device looks and feels is part of the experience.
• If the way a device looks affects the listener’s experience, then an unblinded test is by definition no longer just testing the audio quality of the device. It is also testing how it “makes you feel”. While that is interesting psychologically, it does not justify claims about how the device sounds because audio quality is no longer the only variable.​
b) ...knowing what device is being tested helps the listener know what to listen for.
• The human perceptual system makes a lot of guesses based on what a person expects to see/smell/hear. This makes it highly suggestible, and this applies particularly to hearing. If the listener is expecting to hear more detail through a particular device, they usually will; their brain can add detail that never existed. This confounds the ability of an unblinded test to objectively determine the fidelity (or even pleasing infidelity) of the device's audio output.​
3. Companies wouldn’t sell expensive equipment that makes bold claims about performance if it didn’t make a difference.
◦ As a general rule, companies are profit-motivated, not truth-motivated. They will sell anything they can convince you to buy.​
a) But company X is run by somebody I trust!
• One possibility is that they are not worthy of your trust; a good liar must appear to be speaking in earnest.​
The second possibility is that they are drinking their own kool-aid. Nobody is more motivated to perceive a quality improvement from a piece of equipment than the manufacturer. If they are not testing their equipment using proper measurements and/or double-blind studies, then they are likely to be very susceptible to the psychological effects outlined elsewhere in this FAQ. Even if they are performing scientific tests, they might not be giving the results of those tests appropriate weight.​

4. My significant other (who is not interested in audio) says that they can hear a difference when I change these “audibly transparent” devices/settings/files/cables. Therefore, they must be different.
◦ Your significant other could be picking up on social cues, rather than hearing a real improvement. It is possible that you could get the same response by rustling around behind your A/V setup for half an hour without changing anything and then asking what they thought. If they say it sounds better, that doesn’t mean they are lying; they might really hear a difference, even if the device output hasn't changed.​
Your significant other is just as susceptible to session-to-session test-variance as you are. Additionally, your personal excitement about changed might be infectious. If they know that you are interested in audio, and that you have been playing with your system, they are probably also expecting to hear a change.​

5. Expensive cables can improve a signal.
◦ Everything in your audio chain (with some specific exceptions) reduces fidelity of the signal. This includes your cables. A good cable will at best degrade the signal as little as possible and keep out external noise.​
It is possible for a cable to modify a signal in a way other than adding noise; for example, if the cable has high capacitance it will tend to lose high-frequency information. This is a reduction in fidelity, by definition. If you like that effect, you would be better served by using EQ to get the exact amount of high-end rolloff you want instead of buying an expensive cable that acts as a non-adjustable low-pass filter.​

a) Expensive power cables can improve a signal.
• The socket in your wall is connected to an electrical substation by hundreds of meters of cheap copper wire. This cable picks up RF interference, switching noise from other devices, and who knows what else. Amplifier manufacturers know this, so they put capacitors on the AC input to smooth out this noise. Even if 50 cm of additional copper wire could somehow filter this noise, it would be entirely redundant.​
The entire purpose of having capacitors in the power circuit of an audio device is to ensure that the power system can’t affect the audio signal, positively or negatively.​
b) Expensive digital cables can improve a signal.
• Because digital signals operate at very high frequencies, it is possible to see reduced jitter between cables. However, this can only go so far; the noise that can creep into a system when using even the worst digital cable is almost always inaudible. The reasoning in 5) still applies, because you cannot get a signal that is better than "correct".​
6. Analog audio formats are higher quality than digital audio formats.
◦ A record cutter receives a voltage level and etches a groove on some vinyl in a way which represents that level.​
A tape recorder receives a voltage level and aligns some iron filings on some tape in a way which represents that level.​
A digital recorder receives a voltage level and writes a number in some RAM in a way which represents that level.​
Note the constancy in those 3 scenarios: each type of recorder stores a representation of a voltage level. There is nothing else for a recorder of any kind to store. The only other things available to be recorded are accidental modifications to that voltage level by the recording process: errors such as distortion, noise, and flutter. Ergo, the only thing a particular recording medium can do is reduce the fidelity of the stored signal. That reduction in fidelity might be pleasing, but it is something that a digital file is fully capable of storing because the errors are still just changes in the recorded voltage level.​
If you record playback of a reel-to-reel tape using a decent quality digital audio interface, the resulting file will be audibly identical to the live playback. The reel-to-reel outputs a voltage level, and the audio interface records that voltage level. There is nothing else traveling down the wire to be recorded.​
7. Lots of people believe that something makes a difference in audio, therefore it must be true.
◦ Lots of people believe lots of things, and lots of those people and things are mistaken.​
Think of something lots of people believe that you think is wrong. Now imagine that you were arguing with somebody who believe this thing and they said "I know lots of people who believe this, so it must be true". It is unlikely that you would find this argument compelling.​
In the audio industry specifically, there are many people and companies with a vested interest in spreading misinformation and encouraging anti-scientific thinking (knowingly or accidentally). Once this misinformation spreads to enough people, it can become accepted fact regardless of whether it makes sense. Therefore, the fact that lots of people think that something makes a difference to audio output is not a good argument.​
 
Last edited:

RayDunzl

Grand Contributor
Central Scrutinizer
Joined
Mar 9, 2016
Messages
13,198
Likes
16,981
Location
Riverview FL
The above does not take into account personal taste, which is clearly absurd.

edit suggestion:

The above does not take into account personal taste, which is clearly absurd.

Why?

As written, quick read implies personal taste is absurd.
 

Another Bob

Member
Forum Donor
Joined
Nov 28, 2018
Messages
80
Likes
128
Location
Madison, WI
Good start.

should changes it as little as possible
Typo: "changes" should be "change".

it is far better for you to color your audio in a controlled way using EQ or effects than to buy a box and hope that it is transmitting the best audio for you. This way you can set things up exactly as you want them.
I think this could be stated more clearly. Perhaps something like:
...it is far better to color your audio using EQ or effects that you control and can adjust or defeat based on the needs of the recording or changes in your tastes, than to buy components with coloration permanently baked into their design.
 
OP
nintendoeats

nintendoeats

Member
Joined
Nov 19, 2018
Messages
86
Likes
143
Location
Ottawa, ON, Canada
Good start.

I think this could be stated more clearly. Perhaps something like:
...it is far better to color your audio using EQ or effects that you control and can adjust or defeat based on the needs of the recording or changes in your tastes, than to buy components with coloration permanently baked into their design.

I like that.

I made the change, but shuffled it around so it could be split into two cleaner sentences.
 

PaulD

Senior Member
Joined
Jul 22, 2018
Messages
453
Likes
1,340
Location
Other
I would like to see an additional part on why sighted casual listening is poor for evaluation. There are several articles and papers that could be used to back up the argument. To a novice, they change something and hear a change, the audible result in the mind is evident, but it takes reason and training to understand that it is a bias of some kind, and there are several kinds of bias at work in the hifi industry, for example; confirmation bias, appeals to authority, clustering illusion, expectation bias, dunning-kruger (ok, not really a bias) etc etc. I think the basic experience of many novices needs to be explained to them so that misinformation is not continuously transmitted, because the repetition of misinformation makes people think it is true (illusory truth).

Articles to help:
In Linear Audio there was “Testing One, Two Three” (by SY, he might even contribute it) and “Analogue Hearts, Digital Minds” by Michael Uwins (someone else recently linked to it on ASR, I cannot find it now).

Read Sean Olive’s blog “The Dishonesty of Sighted Listening” http://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html
There is also Floyd Toole’s AES paper “Hearing is Believing vs. Believing is Hearing- Blind vs. Sighted Listening Tests”
 
OP
nintendoeats

nintendoeats

Member
Joined
Nov 19, 2018
Messages
86
Likes
143
Location
Ottawa, ON, Canada
I would like to see an additional part on why sighted casual listening is poor for evaluation. There are several articles and papers that could be used to back up the argument. To a novice, they change something and hear a change, the audible result in the mind is evident, but it takes reason and training to understand that it is a bias of some kind, and there are several kinds of bias at work in the hifi industry, for example; confirmation bias, appeals to authority, clustering illusion, expectation bias, dunning-kruger (ok, not really a bias) etc etc. I think the basic experience of many novices needs to be explained to them so that misinformation is not continuously transmitted, because the repetition of misinformation makes people think it is true (illusory truth).

Articles to help:
In Linear Audio there was “Testing One, Two Three” (by SY, he might even contribute it) and “Analogue Hearts, Digital Minds” by Michael Uwins (someone else recently linked to it on ASR, I cannot find it now).

Read Sean Olive’s blog “The Dishonesty of Sighted Listening” http://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html
There is also Floyd Toole’s AES paper “Hearing is Believing vs. Believing is Hearing- Blind vs. Sighted Listening Tests”

I can see how that isn't quite covered by the existing bullets. Permit me to ponder it, and thanks for the references.
 

Hipper

Addicted to Fun and Learning
Joined
Jun 16, 2019
Messages
753
Likes
625
Location
Herts., England
In principle this seems like a good idea. Firstly though, should this discussion not be in the 'Forum Suggestions and Announcements' forum, although I would agree it might get buried there?

This site does need a FAQ or 'About' page to define what it is about. However I think your attempt is a bit long winded. I would have thought a short set of principles without explanation written in layman's language that would be easy to digest for new readers should be agreed on initially. This would be the first page that would greet new members. That would then be followed by a more in depth explanation that would show why this approach is thought necessary and how it is different from other typical audio forums. Other points about this site, such as the possibility of measuring equipment, could be added.

The core principle is surely that objective measurements are more reliable then subjective opinions when assessing audio equipment.

https://en.wikipedia.org/wiki/Audio_equipment_testing
 

BDWoody

Chief Cat Herder
Moderator
Forum Donor
Joined
Jan 9, 2019
Messages
6,948
Likes
22,625
Location
Mid-Atlantic, USA. (Maryland)
I like the idea of a FAQ, but maybe put it in the form of a list of the actual questions that are typically asked, and in a way that leads them through what would typically be the first 10 or so posts in a thread.

Something like:

"What do you mean all DAC's sound the same!"
Then under that point would be the little plus button to expand that section, In which would be the more complete explanation. You could find the best actual responses from previous threads, with a link to it so they could jump into it and follow from there if they chose.

"But I hear it with my own ears! I know what I hear!"
Same thing...find a great response to that post, quote it, link it, on to the next.

"But I'm\my buddy is a musician/recording engineer/stayed at the Holiday Inn Express last night, so I REALLY know!"

"Well what about cables?"

"Can all these people really be wrong?"

"Why do people say Schiit products are so great when all these measurements show them to be so schitty?"

Tubes!!!!

"Vinyl!!!!!"

Rinse repeat until the FAQ thought line is complete.

Also included in the drop down field could be links to a good video, and maybe a good external academic or scientific reference or two or 10 for those wanting the deeper dive...
 
OP
nintendoeats

nintendoeats

Member
Joined
Nov 19, 2018
Messages
86
Likes
143
Location
Ottawa, ON, Canada
@Hipper So you think this should be shorter, more general and with less detail?
@BDWoody So you think this should be longer, more specific and with more detail?

In my view, a true FAQ for addressing all of the possible arguments lodged against measurement-driven audio is a second step, more like what Hipper is saying. I will look at distilling what is there further, but I am concerned that reducing too much will lead to something that comes across as dogmatic rather than methodical.
 

BDWoody

Chief Cat Herder
Moderator
Forum Donor
Joined
Jan 9, 2019
Messages
6,948
Likes
22,625
Location
Mid-Atlantic, USA. (Maryland)
@Hipper So you think this should be shorter, more general and with less detail?
@BDWoody So you think this should be longer, more specific and with more detail?

In my view, a true FAQ for addressing all of the possible arguments lodged against measurement-driven audio is a second step, more like what Hipper is saying. I will look at distilling what is there further, but I am concerned that reducing too much will lead to something that comes across as dogmatic rather than methodical.

Actually, much shorter in the sense that rather than an essay, I'd propose a list of questions with answers you can choose to see if you hit the drop down. They can look at the list, pick their question and get an answer.
Isn't that what FAQ stands for?
 

BDWoody

Chief Cat Herder
Moderator
Forum Donor
Joined
Jan 9, 2019
Messages
6,948
Likes
22,625
Location
Mid-Atlantic, USA. (Maryland)
@BDWoody That FAQ style looks strangely reminiscent of a company I won't mention hahaha! I don't know if you intended that as satire, or if I'm just tired and imagining things lol.
No hidden agenda...this time...as far as you know.
 

dkinric

Addicted to Fun and Learning
Joined
Nov 21, 2018
Messages
666
Likes
1,446
Location
Virginia, USA
I like the idea of a FAQ, but maybe put it in the form of a list of the actual questions that are typically asked, and in a way that leads them through what would typically be the first 10 or so posts in a thread.

Something like:

"What do you mean all DAC's sound the same!"
Then under that point would be the little plus button to expand that section, In which would be the more complete explanation. You could find the best actual responses from previous threads, with a link to it so they could jump into it and follow from there if they chose.

"But I hear it with my own ears! I know what I hear!"
Same thing...find a great response to that post, quote it, link it, on to the next.

"But I'm\my buddy is a musician/recording engineer/stayed at the Holiday Inn Express last night, so I REALLY know!"

"Well what about cables?"

"Can all these people really be wrong?"

"Why do people say Schiit products are so great when all these measurements show them to be so schitty?"

Tubes!!!!

"Vinyl!!!!!"

Rinse repeat until the FAQ thought line is complete.

Also included in the drop down field could be links to a good video, and maybe a good external academic or scientific reference or two or 10 for those wanting the deeper dive...

The question/answer format would get a lot more reads than a Manifesto.
 

AndrovichIV

Active Member
Forum Donor
Joined
May 25, 2019
Messages
158
Likes
218
This is great! My 0.05 is that a FAQ format (a la Schiit ironically) might be more user friendly. Also, it might be useful to pepper this with some authoritative references (no need to put science lingo in there, just the reference in parentheses if anyone wants to read that)
 

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,298
Location
uk, taunton
Can't be a prope cult without a manifesto, I like the idea and most of the first part before the bullet points . A strong paragraph would carry the most power. People only read so much online .

It won't stop folks signing up just to battle. People seem to need recognition and attention and some get that from being argumentive trolls online.
 

pwjazz

Addicted to Fun and Learning
Forum Donor
Joined
Sep 21, 2018
Messages
507
Likes
746
hear what the artist intended

I would strike the notion of "as the artist intended" from the manifesto. It opens the door to a lot of argumentation about differences between the artists' listening equipment and ours, emotional/chemical state, who is the artist (writer, performer, producer, engineer, ...?) and so on. Also in practice, as I understand it, many artists intend for their stuff to sound good in a variety of contexts and, ultimately, sell well, so it's not like there's some one golden reproduction of the audio that would make an artist say "yes! that's what I intended" :)

I think we can safely define fidelity with respect to the recorded signal, but going further back in the chain than that gets tricky.
 

Ceburaska

Active Member
Forum Donor
Joined
Jan 30, 2018
Messages
250
Likes
301
Location
Gloucestershire, England
Last three lines of first paragraph, I would change order. Actually, I’d delete the last line. And have the other two swap order so it reads
Arguments against judging audio equipment by measurement have been advanced for a century, and more than ever in the last few decades. Just know that the principles outlined here are precisely what set ASR apart from many other audio websites.

In the fourth bullet point I’d rather say “people’s perceptions change” or something similar. Otherwise it might be misconstrued as saying there are changes in who is involved in listening groups.
 
Top Bottom