Master Thread: Are measurements Everything or Nothing?

[ahofer]

Raising the question of ‘beg the question’

One of the phrases most guaranteed to annoy usage traditionalists and purists is beg the question meaning raise the question or evade the question. While raise the question (or invite, elicit, prom…

stancarey.wordpress.com

So you googled, congratulations. Do you endorse this at the end?

The expression is ‘skunked’, to use Garner’s term. Grammarphobia agrees that it’s ‘virtually useless’, and Mark Liberman recommends avoiding it altogether. In formal use I advise caution for this reason, but in everyday use you’ll encounter little or no difficulty or criticism with the raise the question usage.

Eventually the contention may abate enough that the phrase can be used in edited prose without annoying, confounding, or distracting a significant number of readers. Until then each writer must decide for themself based on context, audience, and taste.

 

[birdog1960]

So you googled, congratulations. Do you endorse this at the end?

yes, I decided for myself. you do you. Should we remove modern acronyms and slang from all posts as well? I can imagine a few excellent examples.

 

[ahofer]

yes, I decided for myself. you do you. Should we remove modern acronyms and slang from all posts as well? I can imagine a few excellent examples.

Oh stop being so butt-hurt.

To return to the topic at hand, One thing I certainly did not do is pretend that information available in abundance here does not exist. That would be over the top.

 

[birdog1960]

Oh stop being so butt-hurt.

To return to the topic at hand, One thing I certainly did not do is pretend that information available in abundance here does not exist. That would be over the top.

perfect. an alternative definition of "anal" came to mind. I'll stop now.

 

[voodooless]

Yes, seems to work for me.

Here's 30 sec of left channel of "Blågutten / Quiet Winter Night / Hoff Ensemble" from 24/96 file that was available on http://www.2l.no/hires/ (sadly, not anymore):

sox "2L-087_stereo-96kHz_06.flac" left.flac trim 0 30 remix 1

Using octave we can do FFT:

[X, Fs] = audioread("left.flac");
Y = fft(X);
save left_fft.txt Y;
exit

We get a bunch of complex numbers:

]$ head left_fft.txt
# Created by Octave 5.2.0, Wed Dec 28 20:03:53 2022 CET <danadam@ubuntu>
# name: Y
# type: complex matrix
# rows: 2880000
# columns: 1
 (-331.73065268993378,0)
 (30.937272302294723,-13.218130854760776)
 (25.130550944365645,-54.867404060761906)
 (-19.95362865681485,0.96937455479392065)
 (27.960966137448651,-18.215475665983263)
 ...

Back in octave we can plot amplitude of this FFT:

load left_fft.txt
Fs = 96000;
L = length(Y);

P2 = abs(Y/L);
P1 = P2(1:L/2+1);
P1(2:end-1) = 2*P1(2:end-1);

f = Fs*(0:(L/2))/L;

figure(1)
plot(f, 20 * log(P1) / log(10) )
title('Amplitude Spectrum')
xlabel('Hz')
ylabel('dBFS')
pause

View attachment 252950

Still in octave we can do inverse FFT to recover our waveform:

load left_fft.txt
X2 = ifft(Y);
audiowrite("left_back.flac", X2, 96000, "BitsPerSample", 24);
exit

Finally, using sox we can subtract one file from the other and check peaks:

]$ sox -m -v 1 "left.flac" -v -1 "left_back.flac" "null.flac"
]$ sox null.flac -n stats
DC offset   0.000000
Min level   0.000000
Max level   0.000000
Pk lev dB       -inf
RMS lev dB      -inf
RMS Pk dB       -inf
RMS Tr dB       -inf
...

They are identical. All files available at the mega link mentioned earlier.

Not very surprising. I believe Octave uses double precision FFT calculations.

 

[tomelex]

Uh-oh. I also listen to plenty of vinyl.

I want all the wrong things, apparently.

I know your audio ideas and sources well, you caught me out on that one!

 

[pma]

Transform ==> Inverse Transform must work. If it applies to sampled time function, then the Nyquist condition must be fulfilled. That’s the key and maybe also a key to misunderstandings.

 

[IPunchCholla]

Oh stop being so butt-hurt.

To return to the topic at hand, One thing I certainly did not do is pretend that information available in abundance here does not exist. That would be over the top.

I agree it is in abundance here, it is not well cited or organized. The oft cited audibility thresholds thread relies on uncited (thankfully stated as such) numbers for much of its argument. For everything published experimentally, there is a thread following it often making substantial claims against the data as presented, or it’s implications.

As someone who has been here for about a year, and pretty actively engaged and reading most of the suggested articles as well as well as delving into the current scientific literature at points, I still get pretty lost occasionally.

My math skills are rusty, and I am not in the sciences at all. The closest training I have is archaeology. I am a visual artist who is dipping their toe into audio and trying to understand what is important (initial question was I want to present a recorded sound indistinguishable from a live sound, what would that take?). I’ve strayed far from my initial question while trying to get answers, by going through threads like these just trying to figure out the questions I should be asking. I have hit walls many times, even through best efforts.

So yeah. Sometimes it would be nice to just have certain relevant thresholds spelled out, if there is consensus on them.

 

[Xulonn]

I agree it is in abundance here, it is not well cited or organized.

One big issue I have with ASR is that it searches for keywords only, and does not have boolean search capability. Searchng internet sites with lots of info is very difficult when one is unable to exclude words and/or phrases.

What is Boolean Search?​

Boolean Search uses a combination of keywords and the three main Boolean operators (AND, OR and NOT) to organise and sift through your searches. It produces more accurate and relevant results, allowing you to navigate through appropriate candidates, while disregarding the unrelated.

 

[birdog1960]

Oh stop being so butt-hurt.

I agree it is in abundance here, it is not well cited or organized. The oft cited audibility thresholds thread relies on uncited (thankfully stated as such) numbers for much of its argument. For everything published experimentally, there is a thread following it often making substantial claims against the data as presented, or it’s implications.

As someone who has been here for about a year, and pretty actively engaged and reading most of the suggested articles as well as well as delving into the current scientific literature at points, I still get pretty lost occasionally.

My math skills are rusty, and I am not in the sciences at all. The closest training I have is archaeology. I am a visual artist who is dipping their toe into audio and trying to understand what is important (initial question was I want to present a recorded sound indistinguishable from a live sound, what would that take?). I’ve strayed far from my initial question while trying to get answers, by going through threads like these just trying to figure out the questions I should be asking. I have hit walls many times, even through best efforts.

So yeah. Sometimes it would be nice to just have certain relevant thresholds spelled out, if there is consensus on them.

exactly. Is it real or is it memorex? if that's not possible, what is the closest we can get (taking audibility thresholds and technology constraints into account)? How much would that cost and at what point do diminishing returns kick in (given that there are apparently many transparent, affordable components)? Does xyz piece of equipment (on its own) approach it and how closely? These questions, to my thinking, are designed to dispel snake oil claims.

 

[manisandher]

Yes, seems to work for me.

Never used Octave, so will have to take your word for it. Thanks.

 

[amirm]

One big issue I have with ASR is that it searches for keywords only, and does not have boolean search capability. Searchng internet sites with lots of info is very difficult when one is unable to exclude words and/or phrases.

We actually do support that. "AND" is implied when you put multiple words in there. For "OR," use the | symbol. For exclusion of a term, use - (e.g. topping -DAC shows reviews of topping that do NOT have DAC in them).

 

[amirm]

So, what are those codecs you were waxing lyrical about using?

Codecs use (over)lapped transforms to obviate the need for windowing (they do use windowing but for another purpose). The general term is "MDCT" which stands for modified discrete cosine transforms. The "M" part is the overlapped operation. it is a clever scheme to remove aliasing artifacts in time domain. The other nice thing about the transform is that it produces half as much data.

There are also codecs that are "bit exact" meaning they have inverse transforms that generate identical results, using integer transforms no less.

Net, net, yes we do go from time to frequency and back to time without creating artifacts due to that operation.

 

[amirm]

A point about measurements: we can run hundreds of audio tests if we wanted. And if someone identified something we are not measuring, we can trivially create a measurement for that as well. I only run half a dozen tests in general because more doesn't do more to quantify how linear a device is beyond very end of the diminishing returns tail.

So it is not that "we don't measure everything" but that "we don't need to measure everything" to know how linear and transparent a device is. Yes, we leave 0.0001% to chance. Show us a real test case where that loss matters and I will add a test for it. Indeed, I have done that over the years I have been testing, growing the set to what you see.

Ultimately, we want to get an idea of how well designed and engineered an audio device is. A product that has gone through rigorous process of producing extremely low noise and highly transparent operation, is going to do similarly well across many tests. Conversely for a poorly designed one.

Now, there is a class of audio technology where measurements are rendered mostly useless. The aforementioned lossy codecs is one. Such a pipeline makes decisions to degrade audio on frame by frame basis (a frame is usually from 0.5K to 8K samples) so we can't characterizing it with a simple sine wave. The system has "intelligence" to vary its compression ratio and logic dynamically. So we rely almost exclusively on listening tests for evaluation and development of lossy codecs.

Audio devices in contrast are totally dumb. They have no past history other than say, in a filter. And that filter performs its function statically, not varying its operation based on any logic. There can be some memory effect if you drive a system beyond its bandwidth with things like artificial impulse functions. Such functions though are "illegal" audio signals as they require infinite bandwidth. Net, net, there is little mystery in operation of audio devices as to dispense with measurements and require mandatory listening tests.

But even if they did, you better perform controlled tests. This is very true for developers of audio technology. They can easily get wedded to their own solutions. For this reason, when my codec team at Microsoft developed new encoders or algorithms, they would ask me to listen to it to verify they were not fooling themselves. And for developments of standardized codecs, blind testing was mandatory.

 

[amirm]

I bet the least informed members make up the majority.

We are fortunate enough that what you say is quite wrong. Here is our top contributors: https://www.audiosciencereview.com/forum/index.php?members/&key=most_messages

Plenty of experts in topics we discuss in there. Certainly more than what you bring to the party.

 

[dfuller]

At this point, the only tests for audio gear that I think don't entirely characterize their behavior are speakers and headphones, solely because they're so much further from linear devices. Even the best are a mess of stored energy and nonlinear distortion (compared to electronics).

That, and the current best metrics I know of for speakers don't really take into account dynamic capability nor LF transient behavior (which I am increasingly convinced matters much more than most test suites let on), nor do they necessarily work for heavily treated spaces, where there is a different balance of on-axis/listening window vs sound power.

 

[amirm]

Many here will not criticise side-wall generated distortion which is perceived a widening of the sound scene and increased envelopment not will they criticise wide-directivity speakers as lower fidelity.

Reflections are totally linear effects (assuming no rattles, etc.). To wit, they don't create any new frequencies which means your room transfer function is 100% linear. And of course a perfect reflection doesn't change the noise floor either. So calling them "distortion" is misleading.

What side reflections do -- and why in controlled testing they are preferred -- is that they fix some of the issues with stereo playback. And mono in the case of center speakers. That we don't like a point source of sound as we prefer a more spacious sound. Multi-channel fixes that problem but it is not in common use for music.

People who don't study the research run with lay intuition that "reflections are bad" not realizing that when you listen to people in your house there is a ton of reflections yet that is better than any recording. And of course environments such as concert halls are nothing but reflections (past Dc).

So no, we don't bite our noise despite our face. We follow the science where takes us without some silly rule made up to argue otherwise.

 

[amirm]

That, and the current best metrics I know of for speakers don't really take into account dynamic capability nor LF transient behavior (which I am increasingly convinced matters much more than most test suites let on), nor do they necessarily work for heavily treated spaces, where there is a different balance of on-axis/listening window vs sound power.

There are some tests for dynamics but I don't like them. They make the smallest attempt at audibility assessments. As you know I listen instead as that is so much easier but admittedly, not precise or very comparative.

 

[antcollinet]

Don’t use only 32 bit…

Exactly 32 bit FP only has a precision of 23 bits.

 

[MattHooper]

A point about measurements: we can run hundreds of audio tests if we wanted. And if someone identified something we are not measuring, we can trivially create a measurement for that as well. I only run half a dozen tests in general because more doesn't do more to quantify how linear a device is beyond very end of the diminishing returns tail.

So it is not that "we don't measure everything" but that "we don't need to measure everything" to know how linear and transparent a device is. Yes, we leave 0.0001% to chance. Show us a real test case where that loss matters and I will add a test for it. Indeed, I have done that over the years I have been testing, growing the set to what you see.

Ultimately, we want to get an idea of how well designed and engineered an audio device is. A product that has gone through rigorous process of producing extremely low noise and highly transparent operation, is going to do similarly well across many tests. Conversely for a poorly designed one.

Now, there is a class of audio technology where measurements are rendered mostly useless. The aforementioned lossy codecs is one. Such a pipeline makes decisions to degrade audio on frame by frame basis (a frame is usually from 0.5K to 8K samples) so we can't characterizing it with a simple sine wave. The system has "intelligence" to vary its compression ratio and logic dynamically. So we rely almost exclusively on listening tests for evaluation and development of lossy codecs.

Audio devices in contrast are totally dumb. They have no past history other than say, in a filter. And that filter performs its function statically, not varying its operation based on any logic. There can be some memory effect if you drive a system beyond its bandwidth with things like artificial impulse functions. Such functions though are "illegal" audio signals as they require infinite bandwidth. Net, net, there is little mystery in operation of audio devices as to dispense with measurements and require mandatory listening tests.

But even if they did, you better perform controlled tests. This is very true for developers of audio technology. They can easily get wedded to their own solutions. For this reason, when my codec team at Microsoft developed new encoders or algorithms, they would ask me to listen to it to verify they were not fooling themselves. And for developments of standardized codecs, blind testing was mandatory.

Very interesting distinctions. Thanks!