Entropy and information; the devil and details

[edahl]

I'm interested in what "detail" is in sound reproduction. On the one hand there's a qualitative and subjective aspect to it, but what interests me right now is how it might relate to something measurable. One candidate is that detail should relate to information retention (inversely, entropy) along a signal chain: if a detail is not reproduced, we could not possibly hear it. We can look at information divergence at any given point in the chain. Roughly, knowing the "true" signal (the audio source), how much does the current signal diverge from the original source? Harmonic distortion is a measure of the extent to which higher harmonics are added to a signal, muddying the signal, and can likely be interpreted as a form of entropy (I haven't done the work to show how, so please let me know if you know how). I’d be interested in research pointing to how notions of entropy might be useful to evaluating high fidelity audio reproduction. E.g., combinatorial notions of entropy through a dac/amp/headphone/mic/ad-converter chain, and so on. Is frequency response and THD+N enough to capture everything we need to evaluate high fidelity audio reproduction, or are there concepts of information theory we’re yet to fully employ? These thoughts are rather fresh to me, and though my background is in mathematics there's no point in me redeveloping at a snail's pace what must already be a rather well understood subject. So I ask rather naïvely and simply: what do we know?

 

[JeffS7444]

I'm interested in what "detail" is in sound reproduction. On the one hand there's a qualitative and subjective aspect to it, but what interests me right now is how it might relate to something measurable.

Mostly frequency response I think. In the case of maximizing intelligibility of the human voice, you can actually have a sound which is low-fidelity yet sufficiently "detailed" to convey the meaning of the spoken word. Here, a narrow bandwidth with a peak at around 5 khz may be very useful, and indeed, this is what you may find in many a small AM radio which has a "crisp" sound. This can be taken to even further extremes when tuning into single sideband shortwave and ham radio broadcasts.

Harmonic distortion is a measure of the extent to which higher harmonics are added to a signal, muddying the signal, and can likely be interpreted as a form of entropy (I haven't done the work to show how, so please let me know if you know how). I’d be interested in research pointing to how notions of entropy might be useful to evaluating high fidelity audio reproduction.

I'm not aware of entropy figuring into sound reproduction.

 

[edahl]

I'm not aware of entropy figuring into sound reproduction.

Entropy figures into pretty much any process where information is handled and conveyed in some way or another.

 

[dc655321]

I was looking into the information-theoretic content of music signals around a year ago (?)...
At the time, I was looking at power spectral estimation as a tool for comparing musical energy distributions.
Found this thesis somewhat useful, as well as this formulation of spectral entropy at the bottom of the linked page.
@j_j may know more about this topic.

 

[j_j]

I'm not aware of entropy figuring into sound reproduction.

It does in several ways. First the actual signal entropy can be measured. Look at "Spectral Flatness Measure" as a measure of redundancy, for instance. This involves determining redundancy and eliminating that by smart processing. For instance, for most waveforms, something like LPC or transform coding attempts to get close to the actual signal entropy. (No, you can't ever get all the way there, nope, without loss.)

There are things like 'zip' and such that can also compress an audio signal (inefficiently, by exploiting some of the redundancy), etc.

Second, it is possible to measure the "perceptual Entropy". Wow that goes back. https://ieeexplore.ieee.org/document/197157 Way back.

But in general, everything has an information rate or total content. And it's often quite counterintuitive how that works out.

Which signal do you think has higher entropy, a sine wave or white noise?

 

[pozz]

If we're talking gear, frequency response captures areas of energy storage (resonances, either as peaks or cancellations), which could be called entropic.

 

[j_j]

If we're talking gear, frequency response captures areas of energy storage (resonances, either as peaks or cancellations), which could be called entropic.

Um, nonlinear modifications reduce information content. Frequency shaping, with noise floor involved, likewise. But information entropy is a very precise measure, please.

 

[paulraphael]

Which signal do you think has higher entropy, a sine wave or white noise?

Well, you've already announced this as a trick question. Let's skip my answer, so you can get straight to the more interesting business of explaining why I'm wrong?

 

[Inner Space]

To me, detail is coherent low-level information intended for reproduction, but which can be masked or obscured by system noise or distortion, and especially by the listening room's noise floor. Thus in itself it's measurable by inference - the lower the cumulative SINAD, and especially the lower the room noise, the more chance you have of hearing it, if it's there. Uneven FR can either help or hurt, by either pushing or burying the region in question. I know nothing about entropy, except the general nature of my life.

 

[pozz]

Um, nonlinear modifications reduce information content. Frequency shaping, with noise floor involved, likewise. But information entropy is a very precise measure, please.

So the term can't be used loosely in that sense? Don't informational outcomes describe entropy, there being a clear relationship for example between (known) electrical signal and final (variable) acoustic measurement? Or is that too imprecise?

 

[dc655321]

Which signal do you think has higher entropy, a sine wave or white noise?

 

[Wes]

There's a Maxwell's Demon joke in here somewhere...

 

[j_j]

Well, you've already announced this as a trick question. Let's skip my answer, so you can get straight to the more interesting business of explaining why I'm wrong?

So what's your answer?

 

[Ron Texas]

You have no idea how much entropy a lawyer can make. No engineer can come close to all the trouble I have caused.

 

[Alice of Old Vincennes]

Same here. I never thought about it in the context of this thread. Four decades in the practice, and, I suppose, a significant contribution. Physicists should include our efforts in their calculations.

 

[Blumlein 88]

White noise is high entropy I do believe.

High entropy passwords are most random, most like noise.

A sine wave is very ordered.

 

[posvibes]

Does the entropy extend as far as our brain and consciousness?

 

[paulraphael]

So what's your answer?

Well, my answer would have been white noise is much higher in entropy, but you suggested this would be counter-intuitive, so I assume I'm wrong.

 

[edahl]

I'm happy to see my question has sparked such lively discussion! I'll give the more technical contributions the time and focus they require as soon as I can (@j_j and @dc655321), but for now let me put two links here, the first of which I found rather interesting and the second more generally explanatory.

https://apps.dtic.mil/sti/pdfs/ADA494612.pdf
https://www.baeldung.com/cs/cross-entropy

 

[j_j]

White noise is high entropy I do believe.

High entropy passwords are most random, most like noise.

A sine wave is very ordered.

Indeed. A very clean sine wave is very low entropy, because it is very, very redundant. A flat spectrum maximizes the entropy.