Monaural Speaker Recordings - Blind Testing Six Speakers for Fun

[GXAlan]

It's time for another blind test! The recordings are not as level matched as I had hoped, but c'est la vie. This is just for entertainment not science.

Background
Six speakers on trial, not all in the same room. Most expensive system was $70,000.

• A "golfing panther" speaker
• A “happily lounging panther" speaker
  
  
• Lifestyle product from a mainstream brand, marketing focuses on sound quality
• Lifestyle product from a mainstream brand, marketing focuses on luxury/experience; music playback is actually minimally advertised.
  
  
• Audiophile speaker, marketing focuses on "realistic instruments"
• Audiophile speaker, marketing focuses on "live sound"

Setup Conditions
A -20 dBFS pink noise tone was used to achieve approximately 75 dB SPL at the listening position. The listening position varied between rooms

• Lifestyle products have very coarse volume controls and may apply volume-dependent processing.
• 75 dB is what is reported by REW when using Check Levels (which reports to the nearest dB only).
  
  You must adjust gain to your preferences.

A MiniDSP UMIK-2 was used as a recording device pointed toward the speakers. The internal noise of the UMIK-2 is still really high, which leads to baseline noise in all recordings. After the recording was made, the calibrated frequency response TXT files was used to apply a corrective EQ to simulate a flat recording. Based upon the listening levels, no clipping occurred.

The UMIK-2 is more directional than the UMIK-1, and as a monaural microphone, what you hear in the recording doesn't truly reflect what you hear in real-life. The challenge with binaural recording or using a portable handheld stereo recorder is a lack of calibration of microphones and variable stereo presentation. Still, this is a way to compare speakers under different conditions using a device with a calibrated frequency response.

An excerpt from one of Beethoven's violin sonatas was used. This was done because the original recording is long enough to allow a reasonable excerpt under Fair Use and because the pianos and violins have the paradox of everyone knowing what a piano and violin should sound like, and yet no two pianos and no two violins sound the same.

The Poll
There are six speakers; the recordings are not well volume matched despite matching of Pink Noise SPLs. Thus, this poll allows you to pick your top two choices. I'm assuming the loudest recording will get picked, but I'm hoping there is a clear loser and the golfing panther speaker ends up being one of the popular choices, but we'll see!

I have added "No Preference" as an option since the UMIK-2's internal noise is so high. The nice thing abot this test is that none of the recordings sound the same due to level differences and speaker differences. They're easily ABX'able -- the question is just which recording you prefer!

I'd like to get ~50 voters (100 responses) before revealing the identity of the speakers.

EDIT: Recordings deleted after one month.

 

[holdingpants01]

To match the volume of the files, here are the changed values in dB so they all equal in volume the D file at -34.1 LUFS-I

 

[holdingpants01]

I would suggest adding a source clip as well

 

[GXAlan]

I would suggest adding a source clip as well

I worry about the legalities of that. My specific source is from about a minute into track three if this Accuphase demo SACD: https://www.discogs.com/release/110...ial-Sound-Selection-3-For-Superior-Equipment-

It is likely from this original recording, although Amazon just has 24/44.1.

https://music.amazon.com/albums/B084GYQDY2?do=play&trackAsin=B084GZD55V&ref=dm_sh_b8cBo0Unoq1ajTABbKB9boAUU

 

[Axo1989]

This one is going to be much harder for me than the 300B vs Topping comparo last time. Classical, eh? You know, I don't really know what a violin "should" sound like. Not even sure about piano really.

Spoiler: first impression

So, listening on AirPods Max (like my first try last time) the first run-through I initially liked E. Struck me as a bit smoother. Repeating now and trying to listen to the instruments I'm unfamiliar with. E may be a bit thin actually. B sounds a bit muffled and maybe some distortion on the treble peaks. Not voting yet, like last time I'm going to try it on the Sony Z1R (which made me appreciate the tube sound a bit more last time so might help with the classical). They're in the office and it's a bit cold this morning to go sit there before breakfast. I'll post again after that.

Incidentally on the clips, the volume isn't much on APM at maximum, had to use noise cancelling to block the incidentals (like the magpies and the cat). Should be better with a headphone amp and the Sonys. Also incidentally, what's going on at 0:25-0:27 and again right at the end? I thought mic handling noise but it's on all the clips so not that ... like a little static sweep or some electro creeping in (no such luck I expect). There does appear to be an extra bass bump around 0:20-0:21 on track B and a fair bit of something like rubbing around that time on track C. Funny really, I found those extraneae a bit more interesting sonically than the program material: musique concrète ftw, right? I'll get used to the music after another listen or six I imagine. I'm even getting a bit of emotion at the small crescendo from 0:10-0:15 already.

 

[GXAlan]

This one is going to be much harder for me than the 300B vs Topping comparo last time. Classical, eh? You know, I don't really know what a violin "should" sound like. Not even sure about piano really.

The speaker identities are so different that I really was hoping that the difference would be striking once recorded. Yet, I admit, when I start comparing them via headphones right now, it's hard to pick a winner.

1) Maybe there is a paralysis of choice. It's easy to pick between two, but hard to pick between six.

2) Maybe this is the whole circle of confusion part. We know what transparency is, and we can judge that. But maybe the reason why audiophiles fall into a certain demographic while music fans are more diverse is that audiophiles focus on transparency, and in these samples, since the focus is more on music (since there isn't a good reference to compare too) and it's the *same* musicians playing, the emotional expression is the same.
(If you listen to the same piece from Beethoven by different artists, there is different levels of speed/intensity, etc.)

3) Maybe @MKR or @aliqaz are willing to give this blind test a try (if they listen to any classical music) since they've been in the market for new speakers and are more used to making big/broad comparisons.

I'm not sure we'll get to 100 votes, but let's see what happens over the next month.

What was surprising is how different the volumes actually are when they play the same 75 dB SPL. This strongly suggests to me that so much of speaker comparisons results in mismatched levels

 

[Axo1989]

Don't be put off by my opening comment, I think people who are into classical can hit the ground running, unlike me with a learning curve. Six is a lot I agree, the traditional Toole-ian speaker comparo recommendation is four. And if I understand it, you've corrected for FR? Scratch that, you mean microphone correction. But still, there are differences. And I found the Spring Rondo track on Apple Music to compare ...

Spoiler: better hide this comment I guess

Those brushing (?) noises are certainly on the original (not the other noises though).

 

[AlfaNovember]

Is there a way to tell Roon what the volume offsets are? I poked around in the UI but didn't see anything to adjust on a track by track basis.

 

[Blumlein 88]

I used cyjanopan's volume corrections. Don't see a lot of point in picking with obvious level differences.

Spoiler

I chose two in the poll. E and F. I preferred F over all the others. I listened over a single mono speaker (JBL LSR 308). Seemed reasonable as it was a mono recording.

I'd suggest like Toole does, that you match levels using pink noise filtered below 500 hz and above 2 khz at a 2nd order roll off. Much of your level issues seem to come from the very low frequencies. I looked at the spectrum after I had listened and made my choices.

 

[holdingpants01]

After a volume matching it was quite easy to pick favourites

Sometimes I liked E more, sometimes F, to me both are the only acceptable, I didn't like any of the rest

 

[Blumlein 88]

Actually, you needed one other choice in the poll. With all the tracks in a DAW while trying them out I accidentally turned all six tracks on. I'm almost convinced that was better than any single track.

 

[IAtaman]

Level matched as per gain settings cyjanopan suggested. 44.1kHz / 24bit versions. Originals were 44.1kHz/32 bit float.

 

[IAtaman]

I have no formal training whatsoever and I do not think I have any particular talent or skill. With that caveat here are my impressions:

Spoiler: Impressions

• I have not looked at what others have preferred not to color my own judgement.
• I listened to them all and made my notes. My favorites were D and F. D was a bit darker but I liked it anyway.
• Then I thought maybe feeling of darker or brighter might be affected by what came just before it, so I randomized the order and listened to them one more time
• Without seeing what is playing this time, I liked D and E better.
• Based on that. Overall I'd say my favorite was D, followed by E and then F.
• A and B sounded like they have been recorded through a band limited media, as if your friend called you from his phone and you recorded the call. B sounded a bit more boxy to me even.
• C had noises in it so I am not sure if it got a fair assessment.

Thank you for putting this together. It was fun.

 

[IAtaman]

Actually, you needed one other choice in the poll. With all the tracks in a DAW while trying them out I accidentally turned all six tracks on. I'm almost convinced that was better than any single track.

Here is all of them level mathced and mixed together into a single track

 

[dasdoing]

used files from post 52 on headphones

Spoiler: my take

a) sound a little "around the corner". as if I am in the hall leading to the symphony hall. especialy the two violin notes around 16s and 19s sound very bad
b) boomy (but without low end) and missing clarity, both a and b bad in it's own way
c) this seam to sound "ok" if you consider it is a recording of a speaker
d) suprised me. doesn't really sound like recording a speaker....more like if a bad mic was used in the hall. missing clarity.....my favourite so far, but still bad
e) best balance so far, though sounds just a little harsh. realness doesn't come close to d though... miss the attack of the piano notes for example which are clearly present in d. the high piano notes right at the beginning almost sound like a flute
f) seams to be shy in the mids

listening again: knowing all now, c) is obviously bad, too...very thin

e) wins for me for balance, but it's not realistic
d) wins for realism, but where is the clarity?

hard to decide, guess I have to vote d) and e)

EDIT: the original would have helped. I don't know how balanced the original was
EDIT2: now I listened to the original and they are all soooo bad lol. I don't know how people like reflective rooms. it's so obvious. but I continue with my rating. e) is really balanced, but the transients are so messed up. d) is really good but that missing clarity is anoying

 

[GXAlan]

Thanks @IAtaman for the level matching.

@dasdoing
I agree. These recordings are *bad* compared to doing a recording with a portable PCM recorder. But a portable PCM recorder doesn’t have a flat FR.

The scary part is that our room correction measurements are done with tools like the UMIK-2.

Since the audio sounds different, I wonder if this means that the calibration is only good for a specific SPL and the mic is more non linear than we think?

That is, you can make analog electrical recordings from playback devices and playing the recording back is pretty transparent. Do it enough times and you get noise.

Here, with one generation of recording speakers the quality drops dramatically— and this is on a UMIK-2! It’s even less transparent if you use a UMIK-1.

I wonder if the omnidirectional nature of the UMIK means that it hears something different than the way our ears experience it. Maybe the UMIK does best with frequency sweeps and it’s less effective with multi tones? Maybe this is why using the UMIK-2 for my pink noise SPL level matching ended up having some incredibly different actual volumes.

I will probably try to buy a Zoom H3-VR at some point.

 

[Blumlein 88]

Thanks @IAtaman for the level matching.

@dasdoing
I agree. These recordings are *bad* compared to doing a recording with a portable PCM recorder. But a portable PCM recorder doesn’t have a flat FR.

The scary part is that our room correction measurements are done with tools like the UMIK-2.

Since the audio sounds different, I wonder if this means that the calibration is only good for a specific SPL and the mic is more non linear than we think?

That is, you can make analog electrical recordings from playback devices and playing the recording back is pretty transparent. Do it enough times and you get noise.

Here, with one generation of recording speakers the quality drops dramatically— and this is on a UMIK-2! It’s even less transparent if you use a UMIK-1.

I wonder if the omnidirectional nature of the UMIK means that it hears something different than the way our ears experience it. Maybe the UMIK does best with frequency sweeps and it’s less effective with multi tones? Maybe this is why using the UMIK-2 for my pink noise SPL level matching ended up having some incredibly different actual volumes.

I will probably try to buy a Zoom H3-VR at some point.

The issue is reflections. Your ears filter out early reflections and you hear mainly direct sound from the speakers (or other sources). This is done due to the delay and angle of incidence among other things between direct vs reflected sound. A recording from a microphone has picked up all those reflections without a filter and when you play them back the recorded reflections come direct from the speaker which prevents your hearing from filtering it out. The microphone is in a sense accurate and your hearing in the way it works prevents you from being able to hear it.

When you use a Umik for measurements the reason you use a sweep is so you can window it in time. Basically if the software is only listening in the narrow range of frequencies and rejecting what comes with a delay from reflections it can mostly not be corrupted by those reflections. It sweeps a filter along with the tonal sweep to accomplish this. In a crude sense it measures more like the way we hear, but mainly it hears the mostly direct response filtering out the room's effects. It is not perfect in this, but more or less successful. Much of the issue with pink noise was the level being corrupted by low frequencies though some of it was due to room reflections building up energy perhaps.

I don't think you'll find recording with a Zoom H3 in a room will be all that much of an improvement. Recording close to the speaker rather than the LP would work better as the direct to reflected sound energy is tilted more toward direct. You need to be far enough the sound from multiple drivers has a chance to mesh. So for many speakers you'll need to be 1.5 to 2 meters from the speaker. The further out you go the more the room ruins your recordings.

Somewhere here back a few years ago, I had a thread where I recorded music in room at different locations and different microphones. I don't know if those files are still there. If so you could listen and get an experience of what I am describing. All this worrying that the Umik is a significant distortion in measurements is a misplaced worry. The Umik 1 is fine and the REW software that lets it work so well is actually quite something maybe more than many realize. As a pure measuring device the somewhat directional at high frequencies Umik 2 was a step backwards. Still it is not a big issue for measurements.

 

[dasdoing]

ince the audio sounds different, I wonder if this means that the calibration is only good for a specific SPL and the mic is more non linear than we think?

were those speakers in-room corrected?

I wonder if the omnidirectional nature of the UMIK means that it hears something different than the way our ears experience it.

for sure it hears diferently. let's not forget that our head is in the way for several angles. also our head has "2 mics".
but I am not convinced that this would cause the recordings to sound like total garbage.
I am on the contrary almost convinced that tuning the speaker recording to sound like the original is a much more reliable way than using rather random preference curves on smoothed graphs.
One way I am testing atm is for example creating an IR of my listening position and than loudness match it via 1/3 octave pink noise to the original.

Your ears filter out early reflections and you hear mainly direct sound from the speakers

sorry, but mine never did this. it is always mixed in the direct sound, and Haas effect comes into play in not letting me identify any reflection.
this only works in big halls with reflections that we then call "reverb". Reverb is heard seperatly, yes. But not ultra fast small room reflections.

 

[dasdoing]

The right way would be to record this with binaural heads.
speaking of wich, if you listen to binaural IRs recorded even in TREATED professional rooms, they still sound colorized as hell...still sound roomy. if the "we hear it diferently" theory was right any binural IR of a good speaker should sound like heaven on headphones.

 

[dasdoing]

and I bet that D was either recorded in a reflection treated room, and/or the speaker has a very controlled and narrow dispersion.