Mark Waldrep's HD audio challenge II

[Blumlein 88]

http://www.realhd-audio.com/?p=6713

If you haven't seen it, a group of 20 tracks originally recorded by Mark Waldrep in 96/24 available for download in various other formats for a test of whether higher resolution audio is audibly different. These are the full tracks not just samples.

So take a look, you might want to take part.

 

[MarcL]

Just finished listening to all of the tracks with two friends and recording the results. After honestly recording results based only on listening, I used analysis software to see how we did. Interestingly, we correctly identified the high res tracks 5, 6 and 7 times (respectively) out of 20 tracks. What this tells me is that all three of us could tell the difference between the tracks around 75% of the time ... but we "incorrectly" picked the low res instead of the high res. Honestly it was really hard, but we did better than 50/50.

 

[JimB]

... we correctly identified the high res tracks 5, 6 and 7 times (respectively) out of 20 tracks. What this tells me is that all three of us could tell the difference between the tracks around 75% of the time ... but we "incorrectly" picked the low res instead of the high res...

Thanks for this. In your test, I'm not following your comment that the three of you could tell a difference (reliably?) 75% of the time. Can you give a little more detail of what you mean? Also, what was the listening equipment and setup? All three listening at the same time, together?

I tried the previous test files, and after reasonable efforts, including repeated trials over time, with different listening situations, I felt that I couldn't reliably discern a distinction, let alone which one was the true high res version. Note that this was not my first impression. At first, I thought there we specific 'tells' in 5 of the 6 files for me. But, after more attempts over time, those initial characterizations failed me. Mark noted that at least someone had gotten all six right. In my brief communications with him, it was clear that all results obtained to that point were random. That is, given enough people (not that many), one or more will get them all right by chance. Similarly, some people got them all wrong. Did that mean they could hear a clear difference, but just 'preferred' the low res files on their system? Not necessarily, as that too is expected to happen about as often as a perfect score, based on random statistics. Some people did say they could not tell a difference in some cases. Taken all together, there was no significant differentiation demonstrated. This lead Mark to want a more substantial study.

 

[Sal1950]

If you haven't seen it, a group of 20 tracks originally recorded by Mark Waldrep in 96/24 available for download in various other formats for a test of whether higher resolution audio is audibly different.

Have you seen the nasty attack a suit from Qobuz made against Mark over his position on High Definition files and all that. Honestly I was quite surprised by the below the belt insults and vulgar name calling by Mr David Solomon of Qobuz.

Nonsense, Trolling, and Personal Attacks
http://secure.campaigner.com/csb/Public/show/566i-1jf7bi--njwes-qy5nph6

 

[amirm]

I am surprised too. I met David once in person (and multiple times at shows) and had never seen this side of him. No doubt entrance of Amazon into high-res market has caused huge stress for them. David (Chesky) told me long time ago that they developed tools to catch empty upsampling. If Qobuz doesn't have that, they better get one.

 

[Sal1950]

David (Chesky) told me long time ago that they developed tools to catch empty upsampling. If Qobuz doesn't have that, they better get one.

What is "empty upsampling"? If it originated from an analog tape or a redbook digital, it's still questionable to distribute as HiRez IMO

 

[amirm]

Some 44.1 kHz content was upsampled to 88/96 kHz. Same thing happened to SACD releases back in the day. HDTracks got a lot of grief on this when people used Audacity to find these.

 

[Sal1950]

Some 44.1 kHz content was upsampled to 88/96 kHz. Same thing happened to SACD releases back in the day. HDTracks got a lot of grief on this when people used Audacity to find these.

Ok but I still confused then.
What's the difference between "empty upsampling" and Frank Sinatra's 1956 This Is Sinatra analog being sold as a 24/192 HiRez recording?
What makes the one "bad" and the other acceptable practice?
https://www.hdtracks.com/this-is-sinatra-197756?___store=default&nosto=nosto-page-search1

 

[amirm]

What's the difference between "empty upsampling" and Frank Sinatra's 1956 This Is Sinatra analog being sold as a 24/192 HiRez recording?

With analog, there is no brickwall limit in bandwidth. High frequencies diminish gradually. You could argue then where the cut off should be.

With empty upsampling, someone takes CD content that is already sharply filtered above 22.05 kHz, and upsamples it and sells it as high-res. In that regard, it is CD filled with a bunch of empty space. Here is an example:

The empty upsampled CD presents no value at all. The high-res capture of analog can have a bit of value depending on how much the high frequencies extended.

 

[Thomas savage]

I am surprised too. I met David once in person (and multiple times at shows) and had never seen this side of him. No doubt entrance of Amazon into high-res market has caused huge stress for them. David (Chesky) told me long time ago that they developed tools to catch empty upsampling. If Qobuz doesn't have that, they better get one.

Qobuz has no chance of survival, Amazon will find everything and anything on them and smear campaign them into nothing.

They must know this , it's desperate times and I'd hate to be them.

Back to the op , I'd totally missed this thread so thanks @Blumlein 88 for posting these.

 

[solderdude]

I read this:

Trolling or Truth Telling?

Earlier this morning, on the Hi-res Digital Audio Discovery group FB page, a member commented about the advantages of 96/24-bit audio over 48/24-bit in reference to a new Coldplay album. He wrote, "I think 24/96 is much better file."

Haven't read the actual response and reasoning of that poster but.. arguably the 96/24 will be the better file. It is closer to the actual recording.
I still think it is possible some people audibly can discern 96kHz over 48kHz but not pass the HD audio challenge II at their own home.
How one may ask ?
Well, simples, and I wrote this before.
When the guy who posted the part about the Coldplay album was listening with a (very) expensive NOS R2R DAC without any reconstruction filter in it chances are he can hear a difference between 48 and 96 (let's leave the bits out because R2R can't resolve 24 bits) simply because the 48 has audible roll-off and 96 won't. Also the actual waveform will be better with 96.
Of course, when the same guy is using a well designed DAC his audible findings are suspect and probably sighted.

However, he may not pass the HD audio challenge II as all of them will play at the highest bitrate so there won't be roll-off with the lower sample-rates in the test.

 

[MarcL]

Have you seen the nasty attack a suit from Qobuz made against Mark over his position on High Definition files and all that. Honestly I was quite surprised by the below the belt insults and vulgar name calling by Mr David Solomon of Qobuz.

Nonsense, Trolling, and Personal Attacks
http://secure.campaigner.com/csb/Public/show/566i-1jf7bi--njwes-qy5nph6

Thanks for this. In your test, I'm not following your comment that the three of you could tell a difference (reliably?) 75% of the time. Can you give a little more detail of what you mean? Also, what was the listening equipment and setup? All three listening at the same time, together?

I tried the previous test files, and after reasonable efforts, including repeated trials over time, with different listening situations, I felt that I couldn't reliably discern a distinction, let alone which one was the true high res version. Note that this was not my first impression. At first, I thought there we specific 'tells' in 5 of the 6 files for me. But, after more attempts over time, those initial characterizations failed me. Mark noted that at least someone had gotten all six right. In my brief communications with him, it was clear that all results obtained to that point were random. That is, given enough people (not that many), one or more will get them all right by chance. Similarly, some people got them all wrong. Did that mean they could hear a clear difference, but just 'preferred' the low res files on their system? Not necessarily, as that too is expected to happen about as often as a perfect score, based on random statistics. Some people did say they could not tell a difference in some cases. Taken all together, there was no significant differentiation demonstrated. This lead Mark to want a more substantial study.

JimB ..... Sorry for some reason my brain was mushy after listening a couple hours and it was hard to explain . My wife, a friend and I sat on the couch and I played the clips. So we listened to the same system at the same time and I played a minute or two of each track and went back and forth two or three times playing the same parts of the tracks for sample A and B in each case. We decided this was more realistic in comparing what we thought we were hearing (i.e. does that vocal crescendo or bass drum hit sound the same in the two samples?) than listening to entire 3 to 8 minute samples in their entirety. We spent an hour, took a break (dinner, wine) and listened to the rest ... probably 2 1/2 hrs listening total, moving faster toward the end. Equipment: Tracks played from JRiver on a computer over HDMI to Emotiva XMC-1 running Dirac Live in Stereo mode. Left/Right speakers are Magneplanar 3.7; Two Outlaw subs (front/rear of the room). Nord One stereo amp (700w/4ohms). Room is well-treated with bass traps and diffusion. Measured response is very uniform with many years of careful speaker placement, Dirac calibration and listening tests. All tracks were played at the same (moderate) listening level. Due to the wide dynamic range of many of the tracks, in retrospect I should have played maybe 5db louder. (BTW, I own many thousands of high-res tracks from 96/24 to ridiculous high rate PCM, and DSD and some SACD. We mostly listen to these, though sometimes listen to a streaming service or older ripped MP3's)

The test says to pick the high-res version from each sample pair. We each used some criteria (without discussing) and marked our results on a score sheet. After results were compiled and my results submitted to Mark, I used Music Scope to determine which were the 96/24. I got 6 "right" out of 20. My wife and friend got 7 and 5 respectively. So .... for what it's worth, we pretty consistently all used our own criteria and it seems we could tell the difference in this single test, under these conditions ... BUT what we picked as the high-res was actually the low-res the majority of the time!

My wife and I did the previous test with similar process in the same room, same equipment and we each got 5 out of 6 correct (actually picking the high-res that time. We both got the first four, the each got one of the last two.

All that said .... the engineer in me says if we did repeated tests our results would be random ... as you found. At best I think it may be possible to tell the difference between samples, but I am amused by the fact that in this trial all three of us picked the low-res file most often. Could it be that there was an unintended, consistent, but unmeasurable difference in the prep of the files?

There are many ways to look at this question. What if instead of this prep process, two identical recorders were used in a live session, one recording 44.1/16 and the other 96/24 from the same feed at the same time with identical levels. No up or down sampling later, or touching the files in any way. Would an A/B/X test just to determine if one can consistently hear a difference (not preference as inferred here) be more valid? I think so. Etc. ... many ways to get to the root of this.

I suspect that technically high quality recordings will be indistinguishable, though there's good reason to make use of the headroom offered by 96/24 in the production process.

SAL1950 ..... I read Mark's response in his blog post yesterday. Solomon was quite harsh and unnecessarily personal in his attack. The audio industry is infested with snake-oil salesman and the streaming market is predictably vicious in tossing every bit of mud and misinformation they can muster to grab their share of this ever-growing pot of gold ... physics and facts be damned.

 

[Soniclife]

JimB ..... Sorry for some reason my brain was mushy after listening a couple hours and it was hard to explain . My wife, a friend and I sat on the couch and I played the clips. So we listened to the same system at the same time and I played a minute or two of each track and went back and forth two or three times playing the same parts of the tracks for sample A and B in each case. We decided this was more realistic in comparing what we thought we were hearing (i.e. does that vocal crescendo or bass drum hit sound the same in the two samples?) than listening to entire 3 to 8 minute samples in their entirety. We spent an hour, took a break (dinner, wine) and listened to the rest ... probably 2 1/2 hrs listening total, moving faster toward the end. Equipment: Tracks played from JRiver on a computer over HDMI to Emotiva XMC-1 running Dirac Live in Stereo mode. Left/Right speakers are Magneplanar 3.7; Two Outlaw subs (front/rear of the room). Nord One stereo amp (700w/4ohms). Room is well-treated with bass traps and diffusion. Measured response is very uniform with many years of careful speaker placement, Dirac calibration and listening tests. All tracks were played at the same (moderate) listening level. Due to the wide dynamic range of many of the tracks, in retrospect I should have played maybe 5db louder. (BTW, I own many thousands of high-res tracks from 96/24 to ridiculous high rate PCM, and DSD and some SACD. We mostly listen to these, though sometimes listen to a streaming service or older ripped MP3's)

The test says to pick the high-res version from each sample pair. We each used some criteria (without discussing) and marked our results on a score sheet. After results were compiled and my results submitted to Mark, I used Music Scope to determine which were the 96/24. I got 6 "right" out of 20. My wife and friend got 7 and 5 respectively. So .... for what it's worth, we pretty consistently all used our own criteria and it seems we could tell the difference in this single test, under these conditions ... BUT what we picked as the high-res was actually the low-res the majority of the time!

My wife and I did the previous test with similar process in the same room, same equipment and we each got 5 out of 6 correct (actually picking the high-res that time. We both got the first four, the each got one of the last two.

All that said .... the engineer in me says if we did repeated tests our results would be random ... as you found. At best I think it may be possible to tell the difference between samples, but I am amused by the fact that in this trial all three of us picked the low-res file most often. Could it be that there was an unintended, consistent, but unmeasurable difference in the prep of the files?

There are many ways to look at this question. What if instead of this prep process, two identical recorders were used in a live session, one recording 44.1/16 and the other 96/24 from the same feed at the same time with identical levels. No up or down sampling later, or touching the files in any way. Would an A/B/X test just to determine if one can consistently hear a difference (not preference as inferred here) be more valid? I think so. Etc. ... many ways to get to the root of this.

I suspect that technically high quality recordings will be indistinguishable, though there's good reason to make use of the headroom offered by 96/24 in the production process.

SAL1950 ..... I read Mark's response in his blog post yesterday. Solomon was quite harsh and unnecessarily personal in his attack. The audio industry is infested with snake-oil salesman and the streaming market is predictably vicious in tossing every bit of mud and misinformation they can muster to grab their share of this ever-growing pot of gold ... physics and facts be damned.

Great post.
Do you think the differences you heard were ever significant? I suppose your thought that repeating the test would yield random results each time gives the answer.

 

[MarcL]

Great post.
Do you think the differences you heard were ever significant? I suppose your thought that repeating the test would yield random results each time gives the answer.

Soniclife ..... The differences - if we even indeed heard differences - were very subtle. We talked afterward about listening for transients, dynamics, distortion .... but very very small differences. And we admitted it was more gut feel than anything we could roll back and point to. Even during the test I wondered if I subconsciously favored the first sample (which is why I alternated starting with A or B). On a different day, if I sat a foot to my left, if someone else clicked the tracks and recorded my answers ... I could easily see different results.

 

[anmpr1]

Have you seen the nasty attack a suit from Qobuz made against Mark over his position on High Definition files and all that. Honestly I was quite surprised by the below the belt insults and vulgar name calling by Mr David Solomon of Qobuz.

I just made a similar comment about reviewer Michael Fremer in another thread. I didn't think it was possible for anyone to come across as badly, but Solomon is right up there with 'mikey' (as his in-group chummy associates like to call him).

Don't people realize that going off like this sets them up for ridicule? Very juvenile.

 

[Frank Dernie]

My laziness is delighted to read this.
I was going to do this comparison if I felt the urge, but having people here where they weren't sure or couldn't tell informs me that, as I have found every time I make the comparison on music (I don't listen to jangling keys as a rule) the differences are small to negligible, and therefore definitely not worth bothering with.
I am on the Dr Aix mailing list and had read the Qobuz response. I shall be cancelling my subscription.

 

[Sal1950]

My laziness is delighted to read this.
I was going to do this comparison if I felt the urge, but having people here where they weren't sure or couldn't tell informs me that, as I have found every time I make the comparison on music (I don't listen to jangling keys as a rule) the differences are small to negligible, and therefore definitely not worth bothering with.
I am on the Dr Aix mailing list and had read the Qobuz response. I shall be cancelling my subscription.

Same here on both counts Frank.
I've taken these tests before with the same results, I can't tell the difference. Now that could be cause my 70 year old combat infantryman ears are too damaged to hear any difference, or that there just isn't any. I go with the science of the issue that tells me I shouldn't be able to hear any difference when everything is done correctly.
As to Qobuz, Mr Solomon's low class retorts has made up my mind for me. I've stuck with Spotify due to the costs of a HD feed. Now with prices falling I will cheer the demise of Qobuz and Tidal. If Spotify is finally forced to open a HD feed I will stay with them, otherwise when my current 1 year sub runs out I'll jump to Amazon. I'm no fan of Amazon but in this case to me it's the better of four evils

 

[MarcL]

Yes Frank and Sal .... I did this one but I knew it would be tough. We three last night applied 65, 66 and 67 year old ears. Afterward I put some tones through the speakers confirming that we could definitely still hear 12kHz, but not a peep above that.

But it's not just high end, of course. I have always suspected that if I could hear the difference between 44/16 and 96/24 it would be something to do with dynamic range on very dynamic music with lots of transients. According to the JRiver dynamic range analysis the samples in this test ranged from 4.8 to 21.1 LU, with most around 10 or lower. The sample pairs measured identically (except for the first two which were off by 0.1 and 0.3 respectively; the higher level being the low-res in the first case and high-res in the second case).

So I still buy high-res if I have a choice, mostly for convenience over CD's and in the hope that there would be no better source. I especially like ChannelClassics, 2L and other labels that offer surround mixes in high-res. I buy (and RIP to FLAC) CD's if there is no download source. For streaming, we listen to Calm Radio a LOT. Lots of classical and jazz and 320k bitrate.

 

[Sal1950]

With analog, there is no brickwall limit in bandwidth. High frequencies diminish gradually. You could argue then where the cut off should be.

Thanks Amir, now I understand the technical difference.
And we all know how much musical info there was on those 1956 original tapes over 20K LOL
Just so happens I own the 24/96 HDTracks 1956 Sinatra-This Is Sinatra

 

[JJB70]

A lot of the music I listen to was recorded decades ago in the analogue era. In many cases the recorded quality is excellent as whatever technical limitations arising from available technology that may be present are balanced by the evident skill and musical sensibilities of the producer and recording engineers. And of course, when listening to the recordings of artists such as Emil Gilels, George Szell, Eugene Ormandy, Thomas Beecham, the older Karajan recordings, Furtwangler, Maria Callas etc etc limitations of recorded quality pale next to the artistic value of the performances. However, as much as I treasure these old recordings, I'd consider hi-res transfers to be utterly pointless, I enjoy them in spite of limitations of recording quality.