Audio Engineering Society 2023 Conference

[MayaTlab]

You can reproduce these plots in Matlab or GNU Octave softwares just by the *.m code script below and the originally downloaded *.csv data files.

As you can see, all data processing is faithful to the original data. No extra selection, filtering or error correction procedures were adopted.

Due to the lack of specific experimental details, only a general boxplot is plotted to describe the distribution of all subjective rating data for each target curve.

No further investigations into the influences of the assessors, the pieces of sound samples and the experiment trials were included in the code and the plots. I am not familiar with Sean Olive's data processing and logical inference procedures either. Happy to see further discussions and your opimions.

clear;clc
table1 = readtable("data\MagnitudeFrequencyResponses.csv");
table2 = readtable("data\PreferenceRatings.csv");

freq0 = table1{1,2:end};
targets = struct;
count = 1;
for i = 2:height(table1)
    targets(count).name = table1{i,1};
    targets(count).freq = freq0;
    targets(count).gain = table1{i,2:end};
    count = count + 1;
end
target_names = [targets(:).name];
figure(231011003);clf(231011003);
for i = 1:length(targets)
    semilogx(targets(i).freq, targets(i).gain);
    grid on;
    hold on;
end
xlim([30,25000]);
xlabel('Freq (Hz)');
ylabel('Gain (dB)');
legend(target_names, 'Location', 'eastoutside');

figure(231011004); clf(231011004);
boxplot(table2.Rating, table2.HeadphoneCurve);
ylabel('Rating');

Thanks a lot for the answer, amazing work. I am not familiar with Matlab, but reading the documentation, am I understanding it correctly that the box plot command produces the median value, as well as the 25th and 75th percentiles ?

 

[ThomasXia]

Thanks a lot for the answer, amazing work. I am not familiar with Matlab, but reading the documentation, am I understanding it correctly that the box plot command produces the median value, as well as the 25th and 75th percentiles ?

Yes, a simple boxplot provides the 25, 50(median), 75 percentile values.

Violinplot is an alternative to get a better view of the actual rating distributions. Here is a violinplot showing the distributions, the means, the medians and the 25 and 75 percentiles of the ratings at the same time.

The HP3 curves remain the most satisfactory targets, regardless of the parameter being considered.

You can reproduce it in Python3.

import matplotlib.pyplot as plt
import pandas as pd

df2 = pd.read_csv("data/PreferenceRatings.csv", delimiter=';')

headphone_curves = df2['HeadphoneCurve'].unique()
fig, ax = plt.subplots(figsize= (24, 8))
violin_data = [df2[df2['HeadphoneCurve'] == curve]['Rating'] for curve in headphone_curves]
violin_parts = ax.violinplot(violin_data,showmeans=True, showextrema=False, showmedians=False, widths=0.85)
ax.set_xticks(range(1, len(headphone_curves) + 1))
ax.set_xticklabels(headphone_curves, rotation=45)

for pc in violin_parts['bodies']:
    pc.set_color('azure')
    pc.set_edgecolor('black')
    pc.set_alpha(1)

quartile1, medians, quartile3 = np.percentile(violin_data, [25, 50, 75], axis=1)
inds = np.arange(1, len(medians) + 1)
ax.scatter(inds, medians, marker='o', color='steelblue', s=50, zorder=3)
ax.vlines(inds, quartile1, quartile3, color='skyblue', linestyle='-', lw=5)

 

[MayaTlab]

Yes, a simple boxplot provides the 25, 50(median), 75 percentile values.

Violinplot is an alternative to get a better view of the actual rating distributions. Here is a violinplot showing the distributions, the means, the medians and the 25 and 75 percentiles of the ratings at the same time.

View attachment 318687

The HP3 curves remain the most satisfactory targets, regardless of the parameter being considered.

You can reproduce it in Python3.

import matplotlib.pyplot as plt
import pandas as pd

df2 = pd.read_csv("data/PreferenceRatings.csv", delimiter=';')

headphone_curves = df2['HeadphoneCurve'].unique()
fig, ax = plt.subplots(figsize= (24, 8))
violin_data = [df2[df2['HeadphoneCurve'] == curve]['Rating'] for curve in headphone_curves]
violin_parts = ax.violinplot(violin_data,showmeans=True, showextrema=False, showmedians=False, widths=0.85)
ax.set_xticks(range(1, len(headphone_curves) + 1))
ax.set_xticklabels(headphone_curves, rotation=45)

for pc in violin_parts['bodies']:
    pc.set_color('azure')
    pc.set_edgecolor('black')
    pc.set_alpha(1)

quartile1, medians, quartile3 = np.percentile(violin_data, [25, 50, 75], axis=1)
inds = np.arange(1, len(medians) + 1)
ax.scatter(inds, medians, marker='o', color='steelblue', s=50, zorder=3)
ax.vlines(inds, quartile1, quartile3, color='skyblue', linestyle='-', lw=5)

Thanks again. I have seen enough box plots that did not adhere to these definitions that I wasn't sure what I was looking at.
Generally speaking, it seems that Harman's method for processing preferential data is to find the mean across all listeners and test tracks.

 

[ThomasXia]

Maybe the answers could only be avaliable in the final paper.

 

[MayaTlab]

Maybe the answers could only be avaliable in the final paper.

There's a lot riding on the methodology used to know how to interpret the data indeed.

The csv file containing the FR curves might have some errors, I and others have noticed that there may be a shift in values on the x axis for some traces. Hopefully this only happened when publishing the csv file to Zenodo and not during the actual study.

 

[ThomasXia]

There's a lot riding on the methodology used to know how to interpret the data indeed.

The csv file containing the FR curves might have some errors, I and others have noticed that there may be a shift in values on the x axis for some traces. Hopefully this only happened when publishing the csv file to Zenodo and not during the actual study.

In fact, these may not be shifts errors in the x axis as they look like, but the actual frequency responses of these headphones!!!!

Maybe it is just the fact that some headphones with responses deviating from usual frequency targets heavily are still preferred by the listeners.

Just for example, HP3 vs APM:

reference: https://www.soundguys.com/apple-airpods-max-review-44975/

Update2, Example2, HP4 vs DT700PX. (Deviations look like a quality control or positioning issue more than a simple horizontal shift on x axis.)

 

[Thomas savage]

One challenge with the conference is that a number of talks are concurrent. So not possible to attend all of them.

On my session, they put one of the top Pro youtubers in parallel with mine!

I'm almost sure that's a unintentional and regrettable oversight in scheduling .., almost .

 

[MayaTlab]

In fact, these may not be shifts errors in the x axis as they look like, but the actual frequency responses of these headphones!!!!

Maybe it is just the fact that some headphones deviating from usual frequency targets heavily are still preferred by the listeners.

Just for example:

View attachment 318719
reference: https://www.soundguys.com/apple-airpods-max-review-44975/

To me it seems that not all the curves are affected, and the APM one indeed isn't.

From what I understand the APHarm targets originated from that thread : https://audiosciencereview.com/foru...easurements-using-brüel-kjær-5128-hats.15352/
Mad_Economist has noticed that the curves in the csv file are shifted on the x axis relative to the curves in that thread, and that they match nearly perfectly when you correct for the shift.

In a similar way, using the list of headphones featured in that study :

I'm starting to think that a plausible matching list would be something like this :

• HP1 = Epos Adapt 600
• HP2 = jabra
• HP3 = apm
• HP4 = 700proX
• HP5 = Aonic
• HP6 = Q45
• HP7 = XM4
• HP8 = B&O HX

For some of these, some recognisable features of their FR, particularly the ones that are quite insensitive to positional variation or fixture, seem shifted a bit.

I don't have the time this WE to spend too much time writing here. But, for example, the DT 700 Pro X features a dip at 4kHz that you'll find on all fixtures and that is quite resilient to positional variation, ex here : https://www.soundguys.com/beyerdynamic-dt-700-pro-x-review-60868/
If it is indeed HP4 (and I think that one would struggle to make the case that it isn't), then this is what we get from the csv file :

For some of the other ones these individual features also match better once you scale / shift the FR on the X axis a bit.

Re the question of the preference scores, I don't have a particularly strong opinion on them as we have no idea of the methodology used, or even, if these shifts happened during the study, of what was actually evaluated.

 

[ThomasXia]

For some of the other ones these individual features also match better once you scale / shift the FR on the X axis a bit.

This difference may simply be due to individual variations in quality control or positioning as in the updated figure above, since one single shift do not promise a perfect match.

Thanks for figuring out all these models, really a good job.

 

[MayaTlab]

This difference may simply be due to individual variations in quality control or positioning as in the updated figure above, since one single shift do not promise a perfect match.

Not for that 4kHz dip . And neither for the response in the feedback range for some of the ANC headphones, or some other individual features.

 

[beefkabob]

One challenge with the conference is that a number of talks are concurrent. So not possible to attend all of them.

On my session, they put one of the top Pro youtubers in parallel with mine!

Totally different audiences.

 

[amirm]

There are 22 people signed up for my session. So I am out of the woods as far as presenting to an empty room.

 

[Travis]

I don't think they have a specific agenda and the work simply depends on interest of the students and faculty.

Highly doubtful. They are a research university, so they rely on funding from Canadian Govt. and private industry for research. The agenda probably depends on what interests those providing the funding. They have an entire lab devoted to nothing but emersive audio,

I’m sure the lab director can guide where the research goes, the next step (and it might even interest faculty and students) but they are accountable to whichever research council is providing them with funds to do research.

There are 22 people signed up for my session. So I am out of the woods as far as presenting to an empty room.

I think you will end up with more than that. I think you drew an excellent time slot and a good track.

Did I read that correctly, all of the presentations on you track are 15 minutes each?

 

[amirm]

Did I read that correctly, all of the presentations on you track are 15 minutes each?

Yes. Indeed majority of the talks are 15 minutes. I think only a couple are 30.

 

[Travis]

Yes. Indeed majority of the talks are 15 minutes. I think only a couple are 30.

Well that will work to your advantage based on your YouTube videos. You can get 2x the info. in 1/2 time as most.

 

[Sean Olive]

Preference ratings and 32 magnitude frequency response curves

This repository comprises two CSV files: PreferenceRatings and MagnitudeFrequencyResponses. The former includes preference ratings obtained from 56 naive assessors (30 Danish---DK, 26 Japanese---JP) of 32 headphone curves over several music programs in several trials. The latter includes the...

zenodo.org

View attachment 318109

It might be easier to interpret if you order the mean ratings from highest to lowest - I did this and removed the 2-3 modifications they made to the 8 headphones as I was less interesting in that , although it might be the focus of a different question,

The graph below is the mean based on all ratings ( all listeners, programs, repeats). They didn't code the data according to location (Dane vs Japanese) so we can't analyze it based on culture.

The ratings of the headphone curves is remarkably similar to our work in terms of Harman Target being preferred, diffuse and free field rated progressively lower. It is amazing to me and one of the co-authors I spoke with that two separate experiments using different subjects and methodologies can some up with similar conclusions.

They did use two of our standard tracks (Jennifer Warnes "bird on a wire" and Tracy Chapman "FastCar" so there was that commonality in methods. They also used a virtual headphone method applying equalization filters to a replicator headphone so that rapid A/B comparisons can be made using a MUSHRA method. They used a closed headphone which to me is surprising because open backs generally eliminate the issue of leakage among subjects.

 

[alumnicesar]

It might be easier to interpret if you order the mean ratings from highest to lowest - I did this and removed the 2-3 modifications they made to the 8 headphones as I was less interesting in that , although it might be the focus of a different question,

View attachment 319570

finally, the original way it was presented is horrible lmao. Thanks

is there a way to isolate how the soundguys graph looks in comparison to hoe the estimated harman 2018 v2 graph looks? Not the preference rating but rather just where it differs in frequency

 

[Sean Olive]

It might be easier to interpret if you order the mean ratings from highest to lowest - I did this and removed the 2-3 modifications they made to the 8 headphones as I was less interesting in that , although it might be the focus of a different question,

The graph below is the mean based on all ratings ( all listeners, programs, repeats). They didn't code the data according to location (Dane vs Japanese) so we can't analyze it based on culture.

The ratings of the headphone curves is remarkably similar to our work in terms of Harman Target being preferred, diffuse and free field rated progressively lower. It is amazing to me and one of the co-authors I spoke with that two separate experiments using different subjects and methodologies can some up with similar conclusions.

View attachment 319570

Here is a table of the means, median, etc. in descending order from highest to lowest mean.

 

[Sean Olive]

AES conference program is finally available. I thought I post it for members to see what the buzz is about and whether there are any talks of interest: https://aesny23.sched.com/

You might see some familiar names there from ASR.

I am especially interested in the talk about B&K 5128 headphone preference curves/listening tests. Author had contacted me in advance about it.

I will be attending.. See you there..

 

[alumnicesar]

Here is a table of the means, median, etc. in descending order from highest to lowest mean.


View attachment 319573

i wish they would have tested df with a 10db tilt