• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Audio Engineering Society 2023 Conference

MayaTlab

Addicted to Fun and Learning
Joined
Aug 15, 2020
Messages
947
Likes
1,570
You can reproduce these plots in Matlab or GNU Octave softwares just by the *.m code script below and the originally downloaded *.csv data files.

As you can see, all data processing is faithful to the original data. No extra selection, filtering or error correction procedures were adopted.

Due to the lack of specific experimental details, only a general boxplot is plotted to describe the distribution of all subjective rating data for each target curve.

No further investigations into the influences of the assessors, the pieces of sound samples and the experiment trials were included in the code and the plots. I am not familiar with Sean Olive's data processing and logical inference procedures either. Happy to see further discussions and your opimions.

Code:
clear;clc
table1 = readtable("data\MagnitudeFrequencyResponses.csv");
table2 = readtable("data\PreferenceRatings.csv");

freq0 = table1{1,2:end};
targets = struct;
count = 1;
for i = 2:height(table1)
    targets(count).name = table1{i,1};
    targets(count).freq = freq0;
    targets(count).gain = table1{i,2:end};
    count = count + 1;
end
target_names = [targets(:).name];
figure(231011003);clf(231011003);
for i = 1:length(targets)
    semilogx(targets(i).freq, targets(i).gain);
    grid on;
    hold on;
end
xlim([30,25000]);
xlabel('Freq (Hz)');
ylabel('Gain (dB)');
legend(target_names, 'Location', 'eastoutside');

figure(231011004); clf(231011004);
boxplot(table2.Rating, table2.HeadphoneCurve);
ylabel('Rating');

Thanks a lot for the answer, amazing work. I am not familiar with Matlab, but reading the documentation, am I understanding it correctly that the box plot command produces the median value, as well as the 25th and 75th percentiles ?
 

ThomasXia

Member
Joined
Mar 6, 2019
Messages
39
Likes
96
Thanks a lot for the answer, amazing work. I am not familiar with Matlab, but reading the documentation, am I understanding it correctly that the box plot command produces the median value, as well as the 25th and 75th percentiles ?
Yes, a simple boxplot provides the 25, 50(median), 75 percentile values.

Violinplot is an alternative to get a better view of the actual rating distributions. Here is a violinplot showing the distributions, the means, the medians and the 25 and 75 percentiles of the ratings at the same time.

output.png


The HP3 curves remain the most satisfactory targets, regardless of the parameter being considered.

You can reproduce it in Python3.
Python:
import matplotlib.pyplot as plt
import pandas as pd

df2 = pd.read_csv("data/PreferenceRatings.csv", delimiter=';')

headphone_curves = df2['HeadphoneCurve'].unique()
fig, ax = plt.subplots(figsize= (24, 8))
violin_data = [df2[df2['HeadphoneCurve'] == curve]['Rating'] for curve in headphone_curves]
violin_parts = ax.violinplot(violin_data,showmeans=True, showextrema=False, showmedians=False, widths=0.85)
ax.set_xticks(range(1, len(headphone_curves) + 1))
ax.set_xticklabels(headphone_curves, rotation=45)

for pc in violin_parts['bodies']:
    pc.set_color('azure')
    pc.set_edgecolor('black')
    pc.set_alpha(1)

quartile1, medians, quartile3 = np.percentile(violin_data, [25, 50, 75], axis=1)
inds = np.arange(1, len(medians) + 1)
ax.scatter(inds, medians, marker='o', color='steelblue', s=50, zorder=3)
ax.vlines(inds, quartile1, quartile3, color='skyblue', linestyle='-', lw=5)
 
Last edited:

MayaTlab

Addicted to Fun and Learning
Joined
Aug 15, 2020
Messages
947
Likes
1,570
Yes, a simple boxplot provides the 25, 50(median), 75 percentile values.

Violinplot is an alternative to get a better view of the actual rating distributions. Here is a violinplot showing the distributions, the means, the medians and the 25 and 75 percentiles of the ratings at the same time.

View attachment 318687

The HP3 curves remain the most satisfactory targets, regardless of the parameter being considered.

You can reproduce it in Python3.
Python:
import matplotlib.pyplot as plt
import pandas as pd

df2 = pd.read_csv("data/PreferenceRatings.csv", delimiter=';')

headphone_curves = df2['HeadphoneCurve'].unique()
fig, ax = plt.subplots(figsize= (24, 8))
violin_data = [df2[df2['HeadphoneCurve'] == curve]['Rating'] for curve in headphone_curves]
violin_parts = ax.violinplot(violin_data,showmeans=True, showextrema=False, showmedians=False, widths=0.85)
ax.set_xticks(range(1, len(headphone_curves) + 1))
ax.set_xticklabels(headphone_curves, rotation=45)

for pc in violin_parts['bodies']:
    pc.set_color('azure')
    pc.set_edgecolor('black')
    pc.set_alpha(1)

quartile1, medians, quartile3 = np.percentile(violin_data, [25, 50, 75], axis=1)
inds = np.arange(1, len(medians) + 1)
ax.scatter(inds, medians, marker='o', color='steelblue', s=50, zorder=3)
ax.vlines(inds, quartile1, quartile3, color='skyblue', linestyle='-', lw=5)

Thanks again. I have seen enough box plots that did not adhere to these definitions that I wasn't sure what I was looking at.
Generally speaking, it seems that Harman's method for processing preferential data is to find the mean across all listeners and test tracks.
 

MayaTlab

Addicted to Fun and Learning
Joined
Aug 15, 2020
Messages
947
Likes
1,570
Maybe the answers could only be avaliable in the final paper.

There's a lot riding on the methodology used to know how to interpret the data indeed.

The csv file containing the FR curves might have some errors, I and others have noticed that there may be a shift in values on the x axis for some traces. Hopefully this only happened when publishing the csv file to Zenodo and not during the actual study.
 

ThomasXia

Member
Joined
Mar 6, 2019
Messages
39
Likes
96
There's a lot riding on the methodology used to know how to interpret the data indeed.

The csv file containing the FR curves might have some errors, I and others have noticed that there may be a shift in values on the x axis for some traces. Hopefully this only happened when publishing the csv file to Zenodo and not during the actual study.

In fact, these may not be shifts errors in the x axis as they look like, but the actual frequency responses of these headphones!!!!

Maybe it is just the fact that some headphones with responses deviating from usual frequency targets heavily are still preferred by the listeners.

Just for example, HP3 vs APM:

untitled.png

reference: https://www.soundguys.com/apple-airpods-max-review-44975/

Update2, Example2, HP4 vs DT700PX. (Deviations look like a quality control or positioning issue more than a simple horizontal shift on x axis.)
untitled02.png
 
Last edited:

Thomas savage

Grand Contributor
The Watchman
Forum Donor
Joined
Feb 24, 2016
Messages
10,260
Likes
16,305
Location
uk, taunton
One challenge with the conference is that a number of talks are concurrent. So not possible to attend all of them.

On my session, they put one of the top Pro youtubers in parallel with mine!
I'm almost sure that's a unintentional and regrettable oversight in scheduling .., almost .
 

MayaTlab

Addicted to Fun and Learning
Joined
Aug 15, 2020
Messages
947
Likes
1,570
In fact, these may not be shifts errors in the x axis as they look like, but the actual frequency responses of these headphones!!!!

Maybe it is just the fact that some headphones deviating from usual frequency targets heavily are still preferred by the listeners.

Just for example:

View attachment 318719
reference: https://www.soundguys.com/apple-airpods-max-review-44975/

To me it seems that not all the curves are affected, and the APM one indeed isn't.

From what I understand the APHarm targets originated from that thread : https://audiosciencereview.com/foru...easurements-using-brüel-kjær-5128-hats.15352/
Mad_Economist has noticed that the curves in the csv file are shifted on the x axis relative to the curves in that thread, and that they match nearly perfectly when you correct for the shift.

In a similar way, using the list of headphones featured in that study :

Screenshot 2023-10-13 at 22.31.40.png

I'm starting to think that a plausible matching list would be something like this :
  • HP1 = Epos Adapt 600
  • HP2 = jabra
  • HP3 = apm
  • HP4 = 700proX
  • HP5 = Aonic
  • HP6 = Q45
  • HP7 = XM4
  • HP8 = B&O HX
For some of these, some recognisable features of their FR, particularly the ones that are quite insensitive to positional variation or fixture, seem shifted a bit.

I don't have the time this WE to spend too much time writing here. But, for example, the DT 700 Pro X features a dip at 4kHz that you'll find on all fixtures and that is quite resilient to positional variation, ex here : https://www.soundguys.com/beyerdynamic-dt-700-pro-x-review-60868/
If it is indeed HP4 (and I think that one would struggle to make the case that it isn't), then this is what we get from the csv file :
Screenshot 2023-10-13 at 22.35.25.png

For some of the other ones these individual features also match better once you scale / shift the FR on the X axis a bit.

Re the question of the preference scores, I don't have a particularly strong opinion on them as we have no idea of the methodology used, or even, if these shifts happened during the study, of what was actually evaluated.
 

ThomasXia

Member
Joined
Mar 6, 2019
Messages
39
Likes
96
For some of the other ones these individual features also match better once you scale / shift the FR on the X axis a bit.
This difference may simply be due to individual variations in quality control or positioning as in the updated figure above, since one single shift do not promise a perfect match.

Thanks for figuring out all these models, really a good job.;)
 
Last edited:

MayaTlab

Addicted to Fun and Learning
Joined
Aug 15, 2020
Messages
947
Likes
1,570
This difference may simply be due to individual variations in quality control or positioning as in the updated figure above, since one single shift do not promise a perfect match.

Not for that 4kHz dip :D. And neither for the response in the feedback range for some of the ANC headphones, or some other individual features.
 

beefkabob

Major Contributor
Forum Donor
Joined
Apr 18, 2019
Messages
1,653
Likes
2,093
One challenge with the conference is that a number of talks are concurrent. So not possible to attend all of them.

On my session, they put one of the top Pro youtubers in parallel with mine!
Totally different audiences. :p
 

Travis

Senior Member
Joined
Jul 9, 2018
Messages
455
Likes
552
I don't think they have a specific agenda and the work simply depends on interest of the students and faculty.
Highly doubtful. They are a research university, so they rely on funding from Canadian Govt. and private industry for research. The agenda probably depends on what interests those providing the funding. They have an entire lab devoted to nothing but emersive audio,

I’m sure the lab director can guide where the research goes, the next step (and it might even interest faculty and students) but they are accountable to whichever research council is providing them with funds to do research.

There are 22 people signed up for my session. So I am out of the woods as far as presenting to an empty room. :)
I think you will end up with more than that. I think you drew an excellent time slot and a good track.

Did I read that correctly, all of the presentations on you track are 15 minutes each?
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,597
Likes
239,667
Location
Seattle Area
Did I read that correctly, all of the presentations on you track are 15 minutes each?
Yes. Indeed majority of the talks are 15 minutes. I think only a couple are 30.
 

Travis

Senior Member
Joined
Jul 9, 2018
Messages
455
Likes
552
Yes. Indeed majority of the talks are 15 minutes. I think only a couple are 30.
Well that will work to your advantage based on your YouTube videos. You can get 2x the info. in 1/2 time as most.
 

Sean Olive

Senior Member
Audio Luminary
Technical Expert
Joined
Jul 31, 2019
Messages
334
Likes
3,063
View attachment 318109
It might be easier to interpret if you order the mean ratings from highest to lowest - I did this and removed the 2-3 modifications they made to the 8 headphones as I was less interesting in that , although it might be the focus of a different question,

The graph below is the mean based on all ratings ( all listeners, programs, repeats). They didn't code the data according to location (Dane vs Japanese) so we can't analyze it based on culture.

The ratings of the headphone curves is remarkably similar to our work in terms of Harman Target being preferred, diffuse and free field rated progressively lower. It is amazing to me and one of the co-authors I spoke with that two separate experiments using different subjects and methodologies can some up with similar conclusions.

They did use two of our standard tracks (Jennifer Warnes "bird on a wire" and Tracy Chapman "FastCar" so there was that commonality in methods. They also used a virtual headphone method applying equalization filters to a replicator headphone so that rapid A/B comparisons can be made using a MUSHRA method. They used a closed headphone which to me is surprising because open backs generally eliminate the issue of leakage among subjects.

1697569323029.png
 
Last edited:

alumnicesar

Active Member
Joined
Jul 17, 2023
Messages
138
Likes
71
It might be easier to interpret if you order the mean ratings from highest to lowest - I did this and removed the 2-3 modifications they made to the 8 headphones as I was less interesting in that , although it might be the focus of a different question,

View attachment 319570
finally, the original way it was presented is horrible lmao. Thanks :)

is there a way to isolate how the soundguys graph looks in comparison to hoe the estimated harman 2018 v2 graph looks? Not the preference rating but rather just where it differs in frequency
 
Last edited:

Sean Olive

Senior Member
Audio Luminary
Technical Expert
Joined
Jul 31, 2019
Messages
334
Likes
3,063
It might be easier to interpret if you order the mean ratings from highest to lowest - I did this and removed the 2-3 modifications they made to the 8 headphones as I was less interesting in that , although it might be the focus of a different question,

The graph below is the mean based on all ratings ( all listeners, programs, repeats). They didn't code the data according to location (Dane vs Japanese) so we can't analyze it based on culture.

The ratings of the headphone curves is remarkably similar to our work in terms of Harman Target being preferred, diffuse and free field rated progressively lower. It is amazing to me and one of the co-authors I spoke with that two separate experiments using different subjects and methodologies can some up with similar conclusions.

View attachment 319570
Here is a table of the means, median, etc. in descending order from highest to lowest mean.


1697571174743.png
 

Sean Olive

Senior Member
Audio Luminary
Technical Expert
Joined
Jul 31, 2019
Messages
334
Likes
3,063
AES conference program is finally available. I thought I post it for members to see what the buzz is about and whether there are any talks of interest: https://aesny23.sched.com/

AES-23_1200x675.png.webp


You might see some familiar names there from ASR. :)

I am especially interested in the talk about B&K 5128 headphone preference curves/listening tests. Author had contacted me in advance about it.
I will be attending.. See you there..
 
Top Bottom