AI Thread Summarization Project (LLMs)

[NicoVtt]

Hello everyone,

we are a team of two computer science master students from Germany and are working on a project summarizing threads in the ASR forum using Large Language Models. We already presented some results to the community in the last couple of weeks (Summarization of threads using LLMs (AI), Improved thread summaries using AI (LLM)).

The project period is coming to an end and we wanted to present the latest results of our proof of concept. Our approach now includes references to posts from which the content of a section of the LLM-generated summary was taken. We were able to drastically improve the length of some of the summaries. This had a positive effect on the quality of longer threads in particular and also allowed us to include more potentially useful information for readers in general.

Our approach:

We use a new iterative approach: We divide large threads into evenly sized chunks to fit into the model context window. A first summary for the first chunk is generated and we use each subsequent chunk of messages to add new (relevant) information to the summary. The summary is supplemented and refined step by step. This seems to be the most promising approach we could find. As time is limited, this is the approach we will continue to improve in order to achieve a reasonably usable quality of summaries.

As this is only a proof of concept, there is of course still some work to do before the system could be integrated into the forum.

Below you will find two examples of generated summaries.

Even though our project is coming to an end, we look forward to your feedback!

Thank you in advance for your opinion!

Fabian (MatrixRave) & Nico (NicoVtt)

Audio-Technica ATH-R70x Review (Headphone) [Llama 70B instruct]

The conversation revolves around headphones, specifically the Audio-Technica R70x model. Users discuss their experiences with the headphones, including comfort, sound quality, and power requirements. Some users express surprise at the high distortion levels in the low frequencies, while others defend the headphones' sound quality, citing their comfortable fit and balanced sound signature. The discussion also touches on the Audio-Technica wing system, which some users find comfortable and others do not. The importance of proper fitting is emphasized, as a poor fit can lead to discomfort and affect sound quality. Users share their experiences with other headphones, including Sennheiser models, and compare them to the R70x. Some users appreciate the R70x's neutral sound signature, while others prefer different models. One user shares their positive experience with the R70x, mentioning that they solved comfort issues by trying different ear pads and recommends specific alternatives. The conversation also includes humorous exchanges, such as a joke about headphones being able to take a bullet for you, and an April Fools' Day prank. Additionally, users appreciate Audio-Technica's willingness to experiment with different tuning options and designs, even if they don't always work out. In terms of technical aspects, the discussion highlights that the R70x has a flat frequency response, making it suitable for reference listening. However, comfort and portability are areas for improvement. The review includes a custom EQ setting to improve the sound quality, and users note that the headphones require a high-impedance amplifier to drive them properly. Some users are concerned about bass distortion, but others argue that it's not audible at normal listening levels.

The conversation also delves into measurement techniques, with users discussing the importance of using Harman correction and GRAS conversion calibration files to get accurate SPL readings. A user shares their experiment with measuring peak values and clarifies any misunderstandings about LzPeak representation.

Furthermore, users discuss amplifier options for driving the R70x headphones, with some considering upgrading from the Scarlett Solo Gen 3 and Behringer HA 400 amplifier setup. They receive advice on choosing an amplifier, warning about the limitations of the Behringer HA 400, and opinions on Samson QH 4, Personus HP4, and Behringer HA 400 amplifiers.

Additionally, users share their initial impressions of the R70X headphones, mentioning some issues with sound quality and build materials. They receive reassurance from other users that their concerns are normal. The discussion also touches on the build quality and appearance of the R70X headphones, with users sharing their experiences and offering advice. The conversation also explores alternative amplifier options, such as the Fiio K7, which is considered a powerful option for driving headphones. Portable DAC/amp options like the Qudelix 5K or Truthear Shio are also discussed as providing similar performance at a lower cost.

Users also discuss making balanced cables for their ATH-R70x headphones, with one user clarifying that the headphones can be wired balanced despite Audio-Technica's statement. The importance of proper maintenance is also emphasized, with users sharing concerns about getting moisture in their headphones and receiving reassurance from others. Finally, some users praise the ATH-R70x for mixing and gaming, finding them comfortable and accurate. Overall, the discussion is focused on the technical aspects of headphones, user experiences, and comparisons between different models. The balance of opinions leans towards a nuanced view of the R70x, acknowledging both its strengths and weaknesses.

64 Audio U12t Review (IEM) [Llama 70B instruct]

The conversation revolves around a review of an IEM (in-ear monitor) called U12T, which costs $2000. Many users express their disappointment and disagreement with the high price tag, citing that similar performance can be achieved at a much lower cost. Some users share their personal experiences with IEMs, mentioning that they are not appealing to them due to hygiene reasons or personal preferences. The discussion also touches on the topic of distortion in IEMs, with some users asking about the audibility of distortion at certain levels and how it affects the listening experience. Others mention alternative IEM options that adhere to the Harman target curve, such as the ThieAudio Monarch Mk2 and Moondrop Variations. Additionally, there are comments about the cost of production and profit margins in the audio industry, with some users sharing their experiences in manufacturing and highlighting the importance of considering factors like packaging, labor, and engineering costs when determining a product's price. The conversation also explores the challenges of testing IEMs, including ergonomics and fit issues, as well as the importance of considering multiple perspectives and data points when making purchasing decisions.

The discussion further delves into the value of the Harman curve for IEMs vs over-ear headphones, with some users arguing that it is less relevant or even dubious for IEMs. There are also concerns about the authenticity and quality of products from certain manufacturers or sellers, as well as praise for Apple's approach to IEM design. Some users express frustration with the current state of IEM designs, citing issues with ergonomics, microphonics, and high prices. Others argue that there are many options available, and that the market has improved significantly in recent years. PeteL disagrees with the notion that IEMs are boring in terms of design, pointing out that there are many variations in crossover systems, filter choices, and driver placements.

Reviewer amirm emphasizes the importance of objective measurements and correction filters to bring IEM responses closer to the Harman curve. They argue that without a standard target curve, audio reproduction is lost, and that statistical methods are essential for technological developments. Other users discuss the limitations of IEM designs, citing issues with ear tips, comfort, and sound quality.
The conversation also touches on the idea of trusting evidence-based opinions over consensus or guru authority, with some users praising Amir for being a "lonely voice" who challenges popular opinions and encourages contention of ideas. There is also discussion about the importance of considering individual preferences when evaluating IEMs, as well as the need for comparisons to be based on evidence and merits rather than consensus or authority. Custom IEMs are also discussed, with users praising their sound quality and value for live musicians on stage. The conversation shifts to the topic of overpricing and competition from Chinese manufacturers, with some users arguing that western brands like 64 Audio are overpriced and don't offer enough value for their high prices. Others defend the existence of western brands, citing the importance of local tech and manufacturing industries.

The discussion also explores the technical aspects of IEMs, including the use of dips around 3kHz to create a wider soundstage, treble peaks adjustments, and labor-intensive manufacturing processes. Users debate the sound quality of different IEMs, with some arguing that expensive IEMs don't necessarily sound better than cheaper ones. Others point out that technical factors like soundstage and imaging prowess can justify higher prices. Finally, users ask for recommendations on which IEMs to buy, with some suggesting that Moondrop variations or Truthear sets can outperform 64 Audio's offerings at a lower price point. Overall, the discussion highlights the complexities and nuances of evaluating audio products, particularly IEMs, and the importance of considering multiple perspectives and data points when making purchasing decisions. The balance of opinions lands on the high price point of some IEMs being questionable, with many users expressing disappointment and disagreement with the cost of the U12T. However, there are also significant minority opinions that argue for the value of certain IEMs and the importance of considering various factors when evaluating their performance. The additional context provided adds more insights into the discussion, particularly on the relationship between price and sound quality in IEMs. Users share their experiences with different IEM models, emphasizing the need for a balanced and informed approach when evaluating sound quality. They caution against relying on marketing hype or expectation bias and encourage each other to seek out objective measurements, blind testing, and open-minded listening experiences.

 

[MRC01]

Neat idea, and pragmatic. This could be useful for forum threads upon reaching a certain length. Yet the summaries themselves should also be limited in length.

 

[PatentLawyer]

After having someone just send me the "I Glued My Balls to My Butthole Again" AI-generated parody song from YouTube, this is a refreshing use of AI.

 

[MatrixRave]

Neat idea, and pragmatic. This could be useful for forum threads upon reaching a certain length. Yet the summaries themselves should also be limited in length.

We currently decided against a limitation in length because we had a lot of trouble with getting useful summaries (no hallucination and stuff like that).
Before this version, we had the problem that summaries were far too short.
But yes, for extremely large threads it would be necessary to limit the length of the summary.

 

[AdamG]

We currently decided against a limitation in length because we had a lot of trouble with getting useful summaries (no hallucination and stuff like that).
Before this version, we had the problem that summaries were far too short.
But yes, for extremely large threads it would be necessary to limit the length of the summary.

Or have a TLDR summary and a deeper dive analysis if the reader wants more meat.

 

[MatrixRave]

Or have a TLDR summary and a deeper dive analysis if the reader wants more meat.

Thats also a nice idea. Will add this to the list of improvements

 

[AdamG]

Can you incorporate hyperlinks that take the reader to the original thread? Did I miss this?

Did you decide to exclude the ASR Community Assessment Vote Results? Maybe as a footnote?

FWIIW, I think many members will like using this and not having to drudge through hundreds of pages in some cases. Excellent work Sirs.

 

[MatrixRave]

Can you incorporate hyperlinks that take the reader to the original thread? Did I miss this?

Did you decide to exclude the ASR Community Assessment Vote Results? Maybe as a footnote?

We added references to the summaries that take the ready to the section of the thread where the information is from. In the screenshot it's the blue text on the right side.
We decided against adding results and images because the LLM simply had to much problems with handling all the input and displaying it correctly. Images aren't event interpreted by llama3 and combining different approaches unfortunately was out of scope for the time we had.

Thank you, much appreciated.

 

[AdamG]

We added references to the summaries that take the ready to the section of the thread where the information is from. In the screenshot it's the blue text on the right side.
We decided against adding results and images because the LLM simply had to much problems with handling all the input and displaying it correctly. Images aren't event interpreted by llama3 and combining different approaches unfortunately was out of scope for the time we had.

Thank you, much appreciated.

Thank you for your responses. Have a wonderful day.

 

[MatrixRave]

Thank you for your responses. Have a wonderful day.

Your are welcome, and thanks for your input.
Have a wonderful day as well

 

[NicoVtt]

This is a version of the summary above of 64 Audio U12t Review (IEM) with hyperlinks included, as implemented in our frontend:

Post #1 to #48
The conversation revolves around a review of an IEM (in-ear monitor) called U12T, which costs $2000. Many users express their disappointment and disagreement with the high price tag, citing that similar performance can be achieved at a much lower cost. Some users share their personal experiences with IEMs, mentioning that they are not appealing to them due to hygiene reasons or personal preferences. The discussion also touches on the topic of distortion in IEMs, with some users asking about the audibility of distortion at certain levels and how it affects the listening experience. Others mention alternative IEM options that adhere to the Harman target curve, such as the ThieAudio Monarch Mk2 and Moondrop Variations. Additionally, there are comments about the cost of production and profit margins in the audio industry, with some users sharing their experiences in manufacturing and highlighting the importance of considering factors like packaging, labor, and engineering costs when determining a product's price. The conversation also explores the challenges of testing IEMs, including ergonomics and fit issues, as well as the importance of considering multiple perspectives and data points when making purchasing decisions.

Post #49 to #85
The discussion further delves into the value of the Harman curve for IEMs vs over-ear headphones, with some users arguing that it is less relevant or even dubious for IEMs. There are also concerns about the authenticity and quality of products from certain manufacturers or sellers, as well as praise for Apple's approach to IEM design. Some users express frustration with the current state of IEM designs, citing issues with ergonomics, microphonics, and high prices. Others argue that there are many options available, and that the market has improved significantly in recent years. PeteL disagrees with the notion that IEMs are boring in terms of design, pointing out that there are many variations in crossover systems, filter choices, and driver placements.

Post #86 to #119
Reviewer amirm emphasizes the importance of objective measurements and correction filters to bring IEM responses closer to the Harman curve. They argue that without a standard target curve, audio reproduction is lost, and that statistical methods are essential for technological developments. Other users discuss the limitations of IEM designs, citing issues with ear tips, comfort, and sound quality.

Post #120 to #138
The conversation also touches on the idea of trusting evidence-based opinions over consensus or guru authority, with some users praising Amir for being a "lonely voice" who challenges popular opinions and encourages contention of ideas. There is also discussion about the importance of considering individual preferences when evaluating IEMs, as well as the need for comparisons to be based on evidence and merits rather than consensus or authority. Custom IEMs are also discussed, with users praising their sound quality and value for live musicians on stage. The conversation shifts to the topic of overpricing and competition from Chinese manufacturers, with some users arguing that western brands like 64 Audio are overpriced and don't offer enough value for their high prices. Others defend the existence of western brands, citing the importance of local tech and manufacturing industries.

Post #139 to #193
The discussion also explores the technical aspects of IEMs, including the use of dips around 3kHz to create a wider soundstage, treble peaks adjustments, and labor-intensive manufacturing processes. Users debate the sound quality of different IEMs, with some arguing that expensive IEMs don't necessarily sound better than cheaper ones. Others point out that technical factors like soundstage and imaging prowess can justify higher prices. Finally, users ask for recommendations on which IEMs to buy, with some suggesting that Moondrop variations or Truthear sets can outperform 64 Audio's offerings at a lower price point. Overall, the discussion highlights the complexities and nuances of evaluating audio products, particularly IEMs, and the importance of considering multiple perspectives and data points when making purchasing decisions. The balance of opinions lands on the high price point of some IEMs being questionable, with many users expressing disappointment and disagreement with the cost of the U12T. However, there are also significant minority opinions that argue for the value of certain IEMs and the importance of considering various factors when evaluating their performance. The additional context provided adds more insights into the discussion, particularly on the relationship between price and sound quality in IEMs. Users share their experiences with different IEM models, emphasizing the need for a balanced and informed approach when evaluating sound quality. They caution against relying on marketing hype or expectation bias and encourage each other to seek out objective measurements, blind testing, and open-minded listening experiences.

 

[amirm]

Wow, you have made massive improvement relative to original prototype! Given the disparity of comments we get in these threads, even a human may have a hard time summarizing the thread.

I only found one technical mistake saying I recommended high-impedance headphone amplifier. It should be the opposite.

 

[RayDunzl]

Can I get it to analyze my posts and then create replies in my style so I don't have to?

 

[amirm]

Now there is an idea... AI OOF!

 

[badie]

haha ... was just fighting my way through the "Fosi V3 Mono Amplifier Review Thread", thinking about a KI summarizing ...

Did a search for "summarizing forum Threads with KI" - one of the first results was the project page https://ki.hs-kl.de/summarizing-asr-forum-threads-using-large-language-models/ ... what the heck - someone developing something and was choosing ASR as their playground?

Great!! Wondering why this didn´t get more attention. This would make life so much easier

 

[ferrellms]

This cool - tracking threads in a forum!

Of course, ChatGPT4o has read ASR and lots more besides, and some of it is more accurate than ASR which has some posts that "hallucinate".

 

[pollock0424]

This is a neat idea, I have been thinking something like this for the tapeheads forum. May be not...I have too much on my plate.

Another idea that I have is to train a multimodal network(images and text) from various audio/electronics forums. A newbie member could interact with this agent before asking a dumb question for the 100th time.

 

[hollis]

Awesome work!
What are you working on next?

 

[Old_School_Brad]

So, is there anyone who would be the superhero we (I) need and compile the repository into an executable?