Difference between revisions of "Hydrogenaudio Listening Tests"

From Hydrogenaudio Knowledgebase
Jump to: navigation, search
(Multiformat Tests: +latest)
(MP3 Tests: - ha.io thread link updated)
(15 intermediate revisions by 6 users not shown)
Line 9: Line 9:
 
All tests are sorted by date, newest first. All dates are in [http://www.cl.cam.ac.uk/~mgk25/iso-time.html ISO8601 format].
 
All tests are sorted by date, newest first. All dates are in [http://www.cl.cam.ac.uk/~mgk25/iso-time.html ISO8601 format].
  
=Multiformat Tests=
+
==Tests and Results==
 +
{|class="wikitable" align="right"
 +
!colspan=2| Result Symbol Key
 +
|-
 +
| > || Better than
 +
|-
 +
| ~ || About the same as
 +
|-
 +
| ~> || About the same, but slightly better than
 +
|}
 +
 
 +
===Multiformat Tests===
  
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
 
|- style="background:#aaffff"
 
|- style="background:#aaffff"
! Date !! Title<br>Samples !! Performed by !! Codecs !! Links
+
! Date !! Title<br>Samples !! Performed by !! Codecs !! Result !!  Links
 
|-
 
|-
 
| 2014-07
 
| 2014-07
 
| 96-kbps<br>General
 
| 96-kbps<br>General
| IgorC
+
| Kamedo2
| [[QuickTime AAC]] (iTunes 11.2.2), [[Opus]] (1.1),<br>[[Vorbis]] ([[aoTuV]] Beta6.03), [[MP3]] ([[LAME]] 3.99.5)
+
| [[Apple AAC]] (iTunes 11.2.2), [[Opus]] (1.1),<br>[[Vorbis]] ([[aoTuV]] Beta6.03), [[MP3]] ([[LAME]] 3.99.5)
| [http://www.hydrogenaud.io/forums/index.php?showtopic=106354 HA Thread]<br>In progress
+
| Opus > Apple ><br />MP3(V5) ~ Vorbis
 +
| [http://www.hydrogenaud.io/forums/index.php?showtopic=106354 HA Thread]<br>[http://listening-test.coresv.net/results.htm Complete Result]
 
|-
 
|-
 
| 2011-04-10
 
| 2011-04-10
Line 25: Line 37:
 
| IgorC
 
| IgorC
 
| [[Vorbis]] [[aoTuV]], [[QuickTime AAC|Quicktime]] HE-[[AAC]], [[Opus]]
 
| [[Vorbis]] [[aoTuV]], [[QuickTime AAC|Quicktime]] HE-[[AAC]], [[Opus]]
 +
| Opus > Apple ><br />Nero ~ Vorbis
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=87785 HA Thread]<br>[http://listening-tests.hydrogenaud.io/igorc/results.html Complete Result]
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=87785 HA Thread]<br>[http://listening-tests.hydrogenaud.io/igorc/results.html Complete Result]
 
|-
 
|-
Line 31: Line 44:
 
| Sebastian Mares
 
| Sebastian Mares
 
| [[Vorbis]] [[aoTuV]], [[Nero AAC]],<br>[[WMA]] Pro
 
| [[Vorbis]] [[aoTuV]], [[Nero AAC]],<br>[[WMA]] Pro
 +
| Nero > WMA Pro ~ Vorbis
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=56397 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-64-1/results.htm Complete Result]
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=56397 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-64-1/results.htm Complete Result]
|-
+
|-  
 
| 2006-11-22
 
| 2006-11-22
 
| 48-kbps<br>General
 
| 48-kbps<br>General
 
| Sebastian Mares
 
| Sebastian Mares
 
| [[Vorbis]] [[aoTuV]], [[Nero AAC]],<br>[[WMA]] Pro, [[WMA]] Standard
 
| [[Vorbis]] [[aoTuV]], [[Nero AAC]],<br>[[WMA]] Pro, [[WMA]] Standard
 +
| Nero > WMA Pro ~ Vorbis > <br /> WMA Standard
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=50408 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-48-1/results.htm Complete Result]
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=50408 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-48-1/results.htm Complete Result]
 
|-
 
|-
Line 43: Line 58:
 
| Sebastian Mares
 
| Sebastian Mares
 
| [[QuickTime AAC]], [[Nero AAC]], [[LAME]] [[MP3]],<br>[[Vorbis]] [[aoTuV]], [[WMA]] Pro
 
| [[QuickTime AAC]], [[Nero AAC]], [[LAME]] [[MP3]],<br>[[Vorbis]] [[aoTuV]], [[WMA]] Pro
 +
| 5-way tie
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=40607&view=findpost&p=357117 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-128-1/results.htm Complete Result]
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=40607&view=findpost&p=357117 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-128-1/results.htm Complete Result]
 
|}
 
|}
  
=Single-format Tests=
+
===[[AAC]] Tests===
 
+
==[[AAC]]==
+
  
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
 
|- style="background:#ffaaaa"
 
|- style="background:#ffaaaa"
! Date !! Title<br>Samples !! Performed by !! Codecs !! Links
+
! Date !! Title<br>Samples !! Performed by !! Codecs !! Result !! Links
 +
|-
 +
| 2020-10-17
 +
| 128-kbps LC<br>General
 +
| guruboolez
 +
| [[Fraunhofer|FhG AAC]], [[Fraunhofer FDK AAC|FDK]], [[Nero AAC|Nero]], [[QuickTime AAC|QuickTime]], FFmpeg/Libavcodec
 +
| QuickTime > FhG, FDK > rest
 +
| [https://hydrogenaud.io/index.php?topic=120062.0 HA Thread]
 
|-
 
|-
 
| 2011-08-20
 
| 2011-08-20
 
| 96-kbps <br>General
 
| 96-kbps <br>General
 
| IgorC
 
| IgorC
| CT, Fraunhofer FhG via Winamp (not [[Fraunhofer FDK AAC|FDK]]), [[Nero AAC|Nero]], [[QuickTime AAC|QuickTime]] TVBR & CVBR
+
| CT, [[Fraunhofer|FhG AAC]] via Winamp (not [[Fraunhofer FDK AAC|FDK]]), [[Nero AAC|Nero]], [[QuickTime AAC|QuickTime]] TVBR & CVBR
 +
| Quicktime CVBR ~> TVBR ~><br /> Fraunhofer > CT > Nero
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=89765 HA Thread]<br>[http://listening-tests.hydrogenaud.io/igorc/aac-96-a/results.html Complete result]
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=89765 HA Thread]<br>[http://listening-tests.hydrogenaud.io/igorc/aac-96-a/results.html Complete result]
 
|-
 
|-
Line 64: Line 86:
 
| Gabriel
 
| Gabriel
 
| 3gpp (v1), CT (v1 & v2), [[Nero AAC|Nero]] (v1 & v2)
 
| 3gpp (v1), CT (v1 & v2), [[Nero AAC|Nero]] (v1 & v2)
 +
| All tie for most sample types;<br /> otherwise any v1 > any v2
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=42679&hl= HA Thread]<br>[http://www.mp3-tech.org/tests/aac_48/results.html Complete Result]
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=42679&hl= HA Thread]<br>[http://www.mp3-tech.org/tests/aac_48/results.html Complete Result]
 
|-
 
|-
Line 70: Line 93:
 
| Ivan Dimkovic
 
| Ivan Dimkovic
 
| [[Nero AAC|Nero]] HE-AAC v1 & v2
 
| [[Nero AAC|Nero]] HE-AAC v1 & v2
 +
|
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=41191&hl= HA Thread]
 
| [http://www.hydrogenaud.io/forums/index.php?showtopic=41191&hl= HA Thread]
 
|}
 
|}
  
==[[MP3]]==
+
===[[MP3]] Tests===
  
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
 
|- style="background:#aaffaa"
 
|- style="background:#aaffaa"
! Date !! Title<br>Samples !! Performed by !! Codecs !! Links
+
! Date !! Title<br>Samples !! Performed by !! Codecs !! Result !! Links
 
|-
 
|-
 
| 2008-10-17
 
| 2008-10-17
 
| 130-kbps<br>General
 
| 130-kbps<br>General
 
| Sebastian Mares
 
| Sebastian Mares
| [[LAME]] 3.97, [[LAME]] 3.98.2, [[iTunes]], [[FhG]], [[Helix]]
+
| [[LAME]] 3.97, [[LAME]] 3.98.2, [[iTunes]], [[Fraunhofer|FhG MP3]], [[Xing|Helix]]
| [http://www.hydrogenaud.io/forums/index.php?showtopic=66564 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mp3-128-1/results.htm Complete Result]
+
|
 +
| {{ha|https://hydrogenaud.io/index.php/topic,66564.0.html|Thread}}<br>[http://listening-tests.hydrogenaud.io/sebastian/mp3-128-1/results.htm Complete Result]
 
|}
 
|}
  
=External Tests=
+
==External Tests==
 
Other people have held listening tests in venues outside of Hydrogenaudio (although the organizers of the tests tend to be HA members :).  The results of these tests are found below:
 
Other people have held listening tests in venues outside of Hydrogenaudio (although the organizers of the tests tend to be HA members :).  The results of these tests are found below:
  
* <s> ff123's site on audio testing and listening tests</s> (most recent test 2002-07-27)
+
* [https://web.archive.org/web/20070720024213/http://www.ff123.net/ ff123's site on audio testing and listening tests] (most recent test 2002-07-27)
* <s> Roberto's Public Listening Tests</s> (most recent test 2004-07-11)
+
* [https://web.archive.org/web/20071014001946/http://www.rjamorim.com/test/ Roberto's Public Listening Tests] (most recent test 2004-07-11)
 
* [http://listening-tests.hydrogenaud.io/sebastian/ Sebastian Mares' Listening Tests] (most recent test updated 2008-11-22)
 
* [http://listening-tests.hydrogenaud.io/sebastian/ Sebastian Mares' Listening Tests] (most recent test updated 2008-11-22)
 
* [http://www.soundexpert.info/ SoundExpert Listening Tests] (continually updated)
 
* [http://www.soundexpert.info/ SoundExpert Listening Tests] (continually updated)

Revision as of 09:16, 6 December 2022

One of the most visible ways that HydrogenAudio members have shown their enthusiasm for audio quality and their respect for scientific principles over the years is by organizing and participating in public blind listening tests of lossy codecs and presenting statistical analyses of the results. These results show the advantages and disadvantages of different formats and encoders and help identify which are "state of the art."

Private listening tests also provide useful data, but the public HydrogenAudio tests are a rarity in having enough participants to both average out variations among individual listeners and still have statistically significant results.

Note that a listening test cannot really compare formats or encoder "brands." Instead, it compares particular versions of particular encoders at particular bitrates. Results will differ with different encoders, newer versions of the same encoders, or different bitrates. Use caution when extrapolating, and use the most recent results when extrapolating to the newest encoder versions.

Note: There would be no point in listening tests of lossless encoders; the decoded results are bit-identical to the original uncompressed recordings. Lossless encoder comparisons instead consider technical aspects such as compression ratio and encode/decode speed.

All tests are sorted by date, newest first. All dates are in ISO8601 format.

Tests and Results

Result Symbol Key
> Better than
~ About the same as
~> About the same, but slightly better than

Multiformat Tests

Date Title
Samples
Performed by Codecs Result Links
2014-07 96-kbps
General
Kamedo2 Apple AAC (iTunes 11.2.2), Opus (1.1),
Vorbis (aoTuV Beta6.03), MP3 (LAME 3.99.5)
Opus > Apple >
MP3(V5) ~ Vorbis
HA Thread
Complete Result
2011-04-10 64-kbps
General
IgorC Vorbis aoTuV, Quicktime HE-AAC, Opus Opus > Apple >
Nero ~ Vorbis
HA Thread
Complete Result
2007-08-16 64-kbps
General
Sebastian Mares Vorbis aoTuV, Nero AAC,
WMA Pro
Nero > WMA Pro ~ Vorbis HA Thread
Complete Result
2006-11-22 48-kbps
General
Sebastian Mares Vorbis aoTuV, Nero AAC,
WMA Pro, WMA Standard
Nero > WMA Pro ~ Vorbis >
WMA Standard
HA Thread
Complete Result
2006-01-13 128-kbps
General
Sebastian Mares QuickTime AAC, Nero AAC, LAME MP3,
Vorbis aoTuV, WMA Pro
5-way tie HA Thread
Complete Result

AAC Tests

Date Title
Samples
Performed by Codecs Result Links
2020-10-17 128-kbps LC
General
guruboolez FhG AAC, FDK, Nero, QuickTime, FFmpeg/Libavcodec QuickTime > FhG, FDK > rest HA Thread
2011-08-20 96-kbps
General
IgorC CT, FhG AAC via Winamp (not FDK), Nero, QuickTime TVBR & CVBR Quicktime CVBR ~> TVBR ~>
Fraunhofer > CT > Nero
HA Thread
Complete result
2006-02-20 48-kbps HE-AAC
General
Gabriel 3gpp (v1), CT (v1 & v2), Nero (v1 & v2) All tie for most sample types;
otherwise any v1 > any v2
HA Thread
Complete Result
2006-02-05 48-kbps HE-AAC
Classical
Ivan Dimkovic Nero HE-AAC v1 & v2 HA Thread

MP3 Tests

Date Title
Samples
Performed by Codecs Result Links
2008-10-17 130-kbps
General
Sebastian Mares LAME 3.97, LAME 3.98.2, iTunes, FhG MP3, Helix Thread on hydrogenaudio
Complete Result

External Tests

Other people have held listening tests in venues outside of Hydrogenaudio (although the organizers of the tests tend to be HA members :). The results of these tests are found below: