Hydrogenaudio Listening Tests: Difference between revisions
(Remove link to Lossless test) |
(Undo revision 35480 by 208.127.232.205 (talk) That's a private listening test) |
||
(59 intermediate revisions by 17 users not shown) | |||
Line 1: | Line 1: | ||
One of the most visible ways that HydrogenAudio members have shown their enthusiasm for audio quality and their respect for [[TOS 8|scientific principles]] over the years is by organizing and participating in public blind listening tests of lossy codecs and presenting statistical analyses of the results. These results show the advantages and disadvantages of different formats and encoders and help identify which are "state of the art." | |||
[[Private Listening Tests|Private listening tests]] also provide useful data, but the public HydrogenAudio tests are a rarity in having enough participants to both average out variations among individual listeners and still have statistically significant results. | |||
Note that a listening test cannot really compare formats or encoder "brands." Instead, it compares particular versions of particular encoders at particular bitrates. Results will differ with different encoders, newer versions of the same encoders, or different bitrates. Use caution when extrapolating, and use the most recent results when extrapolating to the newest encoder versions. | |||
'''Note:''' There | '''Note:''' There would be no point in listening tests of [[lossless]] encoders; the decoded results are bit-identical to the original uncompressed recordings. [[Lossless comparison|Lossless encoder comparisons]] instead consider technical aspects such as compression ratio and encode/decode speed. | ||
All tests are sorted by date, newest first. All dates are in [http://www.cl.cam.ac.uk/~mgk25/iso-time.html ISO8601 format]. | |||
= | ==Tests and Results== | ||
{|class="wikitable" align="right" | |||
!colspan=2| Result Symbol Key | |||
|- | |||
{| | | > || Better than | ||
|- | |||
|- | |- | ||
| ~ || About the same as | |||
|- | |- | ||
| ~> || About the same, but slightly better than | |||
| | |||
|} | |} | ||
===Multiformat Tests=== | |||
== | |||
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;" | {| border="1" cellpadding="2" cellspacing="0" style="text-align:center;" | ||
|- style="background:# | |- style="background:#aaffff" | ||
! Date !! Title<br>Samples !! Performed by !! | ! Date !! Title<br>Samples !! Performed by !! Codecs !! Result !! Links | ||
|- | |- | ||
| 2014-07 | |||
| [[ | | 96-kbps<br>General | ||
| Kamedo2 | |||
| [[Apple AAC]] (iTunes 11.2.2), [[Opus]] (1.1),<br>[[Vorbis]] ([[aoTuV]] Beta6.03), [[MP3]] ([[LAME]] 3.99.5) | |||
| Opus > Apple ><br />MP3(V5) ~ Vorbis | |||
| [http://www.hydrogenaud.io/forums/index.php?showtopic=106354 HA Thread]<br>[http://listening-test.coresv.net/results.htm Complete Result] | |||
|- | |- | ||
| 2011-04-10 | |||
| [[ | | 64-kbps<br>General | ||
| IgorC | |||
| [[Vorbis]] [[aoTuV]], [[QuickTime AAC|Quicktime]] HE-[[AAC]], [[Opus]] | |||
| Opus > Apple ><br />Nero ~ Vorbis | |||
| [http://www.hydrogenaud.io/forums/index.php?showtopic=87785 HA Thread]<br>[http://listening-tests.hydrogenaud.io/igorc/results.html Complete Result] | |||
|- | |- | ||
| 2007-08-16 | |||
| [[ | | 64-kbps<br>General | ||
| Sebastian Mares | |||
| [[Vorbis]] [[aoTuV]], [[Nero AAC]],<br>[[WMA]] Pro | |||
| Nero > WMA Pro ~ Vorbis | |||
| [http://www.hydrogenaud.io/forums/index.php?showtopic=56397 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-64-1/results.htm Complete Result] | |||
|- | |||
| 2006-11-22 | |||
| 48-kbps<br>General | |||
| Sebastian Mares | |||
| [[Vorbis]] [[aoTuV]], [[Nero AAC]],<br>[[WMA]] Pro, [[WMA]] Standard | |||
| Nero > WMA Pro ~ Vorbis > <br /> WMA Standard | |||
| [http://www.hydrogenaud.io/forums/index.php?showtopic=50408 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-48-1/results.htm Complete Result] | |||
|- | |- | ||
| 2006-01-13 | |||
| [[LAME]] | | 128-kbps<br>General | ||
| Sebastian Mares | |||
| [[QuickTime AAC]], [[Nero AAC]], [[LAME]] [[MP3]],<br>[[Vorbis]] [[aoTuV]], [[WMA]] Pro | |||
| 5-way tie | |||
| [http://www.hydrogenaud.io/forums/index.php?showtopic=40607&view=findpost&p=357117 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-128-1/results.htm Complete Result] | |||
|} | |} | ||
===[[AAC]] Tests=== | |||
==[[ | |||
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;" | {| border="1" cellpadding="2" cellspacing="0" style="text-align:center;" | ||
|- style="background:# | |- style="background:#ffaaaa" | ||
! Date !! Title<br>Samples !! Performed by | ! Date !! Title<br>Samples !! Performed by !! Codecs !! Result !! Links | ||
|- | |- | ||
| 2011-08-20 | |||
| | | 96-kbps <br>General | ||
| IgorC | |||
| CT, [[Fraunhofer|FhG AAC]] via Winamp (not [[Fraunhofer FDK AAC|FDK]]), [[Nero AAC|Nero]], [[QuickTime AAC|QuickTime]] TVBR & CVBR | |||
| Quicktime CVBR ~> TVBR ~><br /> Fraunhofer > CT > Nero | |||
| [http://www.hydrogenaud.io/forums/index.php?showtopic=89765 HA Thread]<br>[http://listening-tests.hydrogenaud.io/igorc/aac-96-a/results.html Complete result] | |||
|- | |- | ||
| 2006-02-20 | |||
| | | 48-kbps HE-AAC<br>General | ||
| Gabriel | |||
| 3gpp (v1), CT (v1 & v2), [[Nero AAC|Nero]] (v1 & v2) | |||
| All tie for most sample types;<br /> otherwise any v1 > any v2 | |||
| [http://www.hydrogenaud.io/forums/index.php?showtopic=42679&hl= HA Thread]<br>[http://www.mp3-tech.org/tests/aac_48/results.html Complete Result] | |||
|- | |- | ||
| 2006-02-05 | |||
| 48-kbps HE-AAC<br>Classical | |||
| Ivan Dimkovic | |||
| [[Nero AAC|Nero]] HE-AAC v1 & v2 | |||
| | |||
| [http://www.hydrogenaud.io/forums/index.php?showtopic=41191&hl= HA Thread] | |||
|} | |} | ||
===[[MP3]] Tests=== | |||
==[[ | |||
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;" | {| border="1" cellpadding="2" cellspacing="0" style="text-align:center;" | ||
|- style="background:# | |- style="background:#aaffaa" | ||
! Date !! Title<br>Samples !! Performed by !! | ! Date !! Title<br>Samples !! Performed by !! Codecs !! Result !! Links | ||
|- | |- | ||
| 2008-10-17 | |||
| [[ | | 130-kbps<br>General | ||
| Sebastian Mares | |||
| [[LAME]] 3.97, [[LAME]] 3.98.2, [[iTunes]], [[Fraunhofer|FhG MP3]], [[Xing|Helix]] | |||
| | |||
| {{ha|https://hydrogenaud.io/index.php/topic,66564.0.html|Thread}}<br>[http://listening-tests.hydrogenaud.io/sebastian/mp3-128-1/results.htm Complete Result] | |||
|} | |} | ||
==External Tests== | |||
Other people have held listening tests in venues outside of Hydrogenaudio (although the organizers of the tests tend to be HA members :). The results of these tests are found below: | |||
* [https://web.archive.org/web/20070720024213/http://www.ff123.net/ ff123's site on audio testing and listening tests] (most recent test 2002-07-27) | |||
* [https://web.archive.org/web/20071014001946/http://www.rjamorim.com/test/ Roberto's Public Listening Tests] (most recent test 2004-07-11) | |||
* [http://listening-tests.hydrogenaud.io/sebastian/ Sebastian Mares' Listening Tests] (most recent test updated 2008-11-22) | |||
* [http://www.soundexpert.info/ SoundExpert Listening Tests] (continually updated) | |||
[ | |||
[[Category:Listening Tests]] | |||
Latest revision as of 05:18, 5 March 2024
One of the most visible ways that HydrogenAudio members have shown their enthusiasm for audio quality and their respect for scientific principles over the years is by organizing and participating in public blind listening tests of lossy codecs and presenting statistical analyses of the results. These results show the advantages and disadvantages of different formats and encoders and help identify which are "state of the art."
Private listening tests also provide useful data, but the public HydrogenAudio tests are a rarity in having enough participants to both average out variations among individual listeners and still have statistically significant results.
Note that a listening test cannot really compare formats or encoder "brands." Instead, it compares particular versions of particular encoders at particular bitrates. Results will differ with different encoders, newer versions of the same encoders, or different bitrates. Use caution when extrapolating, and use the most recent results when extrapolating to the newest encoder versions.
Note: There would be no point in listening tests of lossless encoders; the decoded results are bit-identical to the original uncompressed recordings. Lossless encoder comparisons instead consider technical aspects such as compression ratio and encode/decode speed.
All tests are sorted by date, newest first. All dates are in ISO8601 format.
Tests and Results
Result Symbol Key | |
---|---|
> | Better than |
~ | About the same as |
~> | About the same, but slightly better than |
Multiformat Tests
Date | Title Samples |
Performed by | Codecs | Result | Links |
---|---|---|---|---|---|
2014-07 | 96-kbps General |
Kamedo2 | Apple AAC (iTunes 11.2.2), Opus (1.1), Vorbis (aoTuV Beta6.03), MP3 (LAME 3.99.5) |
Opus > Apple > MP3(V5) ~ Vorbis |
HA Thread Complete Result |
2011-04-10 | 64-kbps General |
IgorC | Vorbis aoTuV, Quicktime HE-AAC, Opus | Opus > Apple > Nero ~ Vorbis |
HA Thread Complete Result |
2007-08-16 | 64-kbps General |
Sebastian Mares | Vorbis aoTuV, Nero AAC, WMA Pro |
Nero > WMA Pro ~ Vorbis | HA Thread Complete Result |
2006-11-22 | 48-kbps General |
Sebastian Mares | Vorbis aoTuV, Nero AAC, WMA Pro, WMA Standard |
Nero > WMA Pro ~ Vorbis > WMA Standard |
HA Thread Complete Result |
2006-01-13 | 128-kbps General |
Sebastian Mares | QuickTime AAC, Nero AAC, LAME MP3, Vorbis aoTuV, WMA Pro |
5-way tie | HA Thread Complete Result |
AAC Tests
Date | Title Samples |
Performed by | Codecs | Result | Links |
---|---|---|---|---|---|
2011-08-20 | 96-kbps General |
IgorC | CT, FhG AAC via Winamp (not FDK), Nero, QuickTime TVBR & CVBR | Quicktime CVBR ~> TVBR ~> Fraunhofer > CT > Nero |
HA Thread Complete result |
2006-02-20 | 48-kbps HE-AAC General |
Gabriel | 3gpp (v1), CT (v1 & v2), Nero (v1 & v2) | All tie for most sample types; otherwise any v1 > any v2 |
HA Thread Complete Result |
2006-02-05 | 48-kbps HE-AAC Classical |
Ivan Dimkovic | Nero HE-AAC v1 & v2 | HA Thread |
MP3 Tests
Date | Title Samples |
Performed by | Codecs | Result | Links |
---|---|---|---|---|---|
2008-10-17 | 130-kbps General |
Sebastian Mares | LAME 3.97, LAME 3.98.2, iTunes, FhG MP3, Helix | Thread on hydrogenaudio Complete Result |
External Tests
Other people have held listening tests in venues outside of Hydrogenaudio (although the organizers of the tests tend to be HA members :). The results of these tests are found below:
- ff123's site on audio testing and listening tests (most recent test 2002-07-27)
- Roberto's Public Listening Tests (most recent test 2004-07-11)
- Sebastian Mares' Listening Tests (most recent test updated 2008-11-22)
- SoundExpert Listening Tests (continually updated)