FLAC decoder testbench: Difference between revisions

Latest revision as of 06:52, 3 September 2024

To further the use and development of FLAC, a testbench containing files with different parameters was constructed. The testbench consists of 64 FLAC files, each one having a different combination of characteristics (like samplerate, bitdepth and number of channels) and FLAC features (like blocksize, maximum partition order, maximum predictor order, maximum predictor precision etc.) However, all combinations fall within the so called subset: a limited set of FLAC parameters that fit in the frame header and are supposed to be 'streamable'. Please be aware that certain tested features (such as blocksize 16 or a samplerate of 38457Hz) are pretty much useless in day-to-day usage, and that failure to properly decode such a file is mostly of interest to developers than to users.

The testbench can be downloaded here (revision 7) and is maintained here. It is licensed under CC0 1.0.

Revision 6 adds four new files (60 to 63); files 1 through 59 are the same as in revision 5. For more information on which file tests what characteristics and features, please refer to the readme.txt file. Revision 7 replaces file 63 (which was faulty in revision 6) and adds file 64.

	CDDA (44.1kHz/16-bit and less)					High-res (96kHz/24-bit and more)					Multichannel		Metadata			Various
Decoder	Blocksizes	Features	Samplerates	Bitdepths	Var. bl.size	General	Blocksize	Features	Samplerates	Bitdepths	General	Features	Missing	Extreme	MIME	Mono	Overfl.	Zero esc.	Remarks
Files in test set	1-10	11-18	19-21	22-23	24-27	28	29-30	31-32, 34	33-36	37	38-43	44	45-47	48-55	56-59	60	61-63	64
Albrecht DR 460 C	Plays	Plays	Fails	Fails	Plays	Skips					Fails	Skips	Plays	Fails	Plays	Skips	Fails	Skips
Android	Fails	Plays	Plays	Plays	Skips	Plays	Plays	Plays	Plays	Fails	Skips	Skips	Plays	Plays	Plays	Plays	Fails	Fails	Android 13 with Google Play updates of 1st of Febuary 2023 on Samsung XCover 5
Chromecast	Fails	Plays	Fails	Plays	Fails	Plays	Plays	Plays	Fails	Plays	Plays	Plays	Plays	Plays	Skips	Plays	Skips	Plays	Tested with system firmware 275994 and cast firmware 1.56.275994, testbench revision 7
Citroën DS5	Plays	Plays	Plays	Fails	Plays	Plays	Plays	Plays	Plays	Fails	Plays	Plays	Plays	Fails	Plays	Plays	Plays	Plays	Tested with system firmware 7.1.5-r1, model year 2018, testbench revision 7
Cowon iAudio 7	Freezes	Plays	Fails	Rejects	Rejects	Rejects	Rejects	Rejects	Rejects	Rejects	Freezes	Rejects	-	-	Plays	Plays	Rejects	Plays	Device doesn't support FLAC with bitdepth other than 16 and samplerate higher than 44.1 kHz. So actual features of files 45-54 and 62-63 cannot be tested. File 55 plays, but takes long time to start.
Ford SYNC	Plays	Plays	Plays	Skips	Plays	Plays	Plays	Plays	Plays	Skips	Plays	Plays	Plays	Skips	Plays	-	-	-	Tested with revision 5, SYNC version 3.3, build 18201_PRODUCT
JVC KD-R871BT	Fails	Plays	Fails	Plays	Rejects	Plays	Freezes	Reboots	Fails	Rejects	Rejects		Rejects	Reboots	Rejects	-	-	-	Heavy stutter with blocksize 16. Stretches non-standard samplerates. Samplerates above 96kHz, variable blocksize files and multichannel files 'not supported'. Freezes on blocksize 16384 and 13456. 192kHz is played back very slow. Skips all multichannel, files with cover image, extreme metadata. Reboot loop on extreme large VORBISCOMMENT
Kenwood KMM-BT304	Fails	Plays	Rejects	Plays	Rejects	Plays	Rejects	Rejects	Rejects	Plays	Rejects	Rejects	Plays	Reboots	Plays	-	-	-	File 03 plays extremely noisily, barely recognizable. File 51 produces a "NO DEVICE" error and a reboot. File 54 produces a reboot with no message. These files produce a "NA FILE" error and the player skips to the next: 19 21 24 25 26 27 29 30 31 34 35 36 38 39 40 41 42 43 44 45 46 48 55. (firmware: te1706k.krm from 2020-03-05)
Lenco Xemio-240	Plays	Plays	Freezes	Plays	Rejects								Plays	Plays	Plays	Plays	Fails	-	Tested with revision 6, firmware version V5.15, dated 2020-04-24
Mazda Connect	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Tested with revision 7, CX-90 MY2024
Onkyo TX-NR686	Fails	Plays	Rejects	Plays	Rejects	Plays	Plays	Plays	Rejects	Plays	Rejects		Plays	Unresp.	Plays	Plays	Plays	Plays	Tested with revision 6, firmware version 1071-4000-1070-0011.
Panasonic TX-60CX750E	Plays	Plays	Fails	Plays	Plays	Plays	Plays	Plays	Fails	Plays	Plays	Fails	Plays	Fails	Plays	Plays	Plays	-	Based on libFLAC 1.2.1, according to copyright notice. Firmware version 3.228, testbench revision 6
Pioneer BDP-150	Freezes	Fails	Fails	Plays	Freezes	Plays	Freezes	Fails	Fails	Plays	Freezes		Freezes	Freezes	Plays	Plays	Fails	-	Firmware v1.08, testbench revision 6
Pioneer DEH-1900UB	Skips	Plays	Skips	Skips	Skips	Skips							Plays	Other	Plays	-	-	-	Testbench revision 5
Rockbox	Plays	Plays	Plays	Plays	Plays	Plays	Skips	Skips	Plays	Plays	Freezes		Freezes	Fails	Plays	Plays	Fails	Fails	Tested with Rockbox 3.15 (latest stable) on Sansa Fuze with testbench revision 7
Samsung F5100WWB	Crashes	Fails	Fails	Plays	Freezes	Plays	Skips	Fails	Fails	Plays	Freezes		Plays	Freezes	Plays	Plays	Fails	Plays	Testbench revision 7
Sony BDP-1700	Freezes	Fails	Fails	Plays	Freezes	Plays	Freezes	Fails	Fails	Plays	Freezes		Skips	Plays	Plays	Plays	Fails	-	Firmware m30.R.0527, testbench revision 6
Toyota Entune 3.0	Plays	Plays	Skips	Plays	Plays	Plays	Plays	Plays	Skips	Plays	Skips	Skips	Skips	Skips	Plays	Plays	Plays	Plays	Testbench revision 7
Software players
AIMP 5.03 build 2398 (Windows)	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Fails	Testbench revision 7.
Album Player 2.114 (Windows)	Plays	Plays	Plays	Fails	Plays	Plays	Plays	Plays	Plays	Rejects	Rejects	Rejects	Plays	Plays	Plays	Plays	Rejects	Plays	Testbench revision 7.
Amarok (Windows)	Fails	Plays	Plays	Crashes	Plays	Plays	Fails	Plays	Skips	Fails	Plays	Plays	Plays	Plays	Plays	Plays	Fails	-	Amarok version 2.8.0 with testbench revision 5 and 60-63
Cog	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	-	version 2169 (2169-gbf54c452) with testbench revision 6
DeaDBeeF 1.9.3 (Windows)	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	(not tested - )		Plays	Plays	Plays	Plays	Plays	Plays	Plays	deadbeef 1.9.3 [static] [portable]. Testbench revision 7
Exaile (Windows)	Plays	Plays	Plays	Plays	Fails	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	-	First tested version 4.1.1 (GStreamer: 1.18.4) with testbench revision 5. Then updated with 4.1.2-beta1 (GStreamer: 1.18.5) for variable blocksize and 60-63
ffplay N-108161-gaaf6d85283-20220916 (Windows)	Fails	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Testbench revision 7. File 03 still broken in versions N-109368-gcc11afe502-20221206, 6.1-essentials_build-www.gyan.dev and 7.0-essentials_build-www.gyan.dev
foobar2000 1.6.14 and 2.0 beta 7 [x86]	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Testbench revision 7
GoneMAD (Android)	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Crashes	Plays	Hangs	Plays	-	-	-	testbench revision 5
JRiver Media Center (Windows 64; portable)	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Windows version 31.0.84 64-bit with testbench version 7.1. (Version 28.0.88 would reject number 54.)
Mediamonkey 5	Plays	Plays	Plays	Skips	Plays	Plays	Plays	Plays	Plays	Silence	Fails	Plays	Plays	Plays	Plays	Fails	Freezes	Plays	Updated with version 5.0.4.2690 testing files 60-64 and the ones 5.0.1.2433 failed (22) / froze (54-55) upon testing revision 5.
MacOS	Skips	Plays	Plays	Skips	Skips	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Testbench revision 7, tested with Quicktime and Safari on MacOS 13
Mplayer (Windows)	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Fails	Plays	Plays	Plays	Unresp.	Plays	Plays	Fails	Plays	SMPlayer-bundled version Redxii-SVN-r37955-6.2.0 (x86_64) (using ffmpeg N-87137-g6ccd32c367) with testbench revision 5 & files 60-64
mpv	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Tested both mpv v0.38.0-633-ge509ec0a for Windows and v0.38.0-432-g112fa549be, testbench revision 7
MusicBee 3.4.8033 (Windows; portable)	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Testbench revision 7. Only tested on headphones, does left/right correctly on multichannel.
Quod Libet (Windows)	Plays	Plays	Plays	Plays	Fails	Plays	Plays	Plays	Skips	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	-	version 4.4.0 (portable version) with testbench revision 5 and 4.5.0 (portable) with revision 6.
Resoniq 0.9.3 (portable, Windows)	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Fails?	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Apparently uses FLAC 1.3.2. Testbench revision 7.
Spotify (Windows)	Plays	Plays	Plays	Skips	Plays	Plays	Plays	Plays	Skips	Skips	Skips (?)	Skips (?)	Plays	Rejects	Plays	Plays	Plays	-	Spotify for Windows version 1.1.72.439.gc253025e, testbench revision 5; files 60-63 with version 1.1.83.956.gbe7ea9a7. Note, tested a stereo soundcard only (hence the question mark on multichannel).
VLC	Plays	Plays	Plays	Plays	Fails	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Crashes	Plays	Plays	Plays	Plays	VLC 3.0.16 on testbench revision 5 VLC 3.0.17.4 on files 60 to 63. VLC 3.0.16 on file 64
VUPlayer 4.15	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Testbench revision 7.
Winamp 5.8	Plays	Plays	Plays	Freezes	Plays	Plays	Plays	Plays	Freezes	Plays	Freezes		Plays	Plays	Plays	-	-	-	Winamp 5.8 build 3660, testbench revision 5
Windows Media Player	Plays	Plays	Plays	Rejects	Plays	Plays	Plays	Plays	Plays	Rejects	Plays	Plays	Plays	Rejects	Plays	-	-	-	Tested version 12.0.19041.1288 on testbench revision 5
XMPlay	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	XMPlay 3.8.5.42 with FLAC decoder revision 14 tested with testbench revision 7.
Digital Audio Workstations (DAW)
Adobe Audition	Plays	Plays	Plays	Rejects	Plays	Plays	Plays	Plays	Plays	Rejects	Plays	Plays	Fails	Plays	Plays	Plays	Rejects	Plays	Adobe Audition 1.5 with http://www.vuplayer.com/audition.php plugin, tested with testbench revision 7.

Table legend
Plays	all files in this category are played as intended, without glitches, changes etc.; changed volume tolerated (but indicated)
Rejects	One or more files are rejected by the player, either by skipping the file or mentioning a message like: "file not supported"
Fails	One or more files are not played back as intended, for example by stuttering, garbling or playing at incorrect speed, or by never starting playing (though not skipping, freezing or rebooting)
Freezes	On one or more files, the player crashes on decoding, leaving the unit inoperable until reboot, or the unit reboots by itself
The order of precedence is from bottom to top, so if one file freezes the player, one file is rejected and the other play fine, the table reads 'Freezes' for that category.
For the metadata tests, the (absence of) rendering of the metadata is not a criterium, simply whether its presence is an obstacle to playing

@@ Line 117: / Line 117: @@
 | style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
-| style="background: yellow;" | {{tooltip|Skips|Skips files with a non-standard samplerate, plays 22050Hz}}
-| style="background: orange;" | {{tooltip|Fails|Extreme distortion on playback}}
 | style="background: lime;"   | Plays
-| style="background: yellow;" colspan="7" | {{tooltip|Skips|Skips all high-res and multichannel files}}
+| style="background: orange;" | {{tooltip|Fails|Extreme distortion on playback of 8-bit file, 12-bit file is played back too soft}}
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: orange;" | {{tooltip|Fails|20-bit file is played back too soft}}
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: orange;"    | {{tooltip|Fails|File with large application metadata block results playback of part of another file before playing the proper audio}}
+| style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
-| style="background: red;"    | {{tooltip|Hangs|Files with large seektable are skipped (after long delay) and file with combined extreme metadata causes player to hang, requiring removal of USB stick}}
+| style="background: lime;" | {{tooltip|Plays|20-bit file is played back too soft like file 37, but is otherwise OK}}
 | style="background: lime;"   | Plays
-| -
+| class="mw-collapsible mw-collapsed" | Tested with system firmware 7.1.5-r1, model year 2018, testbench revision 7
-| -
-| -
-|
 |-
 ! Cowon iAudio 7
@@ Line 225: / Line 231: @@
 | -
 | class="mw-collapsible mw-collapsed" | Tested with revision 6, firmware version V5.15, dated 2020-04-24
+|-
+! Mazda Connect
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | {{tooltip|Plays|After playing file 27, the unit does not automatically continue playback of the next track}}
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| class="mw-collapsible mw-collapsed" | Tested with revision 7, CX-90 MY2024
 |-
 ! Onkyo TX-NR686
@@ Line 239: / Line 266: @@
 | style="background: yellow;" colspan="2" | {{tooltip|Rejects|Does not play 4.0, 5.1 and 7.1 files but does play 5.0 and 6.1 as stereo and 3.0 as 3.1}}
 | style="background: lime;"   | Plays
-| style="background: red;"    | {{tooltip|Reboots|Crashes on file with 1000x repeating VORBISCOMMENT}}
+| style="background: red;"    | {{tooltip|Unresp.|Becomes unresponsive on file with 1000x repeating VORBISCOMMENT}}
 | style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
-| -
+| style="background: lime;"   | Plays
-| class="mw-collapsible mw-collapsed" | Tested with revision 6, firmware version 1071-2000-1070-0011.
+| class="mw-collapsible mw-collapsed" | Tested with revision 6, firmware version 1071-4000-1070-0011.
 |-
 ! Panasonic TX-60CX750E
@@ Line 319: / Line 346: @@
 | style="background: lime;"   | Plays
 | style="background: orange;" | {{tooltip|Fails|Plays 20-bit and 24-bit file but with loud bursts of static}}
-| -
+| style="background: orange;" | {{tooltip|Fails|PLays, but with short ticks}}
-| class="mw-collapsible mw-collapsed" |  Tested with Rockbox 3.15 (latest stable) on Sansa Fuze with testbench revision 6
+| class="mw-collapsible mw-collapsed" |  Tested with Rockbox 3.15 (latest stable) on Sansa Fuze with testbench revision 7
 |-
 ! Samsung F5100WWB
@@ Line 383: / Line 410: @@
 | class="mw-collapsible mw-collapsed" | Testbench revision 7
 |-
-! colspan="16" | Software players
+! colspan="20" | Software players
 |-
 ! AIMP 5.03 build 2398 (Windows)
@@ Line 529: / Line 556: @@
 | style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
-| class="mw-collapsible mw-collapsed" | Testbench revision 7. File 03 still broken in ffplay version N-109368-gcc11afe502-20221206.
+| class="mw-collapsible mw-collapsed" | Testbench revision 7. File 03 still broken in versions N-109368-gcc11afe502-20221206, 6.1-essentials_build-www.gyan.dev and 7.0-essentials_build-www.gyan.dev
 |-
 ! foobar2000 1.6.14 and 2.0 beta 7 [x86]
@@ Line 573: / Line 600: @@
 | class="mw-collapsible mw-collapsed" | testbench revision 5
 |-
-! JRiver Media Center
+! JRiver Media Center (Windows 64; portable)
 | style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
@@ Line 581: / Line 608: @@
 | style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | {{tooltip|Plays|lowers volume on 20 bit}}
 | style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
@@ Line 587: / Line 617: @@
 | style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
-| style="background: yellow;" | {{tooltip|Rejects|rejects (with error message) extremely large VORBISCOMMENT. Still plays the all-combined 55.}}
+| style="background: lime;"   | {{tooltip|Plays|lowers volume on 20 bit}}
 | style="background: lime;"   | Plays
-| -
+| class="mw-collapsible mw-collapsed" | Windows version 31.0.84 64-bit with testbench version 7.1. (Version 28.0.88 would reject number 54.)
-| -
-| -
-| class="mw-collapsible mw-collapsed" | version 28.0.88 64-bit with testbench revision 5
 |-
 ! Mediamonkey 5
@@ Line 657: / Line 684: @@
 | class="mw-collapsible mw-collapsed" | SMPlayer-bundled version Redxii-SVN-r37955-6.2.0 (x86_64) (using ffmpeg N-87137-g6ccd32c367) with testbench revision 5 & files 60-64
 |-
-! mpv (Windows)
+! mpv
 | style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
@@ Line 671: / Line 698: @@
 | style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
-| style="background: red;"    | {{tooltip|Hangs|Hangs on files with 1000x repeating vorbiscomments}}
 | style="background: lime;"   | Plays
 | style="background: lime;"   | Plays
-| style="background: lime;" | {{tooltip|Plays|Fixed during November 2022; version mpv-x86_64-20211031-git-4a80de9 (Windows) did stutter on 24-bit predictor overflow check}}
 | style="background: lime;"   | Plays
-| class="mw-collapsible mw-collapsed" | mpv-0.35.0-x86_64 for Windows, testbench revision 7; this build (November 14) fixes one more file than mpv-x86_64-20211031-git-4a80de9.
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| class="mw-collapsible mw-collapsed" | Tested both mpv v0.38.0-633-ge509ec0a for Windows and v0.38.0-432-g112fa549be, testbench revision 7
 |-
 ! MusicBee 3.4.8033 (Windows; portable)
@@ Line 865: / Line 892: @@
 | style="background: lime;"   | Plays
 | class="mw-collapsible mw-collapsed" | XMPlay 3.8.5.42 with FLAC decoder revision 14 tested with testbench revision 7.
+|-
+! colspan="20" | Digital Audio Workstations (DAW)
+|-
+! Adobe Audition
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: yellow;" | {{tooltip|Rejects|Rejects 12 bit file}}
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: yellow;" | {{tooltip|Rejects|Rejects 20 bit file}}
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: orange;" | {{tooltip|Fails|Imports silence on file with no total number of samples set}}
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: yellow;" | {{tooltip|Rejects|Rejects 20 bit file}}
+| style="background: lime;"   | Plays
+| class="mw-collapsible mw-collapsed" | Adobe Audition 1.5 with http://www.vuplayer.com/audition.php plugin, tested with testbench revision 7.
 |-}
 |-}
@@ Line 872: / Line 923: @@
 ! colspan="2" | Table legend
 |-
-| style="background: lime;"   | Plays   || all files in this category are played as intended, without glitches, changes etc.
+| style="background: lime;"   | Plays   || all files in this category are played as intended, without glitches, changes etc.; changed volume tolerated (but indicated)
 |-
 | style="background: yellow;" | Rejects || One or more files are rejected by the player, either by skipping the file or mentioning a message like: "file not supported"