FLAC decoder testbench
To further the use and development of FLAC, a testbench containing files with different parameters was constructed. The testbench consists of 63 FLAC files, each one having a different combination of characteristics (like samplerate, bitdepth and number of channels) and FLAC features (like blocksize, maximum partition order, maximum predictor order, maximum predictor precision etc.) However, all combinations fall within the so called subset: a limited set of FLAC parameters that fit in the frame header and are supposed to be 'streamable'. Please be aware that certain tested features (such as blocksize 16 or a samplerate of 38457Hz) are pretty much useless in day-to-day usage, and that failure to properly decode such a file is mostly of interest to developers than to users.
The testbench can be downloaded here (revision 6) and is maintained here. It is licensed under CC0 1.0.
Revision 6 adds four new files (60 to 63); files 1 through 59 are the same as in revision 5. For more information on which file tests what characteristics and features, please refer to the readme.txt file.
| CDDA (44.1kHz/16-bit and less) | High-res (96kHz/24-bit and more) | Multichannel | Metadata | Various | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Decoder | Blocksizes | Features | Samplerates | Bitdepths | Var. bl.size | General | Blocksize | Features | Samplerates | Bitdepths | General | Features | Missing | Extreme | MIME | Mono | Overfl. | Remarks |
| Files in test set | 1-10 | 11-18 | 19-21 | 22-23 | 24-27 | 28 | 29-30 | 31-32, 34 | 33-36 | 37 | 38-43 | 44 | 45-47 | 48-55 | 56-59 | 60 | 61-63 | |
| Android | Fails | Plays | Plays | Plays | Skips | Plays | Plays | Plays | Plays | Fails | Fails | Skips | Plays | Plays | Plays | - | - | Android 11 with Google Play updates of 1st of August 2021 on Samsung XCover 5 |
| Citroën DS5 | Plays | Plays | Skips | Fails | Plays | Skips | Plays | Hangs | Plays | - | - | |||||||
| Chromecast | Fails | Plays | Fails | Plays | Fails | Plays | Fails | Fails | Fails | Plays | Rejects | Plays | Reboots | Rejects | - | - | Heavy stutter with blocksize 16. Mangles non-standard samplerates. Doesn't start playing on old variable blocksize format, blocksizes 16384 and 13456, predictor order 32, 384kHz. Skips multichannel and image/avif. Crashes on file with 1000x repeating VORBISCOMMENT | |
| Cowon iAudio 7 | Fails | Plays | Fails | Rejects | Rejects | Rejects | Rejects | Rejects | Rejects | Rejects | Freezes | - | - | - | - | - | Freezes on blocksizes 254, 725; skips file with blocksize 1937. 38457Hz and 41kHz are speeded up; 22050 Plays. All files with bitdepth other than 16 are skipped. Files with variable block size are skipped. 3.0 plays as silence, others 16/44 MCH files make freeze. | |
| Ford SYNC | Plays | Plays | Plays | Skips | Plays | Plays | Plays | Plays | Plays | Skips | Plays | Plays | Plays | Skips | Plays | - | - | Tested with revision 5, SYNC version 3.3, build 18201_PRODUCT |
| JVC KD-R871BT | Fails | Plays | Fails | Plays | Rejects | Plays | Freezes | Reboots | Fails | Rejects | Rejects | Rejects | Reboots | Rejects | - | - | Heavy stutter with blocksize 16. Stretches non-standard samplerates. Samplerates above 96kHz, variable blocksize files and multichannel files 'not supported'. Freezes on blocksize 16384 and 13456. 192kHz is played back very slow. Skips all multichannel, files with cover image, extreme metadata. Reboot loop on extreme large VORBISCOMMENT | |
| Kenwood KMM-BT304 | Fails | Plays | Rejects | Plays | Rejects | Plays | Rejects | Rejects | Rejects | Plays | Rejects | Rejects | Plays | Reboots | Plays | - | - | File 03 plays extremely noisily, barely recognizable. File 51 produces a "NO DEVICE" error and a reboot. File 54 produces a reboot with no message. These files produce a "NA FILE" error and the player skips to the next: 19 21 24 25 26 27 29 30 31 34 35 36 38 39 40 41 42 43 44 45 46 48 55. (firmware: te1706k.krm from 2020-03-05) |
| Lenco Xemio-240 | Plays | Plays | Freezes | Plays | Rejects | Plays | Plays | Plays | Plays | Fails | Tested with revision 6, firmware version V5.15, dated 2020-04-24 | |||||||
| Onkyo TX-NR686 | Fails | Plays | Rejects | Plays | Rejects | Plays | Plays | Plays | Rejects | Plays | Rejects | Plays | Reboots | Plays | Plays | Plays | Tested with revision 6, firmware version 1071-2000-1070-0011. | |
| Panasonic TX-60CX750E | Plays | Plays | Fails | Plays | Plays | Plays | Plays | Plays | Fails | Plays | Plays | Fails | Plays | Fails | Plays | Plays | Plays | Based on libFLAC 1.2.1, according to copyright notice. Firmware version 3.228, testbench revision 6 |
| Pioneer BDP-150 | Freezes | Fails | Fails | Plays | Freezes | Plays | Freezes | Fails | Fails | Plays | Freezes | Freezes | Freezes | Plays | Plays | Fails | Firmware v1.08, testbench revision 6 | |
| Pioneer DEH-1900UB | Skips | Plays | Skips | Skips | Skips | Skips | Plays | Other | Plays | - | - | Testbench revision 5 | ||||||
| Rockbox | Plays | Plays | Plays | Plays | Plays | Plays | Skips | Skips | Plays | Plays | Freezes | Freezes | Fails | Plays | - | - | Tested with Rockbox 3.15 (latest stable) on Sansa Fuze with testbench revision 5 | |
| Software players | ||||||||||||||||||
| Foobar 2000 | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Foobar2000 version 1.6.8 with testbench revision 5, version 1.6.11 with revision 6 updates only. |
| VLC | Plays | Plays | Plays | Plays | Fails | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Crashes | Plays | Plays | Plays | VLC 3.0.16 on testbench revision 5 VLC 3.0.17.4 on files 60 to 63. |
| Winamp 5.8 | Plays | Plays | Plays | Freezes | Plays | Plays | Plays | Plays | Freezes | Plays | Freezes | Plays | Plays | Plays | - | - | Winamp 5.8 build 3660, testbench revision 5 | |
| Windows Media Player | Plays | Plays | Plays | Rejects | Plays | Plays | Plays | Plays | Plays | Rejects | Plays | Plays | Plays | Rejects | Plays | - | - | Tested version 12.0.19041.1288 on testbench revision 5 |
| mpv | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Hangs | Plays | Plays | Fails | mpv-x86_64-20211031-git-4a80de9 on Windows, testbench revision 5 & files 60-63 |
| Mediamonkey 5 | Plays | Plays | Plays | Crashes | Plays | Plays | Plays | Plays | Plays | Silence | Breaks | Plays | Plays | Unresp. | Plays | - | - | Version 5.0.1.2433 with testbench revision 5 |
| Mplayer (Windows) | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Fails | Plays | Plays | Plays | Unresp. | Plays | Plays | Fails | Version Redxii-SVN-r37955-6.2.0 (x86_64) (using ffmpeg N-87137-g6ccd32c367) with testbench revision 5 & files 60-63 |
| ffplay (Windows) | Fails | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Unresp. | Plays | - | - | version 4.4.1-essentials_build-www.gyan.dev with testbench revision 5 |
| Spotify (Windows) | Plays | Plays | Plays | Skips | Plays | Plays | Plays | Plays | Skips | Skips | Skips (?) | Skips (?) | Plays | Rejects | Plays | - | - | Spotify for Windows version 1.1.72.439.gc253025e tested a stereo soundcard only (hence the question mark on multichannel), with testbench revision 5 |
| JRiver Media Center | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Rejects | Plays | - | - | version 28.0.88 64-bit with testbench revision 5 |
| Amarok (Windows) | Fails | Plays | Plays | Crashes | Plays | Plays | Fails | Plays | Skips | Fails | Plays | Plays | Plays | Plays | Plays | Plays | Fails | Amarok version 2.8.0 with testbench revision 5 and 60-63 |
| Exaile (Windows) | Plays | Plays | Plays | Plays | Rejects | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | - | - | version 4.1.1 (GStreamer: 1.18.4) with testbench revision 5 |
| Quod Libet (Windows) | Plays | Plays | Plays | Plays | Fails | Plays | Plays | Plays | Skips | Plays | Plays | Plays | Plays | Plays | Plays | - | - | version 4.4.0 (portable version) with testbench revision 5 |
| Cog | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | version 2169 (2169-gbf54c452) with testbench revision 6 |
| GoneMAD (Android) | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Plays | Crashes | Plays | Hangs | Plays | - | - | testbench revision 5 |
| Table legend | |
|---|---|
| Plays | all files in this category are played as intended, without glitches, changes etc. |
| Rejects | One or more files are rejected by the player, either by skipping the file or mentioning a message like: "file not supported" |
| Fails | One or more files are not played back as intended, for example by stuttering, garbling or playing at incorrect speed, or by never starting playing (though not skipping, freezing or rebooting) |
| Freezes | On one or more files, the player crashes on decoding, leaving the unit inoperable until reboot, or the unit reboots by itself |
| The order of precedence is from bottom to top, so if one file freezes the player, one file is rejected and the other play fine, the table reads 'Freezes' for that category. | |
| For the metadata tests, the (absence of) rendering of the metadata is not a criterium, simply whether its presence is an obstacle to playing | |