Difference between revisions of "FLAC decoder testbench"

From Hydrogenaudio Knowledgebase
Jump to: navigation, search
m (Correction on Amarok and addition to Pioneer DEH-1900UB)
(Update chromecast entry (which has improved), update testbench to r7 and add column for file 64)
(18 intermediate revisions by 3 users not shown)
Line 1: Line 1:
To further the use and development of [[Free Lossless Audio Codec|FLAC]], a testbench containing files with different parameters was constructed. The testbench consists of 59 FLAC files, each one having a different combination of characteristics (like samplerate, bitdepth and number of channels) and FLAC features (like blocksize, maximum partition order, maximum predictor order, maximum predictor precision etc.) However, all combinations fall within the so called subset: a limited set of FLAC parameters that fit in the frame header and are supposed to be 'streamable'. Please be aware that certain tested features (such as blocksize 16 or a samplerate of 38457Hz) are pretty much useless in day-to-day usage, and that failure to properly decode such a file is mostly of interest to developers than to users.
+
To further the use and development of [[Free Lossless Audio Codec|FLAC]], a testbench containing files with different parameters was constructed. The testbench consists of 63 FLAC files, each one having a different combination of characteristics (like samplerate, bitdepth and number of channels) and FLAC features (like blocksize, maximum partition order, maximum predictor order, maximum predictor precision etc.) However, all combinations fall within the so called subset: a limited set of FLAC parameters that fit in the frame header and are supposed to be 'streamable'. Please be aware that certain tested features (such as blocksize 16 or a samplerate of 38457Hz) are pretty much useless in day-to-day usage, and that failure to properly decode such a file is mostly of interest to developers than to users.
  
For more information on which file tests what characteristics and features, please refer to the readme.txt file that is included in the testbench. This testbench is licensed under [https://creativecommons.org/publicdomain/zero/1.0/ CC0 1.0].
+
The testbench can be [https://drive.google.com/file/d/1ECjkAyqV56xd9IZV-6zccG1Y-iE_hDMj/view?usp=sharing downloaded here (revision 7)] and [https://github.com/ietf-wg-cellar/flac-test-files is maintained here]. It is licensed under [https://creativecommons.org/publicdomain/zero/1.0/ CC0 1.0].
  
The testbench can be [https://drive.google.com/uc?id=1ahwxfe3V78X4Dai1i844tzkZ-3chSmeS downloaded here] and [https://github.com/ietf-wg-cellar/flac-test-files is maintained here].
+
Revision 6 adds four new files (60 to 63); files 1 through 59 are the same as in revision 5. For more information on which file tests what characteristics and features, please refer to the [https://github.com/ietf-wg-cellar/flac-test-files/blob/main/subset/README.txt readme.txt file]. Revision 7 replaces file 63 (which was faulty in revision 6) and adds file 64.
  
 
{| class="wikitable" cellspacing="2"
 
{| class="wikitable" cellspacing="2"
Line 11: Line 11:
 
! colspan="2" | Multichannel
 
! colspan="2" | Multichannel
 
! colspan="3" | Metadata
 
! colspan="3" | Metadata
 +
! colspan="3" | Various
 
!
 
!
 
|-
 
|-
 
!'''Decoder'''
 
!'''Decoder'''
! Block­size
+
! Block­sizes
 
! Features
 
! Features
 
! Sample­rates
 
! Sample­rates
 
! Bit­depths
 
! Bit­depths
! Variable blocksize
+
! {{tooltip|Var. bl.size|Variable blocksize}}
 
! General
 
! General
 
! Block­size
 
! Block­size
Line 24: Line 25:
 
! Sample­rates
 
! Sample­rates
 
! Bit­depths
 
! Bit­depths
! Channel lay-outs
+
! General
 
! Features
 
! Features
 
! Missing
 
! Missing
 
! Extreme
 
! Extreme
 
! MIME
 
! MIME
 +
! Mono
 +
! Overfl.
 +
! Zero esc.
 
! Remarks
 
! Remarks
 
|-
 
|-
Line 47: Line 51:
 
| 48-55
 
| 48-55
 
| 56-59
 
| 56-59
 +
| 60
 +
| 61-63
 +
| 64
 
|-
 
|-
 
! Android
 
! Android
Line 64: Line 71:
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 +
| -
 +
| -
 +
| -
 
| class="mw-collapsible mw-collapsed" | Android 11 with Google Play updates of 1st of August 2021 on Samsung XCover 5
 
| class="mw-collapsible mw-collapsed" | Android 11 with Google Play updates of 1st of August 2021 on Samsung XCover 5
 
|-
 
|-
Line 76: Line 86:
 
| style="background: red;"    | {{tooltip|Hangs|Files with large seektable are skipped (after long delay) and file with combined extreme metadata causes player to hang, requiring removal of USB stick}}
 
| style="background: red;"    | {{tooltip|Hangs|Files with large seektable are skipped (after long delay) and file with combined extreme metadata causes player to hang, requiring removal of USB stick}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 +
| -
 +
| -
 +
| -
 
|  
 
|  
 
|-
 
|-
 
! Chromecast
 
! Chromecast
| style="background: orange;" | Fails
+
| style="background: orange;" | {{tooltip|Fails|Heavy stutter with blocksize 16}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| style="background: orange;" | Fails
+
| style="background: orange;" | {{tooltip|Fails|Mangles non-standard samplerates}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| style="background: orange;" | Fails
+
| style="background: orange;" | {{tooltip|Fails|Never starts playing on old format blocksize}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| style="background: orange;" | Fails
 
| style="background: orange;" | Fails
 
| style="background: orange;" | Fails
 
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| style="background: yellow;" colspan="2" | Rejects
 
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| style="background: red;"   | Reboots
+
| style="background: orange;" | {{tooltip|Fails|Mangles non-standard samplerates}}
| style="background: yellow;" | Rejects
+
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | Heavy stutter with blocksize 16. Mangles non-standard samplerates. Doesn't start playing on old variable blocksize format, blocksizes 16384 and 13456, predictor order 32, 384kHz. Skips multichannel and image/avif. Crashes on file with 1000x repeating VORBISCOMMENT
+
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: yellow;" | {{tooltip|Skips|Skips file with avif embedded picture}}
 +
| style="background: lime;"  | Plays
 +
| style="background: yellow;" | {{tooltip|Skips|Skips file with 24-bit overflow check}}
 +
| style="background: lime;"  | Plays
 +
| class="mw-collapsible mw-collapsed" | Tested with system firmware 275994 and cast firmware 1.56.275994, testbench revision 7
 
|-
 
|-
 
! Cowon iAudio 7
 
! Cowon iAudio 7
Line 107: Line 124:
 
| style="background: yellow;" | Rejects
 
| style="background: yellow;" | Rejects
 
| style="background: red;" colspan="2" | Freezes
 
| style="background: red;" colspan="2" | Freezes
 +
| -
 +
| -
 +
| -
 
| -
 
| -
 
| -
 
| -
Line 128: Line 148:
 
| style="background: yellow;"    | {{tooltip|Skips|Files with large seektable are skipped (after long delay) and file with combined extreme metadata causes player to hang, requiring removal of USB stick}}
 
| style="background: yellow;"    | {{tooltip|Skips|Files with large seektable are skipped (after long delay) and file with combined extreme metadata causes player to hang, requiring removal of USB stick}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 +
| -
 +
| -
 +
| -
 
| class="mw-collapsible mw-collapsed" |  Tested with revision 5, SYNC version 3.3, build 18201_PRODUCT
 
| class="mw-collapsible mw-collapsed" |  Tested with revision 5, SYNC version 3.3, build 18201_PRODUCT
 
|-
 
|-
Line 145: Line 168:
 
| style="background: red;"    | Reboots
 
| style="background: red;"    | Reboots
 
| style="background: yellow;" | Rejects
 
| style="background: yellow;" | Rejects
 +
| -
 +
| -
 +
| -
 
| class="mw-collapsible mw-collapsed" | Heavy stutter with blocksize 16. Stretches non-standard samplerates. Samplerates above 96kHz, variable blocksize files and multichannel files 'not supported'. Freezes on blocksize 16384 and 13456. 192kHz is played back very slow. Skips all multichannel, files with cover image, extreme metadata. Reboot loop on extreme large VORBISCOMMENT
 
| class="mw-collapsible mw-collapsed" | Heavy stutter with blocksize 16. Stretches non-standard samplerates. Samplerates above 96kHz, variable blocksize files and multichannel files 'not supported'. Freezes on blocksize 16384 and 13456. 192kHz is played back very slow. Skips all multichannel, files with cover image, extreme metadata. Reboot loop on extreme large VORBISCOMMENT
 
|-
 
|-
Line 163: Line 189:
 
| style="background: red;"    | Reboots
 
| style="background: red;"    | Reboots
 
| style="background: lime;" | Plays
 
| style="background: lime;" | Plays
 +
| -
 +
| -
 +
| -
 
| class="mw-collapsible mw-collapsed" | File 03 plays extremely noisily, barely recognizable. File 51 produces a "NO DEVICE" error and a reboot. File 54 produces a reboot with no message. These files produce a "NA FILE" error and the player skips to the next: 19 21 24 25 26 27 29 30 31 34 35 36 38 39 40 41 42 43 44 45 46 48 55. (firmware: te1706k.krm from 2020-03-05)
 
| class="mw-collapsible mw-collapsed" | File 03 plays extremely noisily, barely recognizable. File 51 produces a "NO DEVICE" error and a reboot. File 54 produces a reboot with no message. These files produce a "NA FILE" error and the player skips to the next: 19 21 24 25 26 27 29 30 31 34 35 36 38 39 40 41 42 43 44 45 46 48 55. (firmware: te1706k.krm from 2020-03-05)
 +
|-
 +
! Lenco Xemio-240
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: red;"    | {{tooltip|Freezes|Freezes on non-standard sample rates (34567Hz and 39kHz)}}
 +
| style="background: lime;"  | Plays
 +
| style="background: yellow;"  colspan="8" | {{tooltip|Rejects|Returns message 'File format not supported!'}}
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | {{tooltip|Plays|Player is unresponsive during loading, which takes a while}}
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: orange;" | {{tooltip|Fails|20 bit and 24 bit files are played back with shorts bursts of loud static}}
 +
| -
 +
| class="mw-collapsible mw-collapsed" | Tested with revision 6, firmware version V5.15, dated 2020-04-24
 
|-
 
|-
 
! Onkyo TX-NR686
 
! Onkyo TX-NR686
Line 180: Line 223:
 
| style="background: red;"    | {{tooltip|Reboots|Crashes on file with 1000x repeating VORBISCOMMENT}}
 
| style="background: red;"    | {{tooltip|Reboots|Crashes on file with 1000x repeating VORBISCOMMENT}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | Tested with revision 5, firmware version 1071-2000-1070-0011.
+
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| -
 +
| class="mw-collapsible mw-collapsed" | Tested with revision 6, firmware version 1071-2000-1070-0011.
 
|-
 
|-
 
! Panasonic TX-60CX750E
 
! Panasonic TX-60CX750E
Line 198: Line 244:
 
| style="background: orange;" | {{tooltip|Fails|Does not start playing on extremely large vorbiscomment}}
 
| style="background: orange;" | {{tooltip|Fails|Does not start playing on extremely large vorbiscomment}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | Based on libFLAC 1.2.1, according to copyright notice. Firmware version 3.228, testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| -
 +
| class="mw-collapsible mw-collapsed" | Based on libFLAC 1.2.1, according to copyright notice. Firmware version 3.228, testbench revision 6
 
|-
 
|-
 
! Pioneer BDP-150
 
! Pioneer BDP-150
Line 208: Line 257:
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: red;"    | {{tooltip|Freezes|Freezes on non-standard blocksize (13456)}}
 
| style="background: red;"    | {{tooltip|Freezes|Freezes on non-standard blocksize (13456)}}
| style="background: orange;" | {{tooltip|Fails| Incorrect decoding (clicks) of RICE2 partitions with escape codes}}
+
| style="background: orange;" | {{tooltip|Fails|Incorrect decoding (clicks) of RICE2 partitions with escape codes}}
 
| style="background: orange;" | {{tooltip|Fails|Stretches non-standard samplerates to 44.1kHz. Stutters on 192kHz with 32th order predictors}}
 
| style="background: orange;" | {{tooltip|Fails|Stretches non-standard samplerates to 44.1kHz. Stutters on 192kHz with 32th order predictors}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
Line 215: Line 264:
 
| style="background: red;"    | {{tooltip|Freezes|Freezes on files with 1000x repeated vorbiscomments, rejects files with too many seekpoints}}
 
| style="background: red;"    | {{tooltip|Freezes|Freezes on files with 1000x repeated vorbiscomments, rejects files with too many seekpoints}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | Firmware v1.08, testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: orange;" | {{tooltip|Fails|Plays but skips parts of 24-bit file}}
 +
| -
 +
| class="mw-collapsible mw-collapsed" | Firmware v1.08, testbench revision 6
 
|-
 
|-
 
! Pioneer DEH-1900UB  
 
! Pioneer DEH-1900UB  
Line 227: Line 279:
 
| style="background: yellow;" | {{tooltip|Other|Plays, but takes about a minute time to start playing. During this wait, track can be skipped, unit does not become unresponsive}}
 
| style="background: yellow;" | {{tooltip|Other|Plays, but takes about a minute time to start playing. During this wait, track can be skipped, unit does not become unresponsive}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 +
| -
 +
| -
 +
| -
 
| class="mw-collapsible mw-collapsed" | Testbench revision 5
 
| class="mw-collapsible mw-collapsed" | Testbench revision 5
 
|-
 
|-
Line 244: Line 299:
 
| style="background: orange;"| {{tooltip|Fails|Plays extremely slowly on file with APPLICATION metadata block}}
 
| style="background: orange;"| {{tooltip|Fails|Plays extremely slowly on file with APPLICATION metadata block}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" |  Tested with Rockbox 3.15 (latest stable) on Sansa Fuze with testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: orange;" | {{tooltip|Fails|Plays 20-bit and 24-bit file but with loud bursts of static}}
 +
| -
 +
| class="mw-collapsible mw-collapsed" |  Tested with Rockbox 3.15 (latest stable) on Sansa Fuze with testbench revision 6
 +
|-
 +
! Sony BDP-1700
 +
| style="background: red;"    | {{tooltip|Freezes|Freezes on non-standard blocksizes (254, 725, 1000 and 1937)}}
 +
| style="background: orange;" | {{tooltip|Fails|Garbled audio when playing file with LPC with qlp precision of 2}}
 +
| style="background: orange;" | {{tooltip|Fails|Stretches non-standard samplerates to 44.1kHz}}
 +
| style="background: lime;"  | Plays
 +
| style="background: red;"    | {{tooltip|Freezes|Freezes on variable blocksize files with non-standard blocksizes, stutters on variable blocksize files with standard blocksizes}}
 +
| style="background: lime;"  | Plays
 +
| style="background: red;"    | {{tooltip|Freezes|Freezes on non-standard blocksize (13456)}}
 +
| style="background: orange;" | {{tooltip|Fails|Incorrect decoding (clicks) of RICE2 partitions with escape codes}}
 +
| style="background: orange;" | {{tooltip|Fails|Stretches non-standard samplerates to 44.1kHz. Stutters on 192kHz with 32th order predictors}}
 +
| style="background: lime;"  | Plays
 +
| style="background: red;" colspan="2" | {{tooltip|Freezes|Freezes on files with more than 6 channels}}
 +
| style="background: yellow;" | {{tooltip|Skips|Skips file with an unknown number of total samples}}
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: orange;" | {{tooltip|Fails|Plays but skips parts of 24-bit file}}
 +
| -
 +
| class="mw-collapsible mw-collapsed" | Firmware m30.R.0527, testbench revision 6
 
|-
 
|-
 
! colspan="16" | Software players
 
! colspan="16" | Software players
Line 264: Line 342:
 
| style="background: lime;" | {{tooltip|Plays|version 1.6.8 plays. 1.6.7: Error message: Unable to open item for playback (bad allocation)}}
 
| style="background: lime;" | {{tooltip|Plays|version 1.6.8 plays. 1.6.7: Error message: Unable to open item for playback (bad allocation)}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | Foobar2000 version 1.6.8 with testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| -
 +
| class="mw-collapsible mw-collapsed" | Foobar2000 version 1.6.8 with testbench revision 5, version 1.6.11 with revision 6 <i>updates</i> only.
 
|-
 
|-
 
! VLC
 
! VLC
Line 282: Line 363:
 
| style="background: red;"    | {{tooltip|Crashes|Crashes occasionally on files with extremely large VORBISCOMMENT}}
 
| style="background: red;"    | {{tooltip|Crashes|Crashes occasionally on files with extremely large VORBISCOMMENT}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | VLC 3.0.16 on testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| -
 +
| class="mw-collapsible mw-collapsed" | VLC 3.0.16 on testbench revision 5 <br>VLC 3.0.17.4 on files 60 to 63.
 
|-
 
|-
 
! Winamp 5.8
 
! Winamp 5.8
Line 299: Line 383:
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 +
| -
 +
| -
 +
| -
 
| class="mw-collapsible mw-collapsed" | Winamp 5.8 build 3660, testbench revision 5
 
| class="mw-collapsible mw-collapsed" | Winamp 5.8 build 3660, testbench revision 5
 
|-
 
|-
Line 317: Line 404:
 
| style="background: yellow;" | {{tooltip|Rejects|Rejects all}}
 
| style="background: yellow;" | {{tooltip|Rejects|Rejects all}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 +
| -
 +
| -
 +
| -
 
| class="mw-collapsible mw-collapsed" | Tested version 12.0.19041.1288 on testbench revision 5
 
| class="mw-collapsible mw-collapsed" | Tested version 12.0.19041.1288 on testbench revision 5
 
|-
 
|-
Line 335: Line 425:
 
| style="background: red;"    | {{tooltip|Hangs|Hangs on files with 1000x repeating vorbiscomments}}
 
| style="background: red;"    | {{tooltip|Hangs|Hangs on files with 1000x repeating vorbiscomments}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | mpv-x86_64-20211031-git-4a80de9 on Windows, testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: orange;" | {{tooltip|Fails|Stutters on 24-bit predictor overflow check}}
 +
| -
 +
| class="mw-collapsible mw-collapsed" | mpv-x86_64-20211031-git-4a80de9 on Windows, testbench revision 5 & files 60-63
 
|-
 
|-
 
! Mediamonkey 5
 
! Mediamonkey 5
Line 347: Line 440:
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| style="background: orange;" | {{tooltip|Silence|Plays silence on 20-bit file}}
+
| style="background: orange;" | {{tooltip|Silence|Plays silence on 20-bit file.}}
 
| style="background: orange;" | {{tooltip|Breaks|Only plays first 2 seconds of all spoken multichannel files}}
 
| style="background: orange;" | {{tooltip|Breaks|Only plays first 2 seconds of all spoken multichannel files}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
Line 353: Line 446:
 
| style="background: red;"    | {{tooltip|Unresp.|GUI becomes unresponsive on (and after) playing files with 1000x repeating vorbiscomments}}
 
| style="background: red;"    | {{tooltip|Unresp.|GUI becomes unresponsive on (and after) playing files with 1000x repeating vorbiscomments}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 +
| -
 +
| -
 +
| -
 
| class="mw-collapsible mw-collapsed" | Version 5.0.1.2433 with testbench revision 5
 
| class="mw-collapsible mw-collapsed" | Version 5.0.1.2433 with testbench revision 5
 
|-
 
|-
Line 371: Line 467:
 
| style="background: red;"    | {{tooltip|Unresp.|Took over a minute before starting to playing the '55 - file 48-53 combined.flac'}}
 
| style="background: red;"    | {{tooltip|Unresp.|Took over a minute before starting to playing the '55 - file 48-53 combined.flac'}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | Version Redxii-SVN-r37955-6.2.0 (x86_64) (using ffmpeg N-87137-g6ccd32c367) with testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: orange;" | {{tooltip|Fails|Stutters on 24-bit predictor overflow check}}
 +
| -
 +
| class="mw-collapsible mw-collapsed" | Version Redxii-SVN-r37955-6.2.0 (x86_64) (using ffmpeg N-87137-g6ccd32c367) with testbench revision 5 & files 60-63
 
|-
 
|-
 
! ffplay (Windows)
 
! ffplay (Windows)
Line 389: Line 488:
 
| style="background: red;"    | {{tooltip|Unresp.|Took over a minute before starting to playing the '55 - file 48-53 combined.flac'}}
 
| style="background: red;"    | {{tooltip|Unresp.|Took over a minute before starting to playing the '55 - file 48-53 combined.flac'}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 +
| -
 +
| -
 +
| -
 
| class="mw-collapsible mw-collapsed" | version 4.4.1-essentials_build-www.gyan.dev with testbench revision 5
 
| class="mw-collapsible mw-collapsed" | version 4.4.1-essentials_build-www.gyan.dev with testbench revision 5
 
|-
 
|-
Line 407: Line 509:
 
| style="background: yellow;"    | {{tooltip|Rejects|Does not index any of the 'extremely large' 48-52 & 55}}
 
| style="background: yellow;"    | {{tooltip|Rejects|Does not index any of the 'extremely large' 48-52 & 55}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | Spotify for Windows version 1.1.72.439.gc253025e tested a stereo soundcard only (hence the question mark on multichannel), with testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| -
 +
| class="mw-collapsible mw-collapsed" | Spotify for Windows version 1.1.72.439.gc253025e, testbench revision 5; files 60-63 with version 1.1.83.956.gbe7ea9a7. Note, tested a stereo soundcard only (hence the question mark on multichannel).
 
|-
 
|-
 
! JRiver Media Center
 
! JRiver Media Center
Line 425: Line 530:
 
| style="background: yellow;" | {{tooltip|Rejects|rejects (with error message) extremely large VORBISCOMMENT. Still plays the all-combined 55.}}
 
| style="background: yellow;" | {{tooltip|Rejects|rejects (with error message) extremely large VORBISCOMMENT. Still plays the all-combined 55.}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 +
| -
 +
| -
 +
| -
 
| class="mw-collapsible mw-collapsed" | version 28.0.88 64-bit with testbench revision 5
 
| class="mw-collapsible mw-collapsed" | version 28.0.88 64-bit with testbench revision 5
 
|-
 
|-
Line 437: Line 545:
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: yellow;"  | {{tooltip|Skips|Skips 384 kHz}}
 
| style="background: yellow;"  | {{tooltip|Skips|Skips 384 kHz}}
| style="background: yellow;"  | {{tooltip|Skips|Skips 20 bits}}
+
| style="background: orange;"  | {{tooltip|Fails|Silence on 20 bits}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
Line 443: Line 551:
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | Amarok version 2.8.0 with testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: orange;"  | {{tooltip|Fails|Silence on 20 bits, like on number 37}}
 +
| -
 +
| class="mw-collapsible mw-collapsed" | Amarok version 2.8.0 with testbench revision 5 and 60-63
 
|-
 
|-
 
! Exaile (Windows)
 
! Exaile (Windows)
Line 450: Line 561:
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| style="background: yellow;"    | {{tooltip|Rejects|Skips all variable blocksize files, except: stops upon the old format flake 0.11 (but can resume playback on different file)}}
+
| style="background: orange;"    | {{tooltip|Fails|Version 4.1.2 beta1: File 25 Flake smaller blocksize is played back noisy. File 27 old format mutes after a split second. Version 4.1.1: Skips all variable blocksize files, except: stops upon the old format flake 0.11 (but can resume playback on different file)}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
Line 461: Line 572:
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | version 4.1.1  (GStreamer: 1.18.4) with testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| -
 +
| class="mw-collapsible mw-collapsed" | First tested version 4.1.1  (GStreamer: 1.18.4) with testbench revision 5. Then updated with 4.1.2-beta1 (GStreamer: 1.18.5) for variable blocksize and 60-63
 
|-
 
|-
 
! Quod Libet (Windows)
 
! Quod Libet (Windows)
Line 479: Line 593:
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | version 4.4.0 (portable version) with testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| -
 +
| class="mw-collapsible mw-collapsed" | version 4.4.0 (portable version) with testbench revision 5 and 4.5.0 (portable) with revision 6.
 
|-
 
|-
 
! Cog
 
! Cog
Line 497: Line 614:
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
| class="mw-collapsible mw-collapsed" | version 1755 (1755-g92d29e7a) with testbench revision 5
+
| style="background: lime;"  | Plays
 +
| style="background: lime;"  |{{tooltip|Plays|Versions before 1630 will reject the 20 bit file, because they did not properly handle files with bit depths that aren't a multiple of 8.}}
 +
| -
 +
| class="mw-collapsible mw-collapsed" | version 2169 (2169-gbf54c452) with testbench revision 6
 
|-
 
|-
 
! GoneMAD (Android)
 
! GoneMAD (Android)
Line 515: Line 635:
 
| style="background: red;"    | {{tooltip|Hangs|Hangs on 55. Also unresponsible UI on 51, but plays.}}
 
| style="background: red;"    | {{tooltip|Hangs|Hangs on 55. Also unresponsible UI on 51, but plays.}}
 
| style="background: lime;"  | Plays
 
| style="background: lime;"  | Plays
 +
| -
 +
| -
 +
| -
 
| class="mw-collapsible mw-collapsed" | testbench revision 5
 
| class="mw-collapsible mw-collapsed" | testbench revision 5
|}
+
|-
 +
! VUPlayer
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| style="background: lime;"  | Plays
 +
| -
 +
| class="mw-collapsible mw-collapsed" | VUPlayer 64-bit version 4.13 (BASS 2.4.15.0) with testbench revision 6.
 +
|-}
  
 
{| class="wikitable" cellspacing="2"
 
{| class="wikitable" cellspacing="2"

Revision as of 12:12, 30 July 2022

To further the use and development of FLAC, a testbench containing files with different parameters was constructed. The testbench consists of 63 FLAC files, each one having a different combination of characteristics (like samplerate, bitdepth and number of channels) and FLAC features (like blocksize, maximum partition order, maximum predictor order, maximum predictor precision etc.) However, all combinations fall within the so called subset: a limited set of FLAC parameters that fit in the frame header and are supposed to be 'streamable'. Please be aware that certain tested features (such as blocksize 16 or a samplerate of 38457Hz) are pretty much useless in day-to-day usage, and that failure to properly decode such a file is mostly of interest to developers than to users.

The testbench can be downloaded here (revision 7) and is maintained here. It is licensed under CC0 1.0.

Revision 6 adds four new files (60 to 63); files 1 through 59 are the same as in revision 5. For more information on which file tests what characteristics and features, please refer to the readme.txt file. Revision 7 replaces file 63 (which was faulty in revision 6) and adds file 64.

CDDA (44.1kHz/16-bit and less) High-res (96kHz/24-bit and more) Multichannel Metadata Various
Decoder Block­sizes Features Sample­rates Bit­depths Var. bl.size General Block­size Features Sample­rates Bit­depths General Features Missing Extreme MIME Mono Overfl. Zero esc. Remarks
Files in test set 1-10 11-18 19-21 22-23 24-27 28 29-30 31-32, 34 33-36 37 38-43 44 45-47 48-55 56-59 60 61-63 64
Android Fails Plays Plays Plays Skips Plays Plays Plays Plays Fails Fails Skips Plays Plays Plays - - - Android 11 with Google Play updates of 1st of August 2021 on Samsung XCover 5
Citroën DS5 Plays Plays Skips Fails Plays Skips Plays Hangs Plays - - -
Chromecast Fails Plays Fails Plays Fails Plays Plays Plays Fails Plays Plays Plays Plays Plays Skips Plays Skips Plays Tested with system firmware 275994 and cast firmware 1.56.275994, testbench revision 7
Cowon iAudio 7 Fails Plays Fails Rejects Rejects Rejects Rejects Rejects Rejects Rejects Freezes - - - - - - Freezes on blocksizes 254, 725; skips file with blocksize 1937. 38457Hz and 41kHz are speeded up; 22050 Plays. All files with bitdepth other than 16 are skipped. Files with variable block size are skipped. 3.0 plays as silence, others 16/44 MCH files make freeze.
Ford SYNC Plays Plays Plays Skips Plays Plays Plays Plays Plays Skips Plays Plays Plays Skips Plays - - - Tested with revision 5, SYNC version 3.3, build 18201_PRODUCT
JVC KD-R871BT Fails Plays Fails Plays Rejects Plays Freezes Reboots Fails Rejects Rejects Rejects Reboots Rejects - - - Heavy stutter with blocksize 16. Stretches non-standard samplerates. Samplerates above 96kHz, variable blocksize files and multichannel files 'not supported'. Freezes on blocksize 16384 and 13456. 192kHz is played back very slow. Skips all multichannel, files with cover image, extreme metadata. Reboot loop on extreme large VORBISCOMMENT
Kenwood KMM-BT304 Fails Plays Rejects Plays Rejects Plays Rejects Rejects Rejects Plays Rejects Rejects Plays Reboots Plays - - - File 03 plays extremely noisily, barely recognizable. File 51 produces a "NO DEVICE" error and a reboot. File 54 produces a reboot with no message. These files produce a "NA FILE" error and the player skips to the next: 19 21 24 25 26 27 29 30 31 34 35 36 38 39 40 41 42 43 44 45 46 48 55. (firmware: te1706k.krm from 2020-03-05)
Lenco Xemio-240 Plays Plays Freezes Plays Rejects Plays Plays Plays Plays Fails - Tested with revision 6, firmware version V5.15, dated 2020-04-24
Onkyo TX-NR686 Fails Plays Rejects Plays Rejects Plays Plays Plays Rejects Plays Rejects Plays Reboots Plays Plays Plays - Tested with revision 6, firmware version 1071-2000-1070-0011.
Panasonic TX-60CX750E Plays Plays Fails Plays Plays Plays Plays Plays Fails Plays Plays Fails Plays Fails Plays Plays Plays - Based on libFLAC 1.2.1, according to copyright notice. Firmware version 3.228, testbench revision 6
Pioneer BDP-150 Freezes Fails Fails Plays Freezes Plays Freezes Fails Fails Plays Freezes Freezes Freezes Plays Plays Fails - Firmware v1.08, testbench revision 6
Pioneer DEH-1900UB Skips Plays Skips Skips Skips Skips Plays Other Plays - - - Testbench revision 5
Rockbox Plays Plays Plays Plays Plays Plays Skips Skips Plays Plays Freezes Freezes Fails Plays Plays Fails - Tested with Rockbox 3.15 (latest stable) on Sansa Fuze with testbench revision 6
Sony BDP-1700 Freezes Fails Fails Plays Freezes Plays Freezes Fails Fails Plays Freezes Skips Plays Plays Plays Fails - Firmware m30.R.0527, testbench revision 6
Software players
Foobar 2000 Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays - Foobar2000 version 1.6.8 with testbench revision 5, version 1.6.11 with revision 6 updates only.
VLC Plays Plays Plays Plays Fails Plays Plays Plays Plays Plays Plays Plays Plays Crashes Plays Plays Plays - VLC 3.0.16 on testbench revision 5
VLC 3.0.17.4 on files 60 to 63.
Winamp 5.8 Plays Plays Plays Freezes Plays Plays Plays Plays Freezes Plays Freezes Plays Plays Plays - - - Winamp 5.8 build 3660, testbench revision 5
Windows Media Player Plays Plays Plays Rejects Plays Plays Plays Plays Plays Rejects Plays Plays Plays Rejects Plays - - - Tested version 12.0.19041.1288 on testbench revision 5
mpv Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Hangs Plays Plays Fails - mpv-x86_64-20211031-git-4a80de9 on Windows, testbench revision 5 & files 60-63
Mediamonkey 5 Plays Plays Plays Crashes Plays Plays Plays Plays Plays Silence Breaks Plays Plays Unresp. Plays - - - Version 5.0.1.2433 with testbench revision 5
Mplayer (Windows) Plays Plays Plays Plays Plays Plays Plays Plays Plays Fails Plays Plays Plays Unresp. Plays Plays Fails - Version Redxii-SVN-r37955-6.2.0 (x86_64) (using ffmpeg N-87137-g6ccd32c367) with testbench revision 5 & files 60-63
ffplay (Windows) Fails Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Unresp. Plays - - - version 4.4.1-essentials_build-www.gyan.dev with testbench revision 5
Spotify (Windows) Plays Plays Plays Skips Plays Plays Plays Plays Skips Skips Skips (?) Skips (?) Plays Rejects Plays Plays Plays - Spotify for Windows version 1.1.72.439.gc253025e, testbench revision 5; files 60-63 with version 1.1.83.956.gbe7ea9a7. Note, tested a stereo soundcard only (hence the question mark on multichannel).
JRiver Media Center Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Rejects Plays - - - version 28.0.88 64-bit with testbench revision 5
Amarok (Windows) Fails Plays Plays Crashes Plays Plays Fails Plays Skips Fails Plays Plays Plays Plays Plays Plays Fails - Amarok version 2.8.0 with testbench revision 5 and 60-63
Exaile (Windows) Plays Plays Plays Plays Fails Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays - First tested version 4.1.1 (GStreamer: 1.18.4) with testbench revision 5. Then updated with 4.1.2-beta1 (GStreamer: 1.18.5) for variable blocksize and 60-63
Quod Libet (Windows) Plays Plays Plays Plays Fails Plays Plays Plays Skips Plays Plays Plays Plays Plays Plays Plays Plays - version 4.4.0 (portable version) with testbench revision 5 and 4.5.0 (portable) with revision 6.
Cog Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays - version 2169 (2169-gbf54c452) with testbench revision 6
GoneMAD (Android) Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Crashes Plays Hangs Plays - - - testbench revision 5
VUPlayer Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays Plays - VUPlayer 64-bit version 4.13 (BASS 2.4.15.0) with testbench revision 6.
Table legend
Plays all files in this category are played as intended, without glitches, changes etc.
Rejects One or more files are rejected by the player, either by skipping the file or mentioning a message like: "file not supported"
Fails One or more files are not played back as intended, for example by stuttering, garbling or playing at incorrect speed, or by never starting playing (though not skipping, freezing or rebooting)
Freezes On one or more files, the player crashes on decoding, leaving the unit inoperable until reboot, or the unit reboots by itself
The order of precedence is from bottom to top, so if one file freezes the player, one file is rejected and the other play fine, the table reads 'Freezes' for that category.
For the metadata tests, the (absence of) rendering of the metadata is not a criterium, simply whether its presence is an obstacle to playing