FLAC I/O less efficient than STDIN, Direct file access can be almost twice slower than STDIN Options

[skamp]

I was optimizing caudec when I came across this oddity. Basically, letting /usr/bin/flac access .flac files on a slowish HDD directly for decoding ('flac -d file.flac') was in one particular case almost twice slower than piping the files to /usr/bin/flac via STDIN ('cat file.flac | flac -d -').

I used a double album for testing, made of 37 tracks for a total of about 1 GiB, located on a HDD that tops out at about 70 MB/s. Incidentally, flac decodes on my machine at a similar rate.

I ran caudec twice (figuratively - I repeated the tests many times) with 8 concurrent processes, for decoding those FLAC files to WAV on a ramdisk. I made sure to drop all caches between each run. First run was with direct file access, and completed in 40 seconds. Second run was with piping to STDIN, and completed in 25 seconds.

The difference was much less pronounced, surprisingly, on a USB flash drive that tops out at 35 MB/s, 34 seconds vs. 30 seconds, and non-existant on a RAID 1 array that tops out at 130 MB/s and on a SSD that tops out at 500 MB/s. I experienced similar differences with WavPack.

Does anyone have any idea of what's going on?

[Destroid]

Ok, I see that in your third paragraph. (Pardon my oversight, I posted right before a medical appointment so apparently I was distracted.)

If I read correctly:
FLAC -d [HDD -> RAMdisk] = 40s
FLAC STDIO [HDD -> RAMdisk] = 25s
FLAC -d [flashdrive -> RAMdisk] = 34s
FLAC STDIO [flashdrive -> RAMdisk] = 30s
RAID = no tangible difference
SSD = no tangible difference

You mentioned WavPack having similar behavior so it doesn't seem the binary is the culprit either. I don't suppose using less concurrent threads would improve the performance of FLAC -d but it might be worth checking. It also is unclear how these threads are distributed but I wondered if multiple decode threads caused an unintended bottleneck (especially with fast-decoding formats).

edit: I should also have mentioned I thought an instance of STDIO was limited to one thread per file, but this may be a bad assumption on my part.

This post has been edited by Destroid: Nov 26 2012, 22:08