Parallel FLAC encoding via windows frontend, to use all my CPU cores Options

[jrbamford]

So since getting a new machine my flac encoding is quicker.. could be quicker though.. I use the default flac frontend, but when this is encoding i only ever see 50% CPU usage.. ie. one core is twiddling its thumbs.

Anyone got a good strategy to do two flac files at the same time? Is there an option in the frontend to do more than 1 at a time?

One other thing, did something change in the FLAC format over the years.. in the old days the tagging at the end of the encode task used to be like lightning.. now even with a 5x faster machine its now quite slow taking a second or so per track. Just curious on that.

Cheers

[Egor]

Anyone got a good strategy to do two flac files at the same time? Is there an option in the frontend to do more than 1 at a time?

foobar2000 will launch one encoder per CPU on encoding, it supports Unicode, FLAC/OggFLAC tagging. Just install, unpack flac.exe to the foobar2000 program files dir, and use the built-in Converter feature.

[greynol]

All hail foobar2000! Now wouldn't it be nice to be able to disable this feature as it is a complete joke when trying to convert files stored on optical media.

Back on topic, I personally run two two batch scripts at the same time, but would definitely recommend foobar2000 (just not for stuff stored on optical media). I really don't have a favorable opinion of FLAC frontend, FWIW.

This post has been edited by greynol: Aug 16 2007, 06:35

[jrbamford]

Thanks guys... I've never really sorted out foobar to encode flac, i'll look into it... can't beat the near 2x speed up it should give me.

I don't mind the flac frontend that much.. i always worry with foobar that some noise shaping, dithering etc is on, or something like that.. something you dont have to worry about with the FLAC tools as they're not available options.

[djstone86]

(just not for stuff stored on optical media)

why not?

This post has been edited by djstone86: Aug 17 2007, 01:53

[Tahnru]

(just not for stuff stored on optical media)

why not?

Because of seek times/bandwidth. At some point the bottleneck moves away from the CPU and the bottleneck becomes the source instead. Optical media suffers greatly when doing lots of random seeks across the discs surface, as would be required for reading two files at once.

It would be a wonderful to have finer control over the multiple core encoding - I could specify two seperate sources and have them spit out back to a third unit (two optical drives with the result to a external hard drive, for example).

[djstone86]

oh thanks the explanation, one more q off topic, does flac 1.2 take advantage of sse4?

[Tahnru]

SSE(x) support depends on the compiler and settings used when building the files.

Because the Peryn-cored Intel chips aren't out yet (I think?) and the Intel Compiler v10.0 is really new, I don't think there are any SSE4 capable binaries floating around out there. I am quite ready to be wrong about that statement.

Short finish - if the FLAC binaries available now don't support SSE4, they soon can/will.

[Moitah]

The only problem encoding multiple FLACs at once is they can come out very fragmented. I wrote a quick test app to encode using a configurable number of threads at once. If I only encode 1 at a time, the files are okay (< 10 fragments per file), but with 2 at a time the files come out in hundreds of fragments. I tried foobar2000 also which runs 2 encoders at the same time and the files are just as fragmented. For some reason I had the idea that NTFS would be smart enough to avoid fragmentation when writing multiple files at once, but apparently not. Of course, my drive is 95% full (~12/232 GB free) but I think the free space is mostly consolidated. I'll try it on a freshly defragmented drive with tons of free space and see what happens.

[Tahnru]

It will still fragment. I have the same problem, however, I defragment when I sleep.

[Moitah]

Even on a drive with 108/232 GB free, completely defragmented, and the free space consolidated, the same thing happened. When encoding 2 at a time, the average file came out with 230 fragments.

[Tahnru]

Although I haven't tested to see what is happening, my guess is that the two encoder processes alternate writing blocks to the hard drive. Instead of fragmentation where the files are randomly all over the surface of the drive, I would expect to find that this fragmentation is interleaving the blocks from each output file.

Seek time is the killer with random fragmentation. My thought is that a person may not have to worry about this particular mechanism of fragmentation.

[odyssey]

So since getting a new machine my flac encoding is quicker.. could be quicker though.. I use the default flac frontend, but when this is encoding i only ever see 50% CPU usage.. ie. one core is twiddling its thumbs.

Anyone got a good strategy to do two flac files at the same time? Is there an option in the frontend to do more than 1 at a time?

Hint: Hit "Encode" twice

[Moitah]

I used my own IO callbacks instead of letting libFLAC handle the file writing so I can preallocate the file (SetEndOfFile Win32 or FileStream.SetLength .NET). I allocate it to the size of the raw uncompressed audio data and then trim the file to the correct size when done writing. This completely solves the fragmentation issue. I don't know if I will polish and release this, maybe if anyone really wants it.

[Tahnru]

Perhaps it is a useful patch that could be added to the FLAC encoder as a command-line switch? Something like -pre to preallocate output files if a person wants it.