A multicore FLAC encoder Options

[GeorgeFP]

Hi!

Maybe you already know fpMP3Enc, my multicore MP3 encoder...

This time, I'd like to introduce my multicore FLAC encoder. You can download the binaries here.

The package contains Win32 binaries for XP, Vista and 7, and Win64 binaries for Vista and 7.

The FLAC encoder is part of my "fpStream" project. The command-line to convert a WAV file to FLAC is as follows:

CODE

FPSTREAM.EXE readfile in -f <wav-filename> + wavin wav -s in + fpflacenc enc -s wav [-<compression level>] + writefile out -s enc -f <flac-filename>

The benchmarks so far:
I used the same test suite as in fpMP3Enc (Intel Q9450, 61 WAV files, about 5 hours playing time). The compression level was 5. The files were encoded one after the other, not in parallel.

- flac 1.2.1: 3 min 26 secs
- fpFLAC x64: 1 min 13 secs

So, the speedup was 2.8x. This is quite good if you consider that this version does not contain any SSE optimizations.

What's missing:
- SSE optimizations
- Metadata support (tags, seek points etc.)
- free of bugs

And some limitations:
- Win32 version: The total WAV file size sum must not exceed 400MiB(? - not sure).
- Win64 version: The total WAV file size sum must not exceed 400GiB.
- Input: WAV 16-bit stereo, 44.1kHz

Comments and feedback are welcome!
George

[Chinch]

Alright! I'm impressed. I have to say that I didn't think the improvement would be that great on my system, but here are my results (based on about a 53 minute CD):

original FLAC encoder (-8) = 1 minute, 15 seconds.
and George FP's encoder (-8) = 24 seconds!

i didn't believe it, so I did a bit-compare on the two files:

CODE

All tracks decoded fine, no differences found.
    
    Comparing:
    "X:\encode1.flac"
    "X:\encode2.flac"
    No differences in decoded data found.

and then I did a verify on the FLAC files, to make sure they were identical (audio data-wise):

CODE

Item: "X:\encode1.flac"
    MD5: 66B84449D7A9899065A11FA190C994AC
    CRC32: D199E9A2
    No problems found.
    
    All items decoded successfully.
    
    
    Item: "X:\encode2.flac"
    MD5: 66B84449D7A9899065A11FA190C994AC
    CRC32: D199E9A2
    No problems found.
    
    All items decoded successfully.

finally, to prove that it wasn't EXACTLY THE SAME THING, i have noted the file sizes and hashes of both FLAC files... different! both files are unique.

CODE

File: encode1.flac
    Size: 357MB (374,398,783 bytes)
    CRC-32: fa984f9e
       MD5: 6eb400219c22f1aa1525d13f5f7f1135
     SHA-1: 70aa7e4a65ed635eb78b52ef3d2f1b1b8b435692
    
    File: encode2.flac
    Size: 356MB (374,328,806 bytes)
    CRC-32: eac29087
       MD5: a8d830f2defbe50af368ce60a5a69460
     SHA-1: c356080e0e508d26b8d7b625229af0d2ca2dde7b

so George... you have made a believer of me! empirical evidence, right there for me and everyone else to see.... congrats!
I am about to do a second test on a second system, then I will post those results below.

Results from test #2:

original FLAC encoder (-8 compression) = 1 minute, 07 seconds
and George FP's encoder (-8 compression) = 21 seconds

just for kicks I tried flaCUDA...

flaCUDA encoder (-11 compression) = 58 seconds
flaCUDA encoder (-8 compression) = 16 seconds

CODE

E:\Temp>CUETools.FlaCuda.exe -11 encode.wav -o encode3.flac
      CUETools.FlaCuda, Copyright (C) 2009 Gregory S. Chudov.
    Filename  : encode.wav
      File Info : 44100kHz; 2 channel; 16 bit; 00:52:53.4930000
      Results   : 53.89x; 373021162 bytes in 00:00:58.8901034 seconds;
      
      E:\Temp>CUETools.FlaCuda.exe -8 encode.wav -o encode4.flac
      CUETools.FlaCuda, Copyright (C) 2009 Gregory S. Chudov.
      Filename  : encode.wav
      File Info : 44100kHz; 2 channel; 16 bit; 00:52:53.4930000
      Results   : 198.27x; 373969399 bytes in 00:00:16.0056281 seconds;

but there is a discrepancy.... flaCUDA's compression scheme goes up to 11, and that is what the 58 second result was based on... when I use compression -8, the encoding time ends up only taking 16 seconds. jeez. I'm not sure how the compression numbers crossover though. all 4 output files are identical to the original and each other (same audio MD5's), the two files produced by FLAC and fpFLAC both have a bitrate of 944, whereas the flaCUDA -11 encode shows a 940 bitrate, and the flaCUDA -8 encode shows a 943 bitrate (in foobar2000) -- all end-product FLAC files are sized between 355MB and 357MB, so negligible difference, size wise. *shrug* i have no idea.... already put waaaay too much thought into this as it is haha

anyways, with differing compression schemes, it's hard to make a solid conclusion.... other than that both of these two are VERY FAST encoders!

oh yeah, and of course, I forgot to include what specs I was using to test:

Intel i7-860 (2.8Ghz stock, 3.47Ghz in turbo mode -- it's a Quad-core with Hyper-Threading, so it sees it as 8 CPU's)
4GB DDR3-1600 RAM
Windows7 x64... I used of course the 64 bit version and the SSE4 instructions as George said in the readme
(and video card is Nvidia GeForce 9600 GSO)


This post has been edited by Chinch: Dec 2 2009, 08:18