fpMP3Enc: a multi-core MP3 encoder based upon LAME 3.98.2 Options

[GeorgeFP]

Hello,

I'd like to introduce my project "fpMP3Enc" - a multicore MP3 encoder based on LAME 3.98.2. It is a sample application/case study for my multicore library "Fiber Pool".

The source code can be downloaded here: http://www.fiberpool.de/en/downloads.html

The current version is still mostly single-threaded for single-file encoding but scales for multiple-files encoding.

The multicore tweaks used in this version are:
- parallel/asynchronous conversion/scaling of PCM data to float samples
- parallel/asynchronous computation of replay gain
- I/O ordering (WAV files are read into memory first before MP3 files are written to disk)

With the following test system
- Intel Q9450@2.66GHz (Quad Core)
- 8 GiB RAM
- Windows Vista x64 (Superfetch disabled)
- 61 WAV files (2.99 GiB, about 5 hours play time)

I've got the following results:
LAME 3.98.2 (x32; rarewares): 24.6x
LAME64 3.98 (x64; mp3tech): 20.8x
fpMP3Enc (x64; single): 23.3x
fpMP3Enc (x64; multi): 80.2x

Single-file encoding is slower than the original LAME because I haven't SSE-optimized the code.
The scale factor in multiple-files encoding is 3.3 compared to LAME and 3.4 compared to "fpMP3Enc single", which is quite good for a first version.

The next steps will be to perform the psycho acoustics computation for each frame in parallel, then the MDCT, and so on.

Compile and usage:
- You need Visual Studio 2008 (and obviously Windows) to compile the project.
- SSE4 is enabled by default. To disable: #undef USE_SSE4
- Memory control is disabled by default in this version. This can lead to pagefile swapping if too many files are to be encoded. You can enable memory control by using a different 'MEMORY_MODE' macro (experimental).
- Win32 version: The total WAV file size sum must not exceed 400MiB.
- Win64 version: The total WAV file size sum must not exceed 400GiB.
- Input: WAV 16-bit stereo, 44.1kHz
- Output: MP3 Stereo/Joint-Stereo, 44.1kHz

I'd like to know what you think about this project so contact me if you have any comments or feedback.
George

[GeorgeFP]

Hello, it's me again...

The last three weeks I was working on optimizations of the vbr-new algorithm. You can download the updated version from my web site.

The benchmarks (Vista x64, Intel Q9450@2.66GHz, 8 GiB RAM):
LAME 3.98.2 (x32): 32.3x
fpMP3Enc (x64; single file encoding): 60.3x
fpMP3Enc (x64; multi file encoding): 109.7x

This means that fpMP3Enc is about 87% or 1.87x faster than LAME in single file encoding, while the speedup is 3.4x in multi file encoding.

The following optimizations were performed:
- A frame buffer (20 MiB) is used to hold the work data for about 1000 frames
- Psycho acoustics are computed frame by frame in a separate task without data dependencies
- MDCT is computed frame by frame almost in parallel to psycho acoustics (right after attack detection, which is at the very beginning)
- MDCT (left channel) and MDCT (right channel) are computed in separte tasks (for details see here; English translation will follow)
- A small part of the "VBR_encode_frame" function is split into four tasks

ABR and vbr-old should also benefit from these changes. CBR is disabled in this version because this mode has some data dependencies that I have not handled yet.

To get best results you should not use more than 3 threads on quad cores or higher for single file encoding. The value can be set by using the "--threads" option.

There is still so much to optimize in the LAME code. Let's see how far it can get...

[Alexxander]

The benchmarks (Vista x64, Intel Q9450@2.66GHz, 8 GiB RAM):
LAME 3.98.2 (x32): 32.3x
fpMP3Enc (x64; single file encoding): 60.3x
fpMP3Enc (x64; multi file encoding): 109.7x

This means that fpMP3Enc is about 87% or 1.87x faster than LAME in single file encoding, while the speedup is 3.4x in multi file encoding.

Thanks for sharing your work. I'm no expert on programming but would like some things to be cleared up.

How many threads did you use with the presented results of fpMP3Enc ? I suspect the multi file encoding is done with 4 cores. If so, Lame encoding 4 or more files with one file per core would result in about 4 x 32.3=129.2 times encoding speed.

Also, should a native well designed 64 bits encoder be nearly twice as fast as its 32 bits counterpart or do the Core2Duo chips handle this well through some kind of emulation?

I'm trying to understand what is measured here and where the speed gain is coming from.

[GeorgeFP]

How many threads did you use with the presented results of fpMP3Enc ? I suspect the multi file encoding is done with 4 cores. If so, Lame encoding 4 or more files with one file per core would result in about 4 x 32.3=129.2 times encoding speed.

I used 3 threads in single and 4 threads in multi file encoding for the CPU bound tasks. I/O always needs two threads, one for the I/O scheduler and one for listening to the I/O completion port.

For the reason why it's not possible to get a 129.2x speed with 4xLAME, see my previous post.

Also, should a native well designed 64 bits encoder be nearly twice as fast as its 32 bits counterpart or do the Core2Duo chips handle this well through some kind of emulation?

AFAIK, the CPU intensive parts of LAME are SSE-optimized using 64- and 128-bit registers on both systems, x32 and x64. So, I don't think that an x64 encoder would be much faster than a x32 encoder.