A multicore FLAC encoder Options

[GeorgeFP]

Hi!

Maybe you already know fpMP3Enc, my multicore MP3 encoder...

This time, I'd like to introduce my multicore FLAC encoder. You can download the binaries here.

The package contains Win32 binaries for XP, Vista and 7, and Win64 binaries for Vista and 7.

The FLAC encoder is part of my "fpStream" project. The command-line to convert a WAV file to FLAC is as follows:

CODE

FPSTREAM.EXE readfile in -f <wav-filename> + wavin wav -s in + fpflacenc enc -s wav [-<compression level>] + writefile out -s enc -f <flac-filename>

The benchmarks so far:
I used the same test suite as in fpMP3Enc (Intel Q9450, 61 WAV files, about 5 hours playing time). The compression level was 5. The files were encoded one after the other, not in parallel.

- flac 1.2.1: 3 min 26 secs
- fpFLAC x64: 1 min 13 secs

So, the speedup was 2.8x. This is quite good if you consider that this version does not contain any SSE optimizations.

What's missing:
- SSE optimizations
- Metadata support (tags, seek points etc.)
- free of bugs

And some limitations:
- Win32 version: The total WAV file size sum must not exceed 400MiB(? - not sure).
- Win64 version: The total WAV file size sum must not exceed 400GiB.
- Input: WAV 16-bit stereo, 44.1kHz

Comments and feedback are welcome!
George

[Cokemonkey11]

Good stuff, can't wait for the GUI.

[punkrockdude]

Yeah! I love that your make multi core applications out of stuff I use regularly. I too can't wait for a GUI or a version that can accept the standard FLAC commands. Regards

[flacflac]

Wow, this is very interesting. The speed-gain you're describing happens when just converting a single file? Is there any danger of disk-thrashing? How many threads are being calculated at once?
And lastly: is this an open source tool?

Thank you for this development.

[Maurits]

Is there any chance that these improvements can be used 'upstream' in the original FLAC sources? That way the benefits of this are not limited just to the users of one specific operating system.

[hlloyge]

It would be interesting to compare it to CUDA FLAC.

[GeorgeFP]

Wow, this is very interesting. The speed-gain you're describing happens when just converting a single file? Is there any danger of disk-thrashing? How many threads are being calculated at once?
And lastly: is this an open source tool?

Thank you for this development.

This evening I will provide a new package which will also contain SSE optimized versions. The SSE4 version made the test suite in 55s which is a speedup of 3.75x.

The I/O performance was 72.7 MiB/s (3 GiB read, 1 GiB write) when converting single files.

The whole process is stream-based, so there shouldn't be any disk-thrashing.

About the number of threads on a quad-core system:
- 1 main thread for the application (idle most of the time)
- 4 CPU worker threads that do the encoding
- 1 I/O thread for asynchronous I/O
- 1 I/O completion port thread

So, the formula is: numThreads = 3 + numCores

About open source: Yes, it's open source. The code will be available for download soon.

[GeorgeFP]

Is there any chance that these improvements can be used 'upstream' in the original FLAC sources? That way the benefits of this are not limited just to the users of one specific operating system.

This will not be possible. The programming paradigms are different.

Linux versions are planned for Q2/2010.

[Lucho]

CODE

FPSTREAM.EXE readfile in -f <wav-filename> + wavin wav -s in + fpflacenc enc -s wav [-<compression level>] + writefile out -s enc -f <flac-filename>

This is so confusing when it throw an error i dont know which one is argument 1 , argument 2 .... more simplified version will do the trick

Apart from that good work.

[GeorgeFP]

This is so confusing when it throw an error i dont know which one is argument 1 , argument 2 .... more simplified version will do the trick

You can use a simple batch file:

CODE

@fpstream readfile in -f %1 + wavin wav -s in + fpflacenc enc -s wav + writefile out -s enc -f %2

If the name of the batch file is 'doflac.bat' then the cmdline is:

CODE

doflac <wav-filename> <flac-filename>

[GeorgeFP]

OK, I've uploaded a new version. The link is the same as before.

In order to use the SSE versions of 'fpFLAC' you have to copy them from either the 'sse2' or the 'sse4' folder into the main folder.

Edit: Sorry, I found a bug the SSE code (now I know why it was so fast...) . I had to rebuild the project and upload a new version.

This post has been edited by GeorgeFP: Nov 16 2009, 19:22

[GeorgeFP]

It would be interesting to compare it to CUDA FLAC.

Well, the "problem" is that FLAC encoding is already I/O-bound when running on quad-cores - at least on my system. So a CUDA FLAC or a >4-core-system will not run faster than my app.

[flacflac]

Hi GeorgeFP,

thank you for this exciting multicore implementation.

I have a couple of problems using it, perhaps you can help out a bit:

1) Using Win7 x64, most compression settings seem to crash the tool: only if I keep don't specify it or use -8 does it work, and there seems to be a difference in filesize between the two, so I am not sure what the default compression setting is - can you reproduce this problem? (it starts encoding but just doesn't finish)

2) The times I need to encode quickly are either during CD extraction using EAC or when I re-encode existing FLACs as I used lower compression setting before but would like to "upgrade". Can you please tell me whether you are planning to support TRANSCODING existing FLACs? Also, is the compression setting the only original FLAC argument the tool supports?

3) Is it possible to align commands with the original FLAC encoder to be able to simply substitute the current FLAC.exe with your development in EAC?

4) I add padding to my FLACs as I embed album art - unfortunately the padding command as specified in the FLAC documentation seems unsupported (http://flac.sourceforge.net/documentation_tools_flac.html) - implementation should not be a big problem, or am I mistaken?

5) Is tag support a possibility in the near future?

Again, thank you!
ff

[GeorgeFP]

Thanks for your feedback!

1) Using Win7 x64, most compression settings seem to crash the tool: only if I keep don't specify it or use -8 does it work, and there seems to be a difference in filesize between the two, so I am not sure what the default compression setting is - can you reproduce this problem? (it starts encoding but just doesn't finish)

I could reproduce the problem with "-1" and "-4". Both modes use loose mid-stereo which requires special handling. I thought, I made it right, but obviously not. It will be resolved in the next release.
BTW, the default is "-5".

2) The times I need to encode quickly are either during CD extraction using EAC or when I re-encode existing FLACs as I used lower compression setting before but would like to "upgrade". Can you please tell me whether you are planning to support TRANSCODING existing FLACs? Also, is the compression setting the only original FLAC argument the tool supports?

Transcoding is already possible:

CODE

FPSTREAM readfile in -f <old-flac-filename> + flacdec dec -s in + fpflacenc enc -s dec + writefile out -s enc -f <new-flac-filename>

Another option that my tool supports is "--lax" but I don't know what it does. It was easy to port it from the original FLAC code, so I did it.

3) Is it possible to align commands with the original FLAC encoder to be able to simply substitute the current FLAC.exe with your development in EAC?

Yes. I also plan to provide something like a "fpFLAC.exe" with the same command-line options as the original FLAC - for simple encodings.

4) I add padding to my FLACs as I embed album art - unfortunately the padding command as specified in the FLAC documentation seems unsupported (http://flac.sourceforge.net/documentation_tools_flac.html) - implementation should not be a big problem, or am I mistaken?

This is not a problem, the code is already written, only the cmd-line option is missing. Which tool are you using to embed the album art? I'll test it with that.

5) Is tag support a possibility in the near future?

Yes. In the next release you should be able to set title, artist, album, year, track and genre. Just tell me if you need more.

Thanks again for your feedback!

[flacflac]

Hi George, thank you for that detailed reply!

I could reproduce the problem with "-1" and "-4". Both modes use loose mid-stereo which requires special handling. I thought, I made it right, but obviously not. It will be resolved in the next release.
BTW, the default is "-5".

Good to know, I am using 8 anyways, but just thought I'd point that out.

Transcoding is already possible:

CODE

FPSTREAM readfile in -f <old-flac-filename> + flacdec dec -s in + fpflacenc enc -s dec + writefile out -s enc -f <new-flac-filename>

Oh, excellent!!! Will test it out tomorrow!

Another option that my tool supports is "--lax" but I don't know what it does. It was easy to port it from the original FLAC code, so I did it.

This is an option that allows you to generate a rather non-conform FLAC file, which is not really advisable - but hey, any command that you can port is fine with me, just don't encourage people to use this one as they might end up with FLACs that don't play anymore through their streaming hardware.

Yes. I also plan to provide something like a "fpFLAC.exe" with the same command-line options as the original FLAC - for simple encodings.

Great, I think it could make your development widely used, as you could tie it into foobar, eac and anything else without changing every tool around the current command-sequence.

Regarding padding:

This is not a problem, the code is already written, only the cmd-line option is missing. Which tool are you using to embed the album art? I'll test it with that.

Excellent! I actually use www.mp3tag.de to embed, and I just keep a small padding of 64kbyte (= -p 65536). This way no rewrites become necessary when embedding.

Yes. In the next release you should be able to set title, artist, album, year, track and genre. Just tell me if you need more.

Awesome! Perhaps you could implement this support in an "open" fashion, i.e. the way flac.exe does it:

-T FIELD=VALUE
Add a FLAC tag. The comment must adhere to the Vorbis comment spec (which FLAC tags implement), i.e. the FIELD must contain only legal characters, terminated by an 'equals' sign. Make sure to quote the comment if necessary. This option may appear more than once to add several comments. NOTE: all tags will be added to all encoded files.

So, as long as the Vorbis comment spec tag is used, it would work with any of these:

http://age.hobba.nl/audio/mirroredpages/ogg-tagging.html

I am not sure how, but EAC allows for storing of non-standard tags - I personally use the DISCID that comes from CDDB when encoding CD rips made with EAC (-T "GENRE=%m"). If you need some beta testing, let me know.

Again thank you, this could really rock!
ff

[Cokemonkey11]

Yes. In the next release you should be able to set title, artist, album, year, track and genre. Just tell me if you need more.

Is album art possible to be defined as a tag?

[GeorgeFP]

Is album art possible to be defined as a tag?

The option is "--picture=SPECIFICATION" in the original FLAC. I will add it.

[GeorgeFP]

Awesome! Perhaps you could implement this support in an "open" fashion, i.e. the way flac.exe does it:

-T FIELD=VALUE
Add a FLAC tag. The comment must adhere to the Vorbis comment spec (which FLAC tags implement), i.e. the FIELD must contain only legal characters, terminated by an 'equals' sign. Make sure to quote the comment if necessary. This option may appear more than once to add several comments. NOTE: all tags will be added to all encoded files.

Ok

[Cokemonkey11]

Is album art possible to be defined as a tag?

The option is "--picture=SPECIFICATION" in the original FLAC. I will add it.

I meant in relation to my quoted post. You mentioned arist, album, title.. etc

Album Art should be one of those tags, unless I'm incorrect.

[GeorgeFP]

Is album art possible to be defined as a tag?

The option is "--picture=SPECIFICATION" in the original FLAC. I will add it.

I meant in relation to my quoted post. You mentioned arist, album, title.. etc

Album Art should be one of those tags, unless I'm incorrect.

When I add the "-T" option it will be possible to write "-T ALBUMART=pic.png". AFAIK, only the filename will be saved then, not the image data.

[GeorgeFP]

I have uploaded a new version.

The release notes are:
- Fixed problem with adaptive mid-side
- Added cmd-line option "-P <size>" (Adds PADDING block)
- Added cmd-line option "-T <field>=<value>" (Adds VORBIS comment)

I've tested this versions with all possible compression levels without problems.
I've also included a readme.txt that shows some features of the command-line

I've made a new benchmark test. This time I used compression level "-8". The results:

FLAC 1.2.1: 7 mins 44 secs
fpFLAC x64: 2 mins 13 secs

Speedup = 3.5x; Efficiency = 87.5%

[flacflac]

I have uploaded a new version.

The release notes are:
- Fixed problem with adaptive mid-side
- Added cmd-line option "-P <size>" (Adds PADDING block)
- Added cmd-line option "-T <field>=<value>" (Adds VORBIS comment)

I've tested this versions with all possible compression levels without problems.
I've also included a readme.txt that shows some features of the command-line

Hi George, excellent update!! Tagging (even random tags) is working, great job! I have 3 issues:

1) Could you look into supporting the -V tag? (that's capital 'V' or --verify):

-V, --verify
Verify the encoding process. With this option, flac will create a parallel decoder that decodes the output of the encoder and compares the result against the original. It will abort immediately with an error if a mismatch occurs. -V increases the total encoding time but is guaranteed to catch any unforseen bug in the encoding process.

V is very important to many people encoding from WAV as this option increases confidence that nothing got damaged during the encoding process - I don't know how complicated it is to implement that, probably not super easy. It would be great, however!

2) I am having some trouble using the batch conversion mentioned in the readme for transcoding:

CODE

(from my 'doflacmulti.bat')
@fpstream readfile in*i -f "*fp" + flacdec dec*i -s in*i + fpflacenc enc*i -8 -P 65536 --verify -T DISCID=123 -s dec*i + writefile out*i -s enc*i -f "*n.flac"

I keep getting:

ERROR (enc*i): Syntax Error (argument: 16)
ERROR: Cannot create task 'fpflacenc'

I guess I must be missing something, perhaps you can point it out to me?

3) Would it be difficult to have existing tags preserved? Especially for transcoding files this would be a very important feature.


But like I said, great update and I LOVE the speed of this - thank you so much.

[GeorgeFP]

1) Could you look into supporting the -V tag? (that's capital 'V' or --verify):

It's not easy to implement this but there is a way. The biggest problem will be that you will not be able to specify just the -V tag. Instead you will have to add an additional FLAC decoder and a verifier task to the (very long) command line.

2) I am having some trouble using the batch conversion mentioned in the readme for transcoding:

CODE

(from my 'doflacmulti.bat')
@fpstream readfile in*i -f "*fp" + flacdec dec*i -s in*i + fpflacenc enc*i -8 -P 65536 --verify -T DISCID=123 -s dec*i + writefile out*i -s enc*i -f "*n.flac"

I keep getting:

ERROR (enc*i): Syntax Error (argument: 16)
ERROR: Cannot create task 'fpflacenc'

The first problem is the "--verify" option which is not supported.
The second problem is that you haven't specified a file mask (see (7) in readme.txt). The parser will treat "*fp" as a file name.

3) Would it be difficult to have existing tags preserved? Especially for transcoding files this would be a very important feature.

I already have a plan for that

Right now, I'm preparing the web site for the encoder with some explanations about how the algorithm works.

[GeorgeFP]

Here is a screenshot how to configure EAC to use fpFLAC:

Of course, you can set a compression level other than "-8".

On "EAC -> EAC options -> Tools", set the number of "simultaneous external compressor thread(s)" to 1.

[howesz]

For ease of integration with EAC (i.e. so you can copy the text and paste it ) here is the string I used...

readfile in -f %s + wavin wav -s in + tag tag "ARTIST=%a" "TITLE=%t" "ALBUM=%g" "YEAR=%y" "TRACK=%n" "GENRE=%m" + fpflacenc enc -s wav -8 -tag tag + writefile out -s enc -f %d


GeorgeFP,
How hard will it be to add replay gain support? (Please?) With the addition of Verify support (mentioned above), I'll be all set

My usual EAC FLAC 1.2.1b string is:

-8 --replay-gain -V -T "artist=%a" -T "title=%t" -T "album=%g" -T "date=%y" -T "tracknumber=%n" -T "genre=%m" %s

Thanks for the great program!!! Keep up the great work!!!