A multicore FLAC encoder Options

[GeorgeFP]

1) the encoding algorithm is standardized, correct? that is, obviously it is going to provide lossless encoding/decoding, and compress to (at least) the same filesize as one encoded with the original FLAC.exe? i know there were some projects going on trying to increase performance/compression on this codec.... which i don't like, standards are made for reasons

The encoding algorithm is exactly the same (except some SSE optimizations) as the original FLAC.

2) you say the current algorithm for encoding is i/o bound, i'm assuming this means the encoder can encode faster than the hard drive can read/supply the data, is this what you mean? unfortunately, this probably means that on faster systems, the speed improvement (vs. the original encoder) will have a tendency to decline the faster the system gets. fortunately, this also will improve with current and future tech like SSD drives...

The encoder is embarassingly parallel so sooner or later the I/O will become the bottleneck.

[GeorgeFP]

Considering that EAC only passes one file at a time and has its own internal ability to make use of multiple processes, can someone please explain how you derive a benefit from this program with EAC?

If it's possible to utilize all cores with one single file, why would you want to process more than one file in parallel?

[greynol]

Perhaps you could have made it a little more clear as to what your encoder does so that idiots like me wouldn't have to ask such stupid questions.

[GeorgeFP]

Perhaps you could have made it a little more clear as to what your encoder does so that idiots like me wouldn't have to ask such stupid questions.

So, you did NOT visit my web site? Take a look here.

BTW, it seems I've found the bug I've mentioned before - one hour full-speed encoding without a hang.
The new version is online, the names are the same.

[greynol]

Well no. Why didn't you provide that link in the first place?

[jamesbaud]

To use "fpMP3" or "fpFLAC2MP3" you have to download the source code and build the "fpMP3_DLL" project (Blend, SSE2, or SSE4 version). The built DLLs have to be registered with "regsvr32" (admin rights required for that).

I would like to use fpFLAC2MP3. But I'm not able to build the project. I'm just not into this subject.
Maybe somebody smart who owns all the needed software likes to build the project and upload it here in the forum?

That would be very appreciated.

Any chance the built files could be hosted on Rarewares?
The "old" fpMP3Enc is already there.
http://www.rarewares.org/mp3-others.php

[GeorgeFP]

A new version with two bug fixes is available for download:

64-Bit version
32-Bit version (not supported)

Bug#1: Application hung (became idle) sometimes
Bug#2: Memory access violation

Tomorrow I will start work on the -V option and the replay gain feature.

[JacksonGrey]

I keep getting an error that a number of files 'failed to register'. Anyone else experiencing this? Files are installed into the Program Files directory, but any conversion fails in Foobar.

This post has been edited by JacksonGrey: Dec 8 2009, 01:00

[GeorgeFP]

I keep getting an error that a number of files 'failed to register'. Anyone else experiencing this? Files are installed into the Program Files directory, but any conversion fails in Foobar.

The Visual C++ runtimes are not included in the package. You can download them here:

32-Bit
64-Bit

You can also try to register the DLLs manually:
1. Open a command prompt as administrator
2. Go to the installation directory
3. Type

CODE

regsvr32 fpFLAC.dll
regsvr32 fpFLAC_SSE2.dll
regsvr32 fpFLAC_SSE4.dll
regsvr32 fpMP3.dll
regsvr32 fpMP3_SSE2.dll
regsvr32 fpMP3_SSE4.dll
regsvr32 fpStreamLib.dll
regsvr32 fpWave.dll
regsvr32 lFLAC.dll
regsvr32 lVorbis.dll

[GeorgeFP]

I've found the problem with the installer: Several files were missing in the packages so I had to rebuild them again. Sorry for that. The above links still work.

You may get an error when registering fpFLAC_SSE2.dll or fpFLAC_SSE4.dll. This happens when your system does not support SSE2 or SSE4. Just press "Continue" then.

The new installer has updated versions of fpFLAC and fpFLAC2FLAC where you can pass the -V (verify) option.

[greynol]

I'd love to see this done with metaflac so that I can do quick replaygain scanning.

[NuSkooler]

I cannot wait for this to be ready for production so to speak!

Any chance @ GPU support?

[GeorgeFP]

I cannot wait for this to be ready for production so to speak!

Well, the truth is, I don't know if fpFLAC has a future. The feedback is very, very poor.

Any chance @ GPU support?

I will add GPU support into my framework and if the fpFLAC project will be continued it will also support GPUs.

[Bylie]

Well, the truth is, I don't know if fpFLAC has a future. The feedback is very, very poor.

Well to be honest I've been following all of the new projects with a lot of interest and am really impressed by the progress made. Who wouldn't be:

• fpFLAC/MP3 makes much better use of multicore CPU's which are destined to be the future as CPU clockrates are stagnating for quite some time now. I just saw that also GPU support will be integrated, very nice!
• FLACuda already leverages all the GPU processing potential currently being neglected in anything but games.
• TAK which has made some nice improvements compressing audio even further.
• Others which I probably forgot...

But, personally, I'm not very inclined to switch to a new, unproven piece of software even if it incorporates proven code and/or algorithms like FLAC, WavPack, LAME, ...

Maybe a better approach to this would be, to somehow try, to integrate these new projects into the existing, proven, codebase supervised and/or publicly supported by the original developers trying to maintain the same stability and quality of the original software. This might even be a different branch at first but I really think the "umbrella effect" could have a positive effect on the acceptance of the new code/projects by the users which ultimately will result in more useage and feedback.

I know this is sort of an idealistic vision because in reality it might just not be that easy due to a number of realworld problems/conflicts (licensing, time, ...) but I really think it could give these new projects a lot more visibility and it might even give the original software a new momentum.

This post has been edited by Bylie: Dec 24 2009, 15:24

[Gregory S. Chudo...]

Agree. Josh, if you are interested, i'm ready to spend some time helping to incorporate CUDA code.

[skamp]

Josh, if you are interested, i'm ready to spend some time helping to incorporate CUDA code.

GPU encoding in flac through libcuda under linux would be ideal (awesome).

[_m²_]

I cannot wait for this to be ready for production so to speak!

Well, the truth is, I don't know if fpFLAC has a future. The feedback is very, very poor.

Well, there's some feedback I can give you. I installed it and deleted shortly after. I'm not an user, but I can tell you why.

1. For me FLAC is dead, it's too weak, so you came too late. Multithreaded wavpack or mac would probably make more sense.

2. The last command line that I used was
flac -8 -A tukey(0.5) -A flattop -r 0,8 -V
and I considered adding -p, but it was a bit too slow for the gain. Overall it's slightly stronger than -8.

If fpFLAC allowed me to compress slightly better with reasonable speed (and I wouldn't be shifting to TAK), I would use it, though probably with verification done by regular FLAC during the testing period.
However, fpFLAC doesn't accept these switches, so (I guess) it's even weaker than regular FLAC. For me - pointless.

3. Installers are annoying. Regular 7zip package would be better. Can't you make it work w/out registering these dlls? Also, I got error during registration of smth with SSE4 in the name. XP 32bit, SP3, Core 2 Duo. Program worked at least long enough to show help, seeing lacking switches, I didn't even bother to compress something.

4. No Uninstaller. That's extremely annoying. I guessed that I should use the installer, but if I didn't keep it (for these several minutes ), I would just delete the program and have rubbish left in the registry and possibly other places too. And I wouldn't assume that everybody will guess, even considering that the target audience is rather advanced.

5. FLAC is (pendrive) portable, fpFLAC - I don't know. For me, it's very important. The only non-portable program that I found worth using is Visual Studio.

[Qest]

It would be outstanding if the defacto flac tools had support for multiple cores (like fpFLAC) and gpu's (like FLACuda)! That would be the best of all worlds.

[NuSkooler]

1. For me FLAC is dead, it's too weak, so you came too late. Multithreaded wavpack or mac would probably make more sense.

I doubt this is any reason for this particular project to receive minimal attention from outsiders. FLAC is the defacto standard for lossless audio. Sure, WavPack & others have gained some ground (especially around here) but in the "real world" you're going to find FLAC. I think some of the reason is people simply do not know of this. I use FLAC on a every day basis and just found out about this a few days ago. Also, until it can replace flac.exe with confidence, obviously it won't be used by many.

3. Installers are annoying. Regular 7zip package would be better. Can't you make it work w/out registering these dlls? Also, I got error during registration of smth with SSE4 in the name. XP 32bit, SP3, Core 2 Duo. Program worked at least long enough to show help, seeing lacking switches, I didn't even bother to compress something.

I agree. Especially for "utility" type stuff. My suggestion: Statically link everything needed into a fpFLAC.exe. I believe this will make it much more open to the masses.

[Zarggg]

Will this program yield any benefit to a (single core) Pentium 4 with Hyperthreading enabled? My immediate thought would be "no", but I still want to ask.

I apologize if this seems like a dumb question, but it's not very obvious to me and I'm only just starting to learn about how multi-core processors work.

[GeorgeFP]

But, personally, I'm not very inclined to switch to a new, unproven piece of software even if it incorporates proven code and/or algorithms like FLAC, WavPack, LAME, ...

Maybe a better approach to this would be, to somehow try, to integrate these new projects into the existing, proven, codebase supervised and/or publicly supported by the original developers trying to maintain the same stability and quality of the original software. This might even be a different branch at first but I really think the "umbrella effect" could have a positive effect on the acceptance of the new code/projects by the users which ultimately will result in more useage and feedback.

Believe me, if I had the chance to integrate my framework into the existing C code I would have done so. But multi-core programming is very different to serial programming and thus, I had to write fpFLAC from scratch.

Of course, you can wait until the original developers will add multi-core support to their projects. I don't know of any activities going in this direction, neither for FLAC nor for LAME.

[Bylie]

Believe me, if I had the chance to integrate my framework into the existing C code I would have done so. But multi-core programming is very different to serial programming and thus, I had to write fpFLAC from scratch.

Of course, you can wait until the original developers will add multi-core support to their projects. I don't know of any activities going in this direction, neither for FLAC nor for LAME.

I really try to not underestimate the amount of work and research that goes into these projects and I can surely understand that saying something like "integration will yield benefit x and y" might sound much easier than actually doing it!
However, the problem will remain for projects like fpFLAC and others that they are seen more like a proof of concept and hence are more interesting for developers and testers which won't make a large userbase. This might actually not be a problem if the goal is to provide a proof of concept, but if the idea is to get more and more realworld usage/feedback I think somewhere along the line the proof of concept stage has to evolve into something more users will (want to) use. This is where the "umbrella effect" or "public developer support" might play a role as it gives visibility to these projects beyond this forum and also, kind of, assures the large group of endusers that it's OK to also use this for anything else than testing.

This post has been edited by Bylie: Dec 25 2009, 18:46

[Fandango]

Uhm, I would happily replace all single-threaded command line encoders right away, with multi-core + GPU utilizing encoders. It annoys me everytime I transcode a single large file, that more than half of my CPU is idle, while I am wating... waiting... waiting...

[GeorgeFP]

3. Installers are annoying. Regular 7zip package would be better. Can't you make it work w/out registering these dlls? Also, I got error during registration of smth with SSE4 in the name. XP 32bit, SP3, Core 2 Duo. Program worked at least long enough to show help, seeing lacking switches, I didn't even bother to compress something.

I agree. Especially for "utility" type stuff. My suggestion: Statically link everything needed into a fpFLAC.exe. I believe this will make it much more open to the masses.

My "streaming framework" which is similar to DirectShow uses the registry to find the components/filters. I think this is a better approach than putting everything into a single file, especially when you have optimizations for different processor features (SSE, CUDA, Stream, ...)

[GeorgeFP]

I really try to not underestimate the amount of work and research that goes into these projects and I can surely understand that saying something like "integration will yield benefit x and y" might sound much easier than actually doing it!
However, the problem will remain for projects like fpFLAC and others that they are seen more like a proof of concept and hence are more interesting for developers and testers which won't make a large userbase. This might actually not be a problem if the goal is to provide a proof of concept, but if the idea is to get more and more realworld usage/feedback I think somewhere along the line the proof of concept stage has to evolve into something more users will (want to) use. This is where the "umbrella effect" or "public developer support" might play a role as it gives visibility to these projects beyond this forum and also, kind of, assures the large group of endusers that it's OK to also use this for anything else than testing.

I agree with you. For me, it's ok if fpFLAC is seen as a proof of concept and the project can end here right away.