A multicore FLAC encoder Options

[GeorgeFP]

Will this program yield any benefit to a (single core) Pentium 4 with Hyperthreading enabled?

I don't know. I would say "no" or "not much".

[skamp]

Encoding two files at once on my Intel Atom with hyperthreading typically yields a 40-50% improvement, which isn't negligible.

[Fandango]

Will this program yield any benefit to a (single core) Pentium 4 with Hyperthreading enabled?

I don't know. I would say "no" or "not much".

Having previously owned a HT P4, I'd say it will.

[NuSkooler]

My "streaming framework" which is similar to DirectShow uses the registry to find the components/filters. I think this is a better approach than putting everything into a single file, especially when you have optimizations for different processor features (SSE, CUDA, Stream, ...)

The reasons I suggest statically linking everything into a monster executable is portability. flac.exe for example can be dropped on a box and used right away. People like that.

Also, you can still optimize for various processor features in a single executable (e.g. dynamically use funcXXX() vs funcYYY()). Maybe I'm missing what you're trying to convey here?

[GeorgeFP]

The reasons I suggest statically linking everything into a monster executable is portability. flac.exe for example can be dropped on a box and used right away. People like that.

I understand that. Static linking would work for fpFLAC.

One of the next features will be AVI processing which I need for my next project 'fp264'. Because you cannot know how video and audio were encoded in a file you cannot link all codecs statically. So, another solution is needed to check if the system provides the components that are necessary to decode a stream. I've chosen the registry for that.

Another benefit of component based design will be, that it will be possible to write applications/frontends in any language and not only in C++.

[NuSkooler]

Another benefit of component based design will be, that it will be possible to write applications/frontends in any language and not only in C++.

Absolutely. Having everything component based framework and providing one "drop in" fpFLAC.exe binary is the best of both worlds.

As for the codecs, is there not a standard way they that they are already layed out in the Registry? If I install a codec, any application supporting codecs of said type can find them. I don't think there is much to invent there.

[GeorgeFP]

Another benefit of component based design will be, that it will be possible to write applications/frontends in any language and not only in C++.

Absolutely. Having everything component based framework and providing one "drop in" fpFLAC.exe binary is the best of both worlds.

fpFLAC is file based which is not always the best solution. For example, if you want to stream lossless audio (with unknown length) over the network. In this case, it's better to have the codec as a component where input and output can be controlled programmatically - in any language.

As for the codecs, is there not a standard way they that they are already layed out in the Registry? If I install a codec, any application supporting codecs of said type can find them. I don't think there is much to invent there.

You are right but depending on the used framework (VfW, ACM, DirectShow, Windows Media) the codecs are registered in different paths. I do exactly the same.

The first step will be to support legacy VfW or ACM codecs but later I or a 3rd party will provide multi-core-aware codecs.

[GeorgeFP]

BTW, I've decided to add some missing command-line options into fpFLAC (-b, -m, -M, -e, -A, -l, -p, -q, -r and --replay-gain).

[Meeko]

So I'm guessing a multicore FLAC encoder would only be most useful going from WAV or lossless files on a harddrive to FLACs? I've been reading this thread with interest, but I usually rip from CD to FLAC using foobar2000 and my machine easily can convert to FLAC faster than the drive can read the disc. Would I benefit from this at all?

[Fandango]

You are right, multi-threaded encoders are for faster transcoding of already ripped audio.

[clintb]

Any chance someone could whip up a batch file for this bad boy to use fpFLAC2FLAC.exe, somewhat like Synthetic Soul's flac-113.bat? My batch skills are pure rubbish. I have a bunch of files at flac 1.1.x and want to take all to 1.2.1.

Thanks for any help offered!

Example attached:

CODE

@ECHO OFF
CLS

REM FLAC-113.BAT Written by Neil Popham, 2007 (neilpopham@bigfoot.com)

SET batchname=FLAC-113.BAT
SET version=2.2.5
TITLE=%batchname%, Version %version%

REM ######################################################
SET pathToFLAC="C:\Apps\Encoders\FLAC\FLAC.EXE"
SET flacOptions=-8
SET retainListOfFailedFiles=1
SET retainListOfProcessedFiles=0
SET retainListOfSuccessfulFiles=0
SET flacVersion=121
REM ######################################################

REM Check whether a valid file or folder has been passed
IF NOT EXIST %1 DO GOTO NoParams

REM Set counters
SET /A noProcessed=0
SET /A noErrors=0
SET /A noEncoded=0
SET /A noSkipped=0
SET /A srcBytes=0
SET /A dstBytes=0

REM Set path to the list of failed files depending on whether it is to be kept or not
IF [%retainListOfFailedFiles%] EQU [1] (
SET pathForFails=%~dp0%
) ELSE (
SET pathForFails=%TEMP%\
)

REM Set path to the list of successful files depending on whether it is to be kept or not
IF [%retainListOfSuccessfulFiles%] EQU [1] (
SET pathForSuccess=%~dp0%
) ELSE (
SET pathForSuccess=%TEMP%\
)

REM Set path to the list of files to process depending on whether it is to be kept or not
IF [%retainListOfProcessedFiles%] EQU [1] (
SET pathForList=%~dp0%
) ELSE (
SET pathForList=%TEMP%\
)

REM Clear file used to store failed filenames
IF EXIST "%pathForFails%flac-113-failed.txt" DEL "%pathForFails%flac-113-failed.txt"

REM Clear file used to store successful filenames
IF EXIST "%pathForSuccess%flac-113-successful.txt" DEL "%pathForSuccess%flac-113-successful.txt"

REM Clear file used to store the list of files to process
IF EXIST "%pathForList%flac-113-processed.txt" DEL "%pathForList%flac-113-processed.txt"

REM Check whether the parameter is a single file or a folder
IF %~z1 EQU 0 (CALL :FolderAction %1) ELSE (CALL :FileAction %1)

REM Calculated bytes saved
SET /A dffSize=%srcSize%-%dstSize%

REM Report results
ECHO ________________________________________________________________________
ECHO.
ECHO %noProcessed% file(s) processed
IF EXIST "%~dp0FLACGETV.EXE" ECHO %noSkipped% file(s) skipped (already latest version)
ECHO %noEncoded% file(s) encoded
ECHO %noErrors% file(s) returned an error
IF %noErrors% GTR 0 CALL :ReportFiles
ECHO.
ECHO Settings used: FLAC v.%flacVersion% '%flacOptions%'
ECHO Bytes Saved: %dffSize% bytes

REM Pause
ECHO.
ECHO Press any key to exit
PAUSE >NUL

REM End
GOTO:EOF

REM ==========================================
REM FileAction : single file has been passed
REM ==========================================
:FileAction
REM Do we have a FLAC file?
IF /I [%~x1] EQU [.flac] (
CALL :Encode %1
) ELSE (
REM Do we have a TXT file?
IF /I [%~x1] EQU [.txt] (
REM Process the list of files in the file
SET pathForList="%~dp1"
CALL :ProcessList "%~nx1"
) ELSE (
GOTO NoParams
)
)
GOTO:EOF

REM =======================================
REM FolderAction : folder has been passed
REM =======================================
:FolderAction
REM Check folder and subfolders for FLAC files,call ListFiles for each file, and then Encode
ECHO Processing: "%~1"
ECHO.
REM Create temporary list of files to process
ECHO Creating list of files to process
ECHO.
FOR /R %1 %%G IN (*.flac) DO CALL :ListFiles "%%G"
CALL :ProcessList flac-113-processed.txt
GOTO:EOF

REM =============================================
REM ListFiles : Create list of files to process
REM : %1 The FLAC file to add
REM =============================================
:ListFiles
ECHO %1>>"%pathForList%flac-113-processed.txt"
GOTO:EOF

REM ============================================
REM ProcessList : Encode all files in the list
REM : %1 The text list of files
REM ============================================
:ProcessList
REM Encode each file
PUSHD %pathForList%
FOR /F "tokens=* delims=" %%H IN (%~s1) DO CALL :Encode %%H
POPD
IF [%retainListOfFailedFiles%] EQU [1] (
REM ECHO.>>"%pathForList%flac-113-processed.txt"
REM ECHO List created at %DATE% %TIME%>>"%pathForList%flac-113-processed.txt"
) ELSE (
DEL "%pathForList%flac-113-processed.txt"
)
GOTO:EOF


REM =====================================
REM Encode : Encode FLAC file
REM : %1 The FLAC file to Encode
REM =====================================
:Encode
REM Check the vendor string to see if the file is already the latest version
IF EXIST "%~dp0FLACGETV.EXE" (
ECHO Checking Vendor: "%~1"
ECHO ________________________________________________________________________
"%~dp0FLACGETV.EXE" %1
ECHO.
)
IF %ERRORLEVEL% EQU %flacVersion% (
SET /A noSkipped+=1
ECHO File is already the latest version [%flacVersion%]
) ELSE (
REM Test the original
ECHO Testing: "%~1"
ECHO ________________________________________________________________________
%pathToFLAC% -t %1
REM If an error code has been returned increase the error counter
IF ERRORLEVEL 1 (
SET /A noErrors+=1
ECHO %1 [T]>>"%pathForFails%flac-113-failed.txt"
ECHO.
ECHO File failed testing.
) ELSE (
REM Encode the file
ECHO.
ECHO Encoding: "%~1"
ECHO ________________________________________________________________________
%pathToFLAC% %flacOptions% -V -o "%~dpn1-flac-113.flac" %1
REM If an error code has been returned increase the error counter
IF ERRORLEVEL 1 (
SET /A noErrors+=1
ECHO %1 [E]>>"%pathForFails%flac-113-failed.txt"
ECHO.
ECHO File failed encoding.
IF EXIST "%~dpn1-flac-113.flac" DEL "%~dpn1-flac-113.flac"
) ELSE (
SET /A noEncoded+=1
CALL :IncreaseDestinationBytes "%~dpn1-flac-113.flac"
CALL :IncreaseSourceBytes %1
ECHO %1>>"%pathForSuccess%flac-113-successful.txt"
MOVE /Y "%~dpn1-flac-113.flac" %1
)
)
)
ECHO.
ECHO.
REM Increase file counter
SET /A noProcessed+=1
GOTO:EOF

REM ==========================================
REM ReportFiles : List the files that failed
REM ==========================================
:ReportFiles
ECHO.
ECHO Files that failed:
ECHO.
TYPE "%pathForFails%flac-113-failed.txt"
REM Delete the temporary file if not required, or add footer if keeping
IF [%retainListOfFailedFiles%] EQU [1] (
REM ECHO.>>"%pathForFails%flac-113-failed.txt"
REM ECHO List created at %DATE% %TIME%>>"%pathForFails%flac-113-failed.txt"
) ELSE (
DEL "%pathForFails%flac-113-failed.txt"
)
ECHO.
GOTO:EOF

REM =====================================================
REM NoParams : neither a file or folder has been passed
REM =====================================================
:NoParams
ECHO To run this script, please pass a valid file or folder as a parameter.
ECHO.
ECHO Press any key to exit
PAUSE >NUL
GOTO:EOF

:IncreaseSourceBytes
SET /A srcSize+=%~z1
GOTO:EOF

:IncreaseDestinationBytes
SET /A dstSize+=%~z1
GOTO:EOF


REM VERSION HISTORY
REM ===============
REM Version Date Description
REM ================================
REM 1.0.0 2007-01-07 Initial Version
REM 1.1.0 2007-01-07 Now encodes to temporary file and then moves over source
REM 1.1.1 2007-01-07 Minor formatting changes
REM 2.0.0 2007-01-07 Now works on XP (creates a temporary list of files to process)
REM 2.1.0 2007-01-09 Fixed bug with ERRORLEVEL. T(est)/E(ncode) marker added for failed files
REM 2.2.0 2007-01-09 Will skip 1.1.3 files if FlacGetV.exe is same directory as this file
REM 2.2.1 2007-01-10 Added retainListOfFailedFiles and retainListOfProcessedFiles flags
REM 2.2.2 2007-01-12 Added flacVersion
REM 2.2.3 2007-01-28 Added retainListOfSuccessfulFiles
REM 2.2.4 2007-01-28 Now accepts a text list of files as input
REM 2.2.5 2007-02-16 Fixed FlacGetV bug. Now shows bytes saved

This post has been edited by Synthetic Soul: Feb 6 2010, 14:21

[flacflac]

Well, there's some feedback I can give you. I installed it and deleted shortly after. I'm not an user, but I can tell you why.

1. For me FLAC is dead, it's too weak, so you came too late. Multithreaded wavpack or mac would probably make more sense.
etc.

Wow, really? There is close to zero hardware support out there for either of these codecs, so how would other codecs make more sense than the one used by the majority here at HA? Also, what's the rationale behind switching to TAK, a closed-source codec without hardware support? Good luck switching to the next codec that comes along... .

3. Installers are annoying. Regular 7zip package would be better. Can't you make it work w/out registering these dlls? Also, I got error during registration of smth with SSE4 in the name. XP 32bit, SP3, Core 2 Duo. Program worked at least long enough to show help, seeing lacking switches, I didn't even bother to compress something.

Man, you haven't even used it at all? I agree that this program would benefit from not coming with an installer but just an executable you can drop anywhere - but this is a work in progress and imho should not be knocked for that.

The only non-portable program that I found worth using is Visual Studio.

Haha, good one.

@George
I hope you continue your development. I can see about 40% speed increase using the command line to convert from a single WAV to a FLAC, yet I seem to be losing this advantage when turning a large WAV into multiple smaller FLACs using a CUE sheet in Foobar.

My Foobar command line:

CODE

-8 --threads 2 -P 65536 -V -f %s %d

Is it expected to not work well with multiple files? Is this my I/O limiting the speed? I am running a Core2Duo@2GHz.

Also, one more thing: I noticed that when doing a command-line encode there is no mentioning of a "-V"-verification happening - is it going on in the background? Would it be possible to make fpflac a bit more verbose and tell me the result of this verification?

Thanks for this development.

ff

[Synthetic Soul]

Any chance someone could whip up a batch file for this bad boy to use fpFLAC2FLAC.exe, somewhat like Synthetic Soul's flac-113.bat? My batch skills are pure rubbish. I have a bunch of files at flac 1.1.x and want to take all to 1.2.1.

That batch file was superceded by the one in this thread. Thee are some concerns about a few weird characters in filenames, IIRC.

For the record though, I would assume that you'd just need to change the following lives near the very top:

CODE

SET pathToFLAC="C:\Apps\Encoders\FLAC\FLAC.EXE"
SET flacOptions=-8

I haven't been following this thread, so don't know what to suggest as replacements I'm afraid.

That is, if this encoder will handle FLAC files as a source.

[GeorgeFP]

Is it expected to not work well with multiple files? Is this my I/O limiting the speed? I am running a Core2Duo@2GHz.

In foobar2000, you have to set the number of threads to 1:

File -> Preferences -> Advanced -> Tools -> Converter -> Thread count : 1

Also, one more thing: I noticed that when doing a command-line encode there is no mentioning of a "-V"-verification happening - is it going on in the background? Would it be possible to make fpflac a bit more verbose and tell me the result of this verification?

Try to download the latest version from MY site. I've implemented this option some time ago.

[howesz]

BTW, I've decided to add some missing command-line options into fpFLAC (-b, -m, -M, -e, -A, -l, -p, -q, -r and --replay-gain).

Are these options included in fpflac.exe (dated Feb 22, 2010) from the 64-bit v1.0.0.21 installer download? Because I get an error when I use this command line:

-f %s %d -T "ARTIST=%a" -T "TITLE=%t" -T "ALBUM=%g" -T "YEAR=%y" -T "TRACK=%n" -T "GENRE=%m" -8 -V --replay-gain

And again, thanks for your time and efforts!! Multi-core is here to stay...

Zach

[Glenda]

BTW, I've decided to add some missing command-line options into fpFLAC (-b, -m, -M, -e, -A, -l, -p, -q, -r and --replay-gain).

Are these options included in fpflac.exe (dated Feb 22, 2010) from the 64-bit v1.0.0.21 installer download? Because I get an error when I use this command line:

-f %s %d -T "ARTIST=%a" -T "TITLE=%t" -T "ALBUM=%g" -T "YEAR=%y" -T "TRACK=%n" -T "GENRE=%m" -8 -V --replay-gain

And again, thanks for your time and efforts!! Multi-core is here to stay...

Zach

Same here what command line should we use in EAC?

[GeorgeFP]

Same here what command line should we use in EAC?

Sorry for the late reply. I will post screenshots for EAC and foobar2000 configuration soon.

[GeorgeFP]

BTW, I've decided to add some missing command-line options into fpFLAC (-b, -m, -M, -e, -A, -l, -p, -q, -r and --replay-gain).

Are these options included in fpflac.exe (dated Feb 22, 2010) from the 64-bit v1.0.0.21 installer download? Because I get an error when I use this command line:

-f %s %d -T "ARTIST=%a" -T "TITLE=%t" -T "ALBUM=%g" -T "YEAR=%y" -T "TRACK=%n" -T "GENRE=%m" -8 -V --replay-gain

And again, thanks for your time and efforts!! Multi-core is here to stay...

Zach

The --replay-gain option is currently not implemented so if you remove that it should work.

BTW, a new version (x64 only) is available for download which doesn't need an installer anymore, and I'm working on a Linux port. So, the project is not dead, yet

George

[gimp]

BTW, I've decided to add some missing command-line options into fpFLAC (-b, -m, -M, -e, -A, -l, -p, -q, -r and --replay-gain).

Are these options included in fpflac.exe (dated Feb 22, 2010) from the 64-bit v1.0.0.21 installer download? Because I get an error when I use this command line:

-f %s %d -T "ARTIST=%a" -T "TITLE=%t" -T "ALBUM=%g" -T "YEAR=%y" -T "TRACK=%n" -T "GENRE=%m" -8 -V --replay-gain

And again, thanks for your time and efforts!! Multi-core is here to stay...

Zach

The --replay-gain option is currently not implemented so if you remove that it should work.

BTW, a new version (x64 only) is available for download which doesn't need an installer anymore, and I'm working on a Linux port. So, the project is not dead, yet

George

Removed --replay-gain but it still doesn't work for me, even after upgrading to fpStream_1_0_0_23. Any suggestions?

[GeorgeFP]

Removed --replay-gain but it still doesn't work for me, even after upgrading to fpStream_1_0_0_23. Any suggestions?

Thanks for the feedback. I've found a problem when using the -V option. I will upload a fixed version this weekend.

BTW, I've merged all optimized version to one which detects processor features automatically. And the code is ported to VS2010 but I haven't tested yet if it can be compiled with the Express edition.

George

[GeorgeFP]

An updated version is now available on my site.

I've tested fpFLAC with the latest EAC version using the above arguments without errors.

Note 1: During my tests fpFLAC crashed from time to time when it was started by EAC while it ran fine from the command line. It seems (!) to be fixed now, because it didn't crash again in the last test runs.

Note 2: Don't forget to set the numbers of threads to 1 in EAC (EAC -> EAC options -> Tools).

Note 3: The compression performance would be better if EAC would pass all arguments for all files in a single call. In a future version I will add the option to read the arguments from a file.

[Glenda]

The encoder flags errors on every attempt with EAC, possibly not compatible with the latest version of eac

[GeorgeFP]

The encoder flags errors on every attempt with EAC, possibly not compatible with the latest version of eac

I've downloaded "eac-1.0beta1.exe" and configured the fpFLAC arguments with -f %s %d -T "ARTIST=%a" -T "TITLE=%t" -T "ALBUM=%g" -T "YEAR=%y" -T "TRACK=%n" -T "GENRE=%m" -8 -V. I've also added spaces in the directory name. No problems found.

You can try to use fpFLAC from the command-line to check if it works at all, for example: fpFLAC -8 -f <wavfile> <flacfile>

[Glenda]

The encoder flags errors on every attempt with EAC, possibly not compatible with the latest version of eac

I've downloaded "eac-1.0beta1.exe" and configured the fpFLAC arguments with -f %s %d -T "ARTIST=%a" -T "TITLE=%t" -T "ALBUM=%g" -T "YEAR=%y" -T "TRACK=%n" -T "GENRE=%m" -8 -V. I've also added spaces in the directory name. No problems found.

You can try to use fpFLAC from the command-line to check if it works at all, for example: fpFLAC -8 -f <wavfile> <flacfile>

Win7x64:

[GeorgeFP]

It seems that the libraries are of mixed type (32 and 64 bit).

Try the following:
1. Uninstall fpStream or delete the folder.
2. Unpack fpStream_1_0_0_23.7z into a folder of your choice. No installation is required.
3. Run 'fpFLAC'

If you haven't installed the Visual C++ redistributables yet, you can download them here. They are required.