lossyWAV 1.3.0 Delphi to C++ Translation Thread, Added noise linear PCM bitdepth reduction method. Options

[mezenga]

Is there any reason for custom FFT? Why not something like KISSFFT or something?

At Delphi´s development time the reason seemed to be the lack of options.
But this C++ port surely is a good opportunity to review if it remains the better choice.

And here is some small lossyWAV FFT implementation history:
lossyWAV alpha v0.3.12 - Special thanks: Dr. Jean Debord for the use of TPMAT036 uFFT & uTypes units for FFT analysis

lossyWAV v0.6.4 RC1 - Special thanks: Don Cross for the original Pascal source for the FFT algorithm used.

lossyWAV 1.1.0b - Special thanks: Don Cross for the Complex-FFT algorithm used[/url]

Change log 1.1.4b: 14/05/09: FFT (Delphi and assembler) further optimised. Radix-4 FFT implemented in assembler and Delphi and Radix-8 in Delphi. Significant speedup of Delphi FFT throughput.

This post has been edited by mezenga: Sep 8 2012, 04:45

[Nick.C]

I'll have a look at KissFFT - it may be faster.

I also need to try to get to grips with multi-threading to see if there is any way to speed up the processing.

[lvqcl]

There's also PFFFT library: https://bitbucket.org/jpommier/pffft But it is for single-precision floats.

This post has been edited by lvqcl: Sep 8 2012, 13:25

[Nick.C]

I know that single precision would be faster but after trying it out earlier in the development of 1.3.0, I would much prefer to stick to double-precision.

[lvqcl]

SoX uses fft4g.c from Ooura 1D FFT

[Destroid]

Being unaware of KissFFT, I did a brief overview and noticed some factors:

- FFTW has benchmarks showing (on x64) having notably higher performance than KissFFT (the former program being version 3.3.x, the latter program's version I did not observe);

- That KissFFT was not present on all of those benchmarks, possibly having to do with limitations on certain kinds of double-precision inputs;

- That KissFFT is purported to have simpler implementation and/or structure.

I am not an expert in areas of DSP and FFT and wanted to report my impressions and conclusions. I enjoy watching the development process as I also hope to gain knowledge. I congratulate and admire the developers' efforts (and HA, of course).

[mezenga]

- FFTW has benchmarks showing (on x64) having notably higher performance than KissFFT (the former program being version 3.3.x, the latter program's version I did not observe);

FFTW is optionally supported since lossyWAV 1.1.4c (requires libfftw3-3.dll and --fftw switch) but the main discussion here is about the internal FFT function.
Until now FFTW saved 10-20% processing time: enough to became an option but not enough adding 500Kb in the download package.
Lets wait Nick and Tycho get some benchmarks from this C++ port.

[Nick.C]

The C++ executable is *much* bigger than the Delphi one (1716kiB vs. 168kiB).

I am in the process of implementing a radix-4 IFFT while optimising the other FFT routines for speed. I think that there is some speed to be had. Also, the minGW compiler allows optimisation for various different processors and instruction sets. These do make an appreciable difference.

Main target is to get piped I/O working and to track down the accuracy bug previously mentioned.

[tycho]

Even though much of the point of translating to C++ was the possible optimization benefits, I think correctness is still the most important at this point. I believe there are still issues I have mentioned elsewhere that needs to be addressed, or have they been handled? Also, in getParams() an if-statement is missing:

CODE

size_t found = parameters.wavName.find_last_of("/\\");
if (found != std::string::npos)
{
    parameters.WavInpDir = parameters.wavName.substr(0, found + 1);
    parameters.wavName = parameters.wavName.substr(found + 1);
}

For piped IO, make sure to set input/output stream to binary mode on windows.

CODE

#ifdef _WIN32
#include <cstdio>
#include <io.h>
#include <fcntl.h>
#endif
...
if (parameter.STDINPUT || parameter.STDOUTPUT) {
    #ifdef _WIN32
    _setmode(_fileno(stdin), _O_BINARY);
    _setmode(_fileno(stdout), _O_BINARY);
    #endif
    ...
}

This will also make std::cin and std::cout binary, i.e. does not translate linefeed char (LF) to CRLF - leaves it as LF. Works both under g++ and VS.

As for executable size, you may try -Os compiler flag and then "strip lossyWAV.exe". With Visual Studio, the executable doesn't get very large.

[Nick.C]

Thanks for those Tycho.

I've started on piped I/O - it's taking a bit of reconfiguration but will get there in the end.

I will also go back through to check for other previously mentioned issues.

[Nick.C]

Piped I/O is now in testing phase - I processed my 10 album test set in foobar2000 without incident.

I've tracked down two problems in the bit-removal chain but am still getting slightly different results than the "reference" Delphi version. More bug-hunting to do there then.

[Atak_Snajpera]

have you decided yet what multithreading method will you choose for extra speedup?

do you even consider reading wavs to the end file? ( 4gb+ wavs )

This post has been edited by Atak_Snajpera: Sep 13 2012, 18:40

[Nick.C]

I have not started on multi-threading yet. I will go through the debugging process until I am content that the output is a close to the output of the reference version as it can be.

I am considering adding "--ignore-chunk-sizes" functionality, but I don't yet know how best to go about it. It will have to wait until I have finished the debugging process.

[edit] English language fail.... [/edit]

This post has been edited by Nick.C: Sep 13 2012, 19:20

[Atak_Snajpera]

i keep asking about ignore chunk size switch because i plan to add lossywav preprocessing in ripbot264.

This post has been edited by Atak_Snajpera: Sep 13 2012, 20:09

[Nick.C]

Ah! Now it makes sense to me. In that case, I will confirm that I will attempt to develop the --ignore-chunk-sizes functionality (when I've finished initial debugging ).

[Atak_Snajpera]

great and thank you!

[Nick.C]

lossyWAV beta 1.3.0c attached to post #1 in this thread. [edit] Superseded. [/edit]

This post has been edited by Nick.C: Sep 18 2012, 21:56

[Atak_Snajpera]

Speed comparison
Q6600@3Ghz
All read/write done on RAMDISK 2GB

CODE

flac.exe -d "G:\original.flac" --stdout --silent | G:\lossywav.exe - --stdout | G:\flac.exe - -b 512 -8 --silent -o "G:\lossy.flac"

DELPHI version - 11.48x
C++ version - 14.88x

[Nick.C]

That's certainly encouraging !

[Atak_Snajpera]

Ability to process channels independently each in separate thread should give also nice extra speedup. Especially for 6ch movie soundtracks.

I wonder if compiling with Intel Compiler we could get even higher speed than your build.

This post has been edited by Atak_Snajpera: Sep 15 2012, 18:48

[[JAZ]]

I can confirm that higher speed will be possible:

(Core 2 Duo at 1.5GHz)

Delphi : 5.75x
GCC : 7.42x
VC x86 broken : 6.73916x
VC x64 broken : 7.14685x

Delphi FFTW : 7.49x
GCC FFTW : 8.22x
VC x64 FFTW broken : 8.86619x

VC broken means a modified compile of the sources released as v05 by tycho that I've compiled with Visual Studio 2008. x86 compile is with settings: full optimizations and SSE2. (With FFTW, the output is a bit more like GCC, but is still broken.).
Oh, and the size is, between 350KB and 400KB without requiring the msvc runtimes (I've linked it statically. Else it would be around 200KB and require the VS 2008 runtimes).

@Nick: are you waiting to have a completely working version before releasing the C++ sources? I'm asking so that I don't fix things that you've already fixed.
Also, I've seen that the GCC version shows correctly the --bitdist, but --blockdist is completely different. Not sure if that shows a problem in the displaying/storage of those values, or if it is the algorithm. I guess most of the switches you added about statistics should help on debugging this port

[Nick.C]

lossyWAV beta 1.3.0d attached to post #1 in this thread. [edit] Superseded. [/edit]

This post has been edited by Nick.C: Sep 24 2012, 20:27

[mezenga]

lossyWAV beta 1.3.0d attached to post #1 in this thread.

Tested against lossyWAV 1.3.0 delphi version for bit identical output with internal FFT and FFTW.

- standard quality setting: OK (bit identical).

- portable quality setting: KO (not bit identical).

[Nick.C]

.... ok. Thanks for the testing - I'll go bug hunting again.

[punkrockdude]

I really look forward to try this out on Linux one day. I really hope Linux drivers (especially proffessional audio interfaces) will soon be a reality so that I can use only Linux. Since installing an i3 i5 i7 patched kernel I am so impressed with Linux responsiveness. Why do I talk about Linux? Sorry guys