lossyWAV 1.3.0 Delphi to C++ Translation Thread, Added noise linear PCM bitdepth reduction method. Options

[Nick.C]

And he's off!

Tycho has kindly started the translation of lossyWAV from Delphi to C++.

I'll need to learn it to be able to contribute meaningfully....

Changelog:

lossyWAV beta 1.3.0m 25/03/2013

• experimental parameter --feedback introduced to modify bit removal process to include an added noise check. Seems to work well in conjunction with "--maxclips 0" or "--maxclips 1" at lower quality settings.

lossyWAV beta 1.3.0m 25/03/2013

• Removed experimental new parameter --noisebtr.
• --interp-curve reinstated.
• experimental parameter --altfilter introduced to modify adaptive noise shaping method.

lossyWAV beta 1.3.0k3 08/03/2013

• Introduction of experimental new parameter --noisebtr which applies a new method of detemining bits to remove. Well, I say new - SebastianG did point me in this direction *quite* some time ago. Basically, the creation of the noise shaping filter for each channel outputs a numerical value that I have finally used to determine the number of bits to remove from that codec-block channel. It even works on (non-clipping) tonal samples. It should be noted that this is a complete departure from the original method - no searching for the lowest signal in the FFT output bins. Limited tuning so far - however the presets should give approximately the same resultant FLAC output bitrate from --portable up to --insane. Below --portable, I have made the quality settings more aggressive and --extraportable results in a FLAC output bitrate of approximately 283kbit/s. This is quite a drop from the 308 to 311kbit/s previously. If anyone is prepared to spend the time on some testing, that would be very much appreciated. I think that --extraportable is a bit *too* aggressive at the moment, but that is more of a feeling than anything else.
• --interp-curve removed at this time.
• beta 1.3.0k2 issued due to omission of --noisebtr from text output parameter list (used in FACT chunk and log file).
• beta 1.3.0k3 issued due to last minute fix for bits-per-sample not equal to 16.

lossyWAV beta 1.3.0j 08/02/2013

• Further experimental modification to the adaptive noise shaping routine, backed off a little but now works well with >96kHz samplerate

lossyWAV beta 1.3.0i 30/01/2013

• Further experimental modification to the adaptive noise shaping routine. NB: Testing has shown that this does not work well with >96kHz samplerate!!.
• Bug-fix: now processes >16-bit properly - unexpected integer overflow discovered where floating point was thought to be used.

lossyWAV beta 1.3.0h 23/01/2013

• Further experimental modification to the adaptive noise shaping routine. FLAC encode (-5 -b 512) from "-q X -i" results in 302kbit/s (compared to 309 kbit/s for vanilla --extraportable).

lossyWAV beta 1.3.0g 18/01/2013

• Experimental modification to the adaptive noise shaping routine, new parameter -i, --interp-curve; adds optional cubic curve interpolation in lieu of linear interpolation when determining target spectral shape for each noise shaping filter. When encoded in FLAC (-5 -b 512) using the --extraportable preset, my 10 album test set results in 304kbit/s (compared to 309 kbit/s for vanilla --extraportable).

lossyWAV beta 1.3.0f 26/10/2012

• First attempt at FFMPEG compatibility;

lossyWAV beta 1.3.0e 24/09/2012

• More rounding issues found and amended;
• Output is hopefully bit identical with reference output (2nd attempt....);

lossyWAV beta 1.3.0d 18/09/2012

• Bit removal chain error identified: C++ inherently rounds 0.5 differently from Delphi, Delphi uses Banker's rounding (round to even);
• Banker's rounding implemented;
• Output is now bit identical with reference output (caveat: from testing so far). [edit] Found not to be at --portable. Bug hunting continues.... [/edit]

lossyWAV beta 1.3.0c 15/09/2012

• Piped I/O implemented - please report problems;
• Most bugs in bit-removal chain removed - output not yet bit identical;
• Internal FFT routines optimised.

This post has been edited by Nick.C: Mar 30 2013, 15:37

Attached File(s)

 lossyWAV_beta_1.3.0n.zip ( 337K ) Number of downloads: 69

 

[[JAZ]]

Hi Tycho. That sounds nice.

Remember that there are the java and C# ports (although they are from older versions) that could be of help during the conversion:

jLossywav: (based on lossywav 1.1.4) http://www.hydrogenaudio.org/forums/index....st&p=645228
C# (based on 1.0?) : http://www.hydrogenaudio.org/forums/index....showtopic=67418

[Kohlrabi]

Is the port done publicly, like on github or similar? This community has some very knowledgeable C++ hackers, so it might speed the process and help with finding and fixing bugs.

This post has been edited by Kohlrabi: Aug 22 2012, 20:49

[tycho]

JAZ, Thanks, I was aware of the C# translation of version 1.0, but had missed out your java translation. Nice. There are a lot of changes in 1.3 though, and much of the work is to convert the I/O handling and runtime linking with fftw3-3 library. First step is something that compiles with gnu g++ on windows. Next would be VS2010, and then Linux/Mac g++, but that could easily someone else do.

Kohlrabi, I haven't thought about using github yet. My initial thought was just to get most of the code to compile, and then upload the code here. Useful help would be that people browsed through it and verified that the .pas and the .h/.cpp files does the same thing, but I wouldn't mind if people worked on it too.

I think that modern optimized C++ compilers (both 32 and 64 bit) are able to improve on speed a fair bit, but portability and accessibility is as important.

[Nick.C]

Hi Tycho,

How are you getting on? Impatient Inquiring minds would like to know....

[tycho]

It's progressing quite well. Many of the files are files are basically compiling now, but there are still some unresolved functions, e.g. the Delphi Write() function must be emulated, or all the calls to it must be rewritten. Also some of the IO routines need some work, as I mentioned.

I can post the code as it is tomorrow evening, so you can have a feel of it. Later I will come back with few questions for you, and tell a few things I've done during translation.

E.g. You have a few small functions in assembly only, like the ArcTan_Complex(). Shouldn't that return a complex number and not a float? On googling it, I found an implementation: (C++11 has it in the standard lib now)

Complex atan(Complex c)
{
Complex log_v = log( Complex(1.0 - c.imag(), c.real()) / Complex(1.0 + c.imag(), -c.real()) );
return Complex( log_v.imag() * 0.5, -log_v.real() * 0.5 );
}

The float you are returning is maybe the magnitude of the complex number returned here?

[lvqcl]

CODE

function ArcTan_Complex(const X: tDComplex): Extended;
asm
                FLD       X.im
                FLD       X.re
                FPATAN
End;

According to google it is arctg(x.im/x.re) so basically it is C function atan2(x.im, x.re).
...IMHO.

[Nick.C]

Apologies - you've found some of the "dirtier" code in lossyWAV....

The ArcTan_Complex function is, as lvqcl has rightly said, equivalent to a two input atan function.

[Atak_Snajpera]

Once we have C++ version can we expect nice speed up after compiling with Intel Compiler with enabled auto vectorization and auto paralyzation? I must admit that delphi's version is rather slooow. It would be awesome if code was multithreaded and SIMD friendly.

This post has been edited by Atak_Snajpera: Aug 28 2012, 11:56

[tycho]

I haven't studied enough how the data is processed, but I believe that the code must be made more reentrant, so that central functions relies exclusively on own stack data, and not global data structures (except readonly) as today. This opens the door for parallel computation if the algorithm allows for it, which I believe it does.

[Nick.C]

A simple parallelism would be to process each channel concurrently as channels are independent (even if linked then that linking of bits_to_remove would occur after all the FFT analyses have been completed for a given codec block).

[Atak_Snajpera]

A simple parallelism would be to process each channel concurrently as channels are independent (even if linked then that linking of bits_to_remove would occur after all the FFT analyses have been completed for a given codec block).

that would be a good start. 2x faster proccesing on stereo track and even more for typical 5.1 movie sound track (assuming that user has atleast quad core).

[Nick.C]

Another would be to have the FFT analyses run in parallel - each 512 sample codec block (44.1kHz / 48kHz samplerate) has 1x 1024 sample, 16x 64 sample and 32x 32 sample FFT analyses carried out on it (per channel).

[edit] can't count, apparently.... [/edit]

This post has been edited by Nick.C: Aug 28 2012, 20:50

[Atak_Snajpera]

I have noticed that original LossyWav 1.3 does not support .wavs larger than 4GB. My wav file is 5.2GB and lossyWAV sees only 1.2 GB. LossyWav also does not support wave64 format either.


Due to the same reason pipes do not work either with large flacs

CODE

flac.exe -d "E:\Avatar.2009.BluRay.REMUX.1080p.AVC.DTS-HD.MA5.1.flac" --stdout --silent | lossywav.exe - --stdout | flac.exe - -b 512 -8 --silent -o "E\lossy.flac"

i think that new version should have something like --readtoeof or --ignorelength switch like other encoders (Aften AC3 , FhGAACEnc or OpusEnc)

This post has been edited by Atak_Snajpera: Aug 29 2012, 14:36

[Nick.C]

A specification compliant WAV file can be no larger than 4GB (minus a few bytes for header and format information).

Maybe looking at RIFF64 or BWF would be a better way forward to cope with very large files.

[Atak_Snajpera]

Without --ignorelength switch lossyWav is useless for movie sound tracks. I can't even use pipe for direct processing.

BTW. Do you know maybe why your sources won't compile in delphi 7 pro?

This post has been edited by Atak_Snajpera: Aug 29 2012, 18:44

[Nick.C]

You could split the soundtrack into chapters, process with lossyWAV and then recombine using foobar2000.

I don't know about Delphi 7 Pro - I have been using Turbo Delphi Explorer 2006.

[Nick.C]

Tycho,

Are there any specific compiler options that I should be using for compilation of the project?

I have downloaded the Dev-CPP IDE with MinGW 4.6.0.2 - primarily because I wanted a portable mode. Quite liking it and have started "playing" with it to get my head round C++.

On the Write() issue, would fprintf() be better? It would certainly make formatting the output much easier. I have got v0.3 to compile as far as nOutput.cpp.

[tycho]

Nick,
Good to hear you're progressing with c++. Yes, the old ANSI C printf (stdio.h) is in many ways easier and more compact to use, although less type safe (it normally does not check arguments in the printf statements with the formatting string). Personally I would prefer to use the current standard which is the fstream/iostream interface.

To be blunt, DEV Cpp has been dead for years, and it's Windows only (written i Delphi actually). There are better alternatives as I have mentioned earlier. For cross platform I would first recommend Qt Creator (available on Win,Linux,OS X). It is meant for Qt apps, but you may easily develop pure C++ projects without Qt and qmake. It supports many compilers, .e.g Visual Studio (Express) and GNU Mingw. You may download Qt libraries 4.8.2 (not the whole Qt SDK) for the qmake and cross platform GUI ++ libs. As a second alternative you can look at CodeLite, which is a one-man project only, then Code::Blocks which is similar.

I am going to configure Qt Creator myself with Mingw to fix the compilation issues there. It behaves different than VC++. I have currently Mingw 4.7.0 on my machine so there may be differences from 4.6.2. Don't worry about the Write() now, I will take care of that one when we get there.

This is probably information overload, so take it easy.

This post has been edited by tycho: Aug 30 2012, 23:35

[Nick.C]

Updated executable later tonight - huge thanks to Tycho for getting me off my butt to delve into C++ with his translation of the Delphi source code.

A few issues found and corrected, not least the in FFT code - there were a few issues in there (int equates to FLOOR; >> is shift-arithmetic-right, not a simple shift right; log is ln, not log10....).

This post has been edited by Nick.C: Sep 7 2012, 21:28

[Atak_Snajpera]

what is still missing in current c version vs original?

[Nick.C]

A bit of accuracy - there's a bug somewhere in the chain after the FFT analyses that still has to be tracked. Adaptive noise shaping is confirmed to be working though.

[Nick.C]

lossyWAV beta 1.3.0b attached. [edit] Superseded. [/edit]

Changelog:

Internal FFT routines now work.

To do list:

Piped input / output.
Track down accuracy error post FFT analyses.

This post has been edited by Nick.C: Sep 15 2012, 12:04

[Atak_Snajpera]

lossyWAV beta 1.3.0b attached.

Changelog:

Internal FFT routines now work.

To do list:

Piped input / output.
Track down accuracy error post FFT analyses.

after adding pipes could you also add --ignorelength switch?

[mudlord]

Is there any reason for custom FFT? Why not something like KISSFFT or something?