Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: TAK 2.0.0 - Beta release (Read 51170 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

TAK 2.0.0 - Beta release

Beta release 1 of TAK 2.0.0 ((T)om's lossless (A)udio (K)ompressor)

It consists of:

- TAK Applications 2.0.0 Beta 1
- TAK Winamp plugin 2.0.0 Beta 1
- TAK Decoding library 2.0.0 Beta 1

The final release will additionally contain the SDK.

Download:

Download link removed. TAK 2.0.0 Final has been released.

What's new

This release introduces a new file format, which can not be decoded by earlier versions of Tak, Takc, in_tak and tak_deco_lib! But surely it can decode files created by any earlier version.

Improvements:

- Slightly better compression of CD-Audio for any preset (ranging from 0.09 to 0.37 percent for my primary test corpus).
- More than 1 percent better compression for my 8-bit test corpus.
- More than 1.5 percent better compression for my 192 KHz / 24-bit test corpus.
- Up to 0.45 percent better compression for my LossyWav test corpus.
- Higher encoding speed for any basic preset (without addditional evaluation level), higher decoding speed for any preset. Depending on the cpu up to 11 percent faster encoding for -p0 and up to 15 percent faster decoding for -p3 compared with V1.1.2.
- While the new codec is smaller than the previous one, the binaries are a bit bigger because the decoder for the old file format takes up about 18 KB.
- The file format is prepared to support some more future improvements.

Modifications:

- Added xrecode II to the list of applications with TAK support in the readme file.

Known issues:

- If you use pipe decoding and the application reading the pipe is beeing terminated before the whole file has been read, TAKC may get into an endless loop and has to be manually killed with the task manager. I don't think this is a big issue but i will try to fix it in one of the next versions. BTW: Big thanks to shnutils for testing the pipe decoding!
- There seem to be some compatibility issues with pipe decoding to some other applications ("crc1632.exe" has been reported). I will try to fix it in the next release.

Beta testing

The beta version has already gone through extensive testing performed by my automatic scripts. But i haven't performed a noteworthy amount of testing of the new tagging functionality under real world conditions. Please try the beta release and report any bugs in this thread.

I would also be happy about tests of compression efficiency and speed. Because the final release will have identical performance (there may be a speed variation of 1 to 2 percent because of different code alignment of another build), it does make sense to test the beta.

Thanks for testing and have fun

Thomas

TAK 2.0.0 - Beta release

Reply #1
Awesome! Big thank you!!! TAK is the future of lossless compression 
🇺🇦 Glory to Ukraine!

TAK 2.0.0 - Beta release

Reply #2
About decoding speed:

Test file:

Duration : 7:11.198 (19015816 samples)
Sample Rate : 44100 Hz
Channels : 2
Bits Per Sample : 16

***

TAKC 2.0.0 -e -p4m -ihs - %d (960 kbps)

Decoding:

Total length: 35:55.988
Info Read time: 0:00.000
Opening time: 0:00.079
Decoding time: 0:10.404
207.217x realtime

***

TAKC 1.1.2 -e -p4m -ihs - %d (961 kbps)

Decoding:

Total length: 35:55.988
Info Read time: 0:00.000
Opening time: 0:00.079
Decoding time: 0:09.863
218.591x realtime

P.S. Pentium 4 HT 3.7 Ghz
🇺🇦 Glory to Ukraine!

TAK 2.0.0 - Beta release

Reply #3
something interesting...

I've spent test on RightMark test signal and have got the following results:

File Name : Test signal (96 kHz 24-bit).wav
File Size : 24.9MB (26 110 652 bytes)
Duration : 0:45.331 (4351768 samples)
Sample Rate : 96000 Hz
Channels : 2
Bits Per Sample : 24
Bitrate : 4608 kbps
Codec : PCM
Encoding : lossless

***

FLAC 1.2.1 -s --ignore-chunk-sizes -8 - -o %d

Bitrate : 834 kbps

***

TAKC 1.1.2 -e -p4m -ihs - %d

Bitrate : 934 kbps

***

TAKC 2.0.0 -e -p4m -ihs - %d

Bitrate : 1079 kbps

***

Why does FLAC gives more better compression?

P.S. Test_signal.flac

P.P.S. Sorry for my English
🇺🇦 Glory to Ukraine!

TAK 2.0.0 - Beta release

Reply #4
About decoding speed:

TAKC 2.0.0 -e -p4m -ihs - %d (960 kbps)
207.217x realtime

TAKC 1.1.2 -e -p4m -ihs - %d (961 kbps)
218.591x realtime

P.S. Pentium 4 HT 3.7 Ghz

That's possible. TAK does not contain different code variants specifically optimized for particular cpu types. There is exactly one version which works very well with nearly any AMD and INTEL cpu. The only exception is the Pentium 4: It's architecture is so different from most of the other cpus, that optimizations for those cpus often hurt the performance when executed on a Pentium 4.

I've spent test on RightMark test signal and have got the following results:

FLAC 1.2.1 -s --ignore-chunk-sizes -8 - -o %d

Bitrate : 834 kbps

TAKC 1.1.2 -e -p4m -ihs - %d

Bitrate : 934 kbps

TAKC 2.0.0 -e -p4m -ihs - %d

Bitrate : 1079 kbps

Why does FLAC gives more better compression?

Thank you for providing the file!

I can't confirm your bitrates, but the tendency:

FLAC 1.2.1 -8
18.10 percent

TAK -p1m
20.60 percent

Well, this is likely to happen with synthetically generated files, where FLAC indeed sometimes can compress better than TAK. That's the disadvantage of the very fast reduced precision arithmetic TAK is using internally. That's not optimal, but rarely an issue with music or speech.

TAK 1.1.2 contained an additional filter, which was advantageous for such files.

Well, there are also possibilities to tune the current version. A very quick modification of the code brings the compression ratio down to 18.92 percent. I may apply this tuning, which requires a significant amount of work, in a later version.

Thank you for testing!

  Thomas

Addendum: Files affected by this problem usually contain a lot of strong frequency and amplitude modulation.

TAK 2.0.0 - Beta release

Reply #5
Great, thanks.
takc -e -pMax -fsl16384 -wm0 -md5 -v %in %out is the strongest mode, right?
Does pMax set just p4m (possibly something else in the future), or other parameters (I guess that currently only fsl) to the strongest values too?
BTW max fsl 16384 seems low, don't higher values give stronger compression? My quick estimate is that values even 10 times bigger won't noticeable affect seek times on even slow modern PCs....and defining a FLAC-style subset of values allowed for portable use would solve the issue with hardware players.

TAK 2.0.0 - Beta release

Reply #6
Increasing the block size could dramatically increase the memory footprint of TAK.

Well done Thomas on further development of TAK! (although isn't it always the way that you work up to a release and then immediately find some potentially significant changes.... )
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

TAK 2.0.0 - Beta release

Reply #7
Increasing the block size could dramatically increase the memory footprint of TAK.

Indeed. So?

I'm done with testing of the first album (Janis Joplin live, 44.1-16) ...and I'm just amazed.
flac -8 -A tukey(0.5) -A flattop -r 0,8 -V %in %out:
403927323 B
takc -e -pMax -fsl16384 -wm0 -md5 -v %in %out:
380991948 B

I'm checking other albums too, but almost 5% savings are just incredible.

TAK 2.0.0 - Beta release

Reply #8
takc -e -pMax -fsl16384 -wm0 -md5 -v %in %out is the strongest mode, right?
Does pMax set just p4m (possibly something else in the future), or other parameters (I guess that currently only fsl) to the strongest values too?

Yes to both: pMax selects p4m.  Currently there are no hidden options outside of the presets.

-fsl16384 doesn't make a difference, because it's the default value for p4x.

BTW max fsl 16384 seems low, don't higher values give stronger compression? My quick estimate is that values even 10 times bigger won't noticeable affect seek times on even slow modern PCs....and defining a FLAC-style subset of values allowed for portable use would solve the issue with hardware players.

Well, actually -p4 uses a frame size of 11025 samples...

From the readme:

Quote
Frame Size Limit

The Frame size is defined as samples per channel. The encoder first calculates the frame size for the preset's Frame Duration and then compares the resulting sample count with the Frame Size Limit value. If it's bigger, it will be limited. An Example:

Profile 0 is using a Frame Duration of 94 ms. CD-Audio has a sample rate of 441000. 44100 * 0.094 = 4145 samples. This value is bigger than the profile's Frame Size Limit and will therefore be limited to 4096.

Caution: if you increase the frame size limit, you are breaking the profile's specification! Some hardware players may not have enough memory to decode the bigger frames. Decreasing the frame size usually reduces the compression efficiency.

There is only one situation where you are adviced to change the frame size limit: If you want to encode files generated by the high quality LossyWav- preprocessor, Tak's frame size has to match the preprocessors frame size (usually 512 samples).

The frame duration for p4x is 250 ms and 44100 * 0.25 = 11025.

There are several reasons against larger frames:

- It's not necessary. I have tried frame durations of up to 350 ms and on average could not gain even 0.05 percent.
- TAK has a very fast algorithm to partition each frame of each channel into currently up to 5 subframes to adapt the predictor to variations of the signal characteristics. Longer frames would require more subframes and the speed of the partition algorithm decreases fast with any additional subframe. Actually 6 subframes would be optimal for p4, but i have to work on the algorithm to avoid a very significant drop of the encoding speed.
- Larger frames make decoding slower, if they are too big for the cpu cache.
- The decoder needs more memory.


TAK 2.0.0 - Beta release

Reply #9
Well done Thomas on further development of TAK! (although isn't it always the way that you work up to a release and then immediately find some potentially significant changes.... )

Oh yes! 

But i don't want to complain: I am in the lucky position that i can immediately check the effect of possible optimizations. I often thought, how hard it may be for you, because you have to wait for abx results to verify the results of your work...

TAK 2.0.0 - Beta release

Reply #10
I'm checking other albums too, but almost 5% savings are just incredible.

Fine!

Usually you will only achieve such good results for music with a lot of low amplitude parts and/or low complexity (for example: single instrument, mostly voice).

TAK 2.0.0 - Beta release

Reply #11
big thnx!


TAK 2.0.0 - Beta release

Reply #12
Thanks for the great new beta! I've done a few tests myself too:

Results from the older TAK version:

Quote
Touhou 10 Mountain O Faith GST, 1.1.2:

Total encoding time: 0:58.625, 73.23x realtime
Decoding time: 0:22.102
388.515x realtime
560.286.405 bytes


NIN - The Slip 24/96 TAK, 1.1.2:

Total encoding time: 1:07.422, 39.01x realtime
Decoding time: 0:43.322
121.421x realtime
942.195.261 bytes


Carl Orff - Carmina Burana, 1.1.2:

Total encoding time: 0:46.734, 75.89x realtime
Decoding time: 0:17.772
399.156x realtime
222.796.425 bytes


After I tested the en/decoding speed of 1.1.2 for this test corpus I've transcoded the by the 2.0b1 encoder. There will be 2 decoding times each album as I decoded the original 1.1.2 files first and the new ones later:

Quote
Touhou 10 Mountain O Faith GST, 2.0b1:

Total encoding time: 0:58.625, 73.23x realtime
Decoding time: 0:20.605
416.749x realtime
559.617.481 bytes
Decoding time: 0:19.531
439.660x realtime


NIN - The Slip 24/96 TAK, 2.0b1:

Total encoding time: 1:19.954, 32.89x realtime
Decoding time: 0:41.788
125.879x realtime
Decoding time: 0:39.676
940.205.050 bytes
132.580x realtime


Carl Orff - Carmina Burana, 2.0b1:

Total encoding time: 0:55.359, 64.07x realtime
Decoding time: 0:16.474
430.609x realtime
221.149.732 bytes
Decoding time: 0:14.423
491.855x realtime


Encoding became somewhat slower but in return we got both increased decoding speed and better compression - at least for 2 max, the setting I mostly use. Now that I read it back I may messed up the MOF encoding, the two version's speed can't be exactly the same
I'll run another test later on this album... All in all, I see the new beta as a clear improvement over 1.1.2

Test config based on: Core2 duo 6420(conroe)@3.33GHz, 832MHz cl4 ddr2 (GeIL Ultra).
I could see decoding speed increase on the Athlon 64 X2-based machine too.

TAK 2.0.0 - Beta release

Reply #13
big thnx!

Thank you for the data! Nice to see an improvement for hard to compress audio too, i wasn't sure about that.

Encoding became somewhat slower but in return we got both increased decoding speed and better compression - at least for 2 max, the setting I mostly use. Now that I read it back I may messed up the MOF encoding, the two version's speed can't be exactly the same
I'll run another test later on this album... All in all, I see the new beta as a clear improvement over 1.1.2
g speed increase on the Athlon 64 X2-based machine too.

And thank you too!

Encoding of -p2m may be slower with V2.0, only the encoding speed of the basic presets without additional evaluation level has consistently improved. Here the results of my primary sample set for any preset combination:

Code: [Select]
AMD Sempron 2,2 GHz

Preset  Compression            Enco-Speed                 Deco-Speed              
        1.1.2   2.0     Win    1.1.2    2.0       Win     1.1.2    2.0       Win
                                                                          
-p0     58.83   58.74   0.09   264.10   293.53   11.14%   283.52   293.70    3.59%
-p0e    58.67   58.30   0.37   200.29   153.88  -23.17%   284.84   293.96    3.20%
-p0m    58.53   58.22   0.31    88.43    89.72    1.46%   285.66   294.37    3.05%
-p1     57.98   57.84   0.14   193.18   211.09    9.27%   275.80   289.42    4.94%
-p1e    57.86   57.52   0.34   123.55   120.06   -2.82%   276.19   289.83    4.94%
-p1m    57.73   57.42   0.31    64.98    65.90    1.42%   276.48   290.43    5.05%
-p2     57.07   56.90   0.17   131.39   138.73    5.59%   250.36   257.29    2.77%
-p2e    56.89   56.69   0.20    81.92    65.42  -20.14%   250.04   254.78    1.90%
-p2m    56.78   56.59   0.19    44.80    38.14  -14.87%   249.90   256.33    2.57%
-p3     56.52   56.36   0.16    55.97    63.12   12.77%   190.88   218.73   14.59%
-p3e    56.44   56.26   0.18    43.16    43.01   -0.35%   188.53   217.99   15.63%
-p3m    56.37   56.20   0.17    25.44    24.26   -4.64%   187.45   217.41   15.98%
-p4     56.16   56.02   0.14    32.07    35.65   11.16%   166.89   174.47    4.54%
-p4e    56.10   55.93   0.17    18.57    28.04   51.00%   166.27   173.81    4.53%
-p4m    56.07   55.89   0.18    17.81    15.27  -14.26%   166.47   175.26    5.28%


I forgot to mention: I am testing without disk-io.

TAK 2.0.0 - Beta release

Reply #14
OK, I tested a few albums and one track.
TAK / FLAC size comparison, command lines like in my previous post.
Code: [Select]
3 Doors Down   redbook            295943905/305374605 (-3.0%)
Hendrix        Vinyl 24/96        828408576/850258673 (-2.6%)
Metallica      DVD-A 24/96 5.1    .......... Audio format not supported
Bobby McFerrin redbook            165023642/178508766 (-7.6%)
Bolero         redbook             67469479/70235049  (-3.9%)


What's up with this Metallica track?
All other save significantly more than 1.5% that I expected

ADDED: The lowest saving I found yet is on a rather noisy Rob Zombie single, 1.35% on the worst track, others just slightly better.

TAK 2.0.0 - Beta release

Reply #15
I think that TAK only supports 2-channel (so far).
lossyWAV -q X -a 4 -s h -A --feedback 2 --limit 15848 --scale 0.5 | FLAC -5 -e -p -b 512 -P=4096 -S- (having set foobar to output 24-bit PCM; scaling by 0.5 gives the ANS headroom to work)

TAK 2.0.0 - Beta release

Reply #16
Correct. It's on my todo list.

TAK 2.0.0 - Beta release

Reply #17
OK, I tested a few albums and one track.
...

Thank you, i really love to see such nice results...

(although isn't it always the way that you work up to a release and then immediately find some potentially significant changes.... )

And nevertheless -ignoring past experience- i really thought, i wouldn't work on the code for some days following the release...

But Steve Forte Rio's problem file gave me something to think about.

I could easily improve it's compression by modifying the parameters of the algorithm, which converts the parcor into filter (predictor) coefficients. But there are reasons against this:

- It would require an incompatible modification of the new file format.
- I have spent a lot of time tuning the parameters for any file sets i own. The goal was to replace very slow 64-bit by 32-bit multiplications. For this i had to find a subtle balance between the parameters. Any modification could disturb this balance.

I don't think, i should do it. The current parameter set is working perfectly well for any "natural" file i tested. It doesn't make much sense to put much effort into a modification, which potentially would even significantly reduce the speed, only to make TAK perform better on some synthetic problem file.

Maybe i have to accept, that there are some very rare files, which FLAC compresses a bit better...



TAK 2.0.0 - Beta release

Reply #18
Maybe i have to accept, that there are some very rare files, which FLAC compresses a bit better...


Because of injectivity of lossless compressors, there will always be files particularly bad for your codec.
I could take a jpg, attach a wav header, align and ask both codecs to encode it. Even if flac wins...who cares?
I don't think it's beneficial to tune for artificial data, because on something it will hurt. It might be other artificial stuff, but maybe some real things too.

TAK 2.0.0 - Beta release

Reply #19
Usually you will only achieve such good results for music with a lot of low amplitude parts and/or low complexity (for example: single instrument, mostly voice).
Indeed. Some results follows:
Code: [Select]
E:\temp>takc112 -e *.wav
1.wav                              ..........  37.48%  309*
2.wav                              ..........  11.50%  631*
3.wav                              ..........  11.24%  191*
4.wav                              ..........  36.93%  311*
5.wav                              ..........  35.23%  313*
6.wav                              ..........  58.97%  293*
7.wav                              ..........  56.39%  297*
8.wav                              ..........  21.35%  157*
9.wav                              ..........  20.20%  462*
10.wav                              ..........  6.31%  203*

Compression:    25.92 %
Duration:        10.11 sec
Speed:          272.66 * real time

E:\temp>takc112 -d *.tak
1.tak                              ..........  383* Ok
2.tak                              ..........  278* Ok
3.tak                              ..........  217* Ok
4.tak                              ..........  271* Ok
5.tak                              ..........  329* Ok
6.tak                              ..........  262* Ok
7.tak                              ..........  334* Ok
8.tak                              ..........  250* Ok
9.tak                              ..........  667* Ok
10.tak                              ..........  179* Ok

Duration:        9.76 sec
Speed:        282.36 * real time



E:\temp>takc200 -e *.wav
1.wav                              ..........  37.18%  332*
2.wav                              ..........  11.37%  797*
3.wav                              ..........  11.07%  263*
4.wav                              ..........  36.54%  320*
5.wav                              ..........  34.96%  328*
6.wav                              ..........  58.75%  313*
7.wav                              ..........  56.20%  306*
8.wav                              ..........  21.02%  193*
9.wav                              ..........  19.99%  568*
10.wav                              ..........  6.17%  291*

Compression:    25.70 %
Duration:        8.57 sec
Speed:          321.62 * real time

E:\temp>takc200 -d *.tak
1.tak                              ..........  441* Ok
2.tak                              ..........  377* Ok
3.tak                              ..........  295* Ok
4.tak                              ..........  360* Ok
5.tak                              ..........  393* Ok
6.tak                              ..........  367* Ok
7.tak                              ..........  407* Ok
8.tak                              ..........  288* Ok
9.tak                              ..........  350* Ok
10.tak                              ..........  293* Ok

Duration:        7.86 sec
Speed:        350.67 * real time
Something strange with 1.1.2 decode speeds - the results above are from a second run, and the speed improved but it's still not where it should be. In this test hard disk I/O is always a bottleneck.

FYI, this test was on a home studio project with a Beatles-like approach to mulitracking.

edit: A64 2GHz, 2GB, WinXPsp3, WD120
"Something bothering you, Mister Spock?"

TAK 2.0.0 - Beta release

Reply #20
Thomas,

Do you plan to improve performance of  foobar decoder library?

Thank you for open beta testing.








TAK 2.0.0 - Beta release

Reply #21
Thomas,

Do you plan to improve performance of  foobar decoder library?

Thank you for open beta testing.


My tests were run under foobar2k 0.9.6.9 and I could see a nice decoding speed increase this way too
OK, it may have nothing to do with the issue you refer here, but it still impoved

TAK 2.0.0 - Beta release

Reply #22
My 2 cents:
A file was encoded with TAKC 2.0.0 beta -p2e.

Decoding time with foo_input_tak 0.4.3 + tak_deco_lib 2.0.0 b: 13.2 sec.
Decoding time with TAKC 2.0.0: 13.3 sec.

So there is no difference in fb2k and Takc decoding speed.

TAK 2.0.0 - Beta release

Reply #23
Here go my test results of 512 wav files (16-bits 2 channel 1411 kbps, pop/rock/dance) encoded using commandline (so only one core is used) and Windows 7 64-bits on Intel Core 2 Duo at 2,7 GHz:

Code: [Select]
Total Size : 21.9GB (23 559 472 240 bytes)
Duration : 1d 13:05:56.971 (5889862428 samples)

TAK 112 p2
Total Size : 13.6GB (14 625 926 611 bytes)
Avg. Bitrate : 876 kbps
Compression: 62.08 %
Encoding Speed: 182.32 * real time
Decoding Speed: 265.08 * real time

TAK 112 p4m
Total Size : 13.4GB (14 472 102 033 bytes)
Avg. Bitrate : 867 kbps
Compression: 61.43 %
Encoding Speed:  28.66 * real time
Decoding Speed: 217.16 * real time

TAK 200beta1 p2
Total Size : 13.5GB (14 587 439 934 bytes)
Avg. Bitrate : 874 kbps
Compression: 61.92 %
Encoding Speed: 179.94 * real time
Decoding Speed: 269.68 * real time

TAK 200beta1 p4m
Total Size : 13.4GB (14 433 432 998 bytes)
Avg. Bitrate : 865 kbps
Compression: 61.26 %
Encoding Speed:  18.58 * real time
Decoding Speed: 232.28 * real time
For both modes p2 and p4m the compression gain is 2 kbps which imo is not very exciting (FLAC 1.2.1 -8 compresses this set of files to 899 kbps).

The Encoding Speed has decreased but the Decoding Speed has increased. The differences are small for mode p2 but are very big for mode p4m.

I've also done some comparison with foobar2000 pipeline encoding: Mode p2 foobar2000 encoding is much slower compared to commandline encoding, mode p4m produced similar results. I tried figuring out what is happening with mode p2 and noticed that when encoding starts it uses 100% CPU but always falls down to about 65% at some time between 10 and 45 secs after converter start. Hard disks are not the speed bottleneck and I couldn't find answers in Windows 7 logs. This goes beyond my knowledge.

I maintain the 512 wav files several days more, so Thomas, if you want some specific test let me know soon.

Thanks for you hard work for free!

TAK 2.0.0 - Beta release

Reply #24
I've also done some comparison with foobar2000 pipeline encoding: Mode p2 foobar2000 encoding is much slower compared to commandline encoding, mode p4m produced similar results. I tried figuring out what is happening with mode p2 and noticed that when encoding starts it uses 100% CPU but always falls down to about 65% at some time between 10 and 45 secs after converter start. Hard disks are not the speed bottleneck and I couldn't find answers in Windows 7 logs. This goes beyond my knowledge.


How do you know that the reason are not disks? Because I think they are.
fb2k batch converter runs several jobs at once, which is very inoptimal for hard disks and with blitz modes like p2 it's likely to be the reason.