TAK 2.0 Development

Well, it looks as if i will be able to release TAK 2.0 this year... Therefore it's very unlikely that there will be an 1.1.3 release unless a bug fix for 1.1.2 is needed.

There are several good reasons for me to open a TAK 2.0 development thread now:

- Most of the new or improved algorithms have been implemented.
- Therefore i can now provide some performance data which will not differ very much from the final release.
- At some point i may need help from interested users to tune the new algorithms.
- It's easier for me to ask for help or opinions if there is already an appropriate thread.
- With this thread started and my goal stated, i have to stop digging even deeper for further improvements...

What to expect from V2.0:

It will introduce a new codec, but no new features from my todo list. The new codec will be no revolution, but an evolution. When i arrived at hydrogen and published first evaluation versions of YALAC, i was focussing more on maximum compression than on maximum speed. But it didn't take long to get aware of what users liked most of YALAC/TAK: It should be as fast as the fastest existing codecs but provide significantly higher compression. Despite of this clear mission statement, it took quite long for me to fully follow this order. Probably the reduction of the maximum predictor count from 256 to 160 in TAK 1.1.0 marks the last step of this process.

The goal of the TAK 2.0 development can be defined as follows:

Get the most bang for the buck. More specifically:

- Try to make it faster.
- Try to make it stronger (compression wise) but without sacrificing (decoding-) speed.

With these constraints present, i haven't managed to perform something revolutionary. But please don't forget, that many users already regarded the efficiency of the TAK 1.x line as revolutionary...

Ok, now for some data. First the results of my primary sample set. It consists of 46 files taken from rarewares.org, combined in one file. This set has proven (in my evaluations) to stress lossless codecs in many different ways and it's my primary choice if i want a quite representative result covering different genres of music. Surely, i could think of an even more representative set, but this one is nice, because it's publically available.

Here we go with a comparison of V1.1.2 and 2.0:

Test system: AMD Sempron 2,2 GHz


Compression is relative to the original file size, Enco- and Deco-Speed expressed as multiple of real time.

Some remarks:

- Faster decoding for any preset, especially for -p3x.
- Faster encoding for any basic preset (without additional evaluation level).
- Some compression improvements for any preset.
- -p4m is now compressing a bit stronger than any TAK version before, and this despite the reduction of the maximum predictor count from 256 to 160 in V1.1.0.
- Nice improvement of 0.12 percent for the default preset.
- Highest improvements for p0e/p0m,p1e/p1m, which -while only using 4 respectively 12 predictors- will easily (on average) outperform FLAC -8 with 12 predictors.

Now only the basic presets:

AMD Sempron 2.2 GHz


Some speed differences on a Intel Pentium Dual Core 2 GHz:


Compression advantage for other bit depths than 16:


Some remarks:

- Currently there is some regression for 96 KHz. The new algorithms need some fine-tuning.
- In my evaluations only Optimfrog can beat TAK's compression of 192 KHz sample data.

Last but not least some results for LossyWav:


And no, that's not the results of the dedicated LossyWav codec i intend to develop later.

Well, i hope you are not disappointed!

Thomas

I love this codec because of it speed to encode and decode (at least in foobar) and prefer it then FLAC when some compatibility isn't in question.
I didn't read the tables but I guess it will still be fast enc/dec
Only the date?

[edit] just forgot MD5 -

[edit2] I just saw them quickly, and encoding speed graph is little strange. Default -p3 just jumps from the neighbors (why?) Also -p2 looks like a candidate for default setting.

[edit3] well, -p2 is default, I somehow thought it was -p3, nevermind

Great news, Thomas! Nice to see my lossless codec of choice is continuing to be improved upon.

Thank you.

It's closed source until the day it isn't anymore, and maybe that day will never come. Pressuring with each release by remembering extensively the words said looong ago seems to me a desperate scream to get a personal desire fulfilled.

I do agree with the fact that the evolution of TAK doesn't help getting TAK more widely adopted (lack of specifications for developers, lack of multiplatform, not being open source, lack of features FLAC does have, etc.). In fact, I have the impression that with each TAK release less people participate (posting & testing).

But then, Thomas can do what he want, it's his work.

Back to TAK 2.0. Great to see the improvements, how did you got them? You rewrote the encoder and decoder? Do you think there's a room to make it even faster?

When it's out I will compare it to FLAC and TAK 1.1.2, if I have time by that time

I am glad to see a new milestone being set for TAK. However, users fixated on the topic of code availability is becoming very irritating. I think certain members should quit threatening the developer by declaring TAK unusable because of their narrow philosophy. For them I say, "Go write your own open-source lossless program," if only just to get an idea of what it takes to make a codec that is different and better.

Kicking aside the soapbox (and getting back to point of main discussion), I also wanted to throw out other questions: Is TAK looking at adding multichannel support sometime in version 2.x? What other additional tools might 2.x be included in the SDK that programmers can utilize?

Thomas, congratulation on new step of evolution of your codec.
I see that all standard presets -q0 ...-q4 have received enco- and deco- speed improvements while sub-presets "extra" and "max" have gone to more asymmetric balance (slower encoding, faster decoding and better compression). It's great. The CD ripping (EAC) is more slower than encoding to -p0m or -p1m and many time there is a lot of time to encode without rush while decoding speed is important for playback and converting to the lossy formats. I wouldn't mind see even more asymmetric speeds. Maybe new profile ( -p1u ultra?) could bring more compression and/or faster decoding at cost of something like 2-2.5x slower speed comparing to -p0m and -p1m (which is still comparable with speed of FLAC -8)?
Most of FLAC users encode to -8 level. http://www.hydrogenaudio.org/forums/index....c=58731&hl=

Likewise those numbers are ideal as they don't consider I/O bottleneck. The difference between speeds of -p1 and -p1m is considerably smaller in real scenario.

I have replaced something i called "PreFilter", which was present for -p3x and -p4x. The new approach is a lot faster for -p3x.


I think it would be quite easy for me to evoke more responses, if i would release evaluation versions and would frequently ask for opinions on some tuning deceisions like i did in the YALAC-days. But there is far less need for me to do this, because i now know a lot more about the nature of many of TAK's compression techniques and their effect on even extraordinary audio files. This because of the great support of some past testers of YALAC and TAK.

And as i alreaday wrote: The speed-compression-efficiency of the first YALAC/TAK-versions has been regarded as revolutionary by quite some people, but now there is only some -less exciting- evolution happening. Something users possibly are expecting.


What i did:

- Further improvements of the bitcoder for the final residuals. FLAC for instance is using fast but not very efficient rice codes. Monkey's Audio is using range coding: more efficient but noticeable slower. TAK is using codes that are nearly as fast as rice codes and nearly as efficient as range coding. TAK 2 further improves the bitcoder efficiency for very low amplitude signals (here conventional rice codes are very inefficient). You can see a significant effect for 8-bit data, low amplitude parts of for instance classical music and for LossyWav-files.

- I have implemented a simplified variant of the advanced channel decorrelation algorithm used by presets p2 and above. It's about as fast as simple Mid/Side-encoding used by -p0/-p1 in earlier versions. Because of it's speed, it was now possible to bring a good part of the advanced channel decorrelation to the presets p0/p1.

- Presets -p3 and -p4 of the TAK 1.x line are sending each sample through something i called "PreFilter". It's primary effect was the reduction of the space requirements for the storage of the predictor coefficients. Another approach i tried in the YALAC-days was the storage of the predictors as parcor coefficients. Unfortunately this approach was quite slow for high predictor orders, because encoder and decoder have to transform parcor into filter coefficients and this operation required very slow 32*32=64 bit multiplications and the count of those multiplications is growing exponentially with the predictor count.

But now i have managed to modify the algorithm in a way, that it needs only 16*32=32 bit multiplications, what is faster than the PreFilter, if you don't go for extremely high predictor counts > 256.

- I have performed a lot of quite small optimizations to simplify the algorithms and to gain a bit of speed.


I am quite sure i can speed up encoding for the encoder evaluation levels e and m. Regarding decoding, i have my doubts.


Fine!


Yes, i really would like to implement multichannel support. But it's not really easy to do, because i first want to evaluate the possibilities to take advantage of channel correlations of such files.

My todo list for the SDK:

- Encoding capabilitities.
- Unicode support.
- Access to other meta data than APEv2-Tags.


Well, i could implement a brute-force-approach, which possibly would increase compression by 0.05 to 0.1 percent, but might reduce the encoding speed by a factor of 10 to 50...


How true. And p1 is probably only a tiny bit slower than p0. For now i have modified p1 to be nearly as strong p1e while loosing only very little encoding speed (<= 10 %). I will post some results later.

Thomas

Updated results for Version 2.0 Eval 03

Yeah, there is some progress...

Results for my primary sample set:


Compression is relative to the original file size, Enco- and Deco-Speed expressed as multiple of real time.

Some results for LossyWav:


Some remarks:

- Again some tiny compression improvements.
- Again -p0 is encoding significantly faster. Encoding is now nearly as fast as decoding on the Sempron.
- I have deceided, to make -p1 stronger, therefore it's now encoding a bit slower.

Thomas

Any news about release date?

Congrats, Thomas. Great to hear about TAK 2 development progress. Looks really promising.
According to your todo list, there are:


Do you plan to add also brute-force mode before public release ? This mode could still be very useful despite slow encoding speed.
And yes, is there any estimated release date for beta version ?

Monkey's Audio, OptimFROG, LA - they all have better compression ratio than TAK -pMax. But they aren't particularly popular.

There are no DAPs that support TAK. Winamp plugin doesn't support tagging. And yes, no Unicode support. (what else? embedded cover art?) I think it's much more important for TAK.

TAK has one huge advantage over those codecs - very fast decompression.
I can easily live with slower encoding in 'ultra-heavy-brute-force mode', since I compress each album just once. Fast decoding speed stays anyway. So if such mode is possible - why not to have it ?


Yep. But it seems all these features are already on Thomas's todo-list.

2 advantages. Monkey's and FROG have honestly really dumb names that inhibit adoption. I remember when I first found out about lossless encoding, the only 2 codecs I was aware of were FLAC and Monkey's. Before being aware of the other advantages and disadvantages of each (hey, it's lossless after all, I can switch anytime with no quality loss) I chose FLAC because it had a descriptive name with a cool high-tech acronym and file extension, as opposed to littering my hard drive with APE files. TAK is even better because it sounds like the villain from Stephen King's Desperation.

LA reminds me of vintage Roland synthesizers, which is good, but... I don't even know where to get tools for that which would work on my system anymore.

_m²_ & SokilOff :

According to TBeck, Tak 2.0 is supposed to be released before the end of this year.
Needless to say, I'm looking forward for its license.

Now let's keep this on-topic, sauvage78. Don't push your luck.

What do you mean by that? I have a couple of TAK files with Japanese tags and they show up fine on non-JP systems...
Do you mean unicode support for the command line encoder? (Well, file names fed to the encoder)

I believe he does... which is something I've been greatly missing.

"Before christmas" seems to be a reasonable date for the final release. But i don't know, how long the beta testing phase will last. Because i have modified so much code, i may need quite a lot of positive feedback (verification of proper encoding and decoding) before i will feel safe enough to perform a final release... Currently i am not too far away from a beta release (notice: i am calling it "beta" instead of "alpha" because i am quite confident, that it is working well).


A complete brute-force-approach would mean to fully encode any combination of all parameter variations of TAK's filters and to choose the best output. This would be very slow!

If i am right, there is something like a "-secret-totally-impracticable" setting for FLAC, that does something like this. I don't know, how significant the effect on the FLAC compression ratio is, but for TAK it would be very small. And extremely slow!

TAK is using more filters and more variations of their parameters than FLAC does. If you want to calculate the complexity of a complete encode, you have (simplified!) to multiply the possible parameter variations (N) of the filters (fx): N(f1) * N(f2) * N(f3) ... It's a bit like putting 2 pieces of grain on the first field of a checkerboard, 4 on the next, then 8 and so on.

But during the development of the codec i have often calculated the maximum possible compression of selected filter combinations by brute-force and than worked very hard to find a much faster heuristic to estimate the optimal parameters. Surprisingly -after many hundreds of hours- i have always found a heuristic that was extremely close to the brute-force-approach.

Summary: A real brute-force-approach may encode about 50 to 100 times slower and gain less than 0.1 percent of compression.


Good point! I have read the alternative story based upon the TAK character, called "Regulators". I liked it very much and this was one important reason to call the codec TAK.



Yes!

Thomas

Ditto with FLAC. Cf. Omion's test. Even though it only covers decoding speed, you can clearly see that FLAC's --super-secret-totally-impractical-compression-level yields only nominally, neglectably better compression than -8. Even if I can't provide hard figures, I can tell by personal experience that it's indeed totally impractically slow at encoding. Even on a fast machine, encoding occurs at real-time or hardly better.

Apart from that, let me still repeat that - imho - it's a shame that you ditched the -5 compression setting, making -4 the highest preset. TAK's overall speed and efficiency are its well-known advantages, and to me, too, -5 was far from unreasonably slow at encoding, that is, the extra compression was well worth the wait. Just a reminder: almost 2/3 of FLAC's userbase (at least the ones who voted in this poll) use the maximum compression setting. As TAK is a similar, asymmetric codec, I regret that by omitting -5, focus has shifted from the higher to the lower presets.

Just some interesting data, without too much significance:

I use FLAC at -8 but TAK at -p2m and enjoy its incredible encoding and near-flac decoding speed (well, officially it should be even faster but in fb2k it seems a bit different) while its compression is significantly better than flac -8.

Thank you for the answer.


I agree. For me disk space is much more valuable resource than CPU time - I do conversions in background anyway.
The slowest thing that I used (and at the limit of tolerability for me) was wavpack -hhx6... compared to it, p4m is a lightning and I'd be happy to trade some speed to shave a few kilos more.

McGruff the crime dog says, "Kids, don't try this at the breakfast table!" (couldn't resist, lol)

But I did have a serious thought: maybe -p3 and -p4 could get something more aggressive towards compression (i.e. -p3em and -p4em) with the disclaimer that, "This is going to compress reeeally slow for almost an extra 0.04% over -p#m.

is there any beta or so i could test already?

i'm interested in how good this codec will be

Yes, the "focus has shifted from the higher to the lower presets". This was happening from the first presentation of YALAC (as TAK was named before it's first stable release). I have always been affected by user comments at hydrogen. Most users (who posted) wanted it to be as fast as possible. A significant amount of posters (not necessarily users...) kept emphasizing TAK's slower decoding speed than FLAC (especially when using foobar). Maybe i have taken this too serious... Possibly a lot of users indeed would welcome a bit stronger compression in exchange for a bit slower decoding. Possibly i should create a poll?

I have modified the codec to achieve more speed especially for the lower presets. Those modifications may lead to a more significant loss of decoding speed than in previous versions, if i again raise the maximum predictor order for -p4. I will have to try it.


Only a bit better than before.

Currently i am modifying the decoder to be capable to decode files created by later, more sophisticated TAK encoders. Then i can prepare a beta release.

Thomas

I believe this is still true. Most current TAK users are interested in this still, especially the ones who use the default or lower settings.
Decoding speed is definitely a priority, although in this age of computing I think current TAK decoding speeds will be fine for current portables (FLAC decoding speed may be a bit overkill nowadays as compared to 5 years ago.
Why not? I predict the people who use -p3 and up would be in favor of better compression, whereas -p2 users want the fastest encode/decode performance. I am not sure how much die-hard users who want the best compression would care about slower decompression speed. As far as -p3 settings and higher I think the poll might as well ask if decode speed is a priority (i.e. DAW and HTPC don't need to be battery-saving).

Pardon the double-post, but I somehow overlooked this part of the OP. Thomas, can you elaborate about this?

I would say, the strongest side of TAK is efficiency - size/speed rate. It's great to have a codec that has presets with encoding/decoding speed comparable to FLAC and presets with compression levels comparable to APE High.

So I think you shouldn't have hard time choosing "speed or compression". Lower presets are for better speed, higher ones - for better compression

I think you both are right: Let's regard -p0 to -p2 as the speed settings focussing on maximum decoding speed and -p3/-p4 as the power settings which sacrifice some more decoding speed for a bit better compression, even if the proportion may get somewhat insane efficiencywise.

I have alreday raised the maximum predictor count to 256 and will post some results soon.


I am quite confident that i can modify the codec to achieve at least 1 percent better compression for LossyWav-files. Furthermore it may perform significantly better when compressing files with sections where LossyWav hasn't removed any bits and the efficiency falls behind that of a pure lossless encode.

Earlier i wanted to integrate those modifications into the TAK 2.0 codec, but now i plan to create a dedicated codec (the file format supports up to 64 different codecs) . That's a bit easier to do and doesn't stall the 2.0 release.

Thomas

Thomas,

I'm curious on your views of GPGPU and how it could be beneficial for TAK. Do you have any plans to look at this? Mind you I only have a general understanding of how this stuff works but it seems to be getting more interesting now things are going to get more standardized.

The following thread also handles this and even has a proof of concept FLAC implementation based on CUDA.

Here they are:


Compression is relative to the original file size, Enco- and Deco-Speed expressed as multiple of real time.

-p3 is now using 96 instead of 80, -p4 256 instead of 160 predictors.

Some results for other sample rates and bit depths:



Decoding of -p4 is still at least 150 times faster than realtime, what should be acceptable.

That's quite an impressive improvement actually, when compare old p4m to new p4.
Compression ratio from both presets are about the same (0.11 difference) but encoding speed improves 35.93%.

One of us is getting it wrong here. If I understand correctly, compression time drops from 20 times real time to 10 times real time, which means the -p4m compression will take twice as long with TAK 2.0...

Edit: Now I get it, you compared old p4m with the new p4. Sorry, my bad...

I am always interested into new opportunities to optimize TAK and GPGPU is no exception. It's definitely possible to utilize GPGPU for encoding, but i have no practical experience yet. And it will take some time until i will try it. The most important reason:

I am very concerned about the reliability of the encoder. I am worried about failures of the GPU-memory resulting in unrecognized encoder errors. Unless i find a trustable study which shows that GPU-memory isn't failing more often than system memory or unless more GPU's come with ECC for their memory, i really dont wan't to take a risk.

Currently i would prefer to add multicore support to the encoder. It's easier to do, will probably result in a larger speed gain (especially on systems with slow GPU's) and i am feeling more safe regarding the reliability.

Sounds perfectly reasonable to me and it might be the safest approach for new tech, although I think nVidia is actually going to include ECC into their new Fermi architecture. I actually don't know if it will also be made available in normal consumerproducts as marketsegmentation sometimes dictates otherwise. Thanks for the answer and nice to hear that you're looking into multicore CPU support.

As I've had to deal with bit-flip errors with RAM myself, I understand your concern. I also recall reading an article recently about how GPU RAM errors weren't considered critical since one wrong pixel in a video game doesn't matter much.
Here's the thing with lossless codecs though: you can always compare the encode to the source. So, I'm not too worried.

As for the multi-core CPU vs. GPU: think of laptops, HTPCs and generally lower-end computers. They're more likely to have only one or two CPU cores, and a GPU. My new laptop is something of a special case, but a GPU-enabled encoder would run much faster on it: it has an Intel Atom N270 CPU (one core, two threads) and an NVIDIA Ion GPU (GeForce 9400M). The latter's benefit becomes clear when playing high-definition videos, where CPU usage stays below 20% (most of the time, below 10%).

Now, the question is: what machines would benefit most from encoding speed improvements on TAK, which is already quite fast? In my case, my Atom/Ion netbook would certainly benefit more from a GPU-enabled encoder than my quad-core AMD Phenom PC would benefit from a multi-threaded implementation, especially since I can already encode multiple files in parallel.

A quick search with google revealed this interesting page: MemtestG80: A Memory and Logic Tester for NVIDIA CUDA-enabled GPUs. It contains a link to an pdf of the study "Hard Data on Soft Errors: A Large-Scale Assessment of Real-World Error Rates in GPGPU". I had no time to really read or even critically rate the article, but here is an excerpt from the conclusions:




Yes, this seems to be the way to go. I forgot about TAK's verify option, which will decode each frame after encoding and compare the output with the original. Since decoding is so fast, this could be performed by the CPU without sacrificing too much encoding speed.


I agree, this is the question... The answer may require a lot of testing of real implementations. Currently i don't want to put too much effort into it, but i will keep an eye on evaluations of GPU-implementations of similar algorithms as TAK is using.

I'm glac that there are more people like me who are concerned about GPU encoding correctness and it's great that you're one of them.
I've seen somewhere a study done with NVIDIA collaboration that showed that GPU memory errors were indeed an issue...as well as some silicon issues, IIRC GPUs with few processing units damaged passing validation.

But there's a simple solution: Encode on GPU, move to the main memory and then use CPU to verify the results. Retry (on CPU?) in case of a failure.
It should warranty correctness with overhead low enough to still provide a great speed boost.
Verification that parameters are best chosen would be infeasible, which would hurt compression ratio somehow, but I think that the speed improvement is well worth it...especially that there are reliable GPUs from NVIDIA on the way.

Now i am preparing a first release... Probably i will call it a beta.

I am looking at code i have written weeks or months ago to find errors. Today i caught one. Fortunately: Even my quite exhaustive script-based test set possibly wouldn't have caught it because it was based upon very rare conditions. Mathematically possible but extremely rare in practice.

Addditionally i am feeding the encoder with random data to test the decoder regarding features of the encoder, which will later be implemented. I want to be sure, that the TAK 2.0 decoder will decode files created by later, more sophisticated encoders without any problems.

Some bad news for some power hungry users: For now i have again reduced the maximum predictor count from 256 to 160. But the decoder will be laid out to support even up to 320 predictors; this way i am able to add a really insane preset -p5 later ( if i want) and the files will be decodable by the V 2.0 decoder.

I hope, this wasn't boaring. I will release a first version as soon as i am feeling confident enough about the reliability of the new codec.

Thomas

Not boring at all, about the max predictors, would a value like 192 be a reasonable compromise?