I forgot to mention an easy way to get the total sample size to go along with --skip and --until (same as metaflac --show-total-samples).

Thanks Thomas!

Although TAK's decoding speed is already very high in V1.0.3b and probably limited by the IO-speed on most systems, i again have spent some time on tuning.

Here a comparison of the current version with V1.0.3b:


Speed data expressed as multiple of real time on a Pentium 3 with 1000 MHz.

And that's not the end of the line...

I switched lossless format from ape extreme high to TAK -p4.
Though Monkey extreme high is still a little bit better as far as compression ratio is concerned it's not essential to me and I prefer the improved decoding speed compared to Monkey's.

With my collection there's no significant bitrate improvement when going from -p4 to -p5, but real improvement is there when going from -p2 to -p3 to -p4. These settings are very well balanced IMO, and as -p4's encoding speed is still very good for me I use -p4.

Very good work, Thomas.

Any plans to build a rockbox port together with the Rockbox people?
At the moment most of my music on my DAP is compressed via lossyWav+FLAC.
For music with few instruments and/or quiet music however going purely lossless yields the smaller bitrate.
It would be great if I could use TAK on my rockbox armed DAP. At the moment wavPack normal -x5 is the best solution for me (FLAC is a pretty bad performer in many of these cases), but TAK is even better.
Moreover I'd prefer to use lossyWav together with TAK instead of FLAC.

So it would be great to have TAK with Rockbox.
Any plans?

I'm also looking forward to having TAK in Rockbox!

Erm, let's have a linux/Unix port first, please...

Although true for p0 - p2 (or p3), it is still welcome for the higher modes.
Besides it would make it more viable for hardware platforms (if the optimisations are portable).

Now and then i am thinking about dropping -p5... On big representative file sets the advantage over -p4 is usually less than 0.20 percent. But on the other hand some particular files gain up to 2 percent.



There has been a contact with one rockbox developer but it was (and is) too early...

I still have to prepare a source code release in Standard C (at least of the decoder). I am doing it step by step. For TAK 1.0.4 i have replaced some DELPHI specific runtime libraries with portable code. Next step may be the conversion of the code from object orientation to a pure procedural design.

Simultaneously i am working on a slighltly improved codec (a bit faster, a bit stronger and simplier code) which will involve a (small) format change. I have to evaluate some new ideas before i can deceide, if the improvements justify a format change. This has to be done before thinking about a source code release because this should be the first and last format change.


This could happen earlier.


Some are definitely portable, others maybe. But it's difficult to predict the effect on the speed of hardware player implementations.

Thanks to anyone for commenting. It's always very motivating to receive some feedback!

I am thinking about a new preset configuration for -p0 and -p1.

The compression efficiency of new -p1 will lie between old -p0 and -p1.

More important: -p0 will compress much worse than before!

Am i joking?

No.

Welcome in the wonderful world of marketing...

New -p0 is intended to compete with FLAC's -0 to -2. The goal is to make it encode and decode faster than those FLAC settings and -as you would expect from TAK- compression should nevertheless be better (about 0.5 percent) than FLAC.

I doubt, that anyone would use this weak preset, but if want to make TAK the fastest codec, it should also compete well at this low level.

What do you think?

I once use -p4 but some test show me that.. I have a lot of these "particular files (album)" that -p5 offer somewhat noticeable better compression rate than -p4. But if this happen, decoding rate will noticeable slower too. (which is not a big deal) Most of these is quiet music.

TAK is very good with quiet music. Today I find some album that TAK have better compression than the APE "Extra high"

On the another hand.. Sometime (Once.. maybe.. I can't remember), There are another type of "particular files" that made TAK -p3 have better compression rate than -p5 weird eh? And I'm not talking about white noise & some weird stuff too.

And good news on decoding speed up. Any chance to hope about encoding speed up?

About -0,-1 Isn't it fast enough?

Thanks for your hard work TBeck

Well, if you do want to be the encoding speed champion: go ahead.
In a practical sense I can't see how FLAC -0 to -2 can be attractive to anybody. It's hard to imagine that FLAC -5 isn't fast enough while providing a significantly better compression ratio.
Same goes for TAK's -p0 and -p1 attractiveness so it won't hurt if you go this way.

It depends on what goal do you want to reach.

If you want to get first places (or at least beat FLAC -0) in speed tests, then ultrafast presets will be useful. If you plan to get hardware support from some vendor who produce portable players, then you may need these ultrafast presets as well.

But I see no reason to change -p0 and -p1 presets. You may simply add something like -p00 and -p01 presets with faster speed.


But if we're talking about 'real life' usage on PC, then compression means a lot. Personally I use TAK and WavPack with maximum level of compression. And that's the main reason why I prefer these codecs over FLAC.

Encoding speed in this case means not much at all, cause I encode my music only once. As about decoding speed... OK, let's say FLAC -0 has on my PC decoding speed 210x, TAK -p0 has 175x. So what ? How can I see this difference during playback when I have ~0% CPU usage in both cases ?

Once again, it depends on your goals, so you decide. And thanks for great codec btw

On what CPU/RAM? What version of TAK/FLAC libraries? It's interesting that the new FLAC libraries in foobar0.9.4.5.(?) got much faster, now they reach 480-600x decode rate on 44.1/16 stereo music - sometimes they get over 650x (especially on classical music - unaffected by loudness war?)
TAK 1.03 (with foobar 0.9.5 and up) 2 max reaches ~330x-340x on the same machine - about on par or somewhat (~10%) faster than high-bitrate mp3, and definitely faster than wavpack normal (~250x-260x on the same config, same foobar). I don't have any APE files ATM, but AFAIK it's also much slower than TAK.

So the proportions look very different here (FLAC -8 typically leads by ~70% opposed to your ~20%), my system (as a whole hw+sw combined) seems to like FLAC a lot better I dunno how much FLAC settings matter (TAK settings don't change decoding speed very much) but if they do, the difference can be even bigger.
But it doesn't matter anyway, TAK is still very fast and I won't stop supporting it

Specs: core2 duo (conroe 4M core) @3.33GHz, (dual) DDR2 RAM@832MHz/cl4; intel P965 chipset.
Pity that I'm not very familiar with the actual strength of hardware players, but my uneducated guess is that current TAK is no problem, though optimization is always welcome as it increases battery life for sure

If the current code is OO, wouldn't C++ make more sense? Open source doesn't need to be plain C.


I don't think it matters much, (lossless) compression is usually a tradeoff between speed and compression, what you suggest is merely a (small?) shift towards more speed and less compression for p0 p1. I myself use only -p3 and -p4, the compression factor with the achieved decoding speeds is the selling point of TAK (IMO).

Sorry, I was probably a bit too abstract.

My config: Athlon64 3800+ @2.4 GHz, 1 Gb RAM. I got latest foobar v0.9.5.1 beta2 (hope it uses latest FLAC), foo_input_tak dated 9.nov.2007 (is it latest one ?) and foo_benchmark plugin.

Same wav file was compressed with FLAC v1.2.1 using -0 preset and with TAK 1.0.3 using -p0 preset. Decopression speed benchmark is:

FLAC: 352x
TAK: 212x

So yes, latest FLAC is ~1.66x faster than TAK on their fastest presets.

But there is another question: what advantage do I get from this ? Even if some codec decodes @ 1000x, what does it change ? I listen all music @ 1x anyway I get ~0% CPU load with both these codecs. So is there some REAL advantage except nice benchmark results ?

Imo the place where decoding speed is really very important is portable hardware device. Less decoding complexity = work on slower processors and longer playback time. That really makes sense.

But speaking about playback on PC I see no advantages at all: 200x, 400x or even 1000x. It changes nothing.

Except that lossless audio makes little sense on portable devices. It even makes little sense for plain playback on any device, portable or not, unless it serves an alternate purpose at the same time, such as transcoding, archiving, etc (not much point taking up an additional 20 gigs for a lossy archive when a lossless archive is already available)...
The only advantage to fast decoding speeds that I can think of, is in the case of transcoding (less decoding time means more processing power for encoding).

Do you mean FLAC -8 is decoding 70 percent faster than TAK -p2?

Please be aware, that you are not only testing codecs but also how well foobar and the codec are interacting. There have been reports that decoding of TAK files with the foobar plugin is significantly slower than when performed with the TAK application.

Some data comparing the decoding speed of TAK -p2 and FLAC -8 (with md5 turned of):

Josh Coalson's FLAC comparison: 197.84 / 156.47 = 1.26 (FLAC is decoding 26 percent faster)
Synthetic Soul's comparison: 143 / 124 = 1.15 (FLAC is decoding 15 percent faster)

I have my doubts that the level of compression used with TAK will make any difference when it comes to proper playback on a portable.

not many people here use it: http://www.hydrogenaudio.org/forums/index....showtopic=58731
for flac it can be useful for real-time transcoding from unsupported formats (e.g. slimserver->squeezebox).

if it were really that important I could make -0 faster.

3 very good ideas



"plain" C is ways more portable ... go "plain" , please

And transcoding scenario? It's important. It's only CPU results.
I think those benchmark didn't count with real I/O and other conditions. (RAM's and HDD's speed.)

While uncompressed *.wav has a higher CPU speed, TAK and FLAC have smaller data rate (less data to read from HDD). That's why it takes less (real) time to transcode from TAK, FLAC to LAME MP3 (in foobar) than from uncompressed *.wav to same MP3 on some systems. So CPU isn't bottleneck before some limit.

Is it really important to worry about very high decoding speed comparable to FLAC -0 ... -2? When each generation (each 2 years) of CMOS(-like) transistors have a lower power consumption that benefit longer battery life. Current hardware supports FLAC -5 -8 and TAK 1.0.4 -p0 -p1 has the same decoding speed. I wouldn't talk about cuantic electronic revolution that will permits to low power consumption at least hundreds time because someone would hit me. But there are some thoughts that it just the matter of time.
In other hand the consumption of LED display mostly is higher than of decoding itself.
Maybe there is no need to loose compression of -p0 at cost of higher decoding speed.

Agree.

Plain 'C' is better for reading and porting to other language, such as assembly.

It maybe important only for those people who do transcode very big amount of music (20-30 Gb) rather often. Actually I know noone who does so.

For common user, who can transcode 2-3 new albums a time just to put 'em to portable player, there will be no big time win anyway.


Edit: typo

Decoding speed and transcoding: IMO transcoding cannot be the bottleneck there, if the encoder is slower than the decoder. I've seen decoding speed as a bottleneck only in a replaygain scanning scenario...

It's not a question of decoding being a bottleneck. Decoding takes processing power, no matter what, and 5% of the CPU is better than 10%: it leaves 95% for the encoding process, vs. 90%.
Granted, that's irrelevant if you have more than one CPU/core and run only one transcoding process (i.e. only one decoding process and one encoding process), but then you're wasting the other core(s). It becomes relevant again when you run multiple transcoding processes at once.

Personally i think a new preset p0 has faster decode speed and lower compression ratio is good idea,its mean that when system use CPU process power 99% (likely run transcoding or some error process blocked system) my music playback wouldn't be easier to delay especially with ASIO audio driver.

Oh yes, I was wrong about this.

I think when we're talking about formats that can be decoded <1% of the CPU in your PC (probably <1% of one of the cores of the CPU) it's the the OS's fault: it could make good use of a better scheduler that gives a damn about priority... You can try to use longer buffer in the player as a workaround though

edit. umm, typo?

Now that you have switched to solely using numbers rather than a single letter to denote the compression level I see no problem in introducing a few more presets. Therefore, I would gladly see a faster preset that compressed less well, and would gladly keep -p5 (or -p6 or -p7, as it may now be). The wider the range the better I think, to a degree*.

* OptimFROG's range of options are just too confusing IMHO, but I can't see how you can go wrong with a numeric scale (0 = fastest/worse compression, n = slowest/best compression) and the additional option of adding one of two evaluation levels.

I'm dying to get the Linux port...

Me too. What kind of effort would that take?

I have modified -p0 and -p1. New -p0 is currently encoding about 11 percent faster than old -p0. This widens the range at least a bit... I wasn't happy with old p0/p1 because they were so close performance wise.

Here a comparison of the current version with V1.0.3b:


Speed data expressed as multiple of real time on a Pentium 3 with 1000 MHz.

-p0 is now using 4 instead of 8 predictors, -p1 12 instead of 16.

Don't know if it really matters, but i feel better with this configuration. And i see a chance to speed up the -p0 encoding by 15 to 20 percent.

edit: Encoding with -p0 is now another 6 percent faster...

It would take Thomas letting go of his baby and one or two enthusiastic developers who could contribute *their* free time. I fully understand that Thomas has other priorities in life (family (?), work), but refusing help for now makes the point moot. He's been developing his codec for over 10 years, made his first announcement here almost two years ago, and us non-Windows users keep drooling over it, waiting for a port. One year ago we were told to be patient. How much longer? David Bryant found people to port WavPack to linux and write a plugin for XMMS... Now it's even supported by other players.

I'm worried that Thomas may miss the train again because of his reluctance to release the code in its current state, all these feature requests that he's considering, his plans to switch to C++ and Object Oriented Programming, etc... Hashing can be done by piping raw PCM output to a hashing utility. Tagging can be implemented by encoding software, tagging software, or even a wrapper shell script. Cue sheets can be used externally. Artwork is taken care of by APEv2, hence relates to tagging support. More to the point, all of these features have little to do with major format changes, and could easily be developed on a multi-platform codebase. Delaying a souce code release by however much time it would take for a single, already busy person, to implement those features sures sounds like a bad idea to me.

I fear TAK will become yet another Windows codec with the kitchen sink included, and that non-Windows OSes and music players will be 2nd-class citizens (again...). Pushing for WavPack support on unix is already hard as it is ("WakPak? Who the hell uses that thing?"), imagine how hard it's gonna be for TAK! Now that I think of it, I wonder if FLAC didn't get such good support on Free operating systems and playback software in part thanks to Josh's extensive work in documentation and portability. I was drawn to FLAC myself because I was able to implement FLAC metadata parsing in PHP, thanks to his very clear documentation. Later on, I was able to quickly implement the CD-TOC metadata block.

I understand why Thomas would want to wait until the format is final, but then again, people could port the codec as it is now, and with all that work done, subsequent modifications would probably take less time. Also, different people might be interested in working on different things in parallel: one guy might volunteer to port the whole thing, while another might be interested in various modern CPU optimizations, for instance. Look at Vorbis: one guy wanted to make it sound better, and another wanted to make it faster. The result? A faster and better sounding version. Working alone and refusing any outside help is the worst way to go when you are unable to fully dedicate your time and resources (IMO, anyway).

Note that I wouldn't rant about this if Thomas hadn't asked for feedback from the get go. Well, here it is: after a couple of years of waiting, I'm growing impatient.

Often -p5e can cure this.


Ok, only 6 percent more for now. Nevertheless -p0 is now a super fast encoding preset with still very good compression.


Thanks for the input! I will reply when i have a bit more time...

BTW: TAK 1.0.4 is now quite complete. Most important changes:

1) Depending on the preset up to 13 percent faster decoding.
2) New super fast turbo preset.
3) Decoding to StdOut (piping).
4) Most delphi specific libraries replaced with own code.

Probably we will see a release in about 1 to 2 weeks.

Thomas

TAK is going to be a very big monster in terms of being a competitive codec... Thomas work is being amazing... so much implemented in little time... soon his TODO list will be done and then it's just destiny...

As brought up before, beware of preset bloat. The general public does not seem to be waiting for yet faster encoding at the expense of compression ratio: http://www.hydrogenaudio.org/forums/index....showtopic=58731

It's not exactly the same thing. Since FLAC decode rate almost if not all are the same no matter what compression level you use. It's more attractive to use FLAC -8 because... well.. why not?

Another codec are different from this.

Yes, TAK's decoding speed is also affected by different -p settings, but luckily there is no strong increase in decoding effort when using higher -p values.

Anyway the people with compression ratio in focus will use -p 2+. And for the speed folks the new -p 0 and -p 1 settings are welcome. So IMO things are alright for everybody's needs.

I wonder how extra and maximum levels of evaluations will be affected in a new modified -p0 preset.
If new -p0m will have the compression ratio on par with FLAC -8 probably I will stay with it. Or move to p1.
Anyway encode and decode speeds will be more than enough high in a new version.

Thank you!

But i don't think my TODO list will ever get empty...


That's exactly my valuation.



Here the results for my primary test corpus:


Speed data expressed as multiple of real time on a Pentium 3 with 1000 MHz.

or try it yourself: TAK 1.0.4 - Beta release 1

For the priorities: Well, i don't think someone who has also to work a bit for money could spend more time on a project as i on TAK... If you have got the impression, my dedication to TAK has decreased: That's wrong. Possibly one can get this impression, because now there is less discussion about TAK (because i am no longer releasing evaluation versions to test new optimizations).

For the 10 years: It did take so long, because i did so much (several thousand hours), not because of a little priority. TAK contains a couple of very new technologies.


I am preparing a port from an object oriented to a procedural approach (not the other way round) to make an implementation in standard C possible.

For the other features: Users are asking for them and the other lossless compressors have them. Without them TAK will always look bad in feature comparisons and some people will get the feeling, TAK isn't mature.


Absolutely right and -i fear- a very good argument to support my position...

It doesn't make much sense to release source code which isn't clean and lacking a good documentation. Not if you want widespread support. I think there is quite a lot of developers who would be excited when i release the source code. I don't want them to be disappointed because of bad code or documentation. A bad release could kill their motivation and i don't think i would get a second chance to attract developers.

Also very important: I am working as a self employeed developer and if i release bad source code, this will hurt my reputation!


I am convinced a project of some complexity needs some authority (or a heart) to coordinate the actions, at least in the beginning. This task would leave less time for my development work.

Summary:

I a am trying to do what i thinks is the best for TAK's future and also fun for me.

If you look at the latest release (V1.0.4), you find a a good mixture of new features which take into account different demands:

1) Speed optimizations = Much fun for me and important for TAK's reputation as possibly the most efficient (speed * compression) codec.

2) Pipe decoding = Make the users happy

3) Replacement of delphi specific libraries with own code = preparation of the translation into standard C.

That's the way i will do it: A good balance of having fun and considering user demands.

I also have to care more for marketing. BTW: TAK still needs a logo!

I have also raised the priority of "Applications for other operating systems than Windows" on my todo list:

* Tag writing functions for the TAK applications.
* Fast seeking without seektable.
* Raw audio data files as input for the encoder.
* Option to decode only a specific part of a file.
* Unicode support.
* Even more speed and better compression.
* Applications for other operating systems than Windows.
* MD5 audio checksums for verification and identification.
* TAK files as input for the encoder for transcoding purposes.
* Multi channel audio.
* A german version.

This list is dynamic.


Thanks for the feedback!

I have tried my best to explain my position. Because of my quite limited english, my explaination is less detailed and clear than i would like it to be.

Thomas

Tbeck, I TOTALLY agree with your stance, and feel that you should NOT release the source code until you're sure the code is clean, well documented and upto your standards. Furthermore, I think you should not release the source code until you're confident that you will not have to make ANY changes which might break the standard.

I see TAK's current status as an advantage; unlike other "more established" codecs, you have the complete freedom to make changes which might break backwards compatibility but result in a very improved codec.

Please keep up the good work.

I'm not questioning your dedication at all, just pointing out the realities of life: you're not making a living with TAK, hence you cannot spend all of your time on it. Nothing wrong with that. What bugs me, is that things would go much faster if you let other people spend their free time on TAK.


Well, right now it looks bad to me, because there's no unix support (not even MacOS support). Besides, maturity is something that one acquires over *time*. Why can't TAK acquire that maturity while being multi-platform from the start? Why does it need to have everything before you'll let non-Windows users run it?


Alright, but there's a large gap between cleaning up code and writing documentation, and waiting for a full set of features to be developed before even considering releasing the source code. Besides, do you really want to build all these features on your current, I assume messy code, and *then* clean the whole thing up? You've got it backwards!


You have got to be jocking...


Is that it? That's my main gripe about it all: you want TAK to have all of the features other projects added over several years, before you're going to consider other platforms. And you want to do it all by yourself. Will you delay things further if a user comes along and requests, say, a plug-in for Adobe Audition? Oh, and what are you going to say when users bitch about the lack of support for adding cover art directly from TAK.exe? It took years for FLAC to get all that! Did FLAC have a bad reputation before? If TAK had any reputation in the Free Software world, it would be the kind of rep that, hum, closed-source, windows-only software gets. Reminds you of anything? Foobar2000? EAC?


What about chinese? There's literally a billion of them! And what about hindi? That's another billion people!

Look at Monkey's Audio: Windows-centric software, no multichannel support, no piping, no Replay Gain, no inline-tagging, and the list goes on... Yet it's a somewhat popular codec because of its excellent performance. And guess what? It got ported to linux. People wrote plugins for it. Piping support was added through a simple patch. Tagging is supported by third-party software.

Like I said, you want the kitchen sink to be ready before you move to a source code release. I bet I won't see encoding + decoding + playback support under linux before at *least* a[nother] year.
Edit: while we're at it, let me throw in another feature request: multi-threaded encoding. By the time you're finished, we should all have 8-core CPUs (I'm getting a quad-core tomorrow)...


Except he does NOT want to take advantage of that freedom (see his earlier response regarding the ubiquity of MD5).

I thought it was finalized.


This also applies to format specifications and it's a pretty serious issue because you usually can't fix a spec without breaking decoders. Wouldn't it be cool if the smart guys joined forces to actually design something like a new codec? IMHO this makes sense since creating something which is able to outperform existing solutions significantly is very hard to achieve if you're all by yourself.

Cheers,
SG

Well, for instance, he wants to add MD5 hashing support. With FLAC, it's a fixed field in the streaminfo metadata block. I would consider that to be part of the format specification. So, unless he plans on making the MD5 hash a mandatory APEv2 field entry, the TAK format isn't finalized. He also plans on adding multichannel support, which might require some changes.

My reply was rather directed towards other readers who could misinterpret this and get a wrong impression.


Maturity may lie in the eye of the beholder. A software can be rock-stable and feature-rich but nevertheless be percepted as immature. Some example: Earlier i initiated far more discussions about TAK's features, often asking for details on how to do something right. At some point i got the impression this behaviour made some readers doubt, i did know what i was doing and that TAK had already achieved a stable (mature) state. That's one reason why i am now a bit more reluctant.


My code isn't messy and it never was. But long time it was designed to make my many evaluations of new compression methods possible. It served this purpose very well but was unneccessary complex for a production or end user release.

I will definitely only release source code i am feeling comfortable with. Maybe i am a bit too much of a perfectionist, but on the other hand it was exactly this trait of mine that made TAK as strong as it is. I think, you have to live with this as i have to do...

It's still much work to prepare a source code release. Indeed i could speed up the process if i did nothing else. But this would get very annoying for me. Therefore i am trying to balance this work and the fun part.


As i wrote above, i am doing several things simultaneously: a) Adding new features and b) working on a source code release in bearable portions. At some point b) will be finished and this has not to be at the end of the feature list.


Well, i am german... I would like to have a much better documentation for the applications. Since i am not good in writing english, i will do it in german and than ask someone to translate it. The same applies to my homepage.


Hm...

Is there a limit for the quotes per post? Seems i have to split it. To be continued.

And nevertheless i have the impression it is slowly dying... Source code availability is no guarantee for lasting success.

Don't take me wrong: I really like Monkey's audio and it was my codec of choice until the release of TAK 1.0.


Why not ask Josh Coalson to add a stronger hash to FLAC? A new additional meta data structure wouldn't break compatibility. He is possibly be in the position to establish a new standard. I am not.


Given my limited english skills, this reply was much work for me.

I am aware, that more arguments exist - both supporting and against my position.

I don't think anyone can really predict, if it is better to release the source code now in it's current state or later when it meets my demands. I think i have to let my intuition deceide...

Thomas

Ten, IIRC. Annoying, ain't it?

Reasons for using it are quickly going away (IMO), that's for sure.

Somehow this doesn't come as a surprise.

I don't care what people say, to me encoding efficiency is important and you've made a lot of progress in that direction. Thanks Thomas!

In the Free Software world at least (can't speak for Windows or MacOS), it seems the problem was with licensing. FOSS developers claim the license is too ambiguous (the author requires them to ask for permission) and thus refuse to engage work into supporting and including Monkey's Audio in apps and linux distros. I guess it's fundamentally incompatible with the main Free Software licenses (GPL, BSD...).


It used to be too slow for my taste on my old hardware (Athlon XP), but now (Athlon 64 X2) it smokes all other codecs in terms of speed/compression ratio. That was a nice surprise.


FLAC doesn't need a stronger hash. I was merely pointing out your reluctance to break free from what you see as "standards".

If flac doesn't need a stronger hash why would TAK need something stronger than what flac uses?

I don't really have a dog in this race, but it would seem to make sense to use a method that's compatible with at least two other codecs. For those transcoding between formats, md5 would make for pretty easy verification.