TAK Blog 2007-11-28, Report on my current work Options

[TBeck]

I have just successfully converted my first file with foobar2000 by using pipe encoding!

I had to go through a lot of trouble because of some irritating behaviour of the delphi compiler when dealing with stdin/stdout.

Looks as if TAK 1.0.3 will bring support for pipe encoding. Because i have to write some validation tools to check the many modifications of my tak stream read/write functions, which were required for piping support, it might take about 3 weeks until a release.

Other features of the next release:

- Tiny compression improvements for the presets TURBO, FAST and NORMAL.
- About 4 (-p4) to 8 (-p0) percent faster decoding.

That's all for now.

Thomas

[Bourne]

great news! thanks for this excellent work!
please hardware manufacturers, TAK support for heaven's sake!

[footballking3420]

Could somebody explain to me what exactly is pipe encoding? I have heard that recently due to TAK, but have no clue what it actually means and why it's so desired.

What I'm really waiting for as far as TAK goes is Album Art support since I don't believe it has been implemented yet. Once that happens, I'll switch over from FLAC.

[alvaro84]

Could somebody explain to me what exactly is pipe encoding? I have heard that recently due to TAK, but have no clue what it actually means and why it's so desired.

Basically when current version (1.0.2) of tak compresses something it has to be in a wav file, therefore if you'd like to convert something from, for example, flac to tak, you'll have to create a tempfile and give that to the tak compressor. But if the command line compressor supports pipe encoding this step can be omitted: foobar can decode a part of the flac file, send the data to the tak compressor through stdout/stdin without creating a temp file, and repeat it until the end of the file.
When you use pipe encoding, you don't have to have additional storage space for the decoded (pcm wav) data, less fragmentation occures and foobar can draw a nice progress bar while converting a file

(I hope that a pipe encoded file will be exactly the same as a file-to-file encoded one. Some time (and a few versions) ago I made some experiments with flac and it created many additional seek points when converting through the pipe, dunno why

Correct me if I told something wrong

This post has been edited by alvaro84: Nov 28 2007, 07:37

[TBeck]

(I hope that a pipe encoded file will be exactly the same as a file-to-file encoded one. Some time (and a few versions) ago I made some experiments with flac and it created many additional seek points when converting through the pipe, dunno why

No, the file will be different...

The seek table is located at the beginning of the file and therefore sufficient space has to be reserved before writing any audio data. If you don't know the final file size (respectively the duration) in advance, you have to reserve enough space for the biggest file you may encounter.

There are two reasons why the final file size is unknown:

1) When reading from StdIn (the pipe) you can't ask the operating system how large the pipe is.

2) Newer versions of Foobar put a wave file header at the front of the file, but with an invalid size entry.

By default TAK 1.0.3 will create a seek table for a file duration of 8 minutes, which is 2400 Bytes large. There is a new command line options which let's you specify the seek table size in minutes.

If your file is smaller, some space is wasted and the compression ratio isn't optimal. Therefore i wouldn't recommend pipe encoding for users who want to achieve the highest compression possible.

If the final file is longer than 8 minutes, TAK will remove some seekpoints to have space for the additional play time. By default the seek point distance is 1 second. If the table is full, every second seek point will be removed and the seek point distance grows to 2 seconds. This can be repeated to some point.

Another reason for a file difference: Foobar does not neccessarily send the original wave file header and is dropping (rare) additional wave file meta data.

Thomas

[alvaro84]

...by default TAK 1.0.3 will create a seek table for a file duration of 8 minutes, which is 2400 Bytes large. There is a new command line options which let's you specify the seek table size in minutes.

If your file is smaller, some space is wasted and the compression ratio isn't optimal. Therefore i wouldn't recommend pipe encoding for users who want to achieve the highest compression possible.

If the final file is longer than 8 minutes, TAK will remove some seekpoints to have space for the additional play time. By default the seek point distance is 1 second. If the table is full, every second seek point will be removed and the seek point distance grows to 2 seconds. This can be repeated to some point.

Sounds reasonable. I think I can live with about 1kB wasted per multi-megabyte TAK file (~4-5 minutes long tracks assumed here) and I like your solution for longer tracks, I find it quite elegant In the case of flac the file sizes (and the seektable lengths) were much more different.

[skamp]

I never understood why we don't put metadata (not just seektables) at the end of files.

When streaming files, we could send the metadata as a separate chunk, either parallel to or before the audio. Or, the stream format could allow for metadata to be located at either the beginning or the end; the streaming application would reorder data appropriately.

Hard drives would have to seek to the end of the file to load metadata, and then back to the beginning, but I doubt it would pose much of a problem with most files people have, i.e. ranging between 4 and 50 MiB. And if it *does* for whatever reason (I'm thinking of slow rotating media), make *that* the special case. I don't see a justified need for padding and other trickery in most uses. Besides, it seems like we're in a slow but steady transition to Solid State Drives with no seeking involved…

[Synthetic Soul]

By default TAK 1.0.3 will create a seek table for a file duration of 8 minutes, which is 2400 Bytes large. There is a new command line options which let's you specify the seek table size in minutes.
...
If the final file is longer than 8 minutes, TAK will remove some seekpoints to have space for the additional play time. By default the seek point distance is 1 second. If the table is full, every second seek point will be removed and the seek point distance grows to 2 seconds. This can be repeated to some point.

As someone who rips to image, and uses foobar's "Convert to Album Images with Cuesheets or Chapters", I wonder how TAK would cope with a 70 minute file.

[TBeck]

As someone who rips to image, and uses foobar's "Convert to Album Images with Cuesheets or Chapters", I wonder how TAK would cope with a 70 minute file.

If the seek table default size of 8 minutes is beeing used, the seek point distance will grow to about 16 seconds, what means a bit slower seeking on playback.

The table size can be adapted with a new command line option:

CODE

  -sts#      set the seek table size to # minutes (5 to 1440)

Possibly it would be a good idea to define 2 different presets in foobar:

CODE

  -sts5    for small seperate files
  -sts45   for album images

Thomas

This post has been edited by TBeck: Nov 28 2007, 15:16

[Peter]

2) Newer versions of Foobar put a wave file header at the front of the file, but with an invalid size entry.

... and old versions give you values calculated from source file length, which are inaccurate in many cases: truncated source file, input format not signaling accurate length such as WMA lossless or MP3 without LAME-style headers, etc; you run into the same scenario eventually but your encoder works as intended in most cases so commandline encoder developers tend to overlook the issue, leaving users with an unreliable solution. The header you're describing as "invalid" isn't even technically invalid, it just simulates a truncated WAV file scenario.
My preferred solution to this issue (as well as the WAV file size limit) is to add an "ignore WAV length fields" commandline switch to the encoder, so the frontend can tell your encoder to expect an unknown-length stream.
I'm not in position to criticize your designs, but I find it strange that so many popular lossless formats run into seektable allocation trouble when encoding from unknown-length streams. I personally just keep recommending people to use formats that deal with this scenario cleanly (such was WavPack).
There are multiple ways to allow your format to encode a stream without knowing its length in advance: putting seeking information in the bitstream itself rather than maintaining a seektable (Ogg-style), putting seektable at the end of the stream, etc. I guess you can't use any of those now that your bitstream format is frozen.

[TBeck]

2) Newer versions of Foobar put a wave file header at the front of the file, but with an invalid size entry.

... and old versions give you values calculated from source file length, which are inaccurate in many cases: truncated source file, input format not signaling accurate length such as WMA lossless or MP3 without LAME-style headers, etc; you run into the same scenario eventually but your encoder works as intended in most cases so commandline encoder developers tend to overlook the issue, leaving users with an unreliable solution. The header you're describing as "invalid" isn't even technically invalid, it just simulates a truncated WAV file scenario.

This was not meant as criticism!!! I had read some threads dealing with this issue and already knew why foobar does it this way. But since i am not very fluid in writing english, i was too lazy to try to explain it. Sorry if you got this wrong...

My preferred solution to this issue (as well as the WAV file size limit) is to add an "ignore WAV length fields" commandline switch to the encoder, so the frontend can tell your encoder to expect an unknown-length stream.

Surely, TAK 1.0.3 has such a switch.

I'm not in position to criticize your designs, but I find it strange that so many popular lossless formats run into seektable allocation trouble when encoding from unknown-length streams. I personally just keep recommending people to use formats that deal with this scenario cleanly (such was WavPack).
There are multiple ways to allow your format to encode a stream without knowing its length in advance: putting seeking information in the bitstream itself rather than maintaining a seektable (Ogg-style), putting seektable at the end of the stream, etc. I guess you can't use any of those now that your bitstream format is frozen.

Something like "putting seeking information in the bitstream" is defined for the container, but has not been implemented yet.

I can think about one reason for putting the seek table at the beginning of the file:

If you want to encode on a hardware platform with very little memory, you possibly can't keep the whole seek table data in memory until the file end has been reached.

But i don't know how relevant this is.

Thomas

[GeSomeone]

Am I correct in assuming that the default seektable lenght (of 8min.) and --sts are not used when encoding from a file?

[TBeck]

Am I correct in assuming that the default seektable lenght (of 8min.) and --sts are not used when encoding from a file?

Yes, they will be ignored. The same is true if you encode from a pipe and don't (have to) specify the new -ihs (ignore header size) switch.

Thomas

[Bourne]

Could somebody explain to me what exactly is pipe encoding? I have heard that recently due to TAK, but have no clue what it actually means and why it's so desired.

it's really simple... when you transcode from another format, it will generate TAK files directly, "on-the-fly"... it's like drinking and peeing at the same time if you know what I mean... and you don't have to generate a WAV before you encode it to another format.

[TBeck]

... when you transcode from another format, it will generate TAK files directly, "on-the-fly"... it's like drinking and peeing at the same time if you know what I mean...

To cite Frank Zappa: "Why does it hurt when i pee?"

This seems to describe perfectly the trouble i had with the redirection of TAK's output...

This post has been edited by TBeck: Nov 28 2007, 23:06

[Leto Atreides II]

Am I correct in assuming that the default seektable lenght (of 8min.) and --sts are not used when encoding from a file?

Yes, they will be ignored. The same is true if you encode from a pipe and don't (have to) specify the new -ihs (ignore header size) switch.

Thomas

Could you rephrase the second sentence there? I'm confused by the "don't (have to)" phrase...

I think it means --sts is ignored when encoding from a pipe and you don't specify -ihs. But then how is (have to) changing things here?

This post has been edited by Leto Atreides II: Nov 28 2007, 23:44

[Synthetic Soul]

You don't specify -ihs because you don't have to.

This refers to either the user not specifying the switch ("don't") or the application not requiring it ("don't have to").

If you re-read you will see that Thomas confirmed that --sts will be ignored when encoding a file, not from a pipe for which it intended.

Both -ihs and -sts are new switches that may be used when piping.

[ssjkakaroto]

...formats that deal with this scenario cleanly (such was WavPack)....

TBeck, can't you use the same approach as Wavpack?

This post has been edited by ssjkakaroto: Nov 29 2007, 15:23

[TBeck]

...formats that deal with this scenario cleanly (such was WavPack)....

TBeck, can't you use the same approach as Wavpack?

As i wrote above, TAK's container is prepared for this, but i haven't it implemented yet. And i don't plan to do it soon. Reasons:

1) It's a significant amount of work.

2) It will make the code more complex.

3) Currently i can't see a big advantage. The only situation when you will waste a bit of space is pipe encoding (with the ignore header size switch turned on). The loss may be about 0.05 to 0.10 percent, if you haven't choosen a very inappropriate seek table size. And you can reduce the loss by further reducing the seek table size. This may result in a bit slower seeking, but i doubt this will be slower than the WavPack approach.

And if you are a power user who is always encoding with the strongest setting, you may be happy without pipe encoding, because the encoding process anyhow takes as long, that the use of a tempoary file will not matter very much.

My position is different when it comes to streaming; then it makes sense for me to implement the wavpack approach.

Thomas

[jcoalson]

I'm not in position to criticize your designs, but I find it strange that so many popular lossless formats run into seektable allocation trouble when encoding from unknown-length streams. I personally just keep recommending people to use formats that deal with this scenario cleanly (such was WavPack).

btw seek tables are not required for flac either, if you add --no-seektable to the encoding options it will behave just like wavpack.

[skamp]

I wish either of Josh, Thomas or David would answer my post in this thread and tell me what I'm missing.

[graue]

I would assume it's so that when downloading a file, you can see the metadata without waiting for the whole thing to finish.

A lot of old ID3v1-tagged MP3s I have never downloaded fully (however many years ago), hence they have no metadata. Not a good situation. With a lossless format, you might argue that correctness outweighs robustness, but there still is the matter of listening to a FLAC as it's still downloading... which given enough bandwidth, I can currently do. The relevant info is at the beginning.

[skamp]

Erm, I addressed that issue earlier in this thread...

[TBeck]

Erm, I addressed that issue earlier in this thread...

refers to:

I never understood why we don't put metadata (not just seektables) at the end of files.

I can only answer this for TAK:

The recommended tagging standard is APEv2-Tags located at the end of the file. This seems to be most important for me, because tags are the stream component which is most likely to be modified.

If you want to link to an already running audio stream (provided by a server), you usually will have missed the header and will not be able to seek to the stream end to read the tags. In this situation it is necessary to periodically embed meta data into the stream. TAK's info frames are prepared for this. Most of the time neither a header or a footer will help you.

For the seek table: It's located in the file header. But if you don't use pipe encoding with the -ihs switch applied, the size of the table is known in advance, no space is beeing wasted and i don't see any reason why the table should ever be modified.

One of the next versions will support continous embedding of seek data into the audio data stream and then we don't need a seek table for the special case of "pipe encoding with the -ihs switch applied".

Ok, there is a file header in the TAK stream. Maybe that's simply because developers are used to write file headers...

Thomas

This post has been edited by TBeck: Dec 11 2007, 17:58

[skamp]

If you want to link to an already running audio stream (provided by a server), you usually will have missed the header and will not be able to seek to the stream end to read the tags. In this situation it is necessary to periodically embed meta data into the stream.

Or, you can just send the metadata on request on a second channel. No need for fancy stream protocol designs... Doesn't shoutcast already do that?

This post has been edited by skamp: Dec 11 2007, 18:38