Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Yet another lossless audio compressor... (Read 97251 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Yet another lossless audio compressor...

Reply #25
Quote
Quote
Code: [Select]
-----------------------------------+---------+-----------------+---------+
Sum:        58.13   57.94   57.90  |  60.88  |  57.89   57.65  |  --.--  |
EncoTime:   23.37  113.12  247.91  |  --.--  |  28.01   49.20  |  --.--  |
DecoTime:    5.79    6.21    6.28  |  --.--  |  30.83   53.37  |  --.--  |
-----------------------------------+---------+-----------------+---------+
[a href="index.php?act=findpost&pid=377983"][{POST_SNAPBACK}][/a]


That's good. Too good. It's indeed truly remarkable!

What day is today again?
[a href="index.php?act=findpost&pid=378007"][{POST_SNAPBACK}][/a]


No, no, no... I should have carefully looked into my calendar before posting... Bad timing.

But it's true. The whole design of the compressor was made for speed. When i started with the work (1997) my pc was simply too slow to perform the necessary test runs for evaluation in tolerable time. Then i got my first Pentium MMX with 200 MHz.  And that it was: MMX made the things 2 to 3 times faster. So i built the whole compressor for the 16 Bit-Arithmetic, that MMX allowed. To be exact, most calculations are beeing done in 13 Bit.

This choice makes things fast, but the reduced resolution sacrifies some compression efficiency. I too did a comparison with MPEG4 Lossless (mpeg4als). If you use the switch -7, you activate a asymmetric compression mode which is very similar to my Compressor or FLAC. But it's about 0.40 percent better on my test corpus than TAC. One reason seems to be the use of 64-Bit arithmetic, which on the other hand is quite slow.

And to lower my credibility just a little bit more: With the SSE3 instruction set found in newer X86-CPU's, Tac could become even faster, because it provides 8 instead of 4 of the necessary calculations simultaniously...

Should i officially confirm my posts when the 1.4. is over?

  Thomas

Yet another lossless audio compressor...

Reply #26
Quote
But it's true.


Ok, I'm sorry. But cut me some slack. You register here out of nowhere right on April 1st, comes up with a completely groudbreaking innovation in lossles encoding, and still asks "Would it make any sense to release it?"

Quote
Should i officially confirm my posts when the 1.4. is over?[a href="index.php?act=findpost&pid=378017"][{POST_SNAPBACK}][/a]


It's over already in Germany, right?

Yet another lossless audio compressor...

Reply #27
Quote
Quote
But it's true.


Ok, I'm sorry. But cut me some slack. You register here out of nowhere right on April 1st, comes up with a completely groudbreaking innovation in lossles encoding, and still asks "Would it make any sense to release it?"

Quote
Should i officially confirm my posts when the 1.4. is over?[a href="index.php?act=findpost&pid=378017"][{POST_SNAPBACK}][/a]


It's over already in Germany, right?
[a href="index.php?act=findpost&pid=378021"][{POST_SNAPBACK}][/a]



Yes. And i confirm, that it isn't a joke!

I didn't rate my work as groundbreaking. I know for example, that the highest compression modes of OptimFrog give up to 2 percent better compression than TAC. Nevertheless i am a bit proud of the speed of my compressor. That's true.

But i understand, that my post could be a bit suspicious. Possibly a successful troll would even present something a bit surprising instead of something hard to believe. Sigh...

But i initially wanted some motivation to force the release of my compressor. And in this context the need to prove, that my compressor isn't Vaporware, is useful.

  Thomas

Yet another lossless audio compressor...

Reply #28
Quote
Yes. And i confirm, that it isn't a joke!

I didn't rate my work as groundbreaking.

Go ahead, and see how well your baby fares.

Yet another lossless audio compressor...

Reply #29
Quote
Building of the compression engine has been very much work. The creation and promotion of a new (free) format, which seems to be necessary to make the technology useful, would be even more work.

I would agree with that... from my experience, development of the actual compression algorithm takes the least part of time for a successful codec.  algorithm-wise FLAC is not that much different than shorten.  the vast majority of time for me was spent in trying to make it useful (format spec, portable libraries, docs, test suites, all the features people want in a codec like a metadata system, support tools, etc.) and external stuff like project administration, releases, ...

Quote
I'm in doubt that i myself would be able to establish some new standard. And i'm not sure, if it would make sense. It's a pity, that i am too late. Some years ago my work possibly would have had a chance to enrich the development of FLAC.

it still could if it's compatible with the FLAC goals; it's not too late.  your table doesn't have the FLAC decoding times to compare against but if you are getting an extra 10% without more decode complexity that is very promising.

Josh

Yet another lossless audio compressor...

Reply #30
This discussion reminds me of that episode of the Simpsons where Bart had that Dad-at-Large and Homer had that kid-at-large and they were competing against each other for some reason. Anyway, after that big fight Homer had with Bart's "bigger-brother", when Bart and Homer got back together again, and right at the end, when the kid (I think his name was Pepe) and the bigger-brother were sitting at the curb lamenting about how crappy things were for them (Pepe had no dad now, and the bigger-brother had all this food that would go to waste) and they didn't realize that they should partner up since the Pepe kid was perfect with the super-good bigger-brother guy. Then Bart introduced them to each other and there was a happy ending. Let me be like Bart here, and encourage TBeck to join up with the FLAC team, and...... Well you get the idea. 

Yet another lossless audio compressor...

Reply #31
Quote
Furthermore my code for reading the source (wave) files is very rudimentary and could possibly fail on some files.


if you like i can send you a c++-class which can read
Mu/Alaw 8,16,24,32-bit wave,wave-extended and aiff files...

Yet another lossless audio compressor...

Reply #32
IHMO, if you could transform your ideas into some methods / a library for other lossless codec, and thus enhancing them, it would be more appreciate by the user than a brand new codec.

Yet another lossless audio compressor...

Reply #33
Hey, that's really something special! I don't really think your codec will survive in an ongoing lossless competition without some extra features and/or promotion, but you could join up with either of the open-source codec developers to try to merge your ideas into one but very effective codec (if that is possible, of course).
Anyway, good luck with your work.
Infrasonic Quartet + Sennheiser HD650 + Microlab Solo 2 mk3. 

Yet another lossless audio compressor...

Reply #34
Quote
Let me be like Bart here, and encourage TBeck to join up with the FLAC team, and..
[a href="index.php?act=findpost&pid=378087"][{POST_SNAPBACK}][/a]


Oh god, I'm afraided it but why FLAC? Don't think that I'm flaming but I just don't like direct biasing.

Quote
I don't really think your codec will survive in an ongoing lossless competition without some extra features and/or promotion
[a href="index.php?act=findpost&pid=378187"][{POST_SNAPBACK}][/a]


Agree, but partly. Lossless compressors are always have been pieces of software mainly for experinced and interested users so even if your compressor will have only one extra feuture - it will have success, although among quite limited group of users.

For TBeck I can only advise the following:

+ Make your compressor open-source to spread it everywhere without limits

+ First of all develope the format. In ideal it must:
    - support any kind of input files. 8/16/24/32 bit/IEEE float, mono/2 channel/multichannel, any sample rate.
    - support non-audio data embeded into files (RIFF chunks)
    - be error resistant. At least error detection and decode-throught-errors, at most the error correction.
    - provide tagging possibility. APEv2 tags are highly recomended.
    - be asymmetric but providing high compression ratios, same or better than Monkey's Audio.
    - be very fast at decoding and to be hardware friendly
    - provide ReplayGain support
    - provide possibility to improve compression ratios without breaking the backward compatibility
    - provide full Unicode support

+ After you developed the format just provide native reference compilations for
    most well known platforms and also plugins for at least 2-3 famous players on
    such platforms. Also support for burning/editing applications will be just great.

Good luck !

Yet another lossless audio compressor...

Reply #35
Quote
I would agree with that... from my experience, development of the actual compression algorithm takes the least part of time for a successful codec.  algorithm-wise FLAC is not that much different than shorten.  the vast majority of time for me was spent in trying to make it useful (format spec, portable libraries, docs, test suites, all the features people want in a codec like a metadata system, support tools, etc.) and external stuff like project administration, releases, ...

Quote
I'm in doubt that i myself would be able to establish some new standard. And i'm not sure, if it would make sense. It's a pity, that i am too late. Some years ago my work possibly would have had a chance to enrich the development of FLAC.

it still could if it's compatible with the FLAC goals; it's not too late.  your table doesn't have the FLAC decoding times to compare against but if you are getting an extra 10% without more decode complexity that is very promising.

Josh
[a href="index.php?act=findpost&pid=378083"][{POST_SNAPBACK}][/a]


Hello Josh,

very glad to read this "It's not too late..."! To be honest, there is an EMail in my draft box, that should have been sent to the FLAC Dev List, but never found its way...

I'm aware of the really big work needed to make a good format. And to be honest, i wouldn't like to do this. Especially as it would be some reinvention of the wheel and so quite useless.

It has been some time since i last carefully read the flac goals, but if there wasn't a big change, there should be no problem.

I didn't provide speed data for flac in my comparison, cause i was a bit in a hurry. But i will add that soon. It should be no problem to reach the speed advantage needed, cause the fastest of my compression modes allready uses 128 Predictors, which could easily be changed to 64 or 32 too get more speed.

If it would be possible to to add my compression methods (possibly as an alternative to keep backwards compatibility) to FLAC, i would be very happy.

But it would take some time and i would need help.

My code is written in Borland delphi (pascal) and nasm.  I', not a wizzard in c, so it will take some time to translate it. And i don't know nearly nothing about programming for platform compatibility. Last but not least, my skills in writing english aren't very good, so a proper documentation would't be easy. Would there be help from the flac-dev-community? Could it be practicable?

Very exited

  Thomas

Yet another lossless audio compressor...

Reply #36
Quote
For TBeck I can only advise the following:
+ Make your compressor open-source to spread it everywhere without limits
+ First of all develope the format. In ideal it must:
    - support any kind of input files. 8/16/24/32 bit/IEEE float, mono/2 channel/
       multichannel, any sample rate.
    - support non-audio data embeded into files (RIFF chunks)
    - be error resistant. At least error detection and decode-throught-errors, at
       most the error correction.
    - provide tagging possibility. APEv2 tags are highly recomended.
    - be asymmetric but providing high compression ratios, same or better than
      Monkey's Audio.
    - be very fast at decoding and to be hardware friendly
    - provide ReplayGain support
    - provide possibility to improve compression ratios without breaking the
      backward compatibility
+ After you developed the format just provide native reference compilations for
    most well known platforms and also plugins for at least 2-3 famous players on
    such platforms. Also support for burning/editing applications will be just great.

Good luck !
[a href="index.php?act=findpost&pid=378215"][{POST_SNAPBACK}][/a]

One more thing: Please make it Unicode-based, or at least support filenames in multibyte chars. That's important for i18n, ie for the whole world includking China Japan Korea to enjoy your app!!

wavpack and ttaenc didn't like multibyte-char filenames before (the problem was already fixed now tho) and was not able to compress the file when the filename contains a double-byte char whose 2nd half is 0x5C, misunderstanding it as \

I'm looking forward to the Alpha release of this!

Yet another lossless audio compressor...

Reply #37
Quote
it still could if it's compatible with the FLAC goals; it's not too late.  your table doesn't have the FLAC decoding times to compare against but if you are getting an extra 10% without more decode complexity that is very promising.

Josh
[a href="index.php?act=findpost&pid=378083"][{POST_SNAPBACK}][/a]



Here some more comparison now including Timings for FLAC.
Cause i had to use a stop watch for flac, i only tested the
biggest files to reduce the effect of the measuring equipment
(my thumb isn't too accurate...).

The first two columns now include some variations of the HIGH-Mode
with reduced predictor numbers. They could compress better without
increasing decoding time, if they used the same methods for paramter
calculation as Extra or Insane modes. I just wasn't to lazy.

The bottom rows show the speed, that TAK achieves without the use
of MMX-Instructions or general assembler optimizations. This could be
interesting, if the code would be ported to some platform without such
instructions.

Code: [Select]
            TAK     TAK     TAK                       FLAC
Mode:       High    High    High    Extra   Insane |  -8     |
Predictors: 32      64      128     256     384    |         |
---------------------------------------------------+---------+
Song_02     48,96   48,69   48.41   47.87   47.73  |  51.03  |
Song_04     34,19   33,84   33.15   32.59   32.56  |  37.27  |
Song_06     34,30   34,04   33.74   33.34   33.20  |  37.04  |
Song_08     45,74   45,31   44.97   44.56   44.45  |  49.74  |
Song_10     56,84   56,71   56.41   56.00   55.94  |  59.10  |
Song_12     54,13   53,93   53.86   53.33   53.27  |  57.62  |
Song_14     49,14   49,07   48.97   48.51   48.44  |  51.87  |
Song_16     74,17   74,16   74.16   73.82   73.79  |  75.95  |
---------------------------------------------------+---------+
Sum:        48,40   48,15   47.86   47.41   47.33  |  51.35  |
---------------------------------------------------+---------+
Times with the use of MMX:                         |         |
EncoTime:   41,59   45,14   53.01  270.94  595.41  | 191.00  |
DecoTime:   11,35   12,24   13.50   14.90   15.19  |  20.00  |
---------------------------------------------------+---------+
Times without the use of MMX:                      |         |
EncoTime:   63,08   73,87   91,65  638,81 1350,89  | ---.--  |
DecoTime:   16,40   19,46   24,77   31,46   33,38  | ---.--  |
---------------------------------------------------+---------+


To be honest, the comparison of the three leftmost columns of TAK with FLAC isn't
quite fair, because TAK actually doesn't measure the time needed for Disk-IO (my
40 GB Disk is quite slow). This shouldn't significantly affect the validity of
the other two modes, were the calculation overhead is much higher.

Possibly interesting to see, that the MMX-Implementation of the Modes with
low predictor order isn't considerably faster than the implementation without.
That's caused by some overhead introduced by the scaling and other preparations
of the data needed for the use of MMX.

  Thomas

Yet another lossless audio compressor...

Reply #38
Is there any way we can get a sourcecode release of this to examine / integrate in other projects?

Yet another lossless audio compressor...

Reply #39
If I understand your table correctly, you are saying that your compressor (using the 32 predictors) compresses about 3% better than FLAC (-8) and does this with significantly faster encode and decode speeds. And it does this using pure Pascal with no optimized assembly.

Assuming this is correct, I think that your best (and easiest) course of action would simply be to publish a paper on the superb method(s) that you have discovered, become famous, and let others bother with all the unpleasant implemention details. It would obviously be a shame to let this go to waste, especially because the magnitude of the method's improvement (0.5 bits per sample!) could have ramifications across many disciplines.

However, I would stongly suggest that you fix that random bit error issue before getting too excited. It might be that those error bits actually need to be transmitted in the bitstream, wiping out your advantage in the process!

Congratulations!

Yet another lossless audio compressor...

Reply #40
Well, I am currently performing a Lossless Compression Test (described in this thread).

My test system relies on scripts and the computer's timers so there's no thumb-accuracy issue

If you can possibly send me a link (via PM if you prefer) to download your binary then I will objectively contest your algorithm against some others I am testing... my test covers both encoding and decoding tests.

Yet another lossless audio compressor...

Reply #41
Quote
If I understand your table correctly, you are saying that your compressor (using the 32 predictors) compresses about 3% better than FLAC (-8) and does this with significantly faster encode and decode speeds. And it does this using pure Pascal with no optimized assembly.

Assuming this is correct, I think that your best (and easiest) course of action would simply be to publish a paper on the superb method(s) that you have discovered, become famous, and let others bother with all the unpleasant implemention details. It would obviously be a shame to let this go to waste, especially because the magnitude of the method's improvement (0.5 bits per sample!) could have ramifications across many disciplines.

However, I would stongly suggest that you fix that random bit error issue before getting too excited. It might be that those error bits actually need to be transmitted in the bitstream, wiping out your advantage in the process!

Congratulations!
[a href="index.php?act=findpost&pid=378259"][{POST_SNAPBACK}][/a]


Many Thanks!

You are totally right. In the past many revolutionary improvements of my compression efficiency were really bugs. But i did fix the error last night. The maximum reduction in compression efficiency after the fix is 0.01 percent.

I'm not quite sure if this would be the right place to release details of my private life...But...

I'm working self employed (mainly developing software for psychological research) and this is actually not going very well.

I'm suffering from the fact, that i don't have a formal education in informatics and not many references.

So i would like to get some profit out of the publication of my compressor in form of some reputation. I don't know, what could be the best way to achieve this goal.

  Thomas

Yet another lossless audio compressor...

Reply #42
Quote
If I understand your table correctly, you are saying that your compressor (using the 32 predictors) compresses about 3% better than FLAC (-8) and does this with significantly faster encode and decode speeds. And it does this using pure Pascal with no optimized assembly.

Assuming this is correct, I think that your best (and easiest) course of action would simply be to publish a paper on the superb method(s) that you have discovered, become famous, and let others bother with all the unpleasant implemention details. It would obviously be a shame to let this go to waste, especially because the magnitude of the method's improvement (0.5 bits per sample!) could have ramifications across many disciplines.

However, I would stongly suggest that you fix that random bit error issue before getting too excited. It might be that those error bits actually need to be transmitted in the bitstream, wiping out your advantage in the process!

Congratulations!
[a href="index.php?act=findpost&pid=378259"][{POST_SNAPBACK}][/a]


One more:

You read the table right. The reduction of  the calculation accuracy to 13 bit, that was necessary for the MMX-implementation, helps the pascal implemenation too, because the accumulator for the results of the many predictor * sample multiplications only has to be 32 bit wide instead of the 64 bit needed for many other compressors. So the compiler can use the faster x * y = eax instead of x * y = edx:eax instructions. And the use of this instruction allows pipelining (CPU) of the multiplications which isn't possible wit a 64 bit result. So on the pentium 3 you can get a troughput of 1 multiplication per clock cycle with 32 bit instead of 4 clock cycles per every single multiplication for 64 bit.

The higher compression efficiency of my compressor is based on many new or optimized methods. It's the sum of single improvements from about 0.1 to 0.5 percent each.

  Thomas

Yet another lossless audio compressor...

Reply #43
I'm sure many people will wish to donate for using your compressor.  Also, you could always make commercial licenses to the product / codec need to be paid.

Yet another lossless audio compressor...

Reply #44
Quote
So i would like to get some profit out of the publication of my compressor in form of some reputation. I don't know, what could be the best way to achieve this goal.
[a href="index.php?act=findpost&pid=378266"][{POST_SNAPBACK}][/a]

Well, again, a paper would probably be the best path for academic recognition. If you don't feel comfortable writing it yourself, you might team up with someone else in the field like Tilman Liebchen who is right there in Germany. I suspect that he would be interested in talking to you about your methods.

Of course, it would always be a good idea to quickly write them all out in letter form and mail it to yourself, just to be protected (there's probably a more modern way to do that, but you get the idea.)

Yet another lossless audio compressor...

Reply #45
Possibly time for some summary. Especially because it seems impracticable to anwer to every single post.

First thanks for the many encouraging words! They have provided me with some new motivation needed to continue and actually force the progress of my work.

I'm not quite sure what the future will bring. Actually my favourite option would be the inclusion of my technology into an existing free project like FLAC (if possible). But this will take a considerable amount of time (cleaning up and translation of my code to C, writing some documentation).

But the first thing on my priority list is the release of some evaluation version without any gimmicks and without any guarantee for further support of this first incarnation of the format.

I'm feeeling more and more bad because i provide promising results without the public prove of function. So this thing has to be done fast.

And i will enable my email-adress for this forum, if someone wants to send me some personal mail.

It includes the adress of my home page. But be warned. It is only written in german and doesn't contain any audio compression software yet.

Don't take me wrong. This post schould not stop this discussion! I would be really glad to receive further encouragement or constructive critics! This post is only a summary of my present thinking.

  Thomas

Yet another lossless audio compressor...

Reply #46
Quote
Quote
it still could if it's compatible with the FLAC goals; it's not too late.  your table doesn't have the FLAC decoding times to compare against but if you are getting an extra 10% without more decode complexity that is very promising.

Here some more comparison now including Timings for FLAC.
Cause i had to use a stop watch for flac, i only tested the
biggest files to reduce the effect of the measuring equipment
(my thumb isn't too accurate...).

Thomas, this definitely looks promising.  the design of FLAC allows new methods to be added while preserving backwards compatibility.  the main reasons I haven't added any so far is because they all either 1) had too little compresion gain; 2) significantly increased decode complexity; 3) were patent encumbered.

it's hard to say in this crazy world but assuming you're method doesn't run afoul of some patent (will take some research), you seem to have overcome all 3.  there are other methods I have not added to FLAC just because the compression increase wasn't worth it, since the code is running in many devices and it would be some work to get it out to all the manufactures.  but a consistent +10% would be worth it; at the same time I could add all the little stuff I haven't put in yet.

as you know FLAC and libFLAC are meant to be as open and free as possible so if you are OK with that, there is no reason your method cannot be incorporated.  (if you're looking to make some economic benefit from it, that's much harder.  the only I options see for that are to join a giant like dolby or free it and use any success of it as a reference.)

the next step would be to go over the details of your method.  that could be done here or on the flac -dev list where more FLAC devs would see it.                                                                                                                                                                                                       

Josh

Yet another lossless audio compressor...

Reply #47
Quote
Thomas, this definitely looks promising.  the design of FLAC allows new methods to be added while preserving backwards compatibility.  the main reasons I haven't added any so far is because they all either 1) had too little compresion gain; 2) significantly increased decode complexity; 3) were patent encumbered.

it's hard to say in this crazy world but assuming you're method doesn't run afoul of some patent (will take some research), you seem to have overcome all 3.  there are other methods I have not added to FLAC just because the compression increase wasn't worth it, since the code is running in many devices and it would be some work to get it out to all the manufactures.  but a consistent +10% would be worth it; at the same time I could add all the little stuff I haven't put in yet.

as you know FLAC and libFLAC are meant to be as open and free as possible so if you are OK with that, there is no reason your method cannot be incorporated.  (if you're looking to make some economic benefit from it, that's much harder.  the only I options see for that are to join a giant like dolby or free it and use any success of it as a reference.)

the next step would be to go over the details of your method.  that could be done here or on the flac -dev list where more FLAC devs would see it.                                                                                                                                                                                                        

Josh
[a href="index.php?act=findpost&pid=378368"][{POST_SNAPBACK}][/a]


Good News!

The patent thing is a really crazy one! To my knowledge there shouldn't be any problems with my Methods. But who knows.

As i wrote earlier, the improvements are mainly a sum of many optimizations and tricks. So even if there would be a need to remove one or two of them because of patent issues, there should still be a considerable advantage left.

I did try to avoid any patent issues. So i played a bit around with arithmetic encoding, which gave me about 0.9 percent better compression than standard rice. Then i dropped this thing and generated a variation of rice encoding, which gave me at least 0.6 percent more than standard rice. But may be my variation is patented, i don't know for sure.

Actually i'm working hard on a evaluation release. It could be done very soon, cause i dropped my earlier plan to first complete a really usable format.

After this is done, i would like to join the flac-dev-list.

  Thomas

Yet another lossless audio compressor...

Reply #48
If nothing unexpected happens, i will release an evaluation version of my compressor within the next days. It will be a very quick and dirty thing. No error handling, only support for 44 KHz, 16 Bit Stereo and an ugly user interface.

To be honest, i never before did write compressed files to nor did i decompress files from the disk. All this had be done in RAM. So there has been a very little chance, that i faked myself about the compression efficiency. But after some strange errors and much  excitement i could compare the first files and everthing is ok!

Would it be possible to store the program archive (less than 1 MB) at hydrogen audio? My free homepage traffic is very limited, so i would be glad about an alternative.

Very exhausted

But happy

  Thomas

 

Yet another lossless audio compressor...

Reply #49
If none of the moderators object, you can use the Uploads section of HA and give a link to the archive in a post here.

This development is really interesting. Personally, I think a collaboration will benefit both the community and yourself more than doing a one man show. If (for example) FLAC can benefit from your findings, you already have a well-known and quite established format to put in your CV.