Hello,

It took a little longer than expected to release this experimental version, because it kept getting better and better... Now, after some in-depth testing, I decided to release it as it is, although other compression improvements are in the works

What is new in this version (4.600ex):

• the highnew, extranew, bestnew modes and --maximumcompression get slightly better compression (on average 0.1%)
• added --experimental option, enabling advanced experimental compression, which is NOT backward compatible with any previous 4.5xx versions
• use --experimental with any mode for greatly improved compression for some files (on average, 0.60% better compression for Audio CDs)
• extranew mode speed change when using --experimental: encoding < 35% slower, decoding < 11% slower
• --experimental can be also used together with --maximumcompression
• [other] the foobar2000 input plug-in now supports cue sheets (thanks to Artur for suggestions and testing)
• [other] official GUI interface using FroG is ready, it will be included in the upcoming OptimFROG installer (thanks to Daniel, the FroG author; available at http://frog.objective-view.de/ )
• [other] parsing WAV files with invalid headers is in the works

You can get the new experimental version and the foobar2000 input plug-in with cue sheet support from the OptimFROG site, on the Downloads page, at

http://www.LosslessAudio.org/

Please let me know if you find any problems or potential bugs with this experimental version...

Thank you,
Florin Ghido

sounds great!

do you have a plan for the linux releases?

^ ... or the Mac OS X (Darwin) binary?

Uh, Ghido, by "experimental" you don't mean alpha-code that will either (1) cause a BSOD or (2) mangle the compressed waveform, right?

I'm a bit confused now with the command line switches. So:

1. Which switch give the best compression, ratio-wise?
2. Which switch give the best tradeoff between compression & speed?
3. If I want slightly better compression than #2, how do I tweak it?

Thanks for your reply. I like OptimFrog, in fact one of the <10 OptimFrog user in the latest Lossless poll

Edit: Stupid Typo™

No multichannel yet? :/

@xmixahlx, @krmathis: I will build the linux x86 + amd64 and Darwin binaries this weekend. The reason is that I hope to receive feedback until then to include fixes for any small unseen issues, because I don't have linux/amd64 and Darwin/ppc and I use some very slow emulators, therefore building and packaging takes a while.



About OptimFROG developement stages:
- experimental means that the program is tested on my audio corpus, but lacks extended testing. Practically, the most important issue is that there is no guarantee of compatibility with future versions, because it is possible that there will be format changes or improvements until reaching the beta stage (regarding 4.600ex, I have already in work some modified version which should give better compression; while in theory it should not break compatibility with existing experimental files, I cannot be sure until it is really done, so that is why it is marked experimental)
- beta means that the source code is carefully reviewed (including formal analysis of the possible cases for some algorithms), passes torture tests, passes valgrind tests, and behaves well on compression results regression on my 52 GB audio corpus.
- release is almost as beta, but incorporates user feedback and usability issues (mostly commandline and interface related)


The general idea is that --experimental works on top of all existing modes and switches, to give improved compression for them (think about it as an alternate advanced packer for prediction residuals). If you do not use --experimental, the output is backwards compatible, but there are still some small improvements in compression for the *new modes and --maximumcompression.

1#) So, the maximum possible compression would be --maximumcompression --experimental.
2#) In my opinion, the best tradeoff between compression and speed is --mode extranew --experimental.
3#) To slightly increase the compression, without affecting the decoding time, you can use --mode extranew --optimize best --experimental.

For most files, --experimental improvements are small, but for some files, mostly classical music, the improvements can be quite large, up to 12% . The good part is that --experimental works with all existing modes and switches, giving similar improvements, and the decoding time is not increased significantly.

If you really want to play, you can use additionally for all the *new modes and --maximumcompression (especially with --optimize best) an undocumented option, which makes encoding a hog, but strips a few extra kilobytes per song.


Glad you asked . Multichannel is approaching completion, but due to lack of samples, and apparently lack of interest from the users it was not so motivating working on it. It would be *really great* if anyone could send me multichannel samples (you can easily use http://www.losslessaudio.org/SendFile.php, for samples up to 200-300 MB in size it works just great).

Florin

On an electronic rock track : 4600 normal mode = 892 k , normal + experim = 812 k !!

However, I can't play anything encoded with --experimental - I get a blank audio file. Is that normal ?

Nice improvement...

For playing in any supported audio player you have to replace your existing OptimFROG.dll (from player's directory or from windows\system32) with the newly supplied OptimFROG.dll from 4.600ex version. To be sure, the best is to do a search on your HDD for OptimFROG.dll, and see what copies are relevant.

Florin

That worked thanks.

Two other things:

1) When will it be possible to decode ofs+ofc (lossless) with player plugins ?

2) Is it possible to fix OFS.EXE to make correction files through EAC correctly ?

Is there a switch to use when piping input into the optimfrog encoder? Piping seems to work using the foobar2000 converter but the resulting file has quite some issues (using a 4:15 long sample song):

1) Bitrate reported is 21 kbps
2) reported duration is 3:22:53.943 (quite off)
3) the file begins to play fine (though the slider doesn't visibly move because of the wrong duration) but it doesn't play till the end of the song (it stops before the end is reached)

The commandline used:
--encode --mode extranew --experimental --timestamp - --output %d

This is a known issue of foobar2000 generating invalid WAV headers (but there are good reasons for it doing so) when using pipes. [3:22:53.943 == exactly a 4 GB WAV file]. You must replace
--encode --mode extranew --experimental --timestamp - --output %d
with
--encode --mode extranew --experimental --timestamp %s --output %d

to force using a temporary file during conversion, and produce correct WAV headers. OptimFROG fully supports pipes, but support for parsing invalid WAV headers is not yet available (but can be emulated using raw mode switches).

So, another workaround would be, if you use only CD Audio (raw mode defaults to CD Audio), to use the commandline
--encode --mode extranew --experimental --timestamp --raw --headersize 44 - --output %d

Florin

Thank you for your answer! I know that there are no problems when I use a temporary wav file but I'd like the encoder to work with piped input like wavpack.

Using " --raw" seems to do the trick, at least the resulting file plays fine and all properties make sense. I dropped the "--timestamp" switch because I don't think that it makes much sense under these circumstances.

Just one question: What does "--headersize 44" do?

It tells OptimFROG that first 44 bytes of piped data are headers (WAV headers are 44 bytes long). If you remove that switch, these 44 bytes will be treated as audio data (so you'll have about 1ms long click at the begining of every encoded file).

Ok, I see...
I just thought you wanted feedback from as many testers as possible, and hence limiting it to MS Windows *only* is not the best solution.

I will keep an eye on your website, cause the new --experimental mode and GUI sounds interesting.

From my short and not so complete evaluation, I would summarize new version behaviour in the following way:

Non "-...new" modes have not changed in the 4.600 version. Applying the "--experimental" switch to those modes is possible, but the speed penalty is too heavy to be considered worth.

On the other hand "-...new" modes improves for about a 0.1% in 4.600 when not using that "--experimental" switch, but I've seen improvements for a 0.6% on a particular set of files. A speed penalty is included.

Finally "-...new" modes with "--experimental" switch ON show improvements around 0.1% on most files, but huge improvements (like a full 3% or more) on others. On the sample set I used, 3 files over 26 showed such a behaviour, so that average compression ratio on the whole sample set improved for around a 0.6%. A reasonable speed penalty exists but, in my opinion, could be considered acceptable if a 0.6% improvement will be confirmed.

If that all will be confirmed, it could be safe to say that all 4.600 "-...new --experimental" modes can be considered "better" of all 4,520b1 modes. That could possibly put an end to the very close challenge between OFR and LA for higher compression ratios achieved (which so far, in my opinion, was strongly depending on the material used for the evaluation).

I have 130gb OFR files here, thanks for keeping on improving OptimFrog.

Glad to see that development goes further! Especialy I'm very pleased to see that compression have been improved again. Thanks for this release!


The --maximumcompression option is the alias for
--mode ultranew --optimize best --seek slow (0.350x on my AMD64 Venice 3000+)
so you can tweak it further by changing to
--mode ultranew --optimize best --seek min --experimental (0.348x on my AMD64 Venice 3000+)
Unfortunately, the usage of --speedup 1x option have been depricated in this exp-release (and probably thats partly my fault) so previously it was possible to add it also. In some (for me in most) cases it provided tiny compression improvement.
As last solution you can add realy useless (regarding the time), absolutely unrecommended
and undocumented option: --uselessoptimization
So for current release of OptimFROG the best ratio-wise combination is:
--mode ultranew --optimize best --seek min --experimental --uselessoptimization
Such combination is slow as hell. Only 0.048x (zero dot zero four eight realtime!) on my AMD64 Venice 3000+ so it will take about 24 hours to compress a 70 min CD.
Files compressed with such undoc option (without --experimental) are safe regarding the compatibility and seems that this option provides boost of asymmetrical behaviour (like --optimize modes) because the decompression rate
for same CPU is 1.624x which is the same as: --mode ultranew + experimental + <any optimize mode>
Also. This option works with *new modes only and can provide worse results in some cases (I suppose with --optimize fast). At least its happened for one of my sample when being added to --mode extranew --optimize fast. I realy don't recommend to use this option.

@xmixahlx, @krmathis: the linux/x64, linux/amd64, Darwin/ppc versions are now available for download...



Indeed, for the non *new modes, the speed penalty due to --experimental is not negligible, especially if you are looking for speed.

However, I found a way to virtually eliminate this penalty for decoding, and also speed up encoding, with a compression loss of at most 0.04%, without breaking any compatibility This implies that in the next version this will be the default for all the non *new modes. Thanks for the opinions and support...


Thanks for all the details and for the enthusiasm! And congratulations for finding the undocumented option...
As you know, the --maximumcompression --experimental command line (maximum possible compression, regardless of time) is equivalent with
--mode ultranew --optimize best --seek slow --experimental
However, the suggested command line
--mode ultranew --optimize best --seek min --experimental
would work actually worse on average, because --experimental option may loose some improvement opportunities due to very large blocks (--seek min), and also seeking speed is impeded (you have to decode for a seek, on average, half of a block). Of course, IF you manually run both variants on a file, and choose the best it is no problem at all. But, in general, the first variant (i.e. --maximumcompression --experimental) should work better on average.

The --uselessoptimization option indeed exists , and is the undocumented, unrecommended, useless tweak for the *new modes I was talking about. It is important to know that, during encoding, all the --optimize fast/normal/high options use just estimations for parameter optimizations (using 0.1, 0.25, 0.5 of the samples in a block), and only --optimize best option use deterministic parameter optimizations (that is, it guarantees to minimize the size). The --uselessoptimization option depends on the --optimize setting, so IF you decide to play with it, it is recommended to use --optimize best (deterministic), or at least --optimize high. But don't forget that the encoding time increase is huge, and the gains are only a few kB per file...

Florin

Ahha... thanks all for the extreme compression tips. Those answer my #1 and #3 questions.

Now my 2nd question: What is the recommended switches with "acceptable" tradeoff between compression and speed? Of course, the definition of "acceptable" differs between everybody, so I am also intersted with your reasoning.

Thanks!

Thanks a lot!
I am downloading as we speak, and will give it a go...


Edit:
It seems to work fine, at least from my minimal test so far (encoded 4 tracks in 3 different modes).
The "--maximumcompression --experimental" is damn slow on my 1.5GHz PowerBook G4 though!

Test performed with the 29 second sample from here.

No. Thanks to you!


Aha. Truly speaking I thought that --seek <parameter> just controls the amount of so called "seek info" which added to file but seems that I was completely wrong and --seek naturaly controls the size of the blocks. Big gratitude for pointing it out.
By the way, can you tell what values are used by --seek parameters?


Thats surely very arguable and requires some serious investigations but since you tested it on your large corpus then why not to beleive you?


AHA ! One more respect to you for dropping the light on it!

Not necessarily. A WAV header can very well be longer than 44 bytes. It's not very common, but it's perfectly valid.

Yeah, but for PCM it's 44 bytes in 99% of cases, and always when piped from fb2k, right?

Little test I did:
http://uclc.info/LossLess.xls

On this sample LA is still slightly better:
32.885.561 LA -high
32.901.662 OptimFROG --maximumcompression --experimental

I missed this first time... good gawd! 130GB of OFR!!!

*faints imagining the time needed to encode all that audio*