Was just ripping away the other day and suddenly the idea stuck me!
Wouldn’t that be great if there was an encoder out there that would provide us with an option to save the discarded psychoacoustics information that gets stripped away with each MP3 to a file!

I have no idea what the format of this data could be saved in and don't have the faintest clue either but in my mind I think there must be a way that this could be done.
On a technical level (at least) it would make sense that if an encoder knows which information to strip away during the encoding process, IT SHOULD also be able to save this information somehow somewhere?
Now... given the fact that a great encoders (such as lame) produces resulting mp3’s close to dam near cd quality, I don't imagine these extra files would be very big in size but this is where I may be wrong?

The immediate benefit of course would be that we could then reconstruct the original file to its original state (as it was prior to encoding) by combining the resulting mp3 file with the extra file produced by the encoder.
Unless I'm totally off here, we would have ourselves a lossles process then wouldn’t we?

We could also easily store these extra files to a CD-ROM or wherever and bring them back when we needed them, such as when wanting to re-encode a track to a higher bitrate or converting to a different file format etc.

The challenge I see would be to prevent mismatched files from being joined together accidentally, so maybe creating files with same filename as the encoded mp3's would help here?

Just a shot in the dark maybe but still thought it was worth mentioning on this great forum.

For all I know such an encoder might even already exist??

Please feel free to comment!

Cheers!

WavPack hybrid mode and Optimfrog dualstream do what you describe. But the format is no longer mp3. You can manually do what you describe by creating a mp3 and then the difference file and losslessly encoding the difference file. But I don't know if there's a player extension to play this mp3+difference file.

No. Unfortunately, they are big. The total size (the lossy file + the difference file) is slightly larger than what you would obtain via lossless compression.

And actually I don't know why people prefer this over storing a lossy encoded file + a lossless encoded file. Yes the total space consumption would be slightly larger but both of these files would be listenable. Otherwise the difference file is not (you need to locate the lossy file in order to make sense of it). Is the additional space gain big enough to compensate for this seemingly huge inconvenience?

See this thread.

I see... I've briefly browsed over to their sites and unless I'm wrong, these are not mp3 encoders are they?
I still think it would be neat if a great encoder such as lame would provide this option.

Ok this is where I don't get it...
The resulting mp3 is so close to the original wav file, why does the difference file have to be so big??
We just want to preserve the tiny bit of data that was stripped away from the file.
From what I read, it's content that we can hardly hear to start with so I just don't get this part.

Just my 2¢

The MP3 is perceptually close (or even transparent) to the source file, but it is not very similar mathematically. For the purpose of encoding, all that matters is the mathematical relationship between the source and the MP3.

E: Spelling

Compressing a stereo audio track from a standard CD to MP3 @ 128kbps "strips away" over 90% of the original audio data. Compressing it to MP3 @ 320kbps strips away over 75% of the audio data. That's why the difference file would be large, even when losslessly compressed.

think about it like this

lest say you have a wave file 50Mb. If this file was converted to a lossless compressed format you may get a file about 35Mb.

Now if the wave or lossless file were converted to mp3 you could get an mp3 about 5Mb. So the mp3 file stripped away 30Mb or 45Mb. Therefore the stripped information would have to be 30Mb lossless or 45Mb wave!

e.g. 5Mb mp3 + 30Mb lossless stored stripped info = 35Mb original file (lossless)

Unfortunately it is not that simple. You can't just say:

size(lossless) = size(MP3) + size(residu)

The residu that you get after converting a lossless/wav file to MP3 is much harder to encode for a lossless algorithm than the original file. So in general it will look like this:

size(lossless) < size(MP3) + size(residu)

ok maybe i should have said

original(wave) = mp3 + removed(wave)

just taking lossless out of the equation to avoid confusion.

lossy encoders do not remove something, they add narrow band quantisation noise at frequencies where it should be masked by the musical content.
So the difference file would contain blubbering noisy sounds which are hard to encode losslessly.

I don't know if this is right on topic, but you can make a diff file by encoding a wav to mp3, then decode it back to wav. Then you load both the decoded and the original wav into Cool Edit. Select and copy the first file, then change focus to the second file and do Mix Paste:100%:invert.

Such a MP3 + correction file still wouldn't be lossless for everybody because everybody uses different MP3 decoders and different decoders give slightly different output.

I tested with a 20 second clip.

original WAV file: 3205 KiB
lame 3.96 --preset extreme (mp3/apx) : 581 KiB
lame 3.96 --preset 128 (mp3/128): 276 kIb
Difference file (apx): 3205 KiB
Flac (original file) : 2170 KiB
Flac (difference file apx) : 1682 KiB
Flac (difference file 128) : 1949 KiB

So:
Flac (original file) : 2170 KiB
Flac (difference file apx) + mp3/apx 1682 + 581 = 2261 KiB
Flac (difference file 128) + mp3/128 1949 + 276 = 2225 KiB

As you can see, compressing mp3 PLUS the difference, results in two files being a little bigger than compressing the original file.

(Tested only with 1 clip of 20 seconds)

that does not make sence!

ok lets look at audio using binary as everything in computing is based on binary.

lets for a moment assume the following binary is a wave file.

original wave = 101001000011110100101010010011010101001010

i understand that its way too small to be a wave file but lets just say it is for a moment!

after compression lets assume the following is the mp3 binary

compressed mp3 = 1010010

this should then leave you with the following leftover wave binary

leftover or disgarded wave information = 00011110100101010010011010101001010

why is it not possible to add the mp3 binary with the disgarded wave to recreate the original file again? like so

1010010 + 00011110100101010010011010101001010

i guess you would need some extra information in the disgarded information so it knows where the information was disgarded from from the original file.

That is not possible, because data is not "discarded" as such... you make a new audio stream based on the original stream.

So if you have an original wav:
01001001101110100010

and the resulting MP3 as:
100110

What data would you save then? There is just no relation. Storing the original audio stream in a lossless format is the only option.

The problem lies with your conception of "tiny" when it comes to audio. Very simply put: the difference WAV file is low in volume (not audible when the rest of the music is present), however this does not make it any easier to encode. The difference file is actually very close to a random stream of bits which makes it hard to encode (because there's no inherent pattern): To encode an almost random stream of bits of size N losslessly you have to pay about N bits.

Hmmm...just wondering...do it isn't feasible to create special kind of encoder/format just for storing these discarded information? (yeah, it laso would be lossy...but I wonder what results would it give; I wouldn't be surpised if: geberally a little better, but with horrible artifacts here and there)

Anyway, what for when we have Musepack...

There are several formats doing this like Wavpack, but there is already a defined standard using the "from lossy to lossless" philosophy, mosly known as scalable. The most promising one is SLS, the MPEG Scalable Lossless Coding.

SLS uses AAC technology to encode the base layer at 128Kbps. The base layer is lossy but perceptually transparent. SLS is backward compatible with standard AAC and it can be played in AAC enabled devices. The enhancement layer uses IntMDCT to achieve lossless compression at 600 to 700Kbps for two-channel audio. It does not give any remarcable improvement in compression but it is a big improvement in compatibility.

The big disadvantage is that this technology is the same used in lossless WMA, it comes from FhG and its new pal, Microsoft.

Regards,
Oki

IOW a codec's idea is to reduce redundancy, filter out inaudibles, and quantize information. So if we take the "difference file" it's going to be essentially random noise, without much chance for compression (and almost zero chance of compression beyond the average lossless codec's ability).
It'd be easier just to use FLAC in the beginning, and save much unnecessary hassle.

I do not agree with that, the enhancement data is more than random noise, a lot more.

PCM audio data is composed by Perceptible signal + non perceptible signal. Each of those signals is composed by entropy (valuable information including the noise) + redundancy (somehow duplicated information).

The perfect lossless audio codec would keep the entropy of the data (entropy of the perceptible signal+entropy of the non perceptible signal) removing only the redundancy.

The perfect lossy audio codec would keep only the entropy of the perceptible signal, removing the whole non perceptible signal and the entropy of the perceptible signal.

The difference between the perfect lossless codec and the perfect lossy audio codec is the entropy of the non perceptible signal, and that info is a big amount of valuable info, about 75% as Nero said in this thread. 75% of a HQ signal it not only noise but also masked sounds, low volume but with a lot of redundancy and insteresting charasteristics such that SLS can achieve lossless compresion of the non perceptible signal with a ratio of 4 to 1. Pure noise would not allow any compression at all and the final lossless is achieved with a ratio of 2.5 to 1, less than compressing only the non perceived signal. Interesting, isn't it? I will not go deeper into this subject since it would be a veeery long dissertation.

@ Megar

Your test is giving us additional info if you analyse the data and the experiment setup. By doing so you can explain why you obtained those values.

1st. MP3 lacks from a good entropy filter so you are not efficiently removing the redundancy of the perceived signal.
2nd. MP3 decoding adds the non perceptible quatization noise under the perceived signal, adding an undesired non perceived entropy to the signal. IOW: you add entropy from the MP3 decoding to the enhancement layer so, as SunHillow suggested, forget about using the Cooledit sustractive method as a valid step.
3rd. FLAC is not a good codec for the enhancement layer since it does not take into account the caracteristics of the non perceptible signal, it means that it is not good enough in removing redundancy either.

You conclude that MP3 + FLAC > Only FLAC since you are adding noise and you are keeping duplicated redundancy in both parts. A better ilustrative codec can be the SLS, where lossy + lossless complement ~= lossless.

Remember the 75% value that Nero said. This value is realistic since the MPEG SLS encoding uses a AAC @ 128Kbps (25% of the total valuable data) as the lossy core codec, very good for this purpose since it has an entropy filter. The enhancement layer achieves lossless compression adding 472 to 572Kbps (75% of the total valuable data), filling this layer with the entropy of the non perceptible signal with an algorithm designed for removing the redundancy of the non perceptible audio signal. The final size is about the same than a lossless only codec like ALS.

Regards,
Oki

Edit1: Finishing the initial post
Edit2: Conclusions about Megar's test

Don't mean to block your enthusiasm, PTJB, but this idea has been coined numerous times already. Like in this thread, to mention just one.

for the love of the messiah people, do we even search or bother to look for related topics? SLS is neato it uses bitsliced arithmic coding. I skimmed through the paper.

http://www.hydrogenaudio.org/forums/index....34&hl=HotshotGG



The point I made in the thread above .



Yes thank you .



From that thread from last year that I never came across that just wouldn't be possible, however if somebody is able to fully understand how the VQ backend works something like that might be possible, but only for lossy content. Not only would you have the ability to "peel" down bits by dumping the residue you could also, keep the residue in seperate file possible? The problem though is that the encoder is not set up correctly right now to handle that kind of thing (packets aren't ordered correctly). FLAC will always be lossless and that won't change.

Thanks for all your comments guys!
Sorry for not replying any sooner, some of these posts are so technical I had to read them many times over to try to grasp some of their contents.

I kinda see the bigger picture now though.
Guess I just have to keep encoding my albums using lame APS for now!
I do like the mp3 format myself mainly because of its versatility and hardware compatibility, not to mention compatibility with friends and family as well.
I guess I could say that I’m not quite ready for a format makeover just yet.

I realize there are many decent lossless contenders out there but it would simply be too huge of a task for me to start re-ripping all of my CD's from scratch again since I’ve roughly gotten over 1000+ done so far.

Thanks again!

For your example to work MP3 would have to be a very simple dither system based on simply truncating bits from PCM samples. It's not.