I was inspired by TBeck now that he's been testing a new lossless codec.

Does anyone remember the oldskool mods, where the files had samples within them, and the artist created patterns of notes with them that repeated... you cold make a high-quality music file that was hours long (if you wanted to) that was well under a megabyte.

I was wondering if it would be possible to create a special compression routine, particularily optimized for very repetitive music like dance music (where you hear the same beat, and the same pattern repeated numerous times). You could simply recreate the same pattern over and over again, and perhaps small differences (like where you hear the same pattern, but the second time, a new instrument is added) could be added without the cost of the initial pattern or loop.

I'm guessing that most encoding schemes don't operate with this kind of range in mind... they probably wouldn't try to reconstruct the entire song all at once.

Note that I'm rather ignorant about the intricacies of codec design, but from a purely theoretical perspective, is something like this viable?

It's not quite possible to do MOD-alike compression of real music - you would need to extract all the samples, determine the DSPs used, etc. It's often impossible to even extract vocals from music, so I would forget about lossless compressor which creates a MOD from a piece of real music.
But as for repetitive music - have you tried to losslessly compress a WAV decoded from MOD? General compressors like 7zip compress them much better than any lossless audio encoder, that's because MODs consist of identical patterns which often repeat somewhere in the song - lossless audio encoders work by predicting the very next samples, while general compressors work by searching for the same patterns of data somewhere in the whole data stream which works better in such cases.
I once emailed one of the lossless audio encoder's author about that issue, suggesting that it could be useful to implement some kind of pre-scan which would hunt for long repetetive patterns of samples before encoding, but no interest. Bit-to-bit identical patterns of data just don't happen often enough in real music.

Ops, hit reply twice, sorry.

It sort of depends on how much actually periodic information exists at very low frequencies (0.5hz to 2hz). This is how I'm intuitively thinking about it.

If you take a dance track and cut the track into separate tracks for every measure (or every beat), and figure out the common signal in every track, you still need to encode the differences from that common signal to each actual signal. Furthermore, the common signal itself must be primarily low-frequency information, simply because the fundamental is so long (from 500ms-2000ms). Finally, the common signal can't extend much into the midrange or treble, because fewer commonalities will be present at those frequencies. So the bandwidth of the common signal will be small relative to the bandwidth of the original signal. Thus the information encoded in the common signal may not be great enough to make a difference.

In short, you want to remove redundancy, and there is redundancy here, but it's not obvious that there is enough.

Sorry for asking some counterquestions.

Whats wrong with the existing Lossless Codecs?
What makes your Dance Music so special, that it needs a Lossless archiving instead of Lossy.

If the Music (or artist) lacks the courage to use variations, or Samples are mostly redundant, what argues against using a Lossy-Codec?

to be practical you would need an algorithm that could at least detect large-scale repetitions (either identical or similar) within data of length n in O(n) (linear time) which I would venture to say is not possible. any slower than O(n) and it will be too slow for anything but the shortest clip. I don't even think humans could do it in O(n), it would probably be O(nlog(n)) for a human to arrive at endpoints accurate enough for a short-term predictor.

Josh

There's actually a lot more difference in seemingly exact sections of audio than you would suspect. I can fire up Reason 3.0 and Bome's Mouse Keyboard then record a few keyboard hits through a synth and for whatever reason (I know not the mechanics behind this) each one will have subtle differences when viewed in say Audacity or Adobe Audition. A computer keyboard cannot measure the velocity of a keypress so the program has to assign a default velocity (say 90-100) so for the sake of this experiment everything should technically be identical as I understand it. In reality...it never seems to be. Similarly I've heard of artists using multiple samples of everything and alternating them so what seems like the same exact bass drum throughout the song could be a selection of three different samples every so slightly different.

Not being a developer at all I guess in theory it would be possible. However, it would be very, very expensive (in terms of computational effort) to exploit redundancies on that kind of a timescale.

Someone once did research to explout time repetitions in transform based codecs. As far as I remember a small gain with large complexity was realized. Can't really remember more though.

Alright... some countercounter? answers for Lego:

There's nothing "wrong" with existing codecs, but I'm just daydreaming about ways that they could potentially be improved upon. I think that rutra80's comment about how a compression algorithm like 7z works kind of answered my question though. Such a codec would have to spend quite some computational effort and wouldn't be seekable or playable in any practical sense. (i.e. it'd have to be decoded before used).

I shouldn't really have to answer why I'd need lossless over lossy... it's silly to ask that - this is the lossless codec forum. But my short answer is that I like to archive all my music to lossless formats and then transcode to an appropriate format in one easy step at a later time. (I use these songs for DJing). That way when new codec developments come along, I can simply reencode, without reripping. Or maybe that was a sarcastic comment because you don't like dance music?

The thing about artists lacking courage to use variations... the reason that much dance music is highly repetitive is because it's extended for DJ mixing. There are usually four to sixteen measures of a simple beat pattern or loop at the beginning and ends of extended 12" mixes of songs for the DJ to use for mixing it. Generally, these patterns aren't present on the radio edits that average people hear. So that's the point... it has nothing to do with an artist lacking courage...

BTW, it seems that you think there has to be some kind of relationship between the creativity employed in the creation of music and the bitrate you encode at or something... but why is that even relevant to this discussion? Heh... I guess I feel that all music should be encoded equal!


But back to the main discussion, thanks for your input rutra80 and Josh Coalson.
I'm afraid I'm not quite educated enough to understand Josh's response in its entirety, but I get the jist of it. And that is an interesting idea about making mods out of the music or something. That *might* be an idea for a special type of lossy encoder, but probably totally impractical for lossless work.

Perhaps a lossy encoder could do something like analyze the song, break it into patterns, and apply effects or variations on the patterns to do an approximate recreation of the song. It'd have to be quite a simple and highly repeated pattern to do that though. Probably not technically feasible for most music, but maybe those super-extended remixes that go on forever with the same loops could take up 1/4 of the room, because you only need to encode it once, then repeat it. But probably very little music actually would benefit from this. A real "niche", for sure.

The idea of O(n) and O(n.log(n)) is that the time required to calculate this type of thing is of a general order (hence the O) n or log(n). That means, if you want to check for song patterns in, say, a ten minute song, it might take you anywhere from thirty seconds to thirty minutes, or whatever, but the increase would be linear according to the length of the song (double if the song length doubles, in case of O(n), or multiply by 20 if the song length increases ten-fold; by 300 if it multiplies by 100, etc -- in case of O(n.log(n))

The whole point being, as a general concept, efficiency is measured as O(x), or as the order of complexity of the algorithm used. We can say something is more efficient than brute force if it's complexity is lower than O(n); generally, O(log(n)) is strived for, or even O(1) (which solves any given task in constant time). That being, an algorithm which has O(n) or O(n.log(n)) might be pretty good for audio, but since it's not very efficient, the user might think he's waiting too much and neglect a codec which implements this type of search method.

Then again, you could always make it a switchable option, but dance songs aren't most compressible anyways, since (as a previous post said) DSP effects vary over the duration of the song / beats. Often times, subtle changes are made which make such an algorithm useless. At least for now.

I have some ideas to implement. For starters, limit the scanning range to 2 seconds. Then, do a rough FFT analysis, and split the signal in constant parts vs non-constant (in the complete song). Look for constants, and try to synthesize them and match them up (using a 2-or-more-pass system). This will probably be long, though, but might be pretty damn good for compression.

Anyways I have to go to class,
Peace,
Tristan.

I've often though of things like this myself but I doubt there could be any gains unless the two sections were bit accurate - or close to. In reality there's probably slight differences in flanging, and overlayed instruments, echo calculations, phasing etc. This technique is probably more useful on lossless material actually.

I think for this to really work, artists would have to release their music in some kind of format that employs this... i.e. Mod files, or Reason/Rebirth files or the like.

And since that would probably make it too easy to reverse engineer (er... remix, maybe?) the songs, that might not be popular with the music industry. I'm guessing that there isn't a big benefit to the producers in doing this, unless it's a young newbie who wants to be discovered and wants nothing more than to share his passion for music for the world.

Of course, the record companies want nothing more than to make money by "investing" in productive artists that can generate sales revenue. And they are the ones with the power, unfortunately.

One might as well say that another artist lacks the courage to use outright repetition in his compositions... if repetition is part of the style (and clearly for some forms of dance music, amongst others it is) then why not make a lossless codec that improves compression?

On topic, I think rather than expecting exact repetition, using these repetitions and storing the difference (rather like linear prediction used today, but on a grander scale). Possibly this could be particularly useful in minimalism and ambient music, too. Not knowing anything about coding, of course, this could be complete nonsense

Of course ambient music, classical, etc. already compresses fairly well with lossless codecs. I think I've got a Deepak Chopra recording (spoken word) with soft ambient and piano music playing in the background that compresses to just over 400kb/s losslessly. Not that it would be silly to realize even greater gains for it!

Gave this a quick think. Now it wouldn't be a codec, but how about a Lossy/MOD-like hybrid? Run your compressed samples through a MOD-like system, with the wildly variable stuff as one long compressed sample that just plays throughout the song.

Means that music file would have to be authored from the computer they were created on, rather than ripped from CD. But I like the idea.