The Deal With SHN, Too scared to bring this up on STG. Options

[EdBanky]

As far as I have learned through use and reading, SHN is slower on the encode and decode, inferior at compression, has no standalone hardware support, and is unseekable (without appending seek info), compared to FLAC, for example. I understand a relative abundance of support compared to APE, or other formats less Mac-friendly. But why do people insist upon sticking with SHN, even now?

From A Small .SHN and .MD5 FAQ:

"Shorten is currently the standard lossless compression method used in trading communities. Many versions are free, it's usable on nearly all platforms, and it has a large pool of experienced users."

The FAQ also lauds FLAC's "stong future potential."

I guess this FAQ is probably quite outdated, but I continue finding brand new recordings of concerts, and other new encodes, being distributed as SHN. In addition to being larger in size, they are impractical for most applications that I have found.

Is the online bootleg trading community's infatuation with SHN anything more than slow adaptation, or does the format offer advantages I have been unable to find?

[Duble0Syx]

When I first started grabbing bootlegs about 85% were SHN. Now it's about 65% FLAC, so flac is winning, and for good reason, much smaller, much faster to decode. No idea how fast SHN takes to encode, I've never used it, I always converted the bootlegs to FLAC. Are you aware STG is dead? Unless you are referring to whats taken it's place.

This post has been edited by Duble0Syx: Nov 16 2004, 21:52

[EdBanky]

Are you aware STG is dead?  Unless you are referring to whats taken it's place.

Yeah. STG is more symbolic of a reference. Its influence permeates the Web. I should use EasyTree.org as the practical example.

[ffooky]

As far as I can see the only reasons for SHN's continued use are the database of MD5 files at places like etree, possibly the lack of FLAC support for Mac OS 9 but mostly basic human conservatism.

[ssamadhi97]

As far as I have learned through use and reading, SHN is slower on the encode and decode, (...) and is unseekable (without appending seek info)

"All wrong."

Shorten is faster both when encoding and when decoding, for reference see http://members.home.nl/w.speek/comparison.htm

Furthermore, the (in)ability to seek in arbitrary files without external or appended seek info entirely depends on the implementation of the playback plugin / decoder used. It's not a restriction of the format itself.

(for example, the WinAmp and XMMS shn plugins, both of which are based on the reference Shorten decoder, do indeed suffer from the limitation you mentioned; however the foobar2000 shn plugin provides unconstrained seeking functionality on all shn files (at the expense of an occasional little lag when seeking in files without additional seek info))

Is the online bootleg trading community's infatuation with SHN anything more than slow adaptation, or does the format offer advantages I have been unable to find?

Pretty much just slow adaption and the existing md5 database(s), as far as I can tell.

[Duble0Syx]

My mistake about SHN's encoding/decoding speed. Although faster it isn't much. I for one like the instant seeking and extra space flac provices. On a 400mb file set you can average 20-30mb smaller with flac and 20-40mb smaller with wavpack 4.2. SHN isn't such a bad format, but it seems a little out dated now. I am happy to see more flacs coming around. I think the best method for md5sums is to use the WAV files rather than the compressed files. IF you add or edit flacs tags the md5 will not match of course. Be great to have an MD5 app that decompresses and check the wav files rather than the compressed file. Then it wouldn't matter what format you switched to.

[ffooky]

I think the best method for md5sums is to use the WAV files rather than the compressed files.  IF you add or edit flacs tags the md5 will not match of course.  Be great to have an MD5 app that decompresses and check the wav files rather than the compressed file.  Then it wouldn't matter what format you switched to.

shntool's checksum facility, which is confusingly named MD5 as well, does exactly what you want by making a checksum for only the PCM wave data contained in a file in any of the formats it supports. If you then convert the file to any other supported format, change tags, recompress to a higher leve etc.l it will still pass the shntool-md5.
The only current drawback is that there is no Windows app to check shntool-md5 so it has to be done by comparing checksums manually. OS X users can verify automatically with xACT.

[EdBanky]

"All wrong."

Shorten is faster both when encoding and when decoding, for reference see http://members.home.nl/w.speek/comparison.htm

Furthermore, the (in)ability to seek in arbitrary files without external or appended seek info entirely depends on the implementation of the playback plugin / decoder used. It's not a restriction of the format itself.

(for example, the WinAmp and XMMS shn plugins, both of which are based on the reference Shorten decoder, do indeed suffer from the limitation you mentioned; however the foobar2000 shn plugin provides unconstrained seeking functionality on all shn files (at the expense of an occasional little lag when seeking in files without additional seek info))

This was a situation in which I tried to name a few "known" shortcomings for a format that has repeatedly been inconvenient for me to use. I obviously didn't want to post "SHN sucks! Why does anyone use it?" Anyway, it turns out my personal experience has probably been skewed by confounding variables.

When referring to a codec's relative drain on system resources during playback (I mistakenly thought of that as the same general notion as decoding "speed"), is there an objective way to test its "smoothness-of-play"? Part of my perceived experience has been that SHN files have generally appeared to force more "hiccups" while (presumably) the system tries to allocate resources, in the few players I have tried (WinAmp, ZoomPlayer). Again, I've got too many variables for an objective analysis. Any ideas?

As these "observations" seemed to violate at least the spirit of the BIG #8, I tried initially to include what I thought were factual deficiencies of the SHN format.

The intent of this has been to ask "Since, considering a host of factors, SHN seems to be less than ideal for online disk trading, why does it persist as even somewhat popular among the STG types?" I appeciate the need to be accurate in my claims, and I am sorry for the misstatements I made. But instead of devil's advocacy, those of you who understand what I am looking for, can you please comment? If there are other demonstrable areas in which you believe SHN to be inferior, please do mention them, on my behalf. Thanks.

[jcoalson]

The intent of this has been to ask "Since, considering a host of factors, SHN seems to be less than ideal for online disk trading, why does it persist as even somewhat popular among the STG types?"

I think SHN is mostly continuing on momentum. It doesn't really offer anything that other codecs don't, although it does have some use for the times you need a really fast encoder to get some temporary compression.

Josh

[Brink]

the database of MD5 files at places like etree (...)mostly basic human conservatism.

To me, its abosolutely true.

I encode my boots with FLAC for distribution, or APE for archiving, it depends a lot. You can save a good amount of space when using dvd-rs for backups.

But when I receive them in SHN I keep in this format. I like to maintain the originals. A lot of times its the taper who encoded in SHN, so I keep that way.

This post has been edited by Brink: Nov 17 2004, 23:49

[ssamadhi97]

When referring to a codec's relative drain on system resources during playback (I mistakenly thought of that as the same general notion as decoding "speed"), is there an objective way to test its "smoothness-of-play"? Part of my perceived experience has been that SHN files have generally appeared to force more "hiccups" while (presumably) the system tries to allocate resources, in the few players I have tried (WinAmp, ZoomPlayer). Again, I've got too many variables for an objective analysis. Any ideas?

Hardly. "smoothness-of-play" depends on too many variables. Decoding speed and memory consumption while decoding/playing back a file depend entirely on the player software and the decoder/plugin, and then there are tons of other possible influences (like cpu consumption of other programs, decoder/playback thread priority, maybe even harddisk fragmentation, ..)

The only kind of objective measurement you can perform (to the best of my knowledge) is evaluating raw decoding speed itself and hoping that players/plugins are designed in a sane manner..

Generally, there is hardly any format that consumes (or at least "should consume") less resources than Shorten when decoding, so your observation is rather surprising to me.

The intent of this has been to ask "Since, considering a host of factors, SHN seems to be less than ideal for online disk trading, why does it persist as even somewhat popular among the STG types?"  I appeciate the need to be accurate in my claims, and I am sorry for the misstatements I made.  But instead of devil's advocacy, those of you who understand what I am looking for, can you please comment?  If there are other demonstrable areas in which you believe SHN to be inferior, please do mention them, on my behalf.  Thanks.

Like I and others said already, the only things Shorten really has going are its large user base (which indeed seems to be very reluctant to adopt newer formats), and existing infrastructure (md5 databases).

Here's a quick overview over other (theoretical) pros and cons:

pros:
- very fast compression and decompression
- low resource usage (cpu+memory)
- very fast seeking given the availability of proper additional information

cons:
- weak compression performance
- no built-in metadata support
- no built-in file identification/verification/fingerprinting support
- no multichannel support
- hacked-on seeking support
- impossible to cut/glue shn files (inter-frame dependencies)
- no inherent error resilience (neither correction nor detection)
- ugly bitstream format
- playback recovery after bitstream errors nearly impossible / a matter of luck afaik

I only defended the format against your claims because you discredited its only actual advantages there

[EdBanky]

I only defended the format against your claims because you discredited its only actual advantages there

OK, so maybe that was just a little backwards.

Your second answer was exactly what I was looking for. Now I just need to figure out what some of the things you listed actually mean. I am trying to learn. Thanks!

[ssamadhi97]

Your second answer was exactly what I was looking for.  Now I just need to figure out what some of the things you listed actually mean.  I am trying to learn.  Thanks!

Well sorry about going all technical on you there. I co-wrote the Shorten plugin for foobar2000 together with a friend, so I know more than I'd like to know about the inner workings of the format

[buzzy]

I guess this FAQ is probably quite outdated, but I continue finding brand new recordings of concerts, and other new encodes, being distributed as SHN.  In addition to being larger in size, they are impractical for most applications that I have found.

Is the online bootleg trading community's infatuation with SHN anything more than slow adaptation, or does the format offer advantages I have been unable to find?

Yes, that FAQ and many others are outdated.

Mostly it's just inertia and lack of getting the word around.

But shn is very fast. And until very recently, the software tools were better. In real life, flac didn't offer a compelling enough advantage over shn, as much as the codec wonks might dismiss shn. And it's not that the flac tools have gotten better for flac ... it's just that XP SP2 has made them work less well for shn.

I think SHN is mostly continuing on momentum.  It doesn't really offer anything that other codecs don't, although it does have some use for the times you need a really fast encoder to get some temporary compression.

Josh

What a lot of people deep in the codecs here never saw was that there isn't a big enough gain in performance to get people to switch, if it's difficult. And until flac front end with Mike Wren's installer came out, it was too hard to get flac working for most people. So that's only about 12-18 months ago.

And sorry, josh, but you don't seem to have seen the need for better tools for using flac in real life. I know you can't do it all, but seeing the problem would have helped. All the threads at forums.etree.org about the automatic verification and packing list checking issues with ffp, for example - that's a valued feature of md5 that's missed, and ignoring that or saying it's someone else's problem doesn't make it go away. And yes, it really is someone else's problem, but I have to think a little encouragement from you might help. The same on the GUI - sorry, but flac frontend is a nice days work but not really an adequate tool for real, mainstream users.

This post has been edited by buzzy: Dec 10 2004, 15:33

[jcoalson]

And sorry, josh, but you don't seem to have seen the need for better tools for using flac in real life.  I know you can't do it all, but seeing the problem would have helped.  All the threads at forums.etree.org about the automatic verification and packing list checking issues with ffp, for example - that's a valued feature of md5 that's missed, and ignoring that or saying it's someone else's problem doesn't make it go away.  And yes, it really is someone else's problem, but I have to think a little encouragement from you might help.  The same on the GUI - sorry, but flac frontend is a nice days work but not really an adequate tool for real, mainstream users.

I don't know why you think I'm not "seeing the problem". I used to be on the etree and furthurnet forums quite a lot. every time this came up I asked for someone to specify exactly how such a tool should work (since I couldn't figure it out by reading posts). no one ever did.

Josh

[DonP]

I for one see a lot of value in distributing files already tagged. It seems like more and more of the concerts I've BT'd lately *are* in flac format, but they still come untagged with a text file containing track names in formats that vary enough that they generally need a bit of tweeking before running through FB2K's "traders friend" plugin.

So, the tapers are migrating to flac, but aren't enabling the advantages that would make it more worthwhile.

Since I want files in my active collection to be tagged, that involves a second copy as tagging changes the files from the original so they are invalid as far as the bit torrent is concerned.

[jcoalson]

And sorry, josh, but you don't seem to have seen the need for better tools for using flac in real life.  I know you can't do it all, but seeing the problem would have helped.  All the threads at forums.etree.org about the automatic verification and packing list checking issues with ffp, for example - that's a valued feature of md5 that's missed, and ignoring that or saying it's someone else's problem doesn't make it go away.  And yes, it really is someone else's problem, but I have to think a little encouragement from you might help.  The same on the GUI - sorry, but flac frontend is a nice days work but not really an adequate tool for real, mainstream users.

I don't know why you think I'm not "seeing the problem". I used to be on the etree and furthurnet forums quite a lot. every time this came up I asked for someone to specify exactly how such a tool should work (since I couldn't figure it out by reading posts). no one ever did.

OK, another month of silence; this is what I was talking about. I'll just leave it at this, an open invitation for someone to write up a spec for how such a tool should work so I have something to go on.

also if you have some ideas on how to make a better GUI I'd like to hear them.

Josh

[Polar]

Nuff said, Josh. Keep up the good work.

[buzzy]

And sorry, josh, but you don't seem to have seen the need for better tools for using flac in real life.  I know you can't do it all, but seeing the problem would have helped.  All the threads at forums.etree.org about the automatic verification and packing list checking issues with ffp, for example - that's a valued feature of md5 that's missed, and ignoring that or saying it's someone else's problem doesn't make it go away.  And yes, it really is someone else's problem, but I have to think a little encouragement from you might help.  The same on the GUI - sorry, but flac frontend is a nice days work but not really an adequate tool for real, mainstream users.

I don't know why you think I'm not "seeing the problem". I used to be on the etree and furthurnet forums quite a lot. every time this came up I asked for someone to specify exactly how such a tool should work (since I couldn't figure it out by reading posts). no one ever did.

OK, another month of silence; this is what I was talking about. I'll just leave it at this, an open invitation for someone to write up a spec for how such a tool should work so I have something to go on.

also if you have some ideas on how to make a better GUI I'd like to hear them.

Josh

There was a long thread started here about this two years ago. So "nuff said," Polar, about long silences.

http://www.hydrogenaudio.org/forums/index.php?showtopic=6384 - ignore page 2

At this point the reference for usability and integration, for mainstream users anyway, would seem to be iTunes. If there were ways to tweak more settings, and more info about what it's doing, it would be closer to what more advanced users want, too. Obviously that's not to suggest re-creating it; but it's clearly an interesting data point, seeing how integration and usability have driven adoption of iTunes lossless. It's clearly not the codec itself that's driving it.

This post has been edited by buzzy: Feb 12 2005, 20:01

[buzzy]

By the way, it's interesting that among the most popular questions at etree, antsmarching, etc are about how to install (or more exactly fix problems with the install) of mkwACT (the old but not at all forgotten Windows tool for shn).

[jcoalson]

I was on that thread too... but I went back and read it again.

so as far as I can tell the specification is like this:

1. take some file which is a list of lines. each line is colon-separated record, field 1 being filename, field 2 being FLAC md5 sum
2. for each record, check the file against the md5, warn and optionally delete if md5 mismatch, warn if file does not exist.

is this all? this is like 50 lines of perl. but what is not specified is what form this program should take, some shell script language, command-line binary, or integrated into GUI. one thing for sure is it does not belong in flac.exe. (it really belongs in etree's md5sum hack but maybe that's not going to happen for the reasons I mention in that thread.)

as for the GUI thing, even in that thread you had suggestions for speek's front end, but I didn't get that much more out of it. saying that itunes is a good target does not help. itunes is huge. and implying like you did before that such a GUI undertaking should be my responsibility because I developed the codec seems like a stretch.

nevertheless I have been annoyed at the lack of a simple, user-friendly transcoding/transtagging GUI that was F/OSS and crossplatform, so a while back I started on such a thing using my own intuition as to what it should be like. I think I have described it before, search for "fui" here. you can even check out the fui module from the flac cvs repository and play with it. I just have not gotten time to finish it.

Josh

[buzzy]

I was on that thread too...

Right, that was the point. There's some context for this thread, it didn't just come out of the blue.

Even though almost all the specific points raised in that old thread are still an issue for users - the context has changed a lot since then, in terms of users, codecs and tools:

- The transitional period is over for a large chunk of active users. That stuff would have helped to make the transition easier, but for many users that's past, they've overcome the quirks. Even though lots of new users are coming in, there is a large base of flac + flac frontend users now to help them. Which is not to say that ease of use doesn't matter (see below), just that to some degree enough people are over the hurdle to create some momentum; and also, it's history, for better or worse.

- Lots more people seem to be using lossless, maybe enabled by the easier availability of bandwidth, drive space, and burners. People get the idea of lossless, it's more mainstream.

- Related to that, the big players (Apple and Microsoft, as well as some bands like DMB and others) are getting into lossless, with a profit-driven vision of where that's going. There's a fair amount of money at stake, and lots more in the future. They're getting ready for a future where people will buy music by downloading lossless onto a media server, much like people download lossy to their computers / portable today.

So all that changes the picture.

One specific development - thinking about it more, now that BitTorrent has become by far the most popular way for most groups to share live shows (etree, easytree, etc.) - the "packing list issue" is much less of an issue. BT of course verifies that the complete set of files has been transferred before saying that the download has reached 100% complete. For other applications, people are just using an md5 or a par2 file.

So it's more of an example - of how something that's important to some users can seem trivial or unimportant to experienced users; or to those who don't use a codec for a particular application. The application, in this case, being sharing files; those who use a codec for personal use wouldn't have seen this as a problem.

I have been annoyed at the lack of a simple, user-friendly transcoding/transtagging GUI that was F/OSS and crossplatform, so a while back I started on such a thing using my own intuition as to what it should be like.  I think I have described it before, search for "fui" here.  you can even check out the fui module from the flac cvs repository and play with it.  I just have not gotten time to finish it.

Well, that's interesting. Though that may be somewhat of a different issue - since I gather from your posts that you haven't really used mkwACT, shn or even all the features of flac frontend. So, designing something based on your own intuition might or might not address the issues of other types of users. That is, what works for a developer or an old-school HA member might not be what works for the mainstream user. It all depends on the objective.

I've often wondered whether the goal / ideal was to get some other GUI to add flac support, since the codec is just one part of a bigger picture, from the user's point of view. For example, obviously we were all hoping iTunes would just add flac support, though in hindsight it's perhaps no surprise that it didn't fit Apple's way of doing business.

dbPowerAmp has sort of done that, a fair number of people use it for flac and shn by adding the codecs to the install. A decent if not great solution for some.

implying like you did before that such a GUI undertaking should be my responsibility because I developed the codec seems like a stretch.

As I've said before at HA - I can definitely see that codec developers might not at all be interested in adoption of the codec, or tools to encourage adoption. That end of things is probably not the fun part for them , and doesn't give them a chance to advance the state of the art for encoding, and so might not help their careers. (Though it does have a huge effect on the practical state of actual use of lossless and the codec.) If that's the case, fair enough - that's a reasonable point of view.

My assumptions have been that, in general, what's good for flac users is good for flac. And (again an assumption) wider use of flac was something you'd be interested in seeing happen. And presumably you could have more influence than most in encouraging something to happen. Any or all of those assumptions could be wrong.

saying that itunes is a good target does not help.  itunes is huge.

iTunes is a fact of life in the lossless environment. iTunes is out there, it's growing fast, and it's defining what users expect, at least for some users. So for those who are interested in encouraging the use of a codec and the tools that make that possible, iTunes provides a really interesting example.

As noted above, the point isn't to recode iTunes, but to see what it teaches about usability and integration. It's a pretty spectacular lesson. iTunes introduced a whole lot of people to audio encoding, portables, playback on their computer, and so on. And iTunes has and will introduce a lot of people to the idea of lossless encoding.

iTunes gives them a complete solution: ripping, database lookup, tagging, library management, transcoding to lossy, playback, and hardware options. (Presumably most people at HA don't think it's the Apple Lossless codec itself that's attracting users!)

And someone might say, "Well, some (or maybe most - but not all) of those users are users we can't really accommodate" - that's fine, after some real thought. But that's different from the frequent response of just dismissing or ignoring iTunes. Anyone interested in how users use audio or lossless can learn something from iTunes. (But again, that's not to say all codec developers are or should be interested in users. It's their call, of course.)

And there's a similar, though maybe less dramatic lesson, learned from looking at Monkey's Audio. A few years ago, one of the major factors for people adopting MAC was the user interface - a clear advantage over flac in the early days, for those who recall. So that's a good demonstration that it's possible to have some of the benefit of enhanced usability without recreating iTunes.

Lots of people involved in codecs don't really seem to understand usability or integration - presumably because for their needs, they want a maximum of control and flexility; and they have a knowledge base that lowers the barrier to using new tools. The kinds of issues that create barriers to new users seem trivial or unimportant to HA people. But again, ignoring or dismissing those issues doesn't change the reality of what users experience.

In terms of integration, compare the iTunes experience (install, use) to:
- Find out about EAC, learn how to install it, get it to work with your drive, find the Coaster Factory, get the right configuration
- Try to get CDDB to work reliably for you
- Find Case's page on EAC + external encoders, figure out how to enter a big command line for the codec
- Install a front end and/or the codec
- Figure out that it's just a front end, and that Tag and the codec are actually separate programs
- Figure out how Tag works by finding the help file buried in the install directory; find a file renamer to get your filenames into a form Tag likes
- Or kludge WinAmp to do tags for you; or (more feasible recently) use tagging software that handles that codec

That's not a flame against flac or you. The point is to illustrate a reality that most users experience, which a lot of HA people don't see. And again, the point isn't that this is your responsibility, who knows if you're still even reading. But there are some spots that could have been smoothed, and some lessons to be learned, for those who find it relevant.

This post has been edited by buzzy: Feb 24 2005, 16:53

[jcoalson]

I know what you're saying, but I don't have bandwidth to build an itunes. apple has a whole group of people working on that. and why would you expect it from me and not from (for the same reasons), say, monty or bryant or other developers of codecs that would be supported by this itunes clone. I do want flac adoption to increase because it makes the format more useful, so I do what I can, namely provide a library and help out users of the library as much as I can.

the comparison between itunes and eac+coasterfactory+cddb+... is not really apples-to-apples. what you get with eac is high quality ripping and there is a complexity price to pay for that. you could also argue, more justifiably I think, for andre to improve eac. if you are willing to make do with non-secure ripping like itunes there are easier, more push-button solutions. at least for linux there are. on windows afaik fb2k or cdex would fit the bill for ripping+encoding+tagging out of the box.

so if you're arguing for a free open-source itunes-like program I'll go along with that but I don't see how me not being able to do it is an indication that I'm not interested in making things easy for users.

if you're passionate about it and think there is a great need, maybe this is a new project you should start.

Josh

[buzzy]

This is all mostly history and somewhat academic at this point, but history is the topic of this thread - looking at the use of shn and adoption of subsequent lossless formats. The question that started all this was, why did / do people keep using shn so long? As discussed, the answer is in understanding the users.

The point of discussing iTunes was to focus on another, more current example of understanding users, specifically the practical importance of considering usability and integration. As noted above, twice, it wasn't to suggest that anyone recreate iTunes. But it does show how high the bar has been raised. And the contrast with what users need to do to use other lossless formats is useful.

you could also argue, more justifiably I think, for andre to improve eac.

That's a perfect example, except that I was thinking he'd be more likely to pay attention to you than to me. Though since his focus is ripping, it might be tough for anyone to convince him.

But right now lossless has to be shoehorned into EAC right now by users, after the fact. We could certainly wish for better treatment of lossless, with a workable default setup for flac. If anyone can get Andre interested in the doing that, I'm sure people here could give him the specifics on what to do.