Help ripping ~30,000 CDs, Was “Help digitizing […]” ;) Options

[rick.hughes]

- RAID is not backup. RAID is a way to reduce the number of times you need to resort to your backup.

- There is a proprietary solution called UnRAID...

I use unRAID but if your budget forces you to choose between a complete backup and some sort of RAID then you should go for a complete backup, possibly kept offsite.

[LosMintos]

The last R14.2 release should have eliminated this possibility, how ever for true multi drive ripping Batch Ripper was designed for that operation (and has been in use 24/7 for the last 4 years by the largest commercial ripping companies out there).

I never experienced problems with multiple instances (neither dBpoweramp nor EAC). But, just to point it out, would you recommend Batch Ripper even for ordinary PCs with two or three drives? Or is it exclusively bundled with extra hardware?

Then it's just the issue of "how do we back up and make 10TB of data network accessible on the budget of a college radio station?"

Yeah, it's a matter of cost. But, if you do not insist in quite and most energy-saving technology, you'll get 10 TB rather cheap (except hdd's itself, maybe). And, keep in mind, that you don't have to start with 10 TB. You can add HDDs later along with the ripping progress.

but the folders for different types of music is an excellent idea. We get lots of promo material and compilation albums so it's probably a good idea to separate the music into broad categories.

Here's the example folder hierarchy I was thinking of:
\Library\Category\Artist\Album (ID number we add when we get it)\Track number. Artist - Title

or for a real-life example:
Library\Full Albums\Modeselektor\Monkeytown (30458)\04. Modeselektor - Evil Twin.flac

My approach is similar. When it comes to artist's names or titles with exotic characters, you might want to rename/sanitize your folder names. Especially, when you work multiplatform (Linux file server, Windows Client). However, if you don't care so much, you could also use your ID number, probably structured: $left(%ID Number%, 2) / $left(%ID Number%, 3) / %ID Number% / %track% = %artist% = %title%.flac.

It just seems like the biggest hurdle now is to store and back up all these terabytes of data we're going to create by going lossless and still make them accessible to the other computers on our local network.

Again, you don't have to provide the full disk space _now_.

[phofman]

We have been using linux raid in our company for a few years, from 2 to 8 drives configurations. This is just my experience:

Raid is definitely not a backup, I absolutely agree. In fact I was going to ask about the planned backup procedure. However, backing up 10TB of data on low budget is not a trivial task.

I assume the server should run 24/7 since it is the source of music for the radio station. That is why I suggested redundant raid and server hardware (yet inexpensive one).

Linux raid has no problem being used and running synchronization at the same time. Synchronizing 12TB software raid will take many hours, easily a day or two on a mildly loaded server. I agree that raid5 with only one drive of redundancy is not very safe. Six 3TB drives would allow the much safer raid6 of 12TB.

As for the backup, either the rather expensive tape, or IMO a much more flexible solution is a simple desktop PC with large case and motherboard with 6 sata ports, 5 3TB drives in RAID5, one small drive for system with linux (to make life easier) and run rsync every night/week over gigabit ethernet. I do not assume many changes on the main server data array so the synchronization would take just a few seconds. Preferrably the machine should be located in a different building. This solution would have the advantage of being able to take over the file-serving role of the main server quickly in case of hardware failure. It can be booted and shut-down automatically, by bios/halt command, to minimize electricity costs and hard drives wear.

This post has been edited by phofman: Apr 2 2012, 16:00

[krabapple]

Nothing to add except again: go lossless if you can. You can always make mp3 versions of your FLAC files for everyday use, and keep the lossless versions as an archive.

[pdq]

For backup I would suggest AudioSAFE. If you never need to use it then it is completely free. You only get charged if you need to access your backed-up data.

In fact, you might consider backing up the lossless files to AudioSAFE, before you convert them to mp3 and delete the original. If you never want the lossless version then it has cost you nothing to save them, but if your budget at a later time permits then you don't have to rerip.

[rick.hughes]

The original CDs could also be considered the backup. If they are not CD-R then they should have a good lifetime. Depending on how difficult some might be to replace this might be all that is really needed for a backup. Store them at another location in case of disaster.

[phofman]

The original CDs could also be considered the backup. If they are not CD-R then they should have a good lifetime. Depending on how difficult some might be to replace this might be all that is really needed for a backup. Store them at another location in case of disaster.

CDs are definitely a very good backup. It all goes down to the time it took to rip them. If that time spent has lower value than the backup solution, then yes. Considering the time it takes to rip a few thousand CDs which fit onto one harddrive with presumed lifetime of 2 - 3 years (loaded 24/7, regular consumer type HDD), I would tend to prefer building a simple low-cost backup PC. Quality of drives is not going stelar, in fact manufacturers are cutting the warranty period - http://www.tomshardware.com/news/seagate-w...land,14322.html .

[garym]

For backup I would suggest AudioSAFE. If you never need to use it then it is completely free. You only get charged if you need to access your backed-up data.

In fact, you might consider backing up the lossless files to AudioSAFE, before you convert them to mp3 and delete the original. If you never want the lossless version then it has cost you nothing to save them, but if your budget at a later time permits then you don't have to rerip.

excellent ideas!

[Jan S.]

For backup I would suggest AudioSAFE. If you never need to use it then it is completely free. You only get charged if you need to access your backed-up data.

In fact, you might consider backing up the lossless files to AudioSAFE, before you convert them to mp3 and delete the original. If you never want the lossless version then it has cost you nothing to save them, but if your budget at a later time permits then you don't have to rerip.

excellent ideas!

If the files are deleted locally they will be dropped from AudioSAFE after some time. So you have to keep the files locally you want to backup on audioSAFE.

[Porcus]

the rather expensive tape

The tape drives are expensive, but tapes are only slightly more than hard drives, so ... are there rental services? Or backup services where you would show up physically, have your drives copied, and if you are so unlucky you need restoration, show up with tapes and new drives? (I suspect that if such exist, they are at industry-grade pricing.)

[Nessuno]

If the files are deleted locally they will be dropped from AudioSAFE after some time. So you have to keep the files locally you want to backup on audioSAFE.

Actually? That's quite a strange behavior for a backup system. Sounds more like an... asyncronous and delocalized RAID 1!

Well, all in all it depends on how long is "some time"...

This post has been edited by Nessuno: Apr 3 2012, 17:54

[spoon]

Your deletions are kept on audiosafe (the last change) indefinitely.

All we ask is that your account is active, that is a computer logs into audiosafe once every so often.

[Destroid]

Bad news, our budget means we probably won't be able to afford the equipment to go full FLAC. We'll probably stick with V0.

I have to ask again, why insist on this setting? I don't want to derail this thread into the dozens (hundreds?) of other listening threads, but it seems storage costs are relevant. So, I refer to my previous post of -V3...-V5, or, AAC 64/96kbps.

The hardware considerations will add up, plus (as already mentioned) there are redundancy options abound.

[trail]

Bad news, our budget means we probably won't be able to afford the equipment to go full FLAC. We'll probably stick with V0.

I have to ask again, why insist on this setting? I don't want to derail this thread into the dozens (hundreds?) of other listening threads, but it seems storage costs are relevant. So, I refer to my previous post of -V3...-V5, or, AAC 64/96kbps.

The hardware considerations will add up, plus (as already mentioned) there are redundancy options abound.

It's just that I've always found V0 to be the optimum balance between quality and file size for myself. I don't have an aversion to other formats, (except AAC is probably out because we don't own any Macs or use iTunes frequently) but I'm not terribly well-versed in the trade off between file size and audio quality when using different VBR settings. But I did just do an A-B test between a song that I have both FLAC and V5 versions of and couldn't discern any noticeable difference between them. Other formats are definitely a consideration, I just assumed V0 was the happy medium.

[mixminus1]

@Destroid: Yes, the OP's budget is limited, and no lossy codec can truly be an "archival" format.

However, since this is for a music archive at a radio station, we should assume that some form of additional processing and/or lossy transcoding may take place in the broadcasting/webcasting chain.

As such, to try and hit the "sweet spot" of minimizing storage space while providing maximum resistance to downstream artifacts, I think V0 makes the most sense, if AAC is really not an option.

While this test is almost seven years old now, I think it's still useful for the discussion at hand:

http://www.hydrogenaudio.org/forums/index....st&p=282909

That was done with LAME 3.97 a8, so not current, but not exactly "ancient", either, and it fared the worst as a source codec when transcoding to MP3 ABR 128 - and that was with -V0! Going to a lower bitrate certainly wouldn't improve things.

This post has been edited by mixminus1: Apr 5 2012, 21:08

[JJZolx]

If the budget is so limited that the station can't even afford the hard drive space on which to store the music in a lossless format, then I hate to say it, but 'm afraid the project is doomed from the get go. You're likely to patch together a system where the next guy to come in says "Just look at what they left me" and will want to start all over again, perhaps with a more realistic budget.

There is much better software available than foobar or JRiver that is actually designed for radio station programming. You have to take into account things like queueing advertisements, news briefs, pre-recorded shows, not to mention downloading those shows 24x7 as they become available. Then there's the whole side of things where you may want to digitize the outgoing on-air feed and make it available at one or more bitrates in an internet radio feed. Also, you may want to generate a web page displaying your playlist of songs as they're played. Managing all of that at the file level with something like foobar would be a nightmare, if it's even doable.

[shadowking]

Bad news, our budget means we probably won't be able to afford the equipment to go full FLAC. We'll probably stick with V0.

I have to ask again, why insist on this setting? I don't want to derail this thread into the dozens (hundreds?) of other listening threads, but it seems storage costs are relevant. So, I refer to my previous post of -V3...-V5, or, AAC 64/96kbps.

The hardware considerations will add up, plus (as already mentioned) there are redundancy options abound.

I agree. I did 300+ albums with V4 years ago and don't regret it as they mostly sound great . That is 150k vs 250..300k(v0 / itunes) bloat of today. No need for RAID or anything fancy.

This post has been edited by shadowking: Apr 6 2012, 15:50

[Porcus]

OK, so OP cannot afford drives enough for lossless.

Let me assume that the CDs average at 40 minutes. (If they are singles -- much less.) 20 000 hours. 72 million seconds. Want to fit on a single hard drive of 3 TB = 24e+12 bits? Divide out to get one third megabit per second. That's the 320 rate. Might as well use V0.

[trail]

@Destroid: Yes, the OP's budget is limited, and no lossy codec can truly be an "archival" format.

However, since this is for a music archive at a radio station, we should assume that some form of additional processing and/or lossy transcoding may take place in the broadcasting/webcasting chain.

As such, to try and hit the "sweet spot" of minimizing storage space while providing maximum resistance to downstream artifacts, I think V0 makes the most sense, if AAC is really not an option.

While this test is almost seven years old now, I think it's still useful for the discussion at hand:

http://www.hydrogenaudio.org/forums/index....st&p=282909

That was done with LAME 3.97 a8, so not current, but not exactly "ancient", either, and it fared the worst as a source codec when transcoding to MP3 ABR 128 - and that was with -V0! Going to a lower bitrate certainly wouldn't improve things.

Exactly what I was thinking. There is some processing and the like, but it's not any that really takes much away from the music. To put it in perspective, we're a college FM station, and it's safe to assume very few of our listeners are tuning in looking for pristine audio fidelity, you know? I obviously want it to sound as best as I can, but V0 sounds like the optimal trade-off. As a college kid I don't quite have taste, budget, or equipment of an audiophile but V0 hasn't ever disappointed my ears.