Audio Storage, building an inexpensive fileserver Options

[sgreen]

hope this is on topic enough-

i'm planning to build a network fileserver dedicated to storing and serving music (mostly mp3 and flac) across my home network.

i have a few drives available 1x20gb, 1x60gb, 1x80gb and 1x160gb - all drives are regular ata ide. currently the 80gb drive is disconnected and boxed up as a backup.

i don't want the machine to do anything other than allow me to easily store music and stream it over the network. i do not require super fast access times, lots of memory or a fast processor.

i am considering a raid controller to run in a mode that will allow a complete crash of at least one disk without data loss to save me keeping increasingly large drives disconnected for backup purposes. is this wise? what if the largest drive crashed?

the other consideration is big drive support - i know some motherboards/bios have trouble over a 137gb barrier - is this an issue at all with raid?

what might you suppose the minimum motherboard/cpu requirement for a box like this would be? the only other hardware involved would be the raid controller, a network card and memory.

finally for the software i am planning on using a very old 2gb disk for the operating system (running independently to the raid array?) - ideally i'd like the keep the os as simple as possible. can anyone recommend a compact linux distribution that can handle the functions listed above (network/raid/smb) to save me some work?

many thanks for any help offered!

[indybrett]

If you use a hardware RAID card, it should support the largest drives, so it won't matter if the motherboard does.

However, if you want to use a fault tolerant RAID configuraion, like RAID1 or RAID5, you should use drives that are the same size. At least two that are the same size for RAID1 (mirroring), or at least 3 that are the same size for RAID5.

edit: typos

This post has been edited by indybrett: Nov 26 2003, 16:15

[indybrett]

This may help:

http://www.redpathgroup.com/about_raid.html

FWIW, I use a 3Ware IDE RAID card, with 2 mirrored sets, in a Redhat 7.1 Linux server running Samba.

[TwoJ]

I have been thinking of having the same type of computer - one dedicated to servers & storage.
I don't think you need anything too powerful, PII system should suffice, AMD 1200 or better.
Your raid system is a problem because your disks are all odd sizes - i believe that for different size disks the raid will only use the size of the smallest disk in the raid.
You might be able to pull it off in a software raid setup but you'll have to check that out .
Large Drive support you'll need to check the MB manufacturer to see if their BIOS will support it.
What you might find useful is to get a NIC that supports Wake on Lan (wol) so that you can power up that server from any of the other computers on the network - start up/shutdown remotly - nice feature so that you don't have to have it running 24/7 and run across the house if someone turned it off
I would think a redhat edition might be good - not as compact but you also want hardware support and stability.
my $0.02

[sgreen]

thanks for the link indybrett, very helpful.

perhaps raid is not the answer to my problem. for a couple of years now i have been buying increasingly large hard drives for storage. my normal practice is to copy everything from the last hard drive to the new one, then box up the last one as a backup. as you can imagine i have a lot of storage space sitting in boxes (with many duplicated backups)

i am filling the drives with music increasingly quickly, when the 160gb drive is full i will probably get a 300gb+ and use the 160gb as backup - as you can see this situation is going to get very costly and out of hand if drive technology/prices can't keep up!

as raid5 requires 3 hard drives of identical capacity it will not allow me to easily and cheaply increase the capacity of the system, jumping from 3x160 to 3x300 for example

is there any system that meets such a need? or will i have to continue to buy increasingly huge drives, consigning the old ones to boxed backups? this system is effectively doubling the cost per meg everytime i double my capacity as half the drive is filled with data indentical to the backup drive.

i'm sure other people here will have dealt with a similar situation, is there a solution?

thanks again!

[indybrett]

You might consider a DVD burner for backup. That's what I do these days. Not the most efficient way to go, but it does the job.

Edit: For some, the reliabilty of DVD-R media might be a concern. It isn't for me.

This post has been edited by indybrett: Nov 26 2003, 16:55

[Steve]

First of all, skip the RAID idea. You don't need it.

Secondly, you need to break down your data usage for us.

That 160gb drive that's filling up - what's it filling up with? Music? Important documents? Video games and programs?

If I were you, I'd put all of your tunes, archives, and generally static data on the file server you're talking about putting together. Oh, and any old pentium box with 16mb memory has enough power to simply serve files but you might not be able to find one that can support larger hard drives. It's not too much of a problem when you consider that a random pentium 2/k6 box probably supports large hard drives, costs marginally more and yet is still very cheap.

On your main machine, just put your operating system(s), programs, games, working files. Depending on your habits, the drive in your main system could be anywhere from 2gb to 40gb or more - mine would be at about 20gb.

So, in my case, I would have a 20gb drive in my main computer and I would put all the other drives in the file server, and then simply expand the file server as my storage needs increase.

Now backups, that's something else completely.

What do you need to back up? Are you backing up all of your data? Are you backing up stuff you downloaded/ripped and can easily reobtain? Are you backing up important documents?

Each of these conditions have a different reasoning and different best-fit systems. If you were only backing up important documents and other things, you could probably just burn 'em to a few CD/DVD's. If you were backing up unimportant data, you might want to ask yourself why - but, if you really wanted to go through with it and throw economy to the wind, you could buy another replacement drive and continue your I-keep-my-money-in-anti-static-bags idea or start buying tapes.

As far as your ever-increasing need for hard drive storage - what are you storing anyway? A lot of people just store anything and everything they can get their hands on - are you one of them? Is it worth your money to store these things?

Do you really need all that data? It really doesn't sound like these files are work-related or otherwise needing financial protection, but that's only a guess.

This post has been edited by Steve: Nov 26 2003, 17:36

[2Bdecided]

There's a very simple+cheap solution: stop using FLAC!

The current "only back-up half the data" backup solution doesn't sound great. Using an old HDD you might have worn out (OK, unlikely) isn't a great backup either.

FWIW the first piece of advice isn't meant to be stupid: I would love to use lossless, but it's just not practical for me. Maybe you're about to realise the same thing. Using lossy, I've calculated that I can back things up properly (and quite cheaply). Using lossless, I can't even justify the HDD cost for the "in use" version, never mind a back up!

Some people might worry about the occasional (imagined) artefact, or slightly sub standard transcoding. I worry that one day I will need that complete backup, and in the meantime 5x less cost on storage, and 5x quicker backup are very handy.


If you are serious enough to use FLAC, then you probably need to calculate the costs of a real backup solution. An HDD in a box is a good one, but make it a new HDD. Raid stays up to date, but wears out at least 2 HDDs. DVD-R is good, but it's better to swap one HDD rather than 30 DVD-Rs when restoring.

Sorry I can't think of a good, cheap solution for lossless. Maybe there is one, but lossy does it for me. I think

Cheers,
David.

[leland]

Well, there is a good cheap backup solution for lossless - your original cd's.

It will take a long time to rip them again, but if you don't value that time as worth a few hundred dollars, then that's what it is.

I'm thinking of going the RAID route myself (there are a few threads on this in audio hardware, including one I created that died), but the cost is prohibitive.

As for me, I'll probably begin to take the risk of running raid5 and having two drives die at once...Of course, I am undecided whether to shell out the money for a real hardware RAID card. I know that they are worthwhile, but the cost of one is the cost of 2-3 hard drives, which is just ridiculous. I'll have to think about that one.

I don't consider an external hard drive a true backup...RAID can provide redundancy, and the original media is the true backup.

When the time comes to upgrade, you can either buy a new set of drives to expand the array, or, more likely, invest in a whole new server with 1 tb drives or something crazy like that.

[jtclipper]

RAID 5 = expensive SCSI disks and at best 73 Gb each. out of reach and sure not for a simple home server.

You could go IDE RAID 0+1 with an add on card ( promise maybe) and 4 300Gb ( maxtor maybe) HDD's that will be almost 600Gb per stripe and have automatic mirroring ( thus backup ).

In case of expansion you could buy another set like this (IDE card 4 HDD's), but I doubt you will need this for mp3's only , if you really want FLAC use DVD's to burn them , unless you have the original CD's.
The problem will be to find a nice case to place them all inside and a corner in the house out of reach cause disk drives make noise
Don't use an old disk for the OS get a new 20Gb disk instead, if you want to make a server, make a decent one.

I would use Win2k server with NTFS and I sure would buy a P4 ( they are cheap now...) with a board that has a NIC on board a Giga Nic that is and a Giga switch ( netgear maybe is rather cheap for 4 ports..) you might need the gibabit network if a lot of copying will take place to update/manage your collection.

Alternately and just as a fictionary scenario
you could use NAS ( Network attached storage ) if you can handle the cost. This would be ideal because it is tweaked for file sharing has RAID-5 by default and huge capacities

[Fandango]

I would use Win2k server with NTFS and I sure would buy a P4 ( they are cheap now...) with a board that has a NIC on board a Giga Nic that is and a Giga switch ( netgear maybe is rather cheap for 4 ports..) you might need the gibabit network if a lot of copying will take place to update/manage your collection.

Erm, he said inexpensive.

A p-II or even pentium I like cpu + 100mbit is alll you need for handling simple filesharing. I myself use a pentium 120 with 48mb and a 100mbit nic as a file server and internet router running debian linux and samba. This box did cost me 0$, except for the HDs.

His minimum requirements are more likely to be set at the motherboard's HD size limitations.

[Mike Giacomelli]

I have a file server for my music. Its an old P2 that someone was throwing out. I threw in a new disk and a 10/100 NIC. Cost me less then $100 and its really handy. The best part is I can keep throwing disks into it as I run out of space. Right now I've got a total of 3 networked disks plus local storage.

RAID seems exceedingly useless for your purposes, you need effective backup not redudancy (they're not the same). I'd look at buying new disks for your primary machine and the networked one. Then keep a copy of your music on each. This way if you get hit with a worm that flattens your disks, at least you'll still have the fileserver. That or a firewire enclosure.

[jtclipper]

Huge and failsafe storage is not so cheap.
Linux is cool if you are good at it Win2k is easy and powerfull but costs money.
Giga bit networks are the same costwise as 100 these days.

[jrbamford]

I recently lost a lot of valuable information when a harddisk crashed (actually i've since discovered that it was because i had a 150gig drive in win2k which doesn't support larger 48bit disks > 137gig without SP3 AND a registry hack applied) ... this spurred me on to creating a backup machine which i've nearly finished.. I'm using linux, 2x150gig samsung 8meg cache ATA hdds, a promise ultra 100 ATA controller, and a software RAID 1 configuration with a p3 450 machine with 400meg memory...

After figuring stuff out linux isn't too bad.. sofware raid works quite well.. i can't see why you could possibly want to use raid5 when you can use raid 1.... do your really need that good a performance off these drives.. I use normal drives to stream 3.5gig/hour DVD quality captured tv across 100meg lan and its fine... flac, waves etc should be good up to a fair few simultaneous streams...

The only thing to consider is to make sure that if RAID is being employed they are on seperate channels, so although my ata card supports 4 devices on 2 channels in a decent raid config it only supports 2 drives.. this is because with drive failures you can take down an ide channel which may ruin any chance of vital data reaching the backup drive on that channel..

This is where hardware raid cards are good as they usually have seperate channels for each drive... a sata card is mine next with 8 channels perhaps.. but unless the requirements for intense hdd access increases I will use multiple raid mirrors to get a bigger backup serving space...

Another thing of software over hardware RAID is that when a drive fails the array just works.. you can move the array to other linux boxes easily.. also you can put a component drive of a mirror array in a computer on its own and its a valid snapshot the mirror.. hardware RAID usually requires access through the controller only, and if your card dies you may get compatbility problems which could sabotage your backup set...

In short harddisk probs do occur, tho if you have your cds to hand you can obviously rerip.. if its downloaded the biggest problem is losing the actual stuff you had.. if you dont remember what you had you can't reget it easily.. RAID1 should be fine for this kind of usage, and if you are brave non-raid will probably be fine too

[TwoJ]

You could go IDE RAID 0+1 with an add on card ( promise maybe) and 4 300Gb ( maxtor maybe) HDD's that will be almost 600Gb per stripe and have automatic mirroring ( thus backup ).

RAID has NOTHING to do with backup - it is redundency - throw your Raid HDs in the garbage and you'll find out the difference between redundency and backup. Your suggestions are overkill for his needs.

I would agree with others & suggest that if you have the original CDs then keep those as the backups, if not then maybe keep the flacs in an offline storage and keep some form of lossy on your Raid setup.

[sshd]

Here is my suggestion:

You need two servers. One for primary storage and one for backup. You could possible go as low as Pentium75/16 Mb RAM, but I suggest Pentium2/K6-2 with 64-128 Mb RAM as minimum requirement.

The primary server has the 160 Gb drive only. Partition in a 1-2 Gb system partition and the rest as data. Note that only new motherboards can boot from a 160 Gb disk. If motherboard does not support it, you can boot from CD (requires a little know-how). Alternatively you can add a 1-2 Gb hard drive as boot drive. That will not ruin the budget.

The secondary server has the remaining 20, 60 and 80 Gb disks. Notice how 20+60+80 conviently add up to 160. The 20 Gb drive will be your boot drive. Partition 1-2 Gb as system partition and create a RAID0 device using the rest + the other disks. Note: If one disks in the backup fails, you loose everything!

Operating system: Linux is a good choice, if you can manage. Otherwise WinNT Workstation 3.51/4.0 should do nicely on old hardware. You will need a more powerful server if you want Windows2000+ and there is really no need - just a fileserver.

Backup software: rsync works very well under linux. Might work under Windows as well. Otherwise xcopy might work. NT also has a directory replication feature. Maye this will do.

How to use this system: Primary server is always on (or most of the time anyway). Turn on secondary server and sync every 1-2 weeks.

This setup has an important flaw - a lot of stuff can happen in 1-2 weeks. If the primary disk crashes, you might lose data. If you can afford it, add another 160 Gb disk to primary server in RAID1 (mirroring).

There's a very simple+cheap solution: stop using FLAC!
....
Using lossless, I can't even justify the HDD cost for the "in use" version, never mind a back up!
...
Some people might worry about the occasional (imagined) artefact, or slightly sub standard transcoding. I worry that one day I will need that complete backup, and in the meantime 5x less cost on storage, and 5x quicker backup are very handy.

I have a better suggestion. Stop buying new CDs and DVDs. I could buy a new 200 Gb drive every month for the money I save. Buy used if you *must*. This advice is not meant to be immoral, only practical.

There are so many things that can go wrong with a fileserver. RAID1/5 is only useful when a disk crashes. A windows virus, backdoor, plain stupidity, human error, software errors (like metaflac 1.0.4) can easily destroy your collection.

I cannot see why the 5x quicker backup speed should be relevant, unless you live in one small room.

[_Shorty]

RAID 5 = expensive SCSI disks and at best 73 Gb each. out of reach and sure not for a simple home server.

Wrong.

http://www.promise.com/product/product_det...d=84&familyId=7
http://www.promise.com/product/product_det...=112&familyId=2
http://www.promise.com/product/product_det...=111&familyId=2
http://www.promise.com/product/product_det...d=94&familyId=2
http://www.adaptec.com/worldwide/product/p...RAID%2fATA+RAID
http://www.adaptec.com/worldwide/product/p...Serial+ATA+RAID
http://www.3ware.com/products/serial_ata.asp
http://www.3ware.com/products/parallel_ata.asp

And I'm sure there are other companies making them too. Not to mention that you don't even *need* a card like this to use raid5, since you can do it via software with linux and windows. Though obviously the hardware card would be the ideal solution. However, given that in this case he doesn't care about how bad server resources are hit given his intended usage, software would be more than sufficient and would be cheap since nothing would have to be purchased. At least controller-wise. He would still need identical drives to keep from wasting space.

[_Shorty]

i can't see why you could possibly want to use raid5 when you can use raid 1

because raid 1 requires you to buy twice as much storage as you actually want. That's damn expensive. However, raid 5 only requires that you buy one extra drive. If I want 800GB of storage then I only need to buy five 200GB drives with raid 5. With raid 1 I would need to buy eight 200GB drives. Is raid 1 safer? Sure. Is raid 5 safe enough for this type of thing? Sure.

[Audible!]

RAID has NOTHING to do with backup - it is redundency - throw your Raid HDs in the garbage and you'll find out the difference between redundency and backup.

Make a RAID 1 array with two discs. Fill with data.
Remove one of the discs.
Replace the removed disc with another identical drive and rebuild the array.
Put removed disc in safe deposit box.

You now have 100% redundancy and a complete backup.

This is probably too expensive for this situation.

[bawjaws]

If I was doing this (and funnily enough, I am) then I would find a like-minded soul (or two, in fact the more the merrier) and get them to host an off-site mirror of your rips, while you do likewise for them.

[_Shorty]

This is a home server hosting audio files for his own use. This is not a mission critical server running alongside dozens of other mission critical servers in a server room. While certainly not out of the realm of possibilities, chances of losing more than one HD in this environment are pretty slim. I'd have to say raid5 will more than suffice and will be relatively cheap to implement. A little more so if a hardware raid5 controller were used, but this is far from necessary for the job at hand. Software raid5 will do nicely for the load it will incur, and when a drive dies just pick up another one to replace it, running happily all the while. Hot swap won't be a concern, this is a home server holding the guy's audio files. Shut it down, replace the drive, fire it back up, no big deal. Giving the guy advice that makes sense for a mission-critical "It can't go down or we'll lose millions of dollars an hour" situation makes no sense at all for a home audio server. My advice, sell whatever drives you don't need to get money towards picking up similar sized drives and make yourself a software raid5 array with three drives. It will be way faster than you'll ever need, and I honestly don't think you'll ever be in a situation where more than one drive will die at a time. It happens, sure, but not often enough to ever worry about at *home*.

[Dologan]

Sheesh, all those suggestions about three-drive RAID5 and stuff make it seem like HD were disposable, made of crystal or just happened to die every other day for no reason at all. In my 7 or so years of having owned computers and used them somewhat intensively I have never had an HD die on me. Perhaps I've just been lucky, but I also haven't heard of many people with continuously failing HD.
Wouldn't just an external drive of reasonable capacity which spends most of its time offline be a reasonable backup? What would be the half-life of such a drive or the chances of it dying?

This post has been edited by Dologan: Nov 27 2003, 00:45

[indybrett]

This whole thread is utterly amazing, and depressing. All he wanted was some options for a simple, cheap, home file server. Good grief. I'm suprised I haven't seen anyone demand that he needs to replicate his server 1000 miles away in case of a local atomic bomb detonation.

I have exactly what he asked about. An old motherboard, with a $120 IDE RAID card, and some inexpensive IDE hard drives. It works fuggin great.

If he wants a backup system, then that's going to complicate things, but not too much.

Edit: backup part

This post has been edited by indybrett: Nov 27 2003, 02:01

[jtclipper]

Actually I had a disk die on me last week and I had to rebuild the PC from scratch it was a Seagate SATA 120HDD in a RAID0 config. I have all my data also in another PC in my home network so I just copied over 170 Gb of data.
Losing mp3's is something that will make me crazy so I have this failsafe setup 2 PC's and I sync them over the network.
But I like the IDE RAID-5 idea, it seems like a good thing ( if you can afford it ) buy a card pop in a few disks ( like 6 250 Gb HDD's) and stop worrying about it.

[Rotellian]

I think inexpensive lossless can be done. I would use wavpack or optimfrog to squash it losslessly into 2 files - one that plays, but is still quite small (say roughly 4:1 compression) and is all but transparent and the correction file to restore with. This gives reasonable usage of space on the main storage area (whatever that maybe, raid or otherwise). The correction files can be stored elsewhere - possibly cd backup / dvd +-r or maybe purchase an old tape backup like a HP surestore with 10/12 gig tapes, from ebay or somewhere (nice and cheap). This last option could be used for both backups and storage of the restore files. Guess it all depends on how paranoid you are. I think a simple raid1 with this as backup would work quite nicely. In fact, i think might get one of these myself......