Using tar for storing music and backups Options

[simonh]

Are there any thoughts about using tar instead of having the normal directory/audio files? I currently have one directory per disc. i.e

Some Artist - Year - Some Album (Disc 1)/track one.mp3 etc

I use foobar through wine on Linux and downloaded a component that allows reading of archives. If not tar, what about other archive formats? I see the following advantages and disadvantages:

*** Pros ***

Faster transferring of files (small files transfer slower than large ones)
Read only format (I think) - less likelihood of tags getting messed up
Less fragmentation due to less files on the hard drive

*** Cons ***

Lack of support by media players (?)
Slower seek times compared with unarchived files (?)


Anyway, if anyone has any advice or comments that would be great.

Edit: I'm suggesting each disc be tarred: Some Artist - Year - Some Album (Disc 1).tar
So there's no confusion.

This post has been edited by simonh: May 4 2012, 15:42

[polemon]

Faster transferring of files: Your music is already compressed, it makes no sense putting it into an archive for transferring (when you mean not just tar, but also compressed, things like .tar.gz, it makes even less sense, since that introduces additional headers and time for decompression, etc).

Read only format: In case you want to protect your files from being written, by tagging capabilities of the player software, simply remove writing permissions. I gather from your post, that use use some UNIXoidal system, in that case: chmod -w *[mp3|ogg|flac|mp4|m4a]. Having those files inside a .tar archive introduces no level of writing security. The tar libraries often support transparent file handling of contained files in a .tar archive.

Less fragmentation: Has absolutely nothing to do with file itself. Depending on the FS, you either have less fragmentation (for instance: Ext2 and later) or, you have a lot of fragmentation, (FAT32 for instance). Fragmentation occurs at the block level, never at logical file level or something. Defragmentation sometimes even destroys data. Filesystems like Ext3 are fragmenting practically only when writing large files very slowly.

Lack of support by media players: Certainly, since it makes no sense.

Slower seek times compared with unarchived files: It would have no effect on that, except if the archive is compressed. The .tar archive might be understood like an additional layer of FS. Remember: "tar" is an abbreviation for tape archive. It is essentially a filesystem designed for linear access.

Conclusion: I don't want to sound arrogant or something, but it seems that your lack of technical knowledge about those systems made you come to the wrong solutions for - well - nonexistent problems, really. I believe, what you need is a better music library tool. When transferring files over network or on other drives, I suggest using rsync. I usually use rsync -alPvz I sometimes don't use the "z" switch, which filters files through zip compression. Since your music is already compressed, you could only count on file deduplication, which is a feature of the FS.

[simonh]

polemon: thanks for the detailed reply. I'm not suggesting compressed archives. Tar only. I began thinking about it after transferring ~300 mp3's to a memory stick. The transfer rate was about 2mb/sec. After I tarred them (each album), the rate jumped to about 6mb/sec. I'm sure you know more about the reasons for this. My understanding is that larger files transfer quicker and that is my experience. I'm going to do some tests myself. Cheers.

[Porcus]

Well Windows Media Player is known to screw up your media files even if you think you are only playing (=reading), so it might be a point to have stuffed somewhere where WMP cannot reach it. Of course, that is solved as you use *n*x.

[polemon]

polemon: thanks for the detailed reply. I'm not suggesting compressed archives. Tar only. I began thinking about it after transferring ~300 mp3's to a memory stick. The transfer rate was about 2mb/sec. After I tarred them (each album), the rate jumped to about 6mb/sec. I'm sure you know more about the reasons for this. My understanding is that larger files transfer quicker and that is my experience. I'm going to do some tests myself. Cheers.

It depends how you transfer those files. Did you just use cp, or mv? or did you use something else? Try unison or rsync, it should increase transfer speed even more. Also, what filesystem do you use on the USB stick? Try using a FS that is optimized for flash media, maybe even Ext4 (which is a problem when using on Windows computers, mind you). Things like FireWire are optimized for large filestreams, but those are rather specialized, for things like camcorders, etc. where the data is stored on DV tapes, etc (which in turn is a linear data storage of course, with no real FS). You used an USB stick this time, but try an SD card next time or something, you should see a difference even when the SD card is connected via USB to your computer.

This post has been edited by polemon: May 4 2012, 17:01

[mixminus1]

IME, it has to do with directory access, and how slow that is (relatively speaking) on USB flash drives.

The more individual files you write, the more entries have to be made in the directory, and that's what seems to slow down the writing process.

I once had a 1 GB Lexar Firefly drive that I did some tests on as it was much slower than my SanDisk flash drives when doing write or delete operations involving multiple files.

I don't remember the exact numbers now, but simply deleting several dozen files took something like 3 minutes on the Lexar, but only seconds on the SanDisks (they were all formatted as FAT32).

@polemon: Given your comment on Ext4, I'm assuming that at least some of this behavior is due to inefficiencies in the FAT32 file system itself?

[washu]

Another issue causing slow writes on USB media is that usually write caching is disabled. This is because USB keys are often removed without flushing and dismounting the filesystem. If you are careful to always properly dismount your USB keys first then you can enable write caching and speed things up.

[polemon]

@polemon: Given your comment on Ext4, I'm assuming that at least some of this behavior is due to inefficiencies in the FAT32 file system itself?

More likely than not. The problem is, that the flash memory used are very fast when seeking, but passing through large chunks of continuous data is rather slow, when a lot of seeking is involved. SSDs had problems with random writes since their beginning. This problem shows on USB sticks as well. Now, the problem with FAT32 is, that it is very prone to fragmenting, meaning a lot of random writes. Ext4 uses b-trees and not a FAT block. Meaning every block contains the address of the second block, so there's less seeking back to a TOC as it is with FAT (hence file allocation table). Furthermore, Ext4 has some optimizations towards flash media. I don't know exactly what is optimized there, but it's definitely geared towards flash media as in SSDs, etc.

[polemon]

Another issue causing slow writes on USB media is that usually write caching is disabled. This is because USB keys are often removed without flushing and dismounting the filesystem. If you are careful to always properly dismount your USB keys first then you can enable write caching and speed things up.

Yeah, but then it takes forever when unmounting the drives, since those files are written to the memory. Also, when someone is uncareful enough to rip the stick out of the drive, without unmounting it (or when the cache is written only partly to non-volatile memory), it will more often than not, destroy the FS or at least cause some data loss.

[simonh]

I stand corrected:

*** 30 mp3's in two directories ***
simon@Jupiter:~/Test$ time cp audio -R /media/Titan/Incoming/

real 0m6.689s
user 0m0.007s
sys 0m0.433s

*** archives ***
simon@Jupiter:~/Test$ time cp audio -R /media/Titan/Incoming/

real 0m6.792s
user 0m0.008s
sys 0m0.385s
simon@Jupiter:~/Test$

Not significantly slower. Both filesystems ext4.

[skamp]

You probably need to run "sync" to time those commands correctly:

CODE

$ time { cp -R audio /media/Titan/Incoming/; sync; }

This post has been edited by skamp: May 4 2012, 21:26

[polemon]

CODE

$ time { cp -R audio /media/Titan/Incoming/; sync; }

Hmm, when write caching is disabled, it should take zero time to sync. Nice way to quickly test if caching is disabled, actually.