Help - Search - Members - Calendar
Full Version: Parallel Ripping Questions (newb)
Hydrogenaudio Forums > CD-R and Audio Hardware > CD Hardware/Software
maxwellsdemon
Greetings. I'm brand new to Hydrogenaudio and digital audio issues in general, but I am a computer engineer so I'm not a total newbie when it comes to programming and DSP methods. That being said, after looking around online for awhile and reading a bunch of FAQ's to get a handle on all the techno-jargon related to digital audio, I have a number of questions that I'm hoping some people here can answer or point me in the right direction.

Are there any mp3/wav/etc ripping/encoding tools that can be used with multiple optical drives to rip multiple CD's simultaneously?

Are there any ripping/encoding tools that work in parallel (maybe on a Linux cluster)?

I discovered the LAME encoder in my search... is this only an encoder or is a CD ripper as well? Does anyone know where info exists on how to use LAME as the backend encoding for another program? Basically I was thinking of writing some code and I wanted to use LAME instead of having to re-invent the wheel.

Any additional info/FAQs/sites on how to actually create a program to rip a CD (eg how do you interface with and sample from the optical drive, etc)?

Any help is greatly appreciated.

smile.gif
paranoos
Multiple instances of CD rippers will do the trick for you. There isn't anything special about ripping two CDs at the same time using two different drives and two instances of your ripper software. Most modern rippers will allow you to set up an external program to encode the files after or during the ripping process.

For example, in Linux you can use cdparanoia to rip your cd and encode at the same time by using a pipe, or afterwards with simple scripting.
Zoom
QUOTE
Are there any mp3/wav/etc ripping/encoding tools that can be used with multiple optical drives to rip multiple CD's simultaneously?


In my years of ripping I have never ran across such a beast. I imagine one way you could do it is to go SMP and have two versions of the ripping program running on different CPU's. I don't know what the load would be like on the bus, or even if that is practical.

QUOTE
Are there any ripping/encoding tools that work in parallel (maybe on a Linux cluster)?


I know that Grip does ripping and encoding in parallel. I'm not sure if you can set it up with a cluster though.

QUOTE
I discovered the LAME encoder in my search... is this only an encoder or is a CD ripper as well?


Lame is an MP3 encoder. (Well actually it isn't tongue.gif ) It does not rip CDs. (If it does the developers put one by me.)

QUOTE
Does anyone know where info exists on how to use LAME as the backend encoding for another program?


winLame does this. I believe it uses a .dll to implement the encoding.

I guess I kinda answered some of your questions. I don't know, there are plenty of people on here that are more knowledgable than I however.
westgroveg
Depending on you IDE setup you could have transfer errors, Feurio is a very informative tool that will report system problems that could effect audio extraction.
minix
QUOTE(Zoom @ Mar 6 2004, 04:50 AM)
QUOTE
Are there any mp3/wav/etc ripping/encoding tools that can be used with multiple optical drives to rip multiple CD's simultaneously?


In my years of ripping I have never ran across such a beast. I imagine one way you could do it is to go SMP and have two versions of the ripping program running on different CPU's. I don't know what the load would be like on the bus, or even if that is practical.

That can be done with Feurio! since 1998 (and there's no need to open several instances). I guess there must be more programs able to do that...

I remember the author of Feurio saying that LAME is not designed to run 2 instances of it at the same time. I don't know if this is true, or if it's true nowadays.
Tec9SD
Multiple instances of EAC work smoothly (last time I used it was with version 0.9 Beta 4, but newer versions should be the same).
My suggestion is to make a copy of the initial EAC folder and run each session from its own directory, I've found this to be the most suitable way.

With the latest builds of EAC you can suspend the compression queue and either run a couple lame CLI encoders at once while you are away or if you really wanted to you could write a script to merge the various compression (enqueue.dat) queues - restart EAC to reload the queue and encode.

There are many LAME front-ends for every OS and in a wide variety of languages.
What were you thinking of using?

As far as a ripper that works sequentially, I do remember one, but it has been awhile since I last looked into a solution (a friend had 14 Plextors he was hoping to get working in EAC and offered to pay me if I could get EAC to do what he wanted laugh.gif).

It seems like the software I'm thinking of was based on CD Paranoia.
For utmost quality in DAE Exact Audio Copy defeats CD Paranoia, so I suppose it comes down more to which do you need more.

Goodnight, tec

I'd look back in to it tonight but it is quite late. I'll try to check back in on the thread tomorrow.
maxwellsdemon
QUOTE
That can be done with Feurio! since 1998 (and there's no need to open several instances). I guess there must be more programs able to do that...


I downloaded and installed Feurio and it doesn't include an Mp3 encoder. Apparently, they creator of Feurio would have to pay large fees to include an encoder. I suppose I could do a 2 step process of ripping to WAV and then using an encoder like LAME but I don't really want to if I can get around it.


QUOTE
Multiple instances of EAC work smoothly (last time I used it was with version 0.9 Beta 4, but newer versions should be the same).
My suggestion is to make a copy of the initial EAC folder and run each session from its own directory, I've found this to be the most suitable way.


I downloaded and installed EAC but I'm having problems. The main one is that I can't seem to get LAME compiled. The command I'm using from the directory where I extracted all the LAME files is:
CODE
nmake -f Makefile.MSVC comp=msvc asm=no


However I get the following error:
CODE

NMAKE : fatal error U1073: don't know how to make 'lipmp3lame/bitstream.obj'
Stop.


So I'm not sure what's going wrong. I have MS Visual Studio 6 installed on an XP box so maybe someone knows if there's an issue with that setup.

Secondly, EAC seems to be very Beta. None of the tabbed option have titles, etc. Also all error messages that I've encountered simply call up a pop up window with an ! but no text as to what the error is. Finally, I can't seem to get CDDB (or freedb) support working (I just get the !).

I've also read in Feurio's help, that using 2 opticals on the same IDE channel has big problems because the IDE standard has problems (2 simultaneously active devices that are not hard drives fight with each other). I'm not sure if this is true.

Perhaps I should explain why I'm asking all these questions... I'm trying to convert my entire CD collection to Mp3. However, I have in excess of 1300 CD's so doing them one at a time is going to take months. Not that each CD takes a long time to rip but I'm going to have to babysit the computer -- switching CD's, starting the ripping process, making directories for each album, etc. I also have 2 friends in similar situations (one's a DJ and the other works at a record store) - so between us we have almost 4K CD's to rip. However, I have a number of computers and a pile of CD ROM drives (probably around 20 IDE and 8 SCSI). What I want to do is to have multiple opticals ripping simultaneously on a machine so I can load them up and walk away. I also want to automate query CDDB, create artist folder, create album folder. I'm thinking maybe I should apply my coding skills and jump into the digital audio community and start a project on SourceForge for this.

Any thoughts?
Zoom
Oops, I guess we forgot to link to the LAME binaries.
Unless of course for some reason you want to compile lame yourself?

You can get lame here.

AFAIK the best version to use is 3.90.3. The versions after that haven't really been tested greatly up to this point.

And yes EAC is still in beta up to this point.
Pio2001
EAC is labeled beta, but I think that the problem you get comes from a language pack not being properly recognized. Where did you get it and how did you install it ?
Atreus
I'd spend a little more time trying to get EAC to work with your computer. Once you set it up to your liking it will automate all of the tasks that you want to do (query CDDB, create directories, tag and encode to MP3, etc.). Also, before you jump into this project, spend some time reading the FAQs on the forum and searching for your answers - that should help out a lot as you run into problems.

If you have enough room, rip to wav and then every 100 CDs or so just bulk encode them overnight into MP3s. If you don't have enough room, have EAC encode to MP3 during the extraction.

Unfortunately, with 4000 CDs it's going to take a lot of work, no matter how you look at it. Good luck! biggrin.gif

-Aaron
little a
QUOTE(maxwellsdemon @ Mar 5 2004, 20:08) *



Are there any mp3/wav/etc ripping/encoding tools that can be used with multiple optical drives to rip multiple CD's simultaneously?



did you ever resolve this situation? i've got the same q but i'd like to use optical drives in various cps connected over a network.

a
Teknojnky
Mediamonkey supports ripping multiple drives simultaniously, however it doesn't support 'secure' ripping which I'm sure some will consider sacrilege.

It does however support Lame and many other encoder types.

It is multithreaded and you even select which priority a thread runs at by right clicking the thread status at the bottom, so I am guessing that dual core/multi-cpu setups will work even better when ripping several discs simultaniously. (I don't have a dual core/hype-thread so I can't really confirm)
Klyith
QUOTE(little a @ Jul 6 2006, 08:06) *

i've got the same q but i'd like to use optical drives in various cps connected over a network.

That's more difficult. You need to remotely execute the ripping program on each computer, you can't rip via something like a shared drive. That rules out any ripper that can't be fully controlled via command line, unless you really feel like a fool managing them via vnc, that'd be overkill. Plus an added layer of complexity if you want to encode on a different machine than the ripping ones (say one fast smp system encoding for five old P2s). It's possible to pipe programs over network in *nix, but again, overkill.

Finally, I don't see any real motivation for "networked ripping" seeing as you have to be physically at the computer to change cds.
bhoar
QUOTE(Klyith @ Jul 6 2006, 12:41) *

QUOTE(little a @ Jul 6 2006, 08:06) *

i've got the same q but i'd like to use optical drives in various cps connected over a network.

That's more difficult.

...

Finally, I don't see any real motivation for "networked ripping" seeing as you have to be physically at the computer to change cds.


Most (all?) ripping programs use SCSI/ATAPI calls (even over USB/Firewire) to perform the data extraction. So, the ripping programs would either need to support a client/server type arrangement, where there would be puppet/slave I/O modules on the networked computers performing the scsi/atapi commands being controlled by a ripping-puppet-master on a central computer. There are no standard drives that do this under Windows, as far as I know. Maybe iSCSI could be adapted to do it?

However, most compressed formats require a significant amount of CPU, so I suppose the ripping-puppet-master could also have puppet/slave modules running on the networked computers as well. At that point, however, it would make more sense to simply run one instance of an independent ripping program on each machine.

There's sort of a halfway solution: Silex technology produces one or more USB-device-sharing (well, more device-borrowing since it's one user at a time) boxes that allow you to share printers, hard disks and even optical drives over fast ethernet. It uses a local virtual device client to mount/unmount the network USB devices, with the USB deice server running on a small embedded linux client. There may or may not even be some free tools to do things like this.

This is sort of nice if you have a wireless laptop and don't want to drag around a bunch of USB-tethered devices with you: just access them, slowly so slowly, via 802.11g.

...

In terms of multi-drive ripping programs, I know that Riptastic! ($20 shareware, trial available: http://www.riptastic.com/ ) has a batch mode that will automatically eject a disc (if the option is set) and look for any not-yet-ripped discs in other drives or newly inserted discs in any drives to rip. So it works pretty well with multi-drive computers, even though it's only ripping from one drive at a time.

A side effect (or perhaps it's part of the design) is that it works *really* well with the Nakamichi 5-disc SCSI changers that fit in a standard single 5.25" drive bay, since unlike the ATAPI version of the changer, each one of the SCSI changers shows up as five separate drives (via five different LUNs on the SCSI ID). So load a machine up with a stack of standard or compatible changer drives and you are still manually inserting discs, but you have to stop and do it less often. You have to disable auto-eject in this case, however.

Several of us have built 4-7 drive stacks of the nakamichis for unattended ripping.

Also, a robot-aware version of his software is bundled (or sometimes available as a $100-$200 option) for certain robot-driven disc duplicator products, which start at ~$800 and typically cost ~$1200-$4000 for the higher capacity units.

Riptastic! is also the only HTOA aware ripper that I know will extract "track 00" automatically when ripping to tracks. I'm less familiar with its error-detection/correction. Plus, chuck is a pretty cool guy and good about tech support.

Note: I'm not *trying* to sell his product, just mentioning good experiences as they may be applicable here. If that's a TOC violation, let me know.

For pure convenience, I'm setting up two rip-stations now with a few robots, some command-line clients that do specific CD-control/status and robot-control/status jobs, some AutoIT scripting, and a couple of ripping programs (Riptastic! and EAC). In terms of network control, the only network-specific things there will be a) running Remote Desktop so I can look at what's happening without having to sit in front of it and b) writing archives to a NAS.

-brendan
Klyith
QUOTE(bhoar @ Jul 6 2006, 13:24) *
Most (all?) ripping programs use SCSI/ATAPI calls (even over USB/Firewire) to perform the data extraction. So, the ripping programs would either need to support a client/server type arrangement, where there would be puppet/slave I/O modules on the networked computers performing the scsi/atapi commands being controlled by a ripping-puppet-master on a central computer. There are no standard drives that do this under Windows, as far as I know. Maybe iSCSI could be adapted to do it?

Oh god no, you don't have to do anything like that. If you run linux on all the systems, you use nc (netcat) to pipe program IO across the network. A simple example:

CODE
Each slave runs:
cdparanoia -w - | nc -w 3 <IP of master> <unique port>

Master runs (simultaneously):
nc -l -w 6 <port 1> | lame <enc options> - output1.mp3
nc -l -w 6 <port 2> | lame <enc options> - output2.mp3
...
nc -l -w 6 <port N> | lame <enc options> - outputN.mp3

Then you automate it (with much more complexity: the above example only rips whole cds to single files) with shell scripting or maybe some perl or python. The trickiest part is that you have to parse the TOC in your script before you start extracting tracks if you want 1 file per track. Elaborations like auto-ejecting when a cd is done and restarting when a new cd is put in are not as difficult. I'm pretty sure it would be possible to get to the point where all you have to do replace cds close the tray, and everything else is automagical. But I'm not good enough to do it myself...

QUOTE
However, most compressed formats require a significant amount of CPU, so I suppose the ripping-puppet-master could also have puppet/slave modules running on the networked computers as well. At that point, however, it would make more sense to simply run one instance of an independent ripping program on each machine.
I think that using lame, a fast dual core machine could handle the input of at least 4 slaves with modern fast dae drives, or more if they're old machines that can't extract faster than 10x. So if you have a half-dozen old P2-300 systems sitting around...

QUOTE
Several of us have built 4-7 drive stacks of the nakamichis for unattended ripping.
As I see it, you can probably spend the same amount of money and get the robot systems or the "ripping farm". The ripping farm would be (I think) a lot faster, and scales up better as well. However, you have to run around changing discs while it's working. The robot is slow, but you can set it up then leave it and do something else. But I could see someone who had the systems already doing the farm method, it would be free. Scale it to three masters and 15 slaves, you could go through 100 cds an hour or less.
bhoar
Oh god no, I agree, it was a silly idea. smile.gif

Anyway, I guess it all depends upon what you're trying to move, primarily, across the network.

1) general device I/O requests.
2) raw track data by sector
3) cooked, error corrected/checked track data (16-bit 44.1khz WAV data and the toc)
4) final encoded files (wav/flac/mp3 + cue/toc files) to a NAS as a final destination or for further processing.

I was assuming #1 using a) the windows environment and b) the use of a *central* ripping program that supported secure/burst/paranoia/etc. modes talking to remote drives or remote components of the application that handled the drives, which would be silly considering the other options available (as you describe).

Under windows, you'd probably want to do #4 on each machine since windows doesn't lend itself to the ease of network piping that your approach (#3?) would require. However, your approach probably works great on linux (assuming a properly configured cdparanoia, compatible drive(s) and some simple scripts). Of course, you can't currently do anything but #4 if you require EAC, though you can write to FLAC images and then post-process elsewhere.

In terms of what a single computer can do encoding-wise, I'm going with the assumption of the mid-range of the computers out in the field, not the current technology: one or two encoding threads for LAME seems about right. I don't have any dual-core CPU computers here and my last dual-cpu athlon burned out last year to run tests on.

The nakamichi solution wasn't too pricy, they were running ~$20 each on ebay last year. The case was ~$50.

I've moved on to the duplicators since those are just so convenient. If you're lucky, you can get a working 50-disc duplicator via ebay that can be reprogrammed (by you - using docs or an SDK) for $250. A bit more if you want it in guaranteed working order.

Be careful not to purchase a stand-alone duplicator for this (get one that is controlled by a separate PC). The standalone one typically allow you to attach a scsi cable to use the drives, but the robot has no external I/O interface since the box itself contains a cheap computer. If you do end up with one, you'll need to use a analyzer to monitor the serial I/O cable between the CPU board and the robot analog board and then build your own serial I/O interface. Probably best thing to do would be to break the two serial connectiors (on the cpu board, robot analog board) to new external connectors. That way you can easily analyze and, when you want it to work standalone, put a pass through cable on the unit. Should be standard RS-232 levels.

-brendan
BassBinDevil
This is kind of a bump, and an inquiry as to whether there are any new options for doing this. I see that Riptastic is now only available if you buy a particular changer, for example.

Since CDROM and CDRW drives are virtually free now, and I have a server case with a whole lot of drive bays... I'd like to put together a PC with at least 4 CD or DVD drives, which I can then fill with CDs each morning and evening, and let the computer rip them at its leisure. To avoid any I/O conflicts I want the discs to be ripped one at a time. Linux or Windoze is fine. Compression doesn't have to happen "in line".

Multiple instances of EAC might work if I could get them to run one after the other.

Any new suggestions? Thanks! (I can't be the only person that wants to do this...)

bhoar
QUOTE(BassBinDevil @ Jun 12 2008, 07:24) *

This is kind of a bump, and an inquiry as to whether there are any new options for doing this. I see that Riptastic is now only available if you buy a particular changer, for example.

Since CDROM and CDRW drives are virtually free now, and I have a server case with a whole lot of drive bays... I'd like to put together a PC with at least 4 CD or DVD drives, which I can then fill with CDs each morning and evening, and let the computer rip them at its leisure. To avoid any I/O conflicts I want the discs to be ripped one at a time. Linux or Windoze is fine. Compression doesn't have to happen "in line".

Multiple instances of EAC might work if I could get them to run one after the other.

Any new suggestions? Thanks! (I can't be the only person that wants to do this...)


My recommendation is to use dbpoweramp R13 Reference's Batch Ripper. That's what it is for.

Among many great metadata and multi-threading related features, it supports parallel simultaneous rips from multiple drives to one or more target codecs of your choice.

You could set it up for manually loaded drives as mentioned here:

http://forum.dbpoweramp.com/showpost.php?p...mp;postcount=54

Spoon also has drivers for the Sony XL1B changer (200 discs/1 drive) or the Baxter-type 25-disc robots.

In addition, should you have access to a serial-port controlled robotic loader (mediatechnics, mf digital, all pro solutions, ADR, microboards, etc.) I've made drivers available that cover most of them which are in testing or in production at multiple locations across the globe at the moment. smile.gif

-brendan
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2008 Invision Power Services, Inc.