Hello to all,

I'm currently looking into converting a large amount of audio, and for this a normal PC simply isnt enough.

I would like to have a system which can convert from - and to - a lossless audio codec to a lossy audio codec.

Ie : wav - flac - * > mp3, wma, aac, *

I've been googling for a while now, but i guess the keywords i used were not good enough because i found no useful results.

Thanks for your input!

Why won't a PC do?

Even a modest dual core / CPU computer can encode to LAME MP3 at about 35 X real time when using one of the -V quality settings, and --vbr--new

Encoding to FLAC is even faster.

I'm not sure what you mean by "a normal PC won't do." But the poster above hit the nail on the head. The easiest and most cost effective way to accomplish this is with a PC.

Thanks for the replies.

A PC wont do , because i'm not talking about a small volume here.

Also, the solution i'm seeking is more in the "money is not an object" category.

IE: this is not for an amateur , but for enterprise level solution.

To get things more straight: the system has to be able to process audio as fast as possible, as the volume of tracks going through it is rather large - if not huge.
To my current knowlege, a serious server level pc will be outperformed by any kind of hardware which is dedicated for this task.

A PC based system will still be the most effective, considering there are no special hardware systems designed just for this task (to my knowledge atleast). Every system doing tasks like this are PC based.

Throw in a few Quad core Xeons or similar and lots of memory and configure it with the correct software, settings and optimizations and it will fly... Make a cluster just for this task, heck, set up a dedicated server rack for it and it should have zero problems processing lots and lots of files simultaneously.

Ummm... so PCs are pretty much almost the only devices that do that job with any appreciable amount of flexibility anyway...


Pro audio producers use PCs, so that makes no difference.


Just out of curiosity, how many files are you talking about? Not that it matters anyway - in the world of processing, if one PC won't do the trick, you throw more of them at the task until you get the performance you want. Google, for example, does not run on a single server

I mean, if money really is no object then at the very worst you're looking at a server farm for your needs. But you're really not very likely to find any standalone device that can do what you want otherwise.

ok, i guess then that i'll have to go in the software direction...
load balancing and a small dedicated cluster....

Volume will be somewhere in the hundreds of thousands of songs - at an initial stage, and following that "real time" transcoding will be needed, with a maximum delay of 5 seconds or less per track.


So, at this point, whats the best software options i have?
Formats i will need to process are:

WAV > FLAC

FLAC >
mp3
wma
ogg
aac
[pretty much any mainstream audio codec out there]


lol, i find it ironical somehow that the post under mine is "why is there no uncompressed download site" .

the quick answer is:

nobody is willing to give up the master recording so easily-cheaply.

Depending on your volume and long term usage, either dedicated high-end workstation(s) or servers. You would want multiple CPUs and hyperthreading.

Example:
Workstation: Dual/Quad Core Hyperthreading CPUs (highest speed) with 2GB RAM per CPU minimum.
Server: 8+ CPUs (as above) with RAM (as above).

This is depentant on your conversion software. Can it support Hyperthreading? Multiple CPUs and, if so, how many? I have a few different system configurations and dBpowerAMP recognizes all of them. Core Duo, Quad Core, etc, and even utilizes hyperthreading on a single cpu as dual CPUs (as an example).

You're going to have to plan it out. How much area do you have for the hardware? Are multiple high-end workstations better or fewer very large servers? Look at the the requirements for both setups, such as storage, cpus, ram and power (UPS units). If it will be around for a long time, I'd go multi-cpu server. If not, go multiple high-end workstations where they could be cut loose as the workload decreases and stabalizes. Plan it like you would any other server, just focus on cpus, espeically speed as it will be more crucial, and ram. Those will be your primary concerns.

I don't see much else different from any other enterprise setup from the server perspective other than the software requirements, which is the key. I would approach it from that end, focusing on cpus and ram.

Cool... an "encoding" farm. That's one I have not seen yet.

I'm really puzzled by what you're trying to do:
Any decent computer with multiple cores/hyperthreading can encode a hundred thousand files in a couple days or so. A Dual processor machine will do it even faster. You just have to be clever about setting up a script that will fork the appropriate encoding processes.

Real-time encoding is also an interesting thing to want. Just about every encoder out there works above 1x. and going from Wav to <codec> WHILE recording the Wav is codec dependent (and generally not recommended either).

sthayashi: i think that zane understood my problem more properly.

As far as realtime encoding is concerned, i misused this word.
perhaps a better term is on the fly, or simply put -now-. With lots of concurrent sessions. [tens, hundreds, thousands.]
I definitley will go with multi-cpu systems, and there will be at least 2 - 4 gigs of ram on each machine. perhaps even more, it really depends on the final application.

So, the professional version of dBpowerAMP seems quite much like the tool i need, supporting CLI operation.
What other options would i have over that?

Zane: how do you rate that software in usage on production webservers?
you have a WS > script > dbpoweramp > uploadscript [similar] architechture?

I'd agree: a rack with a cluster of multi-core PC-architecture workstation/server machines, using some of the highly-optimized forks of the various codecs (flake, lancer, etc.), seems like the right move. You could then custom script the extraction and encoding distribution processes.

Distributed-Lame ( http://www.agouros.de/dlame/ ) is also something you might want to take a look at.

Also abcde uses distmp3 ( http://packages.debian.org/unstable/sound/distmp3) to distribute encoding threads across many machines.

There's also the RipIT project: http://www.geocities.com/ukcave/ripit.html

Perhaps that'll get you started...let us know what you find.

-brendan

This sounds to me like the OP is looking for a solution to implement hardware for some kind of online music service with on-the-fly live encoding to different formats/settings (as per user's wishes) with as little delay as possible prior to downloading (as has been practiced before by allofmp3.com).

A multi-core cluster with lots of RAM, some Multi-Terabyte RAID storage and serious CPU power is what I see to suit your needs best.

bhoar:

i like that.
it shouldnt be a major hassle to route the request to the proper script and have the most optimized compressor handle the specific transcoding.

anyhow, off from work for today.

thanks to all.

jeanluc: ich bin nicht das Boese. deine intuition ist aber einigermassen korrekt. also machen wir das mal dezent, und mit gute tools, um sachen besser zu machen, fuer alle.

Also something to consider: if this is a commercial environment (i.e. you have money!), and your source is CDs and not some other media, you might be able to satisfy your initial load of data using a mediatechnics or mf-digital robot based solution, such as the one that the Ripfactory folks sell:

http://www.ripfactory.com/robotics.html

The MT4 supports queuing up to 600 CDs per load for processing.

In terms of the on-the-fly adding, do you mean to say "(extract and) encode while playing the audio", or do you mean "able to throw a bunch more CDs or files into the input at any time and have them automagically processed added to the output"?

-brendan

Honestly, I have not yet had the opportunity to see it on production webservers. So, I would have to defer to someone else here who, hopefully, has that information. From what I have seen thus far, I'm impressed.

[http://www.dbpoweramp.com/professional.htm]
Customized Development: dBpowerAMP uses a broad and well tested range of audio related code, contact us to discuss your companies requirements.

Industrial Ripper: The goal is to create an automated CD Ripper (using robotic loaders), ripping large numbers of audio CDs filling the needs of Radio Stations and companies offering a personal ripping service (even offering the uploading of ripped audio to portable devices, such as iPod). The goal of Industrial Ripper is to offer an end to end CD digitizing solution.

Network Encoder / Multi-processor Encoder / Multi Encoder: Often lossy audio formats take huge computer resources to compress, where there is a network of computers available the load can be spread around evenly with obvious speed increases when large numbers of files require encoding. Or for computers with more than one processor, ally a compression task to a specific processor.

I would contact illustrate (and/or Spoon) for more info on dBpowerAMP. I bet they would be very willing to work with you on your requirements. Should dBpowerAMP not meet your requirements, I would start looking at Bhoar's suggestions. Once you've got your software solution, I think the rest will fall into place.

Dang! Now I wanna go build an encoding farm and play! Sorry, can't help myself.

One more topic to add: if a requirement is ripping to CUE/WAV or CUE/FLAC image (instead of tracks) the current version of dbpoweramp (R12) does not do so (though perhaps an older version can?). That feature is slated for sometime between now and R13 (inclusive).

The second best ripper, EAC can do so, but the command-line options are limited. There's now a very good option for harnessing EAC via the command line on windows, which is to use `eaccmd`. The license for this automatically comes with cdtag and it does image ripping:

http://www.hydrogenaudio.org/forums/index....=20&t=49980

It's unlikely you have this requirement, but I just had to mention it.

-brendan

You can't be serious you are going to run a Windows OS on a large clustered server farm. Please contact some educated people with a little more experience in this kind of business.

You can do so, you just need design it so that random server outages, no matter what they were doing, do not cause problems in the workflow. This requires either dedicated monitoring machines with shared responsibilities or distributed monitoring.

Also, never forget the rolling weekly reboots!

...though, in my experience, Server 2003 isn't quite so bad as Server 2000.

I agree, though, that a stripped down non-windows OS (read: linux/bsd/*nix) would probably offer the most stable platform. The only problem is lack of ultra-secure ripping, if the clustered environment needs it.

-brendan

Input has been very interesting, although the DBPowerAmp option seems to be the most suitable to my requirements, supporting scripting and a certain degree of automation.

To hanky: WS stands for WebService, not for WindowsServer. but it could have been WebServer too. But yes, i'll be using a Windows Server.
Why?

http://www.hydrogenaudio.org/forums/index....mp;#entry471276

^^ this is a good answer.

"Half of the world" is using WMA, for whatever drunken reason. so you must support it properly.
And linux audio codec support, such as this propietary stuff, comes in a binary only fashion which is generally a few versions behind than the windows version. [or buggy [hint: ati glrx drivers]]

So since ill be getting binary anyhow, why not just do it the proper way?

Also, to my current knowledge, there is no ditributed ripping system available under *nix/linux , so that would set any project back a few months worth of developing. heck, even distcc has problems sometimes. figure out something that comes bundled with propietary codecs.

not a windows fanboy, but i had my share of experiences.

</rant>


so please, forget the windows/linux discussion, and rather bring it down to "this version of the codec is better than version xyz" . more useful.

greets

I'll say the same thing here I tell my friends and family members in IT about Windows (+MS).... "It gave you a career that puts a nice paycheck in your pocket, so don't completely knock it."

In Sethx's case, it's going to be what best fulfills needs at this time. Long term, someone can write their own flavor of anything. Heck, a major video company wrote their own rendering software because they could not convince the company that owned what they were using for years to modify it for their needs. That and support was going down the tubes (Non-MS BTW).

The professional capabilities weren't the only reason I mentioned dBpowerAMP. I also mentioned them because they appear an enthusiastic company. Meaning they will work with anyone in modifications specific to their needs. That kind of attitude brings better support and product. I consider every aspect before making such decisions, especially if it is for someone else.

[quot]
I'll say the same thing here I tell my friends and family members in IT about Windows (+MS).... "It gave you a career that puts a nice paycheck in your pocket, so don't completely knock it."
[/quot]

That, and if someone wants to flame, there is always usenet/yourfavouriteNG


[quot]
The professional capabilities weren't the only reason I mentioned dBpowerAMP. I also mentioned them because they appear an enthusiastic company. Meaning they will work with anyone in modifications specific to their needs. That kind of attitude brings better support and product. I consider every aspect before making such decisions, especially if it is for someone else.
[/quot]

youre probably helping someone out too. which is not a negative thing, at all.
I looked into their project a bit, and it seems very interesting.
If things evolve, they will recieve an email from me. nohing assured though.

Also, i'll be looking into other projects too, so if anyone has an idea of another product which can provide similar functionality, perhaps with a certain degree of certifications, and which can provide [sigh] drm support, post it here. </me really hopes this DRM crap stops, but, hey, its not us making the rules>

A couple of thoughts - depending on how deep you want to get into the code, you could

- Compile a 64 bit optimized version of LAME. That might give you a performance improvement.

- Consider offloading the LAME math to a high end ATI graphics card. This would requires some additional programming, but has the potential for a large speed increase.

At some point, the limiting factor will be the speed of your disks.

Not sure exactly what you mean. If you are thinking I know someone at illustrate, nope, nada. I heard about the app here on the forum not long ago and have just been testing out the trial for a few weeks now. I also use EAC and a few others, but I prefer apps requiring less configuration. dMc was getting good reviews here, so I thought it worth a look and try.

It does, but time will tell. It's the same for any app. Actually, it took me a few weeks to finally decide to look into it myself.
I wish you luck. You' have a big project to hash out and I hope it all comes together for you.