IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Parallel FLAC encoding via windows frontend, to use all my CPU cores
jrbamford
post Aug 16 2007, 00:21
Post #1





Group: Members
Posts: 308
Joined: 1-December 01
Member No.: 569



So since getting a new machine my flac encoding is quicker.. could be quicker though.. I use the default flac frontend, but when this is encoding i only ever see 50% CPU usage.. ie. one core is twiddling its thumbs.

Anyone got a good strategy to do two flac files at the same time? Is there an option in the frontend to do more than 1 at a time?

One other thing, did something change in the FLAC format over the years.. in the old days the tagging at the end of the encode task used to be like lightning.. now even with a 5x faster machine its now quite slow taking a second or so per track. Just curious on that.

Cheers


--------------------
Binaural recordings of mine: http://binaural.jimtreats.com
Go to the top of the page
+Quote Post
Egor
post Aug 16 2007, 04:33
Post #2





Group: Members
Posts: 826
Joined: 29-September 04
Member No.: 17374



QUOTE (jrbamford @ Aug 16 2007, 06:21) *
Anyone got a good strategy to do two flac files at the same time? Is there an option in the frontend to do more than 1 at a time?

foobar2000 will launch one encoder per CPU on encoding, it supports Unicode, FLAC/OggFLAC tagging. Just install, unpack flac.exe to the foobar2000 program files dir, and use the built-in Converter feature.
Go to the top of the page
+Quote Post
greynol
post Aug 16 2007, 06:33
Post #3





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



All hail foobar2000! Now wouldn't it be nice to be able to disable this feature as it is a complete joke when trying to convert files stored on optical media.

Back on topic, I personally run two two batch scripts at the same time, but would definitely recommend foobar2000 (just not for stuff stored on optical media). I really don't have a favorable opinion of FLAC frontend, FWIW.

This post has been edited by greynol: Aug 16 2007, 06:35


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
jrbamford
post Aug 17 2007, 00:15
Post #4





Group: Members
Posts: 308
Joined: 1-December 01
Member No.: 569



Thanks guys... I've never really sorted out foobar to encode flac, i'll look into it... can't beat the near 2x speed up it should give me.

I don't mind the flac frontend that much.. i always worry with foobar that some noise shaping, dithering etc is on, or something like that.. something you dont have to worry about with the FLAC tools as they're not available options.


--------------------
Binaural recordings of mine: http://binaural.jimtreats.com
Go to the top of the page
+Quote Post
djstone86
post Aug 17 2007, 01:39
Post #5





Group: Members
Posts: 30
Joined: 11-November 05
Member No.: 25697



QUOTE
(just not for stuff stored on optical media)


why not?

This post has been edited by djstone86: Aug 17 2007, 01:53
Go to the top of the page
+Quote Post
Tahnru
post Aug 17 2007, 02:08
Post #6





Group: Members
Posts: 325
Joined: 17-October 05
From: United States
Member No.: 25178



QUOTE (djstone86 @ Aug 16 2007, 19:39) *
QUOTE
(just not for stuff stored on optical media)


why not?


Because of seek times/bandwidth. At some point the bottleneck moves away from the CPU and the bottleneck becomes the source instead. Optical media suffers greatly when doing lots of random seeks across the discs surface, as would be required for reading two files at once.

It would be a wonderful to have finer control over the multiple core encoding - I could specify two seperate sources and have them spit out back to a third unit (two optical drives with the result to a external hard drive, for example).
Go to the top of the page
+Quote Post
djstone86
post Aug 17 2007, 02:18
Post #7





Group: Members
Posts: 30
Joined: 11-November 05
Member No.: 25697



oh thanks the explanation, one more q off topic, does flac 1.2 take advantage of sse4?
Go to the top of the page
+Quote Post
Tahnru
post Aug 17 2007, 02:35
Post #8





Group: Members
Posts: 325
Joined: 17-October 05
From: United States
Member No.: 25178



SSE(x) support depends on the compiler and settings used when building the files.

Because the Peryn-cored Intel chips aren't out yet (I think?) and the Intel Compiler v10.0 is really new, I don't think there are any SSE4 capable binaries floating around out there. I am quite ready to be wrong about that statement.

Short finish - if the FLAC binaries available now don't support SSE4, they soon can/will.
Go to the top of the page
+Quote Post
Moitah
post Aug 17 2007, 03:36
Post #9





Group: Members
Posts: 193
Joined: 5-June 02
From: Virginia Beach, VA
Member No.: 2227



The only problem encoding multiple FLACs at once is they can come out very fragmented. I wrote a quick test app to encode using a configurable number of threads at once. If I only encode 1 at a time, the files are okay (< 10 fragments per file), but with 2 at a time the files come out in hundreds of fragments. I tried foobar2000 also which runs 2 encoders at the same time and the files are just as fragmented. For some reason I had the idea that NTFS would be smart enough to avoid fragmentation when writing multiple files at once, but apparently not. Of course, my drive is 95% full (~12/232 GB free) but I think the free space is mostly consolidated. I'll try it on a freshly defragmented drive with tons of free space and see what happens.
Go to the top of the page
+Quote Post
Tahnru
post Aug 17 2007, 05:20
Post #10





Group: Members
Posts: 325
Joined: 17-October 05
From: United States
Member No.: 25178



It will still fragment. I have the same problem, however, I defragment when I sleep.
Go to the top of the page
+Quote Post
Moitah
post Aug 17 2007, 05:26
Post #11





Group: Members
Posts: 193
Joined: 5-June 02
From: Virginia Beach, VA
Member No.: 2227



Even on a drive with 108/232 GB free, completely defragmented, and the free space consolidated, the same thing happened. When encoding 2 at a time, the average file came out with 230 fragments.
Go to the top of the page
+Quote Post
Tahnru
post Aug 17 2007, 17:02
Post #12





Group: Members
Posts: 325
Joined: 17-October 05
From: United States
Member No.: 25178



Although I haven't tested to see what is happening, my guess is that the two encoder processes alternate writing blocks to the hard drive. Instead of fragmentation where the files are randomly all over the surface of the drive, I would expect to find that this fragmentation is interleaving the blocks from each output file.

Seek time is the killer with random fragmentation. My thought is that a person may not have to worry about this particular mechanism of fragmentation.
Go to the top of the page
+Quote Post
odyssey
post Aug 17 2007, 17:14
Post #13





Group: Members
Posts: 2296
Joined: 18-May 03
From: Denmark
Member No.: 6695



QUOTE (jrbamford @ Aug 16 2007, 01:21) *
So since getting a new machine my flac encoding is quicker.. could be quicker though.. I use the default flac frontend, but when this is encoding i only ever see 50% CPU usage.. ie. one core is twiddling its thumbs.

Anyone got a good strategy to do two flac files at the same time? Is there an option in the frontend to do more than 1 at a time?

Hint: Hit "Encode" twice wink.gif


--------------------
Can't wait for a HD-AAC encoder :P
Go to the top of the page
+Quote Post
Moitah
post Aug 17 2007, 20:18
Post #14





Group: Members
Posts: 193
Joined: 5-June 02
From: Virginia Beach, VA
Member No.: 2227



I used my own IO callbacks instead of letting libFLAC handle the file writing so I can preallocate the file (SetEndOfFile Win32 or FileStream.SetLength .NET). I allocate it to the size of the raw uncompressed audio data and then trim the file to the correct size when done writing. This completely solves the fragmentation issue. I don't know if I will polish and release this, maybe if anyone really wants it.
Go to the top of the page
+Quote Post
Tahnru
post Aug 19 2007, 22:40
Post #15





Group: Members
Posts: 325
Joined: 17-October 05
From: United States
Member No.: 25178



Perhaps it is a useful patch that could be added to the FLAC encoder as a command-line switch? Something like -pre to preallocate output files if a person wants it.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 16th April 2014 - 15:20