IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
CheckWavpackFiles 2.0 BETA released, Free 2-click Wavpack file scanner, now with multi-processor support
gl.tter
post Aug 8 2008, 17:56
Post #1





Group: Members
Posts: 48
Joined: 25-September 05
From: UK
Member No.: 24695



NEW v2.0 BETA2 Version - now with multi-processor/core support!

'CheckWavpackFiles' is a free standalone and easy-to-use Wavpack file scanner for Windows.

* check single Wavpack files, or all .wv files in a file tree or drive for errors.
* easy two-click operation via drag & drop or Explorer shell extension.
* bad files are written to a text file for reference/further processing.
* no command-line skills required (although you can use that too).

http://wavpack.gl.tter.org

This new beta version adds multi-core/processor support for faster scans & updates to Wavpack 4.50 code. Let me know how it works for you.

gl.tter

This post has been edited by gl.tter: Aug 9 2008, 11:43


--------------------
Author of -Wavpack4Wavelab- and -CheckWavpackFiles-
Go to the top of the page
+Quote Post
gl.tter
post Aug 9 2008, 11:43
Post #2





Group: Members
Posts: 48
Joined: 25-September 05
From: UK
Member No.: 24695



Updated to 2.0 Beta2.

Fixes several issues (display corruption, recycle bin folder enumeration & more).

gl.tter


--------------------
Author of -Wavpack4Wavelab- and -CheckWavpackFiles-
Go to the top of the page
+Quote Post
bryant
post Aug 12 2008, 05:38
Post #3


WavPack Developer


Group: Developer (Donating)
Posts: 1287
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



Thanks, gl.tter, this is very cool!

Guess I'll have to get a multicore system to test it thoroughly! smile.gif
Go to the top of the page
+Quote Post
esa372
post Aug 12 2008, 16:50
Post #4





Group: Members (Donating)
Posts: 429
Joined: 5-September 04
From: Los Angeles
Member No.: 16796



Excellent!

Thank you, gl.tter!

biggrin.gif


--------------------
Clowns love haircuts; so should Lee Marvin's valet.
Go to the top of the page
+Quote Post
gl.tter
post Aug 12 2008, 20:11
Post #5





Group: Members
Posts: 48
Joined: 25-September 05
From: UK
Member No.: 24695



Thanks guys. It's kinda fun to watch 4 files being checked at once.

Unfortunately there's a limiting factor, and that's the disk access time. With multiple files (and even worse if the disk is also fragmented), the disk heads are jumping all over the place - it can be slower than the previous single-file version. On the fastest of my SATA disks I get a good 50-80% CPU utilisation (Q6600 quad-core @ 3.4GHz), but on the other there's almost none at all due to the trashing.

One idea I'm looking into for the next version is to cache the file accesses more intelligently. A better solution though would be to multi-processor enable the decoding parts of the library, so that consecutive blocks of a single file are transparently processed in parallel & cached for when they're needed (I've mentioned this before). David, have you ever looked into doing that?


--------------------
Author of -Wavpack4Wavelab- and -CheckWavpackFiles-
Go to the top of the page
+Quote Post
bryant
post Aug 14 2008, 13:33
Post #6


WavPack Developer


Group: Developer (Donating)
Posts: 1287
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (gl.tter @ Aug 12 2008, 12:11) *
One idea I'm looking into for the next version is to cache the file accesses more intelligently. A better solution though would be to multi-processor enable the decoding parts of the library, so that consecutive blocks of a single file are transparently processed in parallel & cached for when they're needed (I've mentioned this before). David, have you ever looked into doing that?

There's no reason why it wouldn't work, although for decoding I wonder if the disk I/O would ever be able to keep up (or justify it), although I guess it would certainly eliminate the thrashing problem.

Also, for encoding (especially the -x modes) doing the same thing would certainly be a gain.

It would probably be a better use of time than going to hand-coded assembler for the single threads (which is another thing I think about), but unfortunately I don't have time at this point to look into either. I am happy to kibitz, however... smile.gif
Go to the top of the page
+Quote Post
gl.tter
post Aug 14 2008, 13:59
Post #7





Group: Members
Posts: 48
Joined: 25-September 05
From: UK
Member No.: 24695



QUOTE (bryant @ Aug 14 2008, 12:33) *
There's no reason why it wouldn't work, although for decoding I wonder if the disk I/O would ever be able to keep up (or justify it)


It will, as I said, I can get 50-80% CPU utilisation scanning 4 files simultaneously on a quad-core @ 3.4GHz (that figure is a across all 4 cores) - the only reason it's not 100% is that I'm disk starved from the trashing. Frequent disk seeks drastically reduce throughput, as we all know when we get pagefile trashing & everything grinds to a halt. With linear single-file streaming I could easily max my CPU and then some.

QUOTE
Also, for encoding (especially the -x modes) doing the same thing would certainly be a gain.


Sure, but probably a little more complex (due to not knowing how much compresses into each block).

QUOTE
It would probably be a better use of time than going to hand-coded assembler for the single threads (which is another thing I think about), but unfortunately I don't have time at this point to look into either. I am happy to kibitz, however... smile.gif


kibitz? : ) I can provide all the threading and CPU code - in fact if you could split the decoding context into independent block-specific ones (ie. ones that have all the decoding state needed to decode an entire block in isolation), I can do the rest. With multi-cores so widespread now, this not only gets you a massive bang-for-buck, but better yet all apps benefit transparently.


--------------------
Author of -Wavpack4Wavelab- and -CheckWavpackFiles-
Go to the top of the page
+Quote Post
bryant
post Aug 15 2008, 06:23
Post #8


WavPack Developer


Group: Developer (Donating)
Posts: 1287
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (gl.tter @ Aug 14 2008, 05:59) *
kibitz? : ) I can provide all the threading and CPU code - in fact if you could split the decoding context into independent block-specific ones (ie. ones that have all the decoding state needed to decode an entire block in isolation), I can do the rest. With multi-cores so widespread now, this not only gets you a massive bang-for-buck, but better yet all apps benefit transparently.

I think the functionality you're looking for already exists in the current library, although it's a little convoluted to get to it. You just need to start up a WavPack decoder context for each thread you want to have, and set the OPEN_STREAMING flag for each one.

The main code would read the WavPack file and parse it into blocks, then pass the blocks (using the WavpackStreamReader callbacks) to the decoding contexts to convert a single block each. Using the OPEN_STREAMING flag makes the contexts not care that the data they suck in is from discontiguous blocks (although your parser must make sure they do get complete blocks). The WavPack DirectShow filter uses this same method (not to use multiple cores but to implement the parsing/seeking separate from the actual decoding).
Go to the top of the page
+Quote Post
gl.tter
post Aug 15 2008, 11:41
Post #9





Group: Members
Posts: 48
Joined: 25-September 05
From: UK
Member No.: 24695



QUOTE (bryant @ Aug 15 2008, 05:23) *
I think the functionality you're looking for already exists in the current library, although it's a little convoluted to get to it. You just need to start up a WavPack decoder context for each thread you want to have, and set the OPEN_STREAMING flag for each one.

The main code would read the WavPack file and parse it into blocks, then pass the blocks (using the WavpackStreamReader callbacks) to the decoding contexts to convert a single block each. Using the OPEN_STREAMING flag makes the contexts not care that the data they suck in is from discontiguous blocks (although your parser must make sure they do get complete blocks). The WavPack DirectShow filter uses this same method (not to use multiple cores but to implement the parsing/seeking separate from the actual decoding).


Cool, I'll look into it.


--------------------
Author of -Wavpack4Wavelab- and -CheckWavpackFiles-
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 19th April 2014 - 14:55