IPB

Welcome Guest ( Log In | Register )

> Upload forum rules

- No over 30 sec clips of copyrighted music. Cite properly and never more than necessary for the discussion.


- No copyrighted software without permission.


- Click here for complete Hydrogenaudio Terms of Service

50 Pages V  « < 3 4 5 6 7 > »   
Closed TopicStart new topic
lossyWAV Development, WAV bit reduction by 2BDecided
Nick.C
post Aug 28 2007, 08:07
Post #101


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



Guru, thanks for the pointer to the samples - links to E16, E36, S10 & S43 are broken.

I have processed the 146 150 files I downloaded (using the Matlab script) and the size results are as follows:

WAV: 252.3MiB; FLAC: 122.2MiB; Matlab Script lossyFLAC:100.9MiB (current lossyWAV.exe: 93.3MiB)

Matlab script output attached.

The Delphi translation is coming on well - the exe will now carry out multiple analyses *without* overlapping FFT analysis. My task for tonight is to reinstate the overlapping FFT analyses, firstly within a 2^n block length and secondly either side of an arbitrary block length.

The crashing bug has been identified as being caused by digital silence and has been fixed.

This post has been edited by Nick.C: Aug 28 2007, 12:16


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
guruboolez
post Aug 28 2007, 11:56
Post #102





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



The four links are corrected. Thanks for these notifications smile.gif
Go to the top of the page
+Quote Post
Nick.C
post Aug 30 2007, 11:52
Post #103


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



Still implementing the script in Delphi - now capable of analysis overlapping the edges of the codec_block for "power-of-2" length codec_blocks. At the moment, the output is not exactly the same as the Matlab script, however the bits removed values have come down to a closer approximation of the script output, indeed over Guru's 150 samples the flossy files are almost exactly the same size in total, if not individually.

Input parameters are being developed by Halb27 as is the wavIO unit - very gratefully received, although not yet selectable by the user - current code only processes 3 analyses, 10, 8, 6 bit fft_lengths, noise_threshold_shift=-3.0, triangular_dither.

I hope to have lossyWAV alpha v0.0.2 available this weekend, which will only be released if it more closely matches the Matlab script output.

Trying to speed optimise the code as well, but that necessarily requires to be delayed until I actually achieve matched output.

This post has been edited by Nick.C: Aug 30 2007, 11:53


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
2Bdecided
post Aug 30 2007, 16:44
Post #104


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



I know it's in this thread somewhere, but which sample was ABXed with noise_threshold=0? What were the other parameters?

I'm not saying it shouldn't happen, just wondering where it did.

Cheers,
David.
Go to the top of the page
+Quote Post
Nick.C
post Aug 31 2007, 19:21
Post #105


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



I think that it may have been atem_lied - but I'm not sure. It was in a spread of 2 or 3 analyses with or without my experimental pereceived noise algorithm - also, I was (at that time) varying the length of the spreading_function.

This post has been edited by Nick.C: Aug 31 2007, 21:15


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
halb27
post Sep 1 2007, 11:48
Post #106





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



I guess it was when I abxed atemlied, but I don't know how what I listened to was related to internal quality settings.


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
Nick.C
post Sep 11 2007, 11:20
Post #107


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



lossyWAV alpha v0.0.2 attached: - superseded by v0.1.0 - see later post.

Command line parameters for quality and codec_block_size over-ride implemented;

Overlapping fft analyses implemented;

Output not yet identical to 2Bdecided's script - examination of low level fft output information required.

Feedback extremely welcome - unless tied to a brick...... wink.gif

Nick.


[edit]
Crashing bug found and fixed (fingers crossed) - v0.0.2b attached. Superseded.
[/edit]

This post has been edited by Nick.C: Dec 4 2007, 09:46


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
halb27
post Sep 11 2007, 19:22
Post #108





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



Hi Nick,

Thank you very much for this version which is already pretty good.

I tried 10 samples most of them known as problematic to wavPack, and out of these 10 I could only abx 2:
badvilbel has a tiny amount of added hiss which I could abx 8/10 at sec. 5.6-7.2. Not tranparent, but a negligible issue IMO.
bibiblolo is worse, and even at low volume there's kind of distortion at sec. 17.1-19.2. No abxing necessary.

Atemlied is fine to me.

This post has been edited by halb27: Sep 11 2007, 19:23


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
Nick.C
post Sep 12 2007, 09:49
Post #109


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



I think that I've found the problem - my poor attempt at a CONV function seems to be the root of the problem. I will start to re-write it tonight.

Also, I think that I've finally determined the cause of the random crashing - a poorly written Magnitude (of a complex number) routine - fixed.

[edit] I've convinced myself that the fft routine that I am using is indeed accurate - so, the next problem *really* is the conv routine. ~sigh~. [/edit]

This post has been edited by Nick.C: Sep 13 2007, 21:02


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
Nick.C
post Sep 14 2007, 12:15
Post #110


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



<minor fanfare> lossyWAV alpha v0.1.0 attached. Superseded.

Closely approximates matlab script by 2Bdecided.

CONV function fixed, window function fixed (was the *real* problem).

Have fun!

[edit] window function properly fixed (finally). Even more closely approximates matlab script. lossyWAV alpha v0.1.0b attached[/edit]

[edit2] v0.1.0b superseded. [/edit2]

This post has been edited by Nick.C: Sep 16 2007, 22:37


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
halb27
post Sep 16 2007, 21:22
Post #111





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



Thanks a lot, Nick. Great News.
I've just come home from visiting friends this weekend and I'm too tired for a listening test right now. Will do it tomorrow.

This post has been edited by halb27: Sep 16 2007, 21:22


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
Nick.C
post Sep 16 2007, 22:36
Post #112


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



QUOTE (halb27 @ Sep 16 2007, 21:22) *
Thanks a lot, Nick. Great News.
I've just come home from visiting friends this weekend and I'm too tired for a listening test right now. Will do it tomorrow.


Don't try *that* one - try *this* one..... wink.gif v0.1.1 - Superseded by v0.1.2

Added :
-w option to suppress warnings,
-q option to reduce screen output.

This post has been edited by Nick.C: Sep 17 2007, 09:14


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
bryant
post Sep 17 2007, 02:06
Post #113


WavPack Developer


Group: Developer (Donating)
Posts: 1287
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



Hey guys, it looks like you're making good progress! smile.gif

I was playing around with alpha v0.1.1 and ran into a couple problems. The first was that I couldn't open files that were write protected. Not a big deal really, but I'm sure that's not how you intended it.

The second problem is that I found several files that caused a range error exception right near the end of the file and truncated the output (I suspect some error in handling the last block). I am attaching two of the failing files.

Cheers!

David
Attached File(s)
Attached File  fail.zip ( 3.1MB ) Number of downloads: 236
 
Go to the top of the page
+Quote Post
Nick.C
post Sep 17 2007, 07:08
Post #114


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



QUOTE (bryant @ Sep 17 2007, 02:06) *
Hey guys, it looks like you're making good progress! smile.gif

I was playing around with alpha v0.1.1 and ran into a couple problems. The first was that I couldn't open files that were write protected. Not a big deal really, but I'm sure that's not how you intended it.

The second problem is that I found several files that caused a range error exception right near the end of the file and truncated the output (I suspect some error in handling the last block). I am attaching two of the failing files.

Cheers!

David


Many thanks - which quality setting were you using for the failed files (although it might not make a difference)?

The read-only file problem is much easier to fix. - Fixed for v0.1.2

Thanks again!

Nick.

[edit] I cannot seem to replicate the range exception failure - what CPU are you using? [/edit]

This post has been edited by Nick.C: Sep 17 2007, 08:43


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
halb27
post Sep 17 2007, 08:42
Post #115





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



I'll look into the wavIO unit tonight for why write-protected files aren't usable.


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
Nick.C
post Sep 17 2007, 09:13
Post #116


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



QUOTE (halb27 @ Sep 17 2007, 08:42) *
I'll look into the wavIO unit tonight for why write-protected files aren't usable.


I've already done it - filemode:=0 before opening infile; filemode:=2 before opening outfile - I'll em you the revised source.

I've changed from extended reals to double reals (which has speeded things up a bit) - v0.1.2 attached. - Superseded by v0.1.3

I've been playing with the spreading function length (again....). For badvilbel, quality 2:

SFL : BTR
1 : 0.4119
2 : 1.2374
3 : 1.6652
4 : 1.9018 (default)
5 : 2.1192
6 : 2.3611
7 : 2.5811
8 : 2.7572

This may be a setting to be changed when defining permanent quality level settings. I may implement a switch to allow the spreading function length to be reduced only.

[edit] Minor bug - required enter to be pressed if an error occured. [/edit]

This post has been edited by Nick.C: Sep 17 2007, 22:59


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
user
post Sep 17 2007, 10:02
Post #117





Group: Members
Posts: 873
Joined: 12-October 01
From: the great wide open
Member No.: 277



btw., do you take care, that lossyFlac cannot be mistaken as *.flac ie. Lossless flac ?
(or exchange flac with any lossless format, which this algorithm is used on)

The difference between lossyFlac and Flac should be recognizeable already by file extension .flac vs. .lfla or whatever.

it is somehow bad wording also, if you call something:

lossy free lossless audio codec.

Either it is lossy or lossless.

This post has been edited by user: Sep 17 2007, 10:06


--------------------
www.High-Quality.ch.vu -- High Quality Audio Archiving Tutorials
Go to the top of the page
+Quote Post
Nick.C
post Sep 17 2007, 10:15
Post #118


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



QUOTE (user @ Sep 17 2007, 10:02) *
btw., do you take care, that lossyFlac cannot be mistaken as *.flac ie. Lossless flac ?
(or exchange flac with any lossless format, which this algorithm is used on)

The difference between lossyFlac and Flac should be recognizeable already by file extension .flac vs. .lfla or whatever.

it is somehow bad wording also, if you call something:

lossy free lossless audio codec.

Either it is lossy or lossless.


A FLAC file is a FLAC file - how the input WAV file has been processed is anybody's guess. At present, how do you know that a FLAC file has not been created from MP3?

Yes, the extension is changed from ".WAV" to ".lossy.WAV", but apart from that there is no change other than to the WAV data itself.

I am probably going to write a test program to determine if a WAV file has been processed with this method - however with variable codec_block_size this may take quite a while to determine with accuracy for a given WAV file.

For me, this processor is for someone (i.e. me, and others) to use on their own lossless files to create processed FLAC files. It is not intended for files to be distributed in this manner - but that would be probably be illegal, wouldn't it.

In the same way as Wavpack has a lossless and lossy mode, this processor allows a user to decide for themself to use a processed WAV file to create a (smaller) FLAC file (than unprocessed, most often) for personal use.

Another option is to create a Wavpack style correction file to allow the processed data to be restored to fully lossless.

This post has been edited by Nick.C: Sep 17 2007, 10:26


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
user
post Sep 17 2007, 10:42
Post #119





Group: Members
Posts: 873
Joined: 12-October 01
From: the great wide open
Member No.: 277



well, I knew, the wavpack lossy/hybrid example would be thrown in.

Difference:

if you create wavpack-lossy with file extension .wv , which could be mistaken as lossless-wavpack (well, even I would prefer another extension, so that lossy-wv files and lossless-wv files are different),
but you get the clear information, if you watch the properties of a wv file. it is written inside the wv file, if encoded as lossy-wv or lossless-wv.

From my very rough understanding, this new "lossyFlac-wav" method is similar, what wavpack-lossy and Optimfrog-DS already do.

The problems I foresee (maybe for me, maybe also for you, if you use lossy-wav-to-flac), that you have Lossless flac files in your system, but also these lossy-wav-to-flac files.
There might come a time, when you yourself cannot remember, which was what.
A program to detect lossy-wav-to-flac files (and to discern from true Lossless) will be another pain, and from current point of view innecessary (work for end-users to select the lossy-wav-to-flac files from true-flac files).

From a user's point of view, I would simply use Wavpack lossy or better hybrid, as obviously users of lossy-wav-to-flac want to circumvent danger of typical lossy-codec artifacts/problems, want to buy a little bit higher bitrate than lossy, but want to buy also lower bitrate than Lossless.
IMO, this is already achived by Optimfrog DS and Wavpack hybrid/lossy. Maybe the new algorithm could be integrated into existing formats, maybe making existing lossy-wavpack/Dualstream work more efficient.

Or as add-on implemented into flac, a flac-lossy/hybrid, but with clear solution of described problem, at least a clear inside information in the file, if it is flac or lossy like in wavpack wv, or maybe better new extensions.

This post has been edited by user: Sep 17 2007, 10:44


--------------------
www.High-Quality.ch.vu -- High Quality Audio Archiving Tutorials
Go to the top of the page
+Quote Post
Nick.C
post Sep 17 2007, 10:51
Post #120


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



QUOTE (user @ Sep 17 2007, 10:42) *
The problems I foresee (maybe for me, maybe also for you, if you use lossy-wav-to-flac), that you have Lossless flac files in your system, but also these lossy-wav-to-flac files.


If the processed files are converted directly, without renaming (i.e. left as filename.lossy.wav) then the resultant FLAC file will be filename.lossy.FLAC - surely that is instantly recognisable?

Yet another option would be to add a metadata tag to the FLAC file indicating that the WAV file from which it was created had been processed.

Unfortunately, to maintain compatibility with hardware / software players the ".FLAC" extension cannot be changed.

This post has been edited by Nick.C: Sep 17 2007, 10:52


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
2Bdecided
post Sep 17 2007, 11:07
Post #121


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



We had this discussion in the original thread...

http://www.hydrogenaudio.org/forums/index....showtopic=55522

In short: it's up to someone to create a FLAC decoder or tool which checks for the wasted_bits feature being used.

If "wasted_bits" changes block by block, then it's almost certainly a lossyFLAC file. If it's the same every block, then it's probably a strange source (very rare). If it's not used at all, then it's not a lossyFLAC file (or it's lossyFLAC, but the pre-processor decided not to remove any bits at all, or just maybe it removed very few and someone changed the FLAC block size so that they didn't ever all fall within one block).


More generally, as Nick has said, you could check for 0s in the LSBs of consecutive (blocks of) samples. There is an issue in not knowing the block size, but even in a generic file editor (set show 16-bit binary values) it's going to be pretty obvious.


There is simply no failsafe way to mark these files as lossy - no more than a decoded mp3, encoded to FLAC, is marked as lossy.

There is also no way to say "this is a different format" because the whole point is that it's not. No more than a .wav created from a decoded mp3 is a different format from a .wav from a CD.

Cheers,
David.
Go to the top of the page
+Quote Post
halb27
post Sep 17 2007, 11:41
Post #122





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



These things are always a matter of taste to some extent, and there will always be people who have strong emotion that FLAC means lossless in an overall sense so that to them using lossy FLAC is not an option.
The problem only starts when they think 'FLAC should mean lossless (in an overall sense) to everybody'. Sure nobody can say what something should mean to everybody, and worse so - as was pointed out already - even if the idea of a lossy FLAC preprocessing would not have come up we never know about the source quality of a FLAC file we receive from somewhere. But we're usually talking about FLAC files produced by ourselves, so this shouldn't be a problem especially as .lossy.flac files are named like that.

The most neutral point of view towards lossy flac IMO is to see it as a welcome usage extension for lossless codecs like FLAC, TAK, wavPack. Nobody needs to use these codecs with a lossy preprocessor, but whoever does gets the benefit of arriving at roughly half the file size with popular music with no audible quality impact (this is the weakest point so far due to limited experience).
The most important usage is with FLAC, as FLAC has a rather good hardware support. This is the major advantage against say OptimFrog DS. wavPack lossy is more interesting because of existing hardware support, and David Bryant has just recently made a big progress due to implementing dynamic noise shaping that works very well, and he seems to be implementing the lossy FLAC idea for a quality control. So in a sense this is the best way to go. Anyway people who can't play wavPack lossy on their hardware player, now have a good alternative when using lossy FLAC.

This post has been edited by halb27: Sep 17 2007, 12:54


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
bryant
post Sep 17 2007, 16:59
Post #123


WavPack Developer


Group: Developer (Donating)
Posts: 1287
Joined: 3-January 02
From: San Francisco CA
Member No.: 900



QUOTE (Nick.C @ Sep 16 2007, 23:08) *
[edit] I cannot seem to replicate the range exception failure - what CPU are you using? [/edit]

Oh, that's surprising. I had guessed it would be easy (which is why I didn't provide more info).

Anyway, my CPU is a Pentium 4 Willamette 1700.

When the error occurs I get a popup with The exception unknown software exception (0x0eedfade) occurred
in the application at location 0x7c57e592
. After I click through I see on the console:

CODE
Processing : 5.08MB; Exception ERangeError in module lossyWav.exe at 0000A1F6.
Range check error.


That's all with v0.1.2. With v0.1.1 the location displayed on the console is 0000A2CA and the location on the popup is the same.

Examining the resulting WAV files I see that Track68.lossy.wav is 36470 bytes short and Track104 is 22470 bytes short.

Maybe it has something to do with uninitialized data being interpreted as FP. I assume there must be some special processing with leftover data at the end of the file...

I verified that the open on read-only files is fixed. smile.gif

David
Go to the top of the page
+Quote Post
Nick.C
post Sep 17 2007, 19:32
Post #124


lossyWAV Developer


Group: Developer
Posts: 1772
Joined: 11-April 07
From: Wherever here is
Member No.: 42400



Hmmm.... No uninitialised data is processed, I think. I will try this evening to replicate the bug and get back to you. Better yet, I'll implement a -d option to switch on a debug mode - which will tell you the number of bits being removed block by block - so the block where it crashes will be easier to determine.

Assuming that you were using the default block size (1024 samples) then 2 channels by 2 bytes per sample by 1024 = 4096 bytes per block, therefore the first crash is 8.9(04) blocks from the end and the second is 5.4(86) blocks from the end. So, unless there are additional WAV chunks after the data then I don't really understand..... Unless, that is, the last data is in the buffer within the wavIO unit. If so, it may well be down to my maths.....

Oh well, back to debugging smile.gif headbang.gif


--------------------
lossyWAV -q X -a 4 --feedback 4| FLAC -8 ~= 320kbps
Go to the top of the page
+Quote Post
halb27
post Sep 17 2007, 21:38
Post #125





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



I've tried the new version (default quaklty) with the samples I used last time.
bibilolo is fine to me now, as are the other samples with the exception of Atemlied. I was able to abx Atemlied 8/10 though it was pretty hard.

I also tried to reproduce the problem with track68 and track104, but with my cpu (AMD mobile Athlon = low power Barton) I couldn't reproduce the problem.
Nick, are you using an AMD cpu?
I'll take lossyWav and the failing tracks with me to work tomorrow where I use an Intel cpu.

This post has been edited by halb27: Sep 17 2007, 21:58


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post

50 Pages V  « < 3 4 5 6 7 > » 
Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 20th April 2014 - 09:50