Hi Folk,

I started migration of data from the old PC to the new one.
This was a little bit a pain, because the system hard disk is nearly
dead (but after some tries I got all data) and I had some stability
issues (the Iomega ZIP drive was the "guilty"), so the copying
stopped from time to time.

This evening I start to sort the data, most (65%) are Audio data
spreaded over all hard disks, other data are some Video data, Images
and Archive data (downloaded and sorted data from the Internet).
I hope the remaining data is less than 4.7 GByte, so I can make a
DVD-backup before continuing sorting.

But this is all straight forward, because of the plenty of hard disk
space. No "tower of hanoi" game.

E-Mail and Web access will then follow. Most difficult will be
the migration of the "1000 little modifications" of the system.
A small program there and there.


The first Musepack related work will be:
- mppenc 1.2 (1.15r + some small additions + bugfixes)
- mppdec 1.96 (1.95z1 + some small additions + bugfixes)
- replaygain 0.89 (support of different sample frequencies than 44.1 kHz, better API, modified application notes)

It follows the file format SV 7.5. Format freezing. Huffman table freezing.
libmpcdec.lib. New, modular decoder.

Is it also possible to make the sources as much platform independent as possible? This would be useful for building binaries by Intel C compiler and XScale, IA-64, PPC support... Maybe someday I could run a decoder on an XScale based Palm device.

I don't know how much source code optimizations in assembly are involved and how difficult this would be. But this would also ensure that MPC will be available on dominant platforms in the long term.

Thanks Frank.

Edit: By the way, maybe we should move the new improvement plans to another thread and keep this thread for pictures. The PC is built and the thread achieved its mission

Frank,

Nice to see you

Anyways, I don't know how much work you want to do, but could you possibly (along with Christian) come up with work for others to do? There's a lot of experience here, and a lot of people who would love to help out, whether that be with testing, porting, or whatever.

Yes, split from Donation for Frank Klemm's new PC. Comments about the PC can go there, comments about things to come can now go here.

I'm already working on this (ok, very slowly). I already make the decoder work under OS X in CVS. I'll work on the encoder tomorrow. Dibrom has some modified code that works but has not provided me with it yet.
I don't have the other compilers though, no Intel at least. But the targets for me are MSCV6, MSVC7 and GCC3.2+. It should work fine with older GCC too. And I think Frank uses a Watcom compiler ?

edit: missing Dibrom's code

mppdec works on amd64 with minor modification to Makefile. Don't know about ppc.

AMD64 is 64 bit x86 so this is not surprising.

@robUx4: In case you're interested in a PalmOS port, the pocket-tunes player developers are quite approachable for plug-in support. Check my recent correspondence here.

Although such issues naturally have lower priority on the to-do list, it would bring an end to the "MPC is not/will never be supported on any portable device" argument. If something is not supported it's mostly due to the platform being closed (it's not MPC developers fault).

amazing

all the small-step advancements are exciting!

some comments:

darin did some work for 1.15r building on OSX. is there anything he can include here? patch the 1.15r-current tree perhaps (if he has made modifications, i dunno, he was sorta cryptic about it)

1.95z67 builds as SV 15.15

is the plan to move mppdec+mppenc to: musepack (encoder/decoder frontend) + libmusepack (library) ?

could 16bit/32bit/dither/etc. support be added into musepack as switched parameters instead of static builds?

alsa support?

and what will be the "official" linux compiler/switches/config/buildprocess?

(so the real comment above is that i think the build process should be revamped for gcc3.2/3.3/3.4+automake,etc)


all i can think of atm
later

and welcome back frank

It's not that easy. There's lot of software that fails because programmers assume wrong pointer sizes and so on. Unfortunately you can't say generally that software that runs on 32Bit x86 will also run on 64Bit x86.

Two more findings:
I just tried it on alpha which works flawlessly too.

ppc works basically but has a flaw. The output is big endian. If you treat the output wave as signed be it sounds fine again.

The version I tested is mppdec-1.1 from Klemm's homepage.

speaking for myself (ie not Frank) I'm not willing to use automake/autoconf. But gcc 3.2+ is probably the one that should be used now. At least for binary distributions.

I don't know when exactly the libmpc will be done (see Frank's post) but that's one of the main goal, so that MPC can be integrated in a wide variety of platform. (Even portable ones )

Should I contact Darin, or he's floating around here ?

If ALSA is also working under OS X (fink) it might be a good move from ESD because now ALSA is in the linux kernel.

For the rest, I don't know.

By 'works', do you mean as a 32bit or 64bit compile?

Autotoolify mppdec shouldn't be too much work. It's only one binary and has two libraries as dependency.

On the other hand there wouldn't be much benefit over not using autotools either.

I was using a native 64 bit build.

I would like to see one binary ( enc & dec ).
I would also like to see correct support for pipelines.

What would be the benefit of one binary only ?

Darin = Dibrom

OK
Anyway I fixed the endianess problems and now the encoder and decoder in CVS work under MSVC and OS X (should be OK with any GCC under Windows & Linux too).

I'll commit the XCode project too and then maybe the whole binary bunch (encoder and decoder for each OS from the code in CVS).

Nice, this fixed the endianess problem here too.

For each os out there? Don't forget that every platform/os combination needs its own binary - have fun!

For mp3 I use lame.exe for both without having to shuffle around with a lot of executables.
Why did the developers of LAME put them together? , I think it is better to have a single binary and I believe that both the encoder and the decoder use some shared code.
Also lame.exe behaves very nicely with pipelines ( | ) and stdin stdout the great majority of mpc binaries have problems with that.

mppenc for Windows uses pipes perfectly for encoding. I use it all the time with foobar's CLI encoder.

I dunno if mppdec is worse, but I've never had a problem with mppenc.

I personally prefer the two separated binaries. Each performs a task well. Throwing them both into a single executable means that everytime you want to decode or encode, there's another switch you have to place on the commandline. It would also mess backwards-compatibility up badly.

AFAIK, some of the newer mppdec versions (1.9[...]) had a bug that broke pipe support.

What *EXACTLY* do not work?

"Maschina nje rabotajet" do not work as error description.

IIRC I've changed some code to support PCM files in mppenc (Encoder!)
to support PCM files between 2 and 4 GByte with tags, but the pipe
code of mppdec should be from very early days.

Please additional respond via board E-Mail system.

MPC is pretty much cross platform. IIRC the changes needed to make it compile on Mac OS X were fairly straight forward, so porting it to another platform or processor shouldn't be too difficult.

ALSA stands for Advanced Linux Sound Architecture, and is pretty much Linux only. The Mac OS X equivalent would be CoreAudio - which uses an entirely different API. Not even OSS is available for Mac OS X, so it's either ESD or Arts - and Arts doesn't work all that well.

Then I would suggest using PortAudio instead of all these APIs. It works under Windows, UNIX, Apple, BeOS, etc. The API is very simple and easy to use.

Still the old question:

What *EXACTLY* do not work?

Don't know if this help (I'm not very technically savvy), but I posted here about my findings of problems with pipe mppdec 1.95e to lame, solved by excanging mppdec 1.95e with 1.93j... (Problem occured in the form that the process of transcoding just stops.)

I asked Case what's broken with pipe support, and he said: "decoder thinks it can seek and tries to write fixed wav header, this produces nasty pop".

I have two serious problems copying my data to the new
system. May be some one of you can help.

OS is Linux.
Data from old partitions is copied to new computer as files between 0.1 and 60 GByte. File system types are: ext2, vfat and reiserfs(3.5).

Problem 1:
File names with characters above 127 on vfat are wrongly interpreted.

"o => -:
"a => ?
~n => +-
"u => ?
"O => ?
´i => ?

It looks like the file names are stored using the MS-DOS
character set, but now interpreted as ISO8859-1 characters.

Mounting using

mount -t vfat -o loop,codepage=437 /media/y/hdf1.image /old/mnt/f

do not change anything. I also tried some other code pages
and iocharsets, nothing changed, still exactly the same
misinterpretation.

FS was created under SuSE 6.4 under the kernel 2.2.15.


Problem 2:
Copying data from the old reiserFS 3.5 crashs the system
and corrupts data. Frequency is around 1 crash per hour.
Under 2.2.15...19 this file system was stable for years.

FS was created under SuSE 6.4 under the kernel 2.2.15.
Images were file system checked again before mounting.

Error messages:

Message from syslogd@elster at Mon Mar 29 02:37:52 2004 ...
elster kernel: vs-13065: update_stat_data: key [895627 896357 0x0 SD], found
item *3.5*[895627 896368 0x1 DIRECT], item_len 275, item_location 2290,
free_space(entry_count) 65535kernel BUG at prints.c:334!

Message from syslogd@elster at Mon Mar 29 03:32:51 2004 ...
elster kernel: vs-13065: update_stat_data: key [895627 896357 0x0 SD], found
item *3.5*[895627 896368 0x1 DIRECT], item_len 275, item_location 2290,
free_space(entry_count) 65535kernel BUG at prints.c:334!

Message from syslogd@elster at Mon Mar 29 04:39:59 2004 ...
elster kernel: vs-13065: update_stat_data: key [925724 942575 0x0 SD], found
item *3.5*[925724 942574 0x1 IND], item_len 8, item_location 2993,
free_space(entry_count) 2219kernel BUG at prints.c:334!


Any hints or ideas?

Problem1 (vfat --> reiserfs):
try an ftp transfer. this should translate different codepages

It's a bug of the Microsoft API.

Current code works like:

unsigned long total = 0;
int samples;
float buffer [ 1152 * 2 ];

write_header ( sample_freq, channels, 0xFFFFFFFF );

while ( ( samples = decode (buffer)) >= 0 ) {
total += samples ;
writesamples ( buffer, samples );
}

if ( seek (0L) == NO_ERROR )
write_header ( sample_freq, channels, total );

The problem is that M$ seek() also return NO_ERROR even when seek fails.
A new FileIO module which must be programmed for every OS should remove
this problem.

class FileIO {
protected:
...
public:
IsReadable ();
IsWritable ();
IsSeekable ();
Read ( void*, size_t);
Write ( const void*, size_t);
Seek ( int64_t offset, enum whence);
} ;

I want to remove ALL this nasty things from the encoder and decoder.

Nice to hear that this will be fixed.

I would like to provide to you some more examples.
Using the binaries found at rarewaves I run into troubles.

------------------------------------------------
Encoder
Version 1.14 with following command line produces a 36 kb mpc file ( basically an empty file )



Version 1.15r with the same syntax works just fine.
------------------------------------------------
Decoder

Both version 1.95e and 1.95z6 produce this result.



Version 1.1 works just fine



---------------------------------------------
I use lame.exe extensively and I never run into problems using pipes it might be a nice ides to look at the code that lame uses.

Frank, does it mean you want to use C++ as well ?

Frank's first problem seems to be caused by the old Linux kernel on his old PC. Files with Umlauts were not saved correctly, and later, this bug was silently fixed and no compatibility switch was introduced. So Kernels 2.0.xx through 2.2.xx don't go along too well with Kernel 2.4.xx and above, concerning certain filenames.

About MPC, Frank is away for easter, he will be back home in 1 1/2 weeks and that's when he starts coding.

That is some nice news on the MusePack front... thanks for the heads up CiTay.

Some short information:

Windows works rock solid on the new computer, but I still have
serious trouble with Linux SuSE 9.0 . The most serious are
data losses on FAT32 partitions. When writing data to a FAT32
partition often the directory entry is not updated at the end.
This results in zero sized files and lost chains of clusters.

FAT32 is the only file system type usable for Windows XP and
Linux.

frank -

you can use captive-ntfs in linux (which uses Windows XP's ntfs drivers to access all ntfs drivers)

http://www.jankratochvil.net/project/captive/

(and i've never had any problems writing to fat16/32)


later

See

http://www.linuxquestions.org/questions/history/154761
http://mailman.linuxtag.org/pipermail/debi...ber/004162.html

This serious bug is fixed in 2.4.25 (search for "128 M").
"FAT: Support large partition (> 128GB)" is a nice circumscription of
"FAT32: Drops files from time to time on >128 GB partitions".

As far as I know captive-ntfs is a part of 2.6.x, but all tries to install 2.6.5
didn't worked. I don't want to spend months of time to built a stable Linux
system. There are several other problems and bugs. This list is currently longer
than 40 entries. 20 of them have the status "don't have any idea what's wrong
or how to solve".

Work for this evening: Try to install standard kernel 2.4.26.

fat32
------
hmm... the fat32 bug needs an overloaded fat32 partition (i.e. > 85% full) and a deprecated linux 2.4.x kernel, so i'd call this problem solved at this point

captive-ntfs:
------
captive-ntfs isn't part of 2.6.5 that i know of (?)

2.6.x includes a rewritten ntfs driver that is more secure and can write to the ntfs partition in a limited fashion (i.e can't write a file bigger than what is on the disk currently)

debian rant:
------
well, for starters, i wouldn't have used a *NIX OS that lacks ability to do actual work on.

perhaps try debian or debian based (libranet, mepis, knoppix) or gentoo/lfs/etc.
http://debian.org

debian unstable (and, if you wish, +experimental +unofficial) = excellent development and user environment with real package management/upgrade/etc. with full support for kernel 2.6 and alsa and all the necessities

it doesn't take a month, it takes a few hours to download and install a complete system...

perhaps if you provide the list of things that don't work for you others can offer solutions?


later

Perhaps it's just me, but if Linux is giving Frank a headache, and Windows is working fine... why not just use Windows entirely? Is there any particular reason *why* Linux must be used? Windows can be secured (i.e., turn off all unnecessary services, don't use Internet Explorer and Outlook/Outlook Express, and install and use a good anti-virus program and firewall), and there are free development environments on Windows, such as Cygwin and GCC, and now the free command-line C/C++ compiler that Microsoft released over the weekend.

Hopefully I don't offend anyone or something, just felt I needed to post what seems to be the easiest solution to me...

Because Klemm started talking about his OS problems. What did you expect?



Captive NTFS uses the WindowsNT/2k/XP ntfs.sys read/write routines. It would only go wrong if the Microsoft driver is wrong.

Or if something is emulated in a wrong way. I have a big experience of using ntfs4win98 and was not always that great.

-Eugene

I never had any problem using Winternals' NTFS98, in both Win98 and WinMe.

Some Remarks:

- I also thought about dropping Linux support. At least this eases System
administration.

- The problem is not prefering Windows or Linux, but I like to support both.
Some (better: a lot of) tools are existing only under Windows, other only
under Linux.

- There are often questions about Mac-OS, there I can't help. I don't want
drop Linux to the same state.

- Win32/Cygwin ist a third plattform, it can't substitute Linux/GCC.
Plattforms I tested in past:
- Linux/GCC
- Linux/ICC (but licence server is not running anymore)
- Windows/MS C++
- Windows/Cygwin (not tested for a longer time)
- Solaris/CC (not tested for a long time)
- AIX/CC (not tested for a long time)

- The problem is not that I'm unable to administrate Windows or Linux.
But I'm not able to fix bugs in Linux, because I'm not a wizzard.
And my impression is that there are more bugs in Linux than Bill Gates
can imagine ...

- There is no systematic system test in Linux. A 128 Gbyte HD bug was fixed
in the last month, although these disks exist for more than 18 months.
With a standard motherboards and RAID (3x 60 GB) for more than 4 years.
Shame on you Mr. Torvald!

- Kernel bugs are not a question of Debian, Redhat or SuSE. It's very unlikely
that they unintentionally fix such bugs.

- There is intentionally reserved space for a second Win and a second Linux
installation (ask Citay). But the current distributions are all with "128 Gbyte
bug inside".

- When I read the Changelog files of Linux, they sound like helloween
documents. A lot of serious errors on critical places very fixed (FS layer).
How many douzens are remaining???

- Any idea when Knoppix 3.4 is released (3.3 has also the 128 GB bug)???
It is Debian based and I use it on some computers. See www.knopper.net

- In the past compiling under Linux AND Windows was a good idea. This
avoid such ugly extremely Windows-specific (MAC) or extremely
Debian 2003-specific (-censored-) source code. BTW it makes often
a lot of problems to port source from one to another Linux distribution.
SuSE-2000 to Debian-2003 or even SuSE-2000 to SuSE-2003.

1. I don't want to start a OS discussion. I only thought dropping a note about my current work is a good idea.

2. Thanks for the information what Captive NTFS is. Using ntfs.sys sounds
more serious and more reliable than just another Linux NTFS hack.

3. I never had problems with NTFS under Windows. Stability on stable hardware
is great. BTW NT 4.0 without SP drives a 128 Gbyte NTFS partition without any
problems. Congratulations! Also congratulations for Dell. A 1997 built Computer
have no problems with a 128 GByte disk (OS is within the first 8 GB).

In one of the last c't Magazines there was a Knoppix 3.4 c't Edition. I think it was Magazine 4/2004, maybe you can get one.

EDIT: I regret, this is not an option. As you said that 128 GB Bug was fixed last month, so this version of Knoppix 3.4 won't have a kernel with that fix. Or if Linux 2.6 doesn't have that bug you could try that, the c't Edition has a boot option for a 2.6 series kernel.

Regards, David

NTFS is a very reliable filesystem, its main problem is write performance. So to gain time during writes, it makes use of any small space that is currently under the HDD's head, without any other consideration. After a while this produces heavy fragmentation and read performance lowers a lot. But this performance is suitable for benchmarks (since they measure short-term filesystem usage only). On any production server, the daily defragmentation process (Diskeeper etc) will constitute about 1/3 of the overall disk accesses !.

I don't think there are any stability issues with filesystems such as Reiserfs. I have a 2TB server in software RAID5 which runs day and night, pushing 250-300MB/s all the time. It operates on millions of very small files, and I've yet to experience a single corruption. Plus the files almost never fragment..

Also Reiserfs 3 is in "maintenance mode" (ie: no patches), and the author says he hasn't received a single bug report for years.

Frank, about Linux support I think it's more important than ever. Sincerely, I agree no OS is perfect and there can be surprising and/or unacceptable bugs, but they are there under Windows too (just hidden). I think more and more people (including myself) have switched or are switching to linux because they're fed up. I agree Microsoft does make some good stuff too, but in general it's a nice cover for something dirty.

It's a pleasure to install software in Gentoo Linux (on my laptop). Everything is snappy, and works great. World's best software is available with a single command. Now it's really starting to be a pleasure to use :-)

I don't think there's any 137GB issue with current kernels - as I said, I use 1.9TB partitions on a Redhat 8 (recently upgraded to Redhat 9) server.

For me the only annoying limitation in Linux was the 2GB issue, and in 2.4 and 2.6 kernels it's a thing of the past :-)

Almost every single thing that can be done under Windows can now be done under linux, plus plenty more, and (very important !!) it can be automated, which is (imho) something crucial to have on a computer . On Win32 everything looks cool, but batch files suck, the console uses obsolete DOS 2.0 codepage, etc.. The regular user doesn't care about it in the short run (because GUI is there), but isn't Musepack more about experienced users and long-term stuff?

By the way, I'd be glad to help for a linux version if necessary.

Cheers !

Edit: As an exemple of a horrible win32 bug, just think about the one which can destroy partitions during a kernel core dump.. It's been there for five years ! Isn't that serious?

OS-preference related discussion split to separate thread

Can the clipping problem be addressed in SV 7.5 instead of SV 8?

Transcoded from SV7.0/7.1 are still clipped, the information is lost.
Native SV7.5 encoder: Problem is solved.

How about the *BSDs? FreeBSD is very good, stable, well documented and organized IMO.

You can use bsd's cc, gcc and others too, even many versions can coexist peacefully without any troubles (example: just call gcc32 instead of gcc31, etc).