I've been having this problem for a while now and I can't find a cure.

I'm running Win2k SP4 with the relevant hotifxes.

Symptoms:
After some hours of uptime, when I start a new program, the GUI becomes totally garbled. This happens most often when opening a video (with media player classic) or starting Matlab. But I've had all kinds of software trigger this behaviour (for example calc.exe, Mozilla).

The mouse pointer seems to be displayed at the correct location, but buttons and windows have moved and are usually not in their original shape.

If I manage to close the software (via alt-f4) things return to normal. The desktop isn't refreshed and still looks garbled, but I can make most of it go away by "minimize all button" (I have to click blindly) and resizing the task bar. Sometimes the Start menu won't work anymore. When I start the same program again, that originally triggered this, the same happens again. Perhaps worse. Sometimes I can start other software after such incidents without trouble.

The problem occurs more frequently, if I run Azureus (Java bit-torrent client). After "the incident" I can't access Azureus anymore at all (the tray icon tool-tip pops up, but the info isn't updated).

The problem can be fixed (at least temporarily until the next time it occurs) by restarting Windows or just by logging out and back in. Perhaps this can give some pointers about services or other things which are restarted at logout/login.

Here is a screenshot (taken with a digital camera, I haven't tried taking a software screenshot, dunno if that would work). Note that my taskbar is usually located on the bottom.

Things I've allready tried:
Reinstalled DirectX (9.0b)
Reinstalled NVidia drivers (official WHQL and unstable versions)
Scanned system with Spybot S&D
Scanned system with F-Prot (command line version, latest signatures)
Checked system log for "out of memory" messages etc.
Deinstalled supicious software (like reclock)

To me it seems to be a Windows resource related problem, but I'm not even sure if the concept still exists on NT derived platforms.

I've googled and searched the MS Knowledge Base but come up with useful results. You're my last chance before a complete reinstall which I loath to do, but this is really starting to annoy me.

You didn't mention this, so it bears asking. Have you made sure that your hardware is not at fault? In other words, are you sure that you don't have memory errors and that your CPU is running in normal temperature ranges?

I am not sure about memory errors. The last memtest I tried wouldn't work with the Athlon64.

With idling desktop the CPU is at 46°C right now (according to speedfan, cool-n-quiet in high-performance mode (ie disabled)) and it is a hot day. I've been running some XviD encodes recently and CPU temp went as high as 57°C which personally makes me feel uncomfortable but should be fine. The XviD encodes and Lame encodes all work out fine and it is said that these two are very stressing and should reveal system instabilities.

Since the trouble exclusively occurs when I start a program, I am inclined to think it is not hardware related. If it were, shouldn't it also appear while just running programs? I guess you can never know.

I have experienced this sort of problem with a Creative labs video card. The NVidia drivers are typically great, however the BIOS on the Creative labs card was crap. I had to fall back to a much older driver revision. You may want to try revision 6.50. Let me know if you cannot find it from a reliable source. I may be able to dig it up for you.

@sthayashi: While I was away, I ran memtest 1.11+ for ca 1:30 hours. I thought it was done (the top progress bar was 100%), but then it started over or startetd a different test. No errors reported in the time it ran. I'll see if I can run a complete scan tomorrow.

@ddrawley: I have an old 6.72 lying around and will try that. Thanks for your offer. I think guru3d has old Detonator versions. I'll check that first.

Since some months nVidia display drivers are over-bloated with GUI-altering stuff (special menus, gadgets, lens, transparency, etc, etc.), so they might be the reason. I'd uninstall them and see if these weird things still happen with native VGA driver. Launching windows in fail-safe mode could give you some clues too.
A year ago I had a problem with explorer.exe crashing and re-launching itself spontaneously every few minutes. I spent a couple of whole days to find that so called Anark Client (something like Flash ActiveX plugin) was guilty. After I uninstalled it, everything came back to normal. So, it's probably some buggy software causing your problems. Find it and get rid of it, hope you won't have to go through the reinstall hell...

Any progress Gecko?