Back in the (g)olden days, back when sodomy was illegal, when Enron reigned supreme, all was well in the land of effortless computing. I could have my cake and eat it too. I could run my favorite distributed computing client, be it Seti@Home or Folding@Home or something else, set it to idle priority, and whenever something more important came up, it would give up the throne. This very obvious and simple behavior is now all but impossible with Hyperthreading, at least insofar as it's implemented in Windows XP.

Let's say I have Folding@Home running on CPUs 0 & 1. Two instances, each in a different folder. Both are idle priority. I start up a a divx encode. Put its priority at normal. Hell, even realtime. What does Windows do? Instead of doing the perfectly obvious thing, which is to kill off access to the CPU from all 'idle' priority programs, it shares the CPU 50/50. 50% going to the divx process (all of CPU0), 50% going to the FAH processes (25%+25% on CPU1). No matter that the FAH processes are clearly marked as "Idle."

Now, why is this a problem? Because I'm a lazy bastard, that's why. I should not have to take extra steps to make sure a process stops computing when something more important comes along when I've already told Windows that it's an "Idle" process.

John

Is your divx encoding program multi-threaded? If it's not, then you're not going to see it take up both thread paths of a hyperthreaded processor.
Have you seen similar results in other OS's with or without true dual processing? This problem is probably not a hyperthreading problem so much as it is a Windows thread priorities problem.

Try killing off the idle (FAH) process entirely, start up a DivX encode, and see if the DivX encode takes up 100% of both CPUs, or if it (as I suspect) leaves that "25%+25%" to run on "System Idle Process". If SystemIdleProcess is getting CPU time, then the DivX encoding is not fully utilizing both CPUs, and therefore it's perfectly reasonable for FAH to use the unused CPU time.

You can use "set affinity..." in the task manager to assign both fl00ding@home and divx encoding to the same virtual CPU.

That takes work.

You shouldn't complain then if you aren't going to investigate and determine where the problem lies. Especially if you imply that Hyperthreading is at fault (i.e. a hardware problem), as your subject suggests.

I think I made quite clear in my original post that the fault lies with Windows. DivX is multi-threaded but still doesn't make full use of the second virtual processor. Real CPU cycles are still given to the Idle program, when they should be given to the non-Idle program. Non-100% CPU-utilization does NOT imply that in a single-processor system it wouldn't be using 100% of the CPU. It only implies that one thread might be, and that the other threads aren't CPU intensive at all.

I understand how hyperthreading works. The problem as I have laid out in the original post is indeed the fact of the matter. Hyperthreading support in Windows XP isn't ideally laid out. My reply "That takes work" is consistent with my original post. In a Hyperthreading system, an idle process should not be consuming resources under the conditions laid out in the original post.

you're assuming that each thread is going to want as much cpu as possible, when that very likely is the furthest from fact. You didn't mention which program you're using to do your video encoding, but a fair guess would be that at most there is a decode thread and an encode thread. The decode thread isn't going to use up any noticable amount of CPU, since the encode thread is going to take a long time to chew through the data it is being fed from the decode thread. So obviously any other programs running are still going to get a lot of CPU, as you can't run one thread on more than one cpu, or virtual cpu in hyperthreading's case.

Upgrade to Win2003 Server. Windows XP has clear bugs in schedulter+hyperthreading despite being marketed as 'hyperthreading optimized' by MS.

The problem is that Windows XP doesn't know Hyperthreading. It thinks it's a 2 CPU SMP system. Your DivX encoder is scheduled to one of the CPUs. For Windows, the other CPU (it doesn't know it is a 'virtual' CPU) is idle, so it schedules the other background process to it. Because both processes are not competing for the same CPU (in the Windows scheme of things), the priority does not matter -- priority matters only when two processes/threads are competing for the *same* CPU so Windows can share processor time. Well, we know Hyperthreading is not SMP, so both processes compete for CPU resources such as cache, etc. and end up using 50% of CPU each.

Linux kernel 2.4 has the same problems, the next version of the kernel (2.6) will be Hypertheading aware. I don't know the timeline for Windows.

Edit: typo/grammar.

Hi Garf,

Thanks for the advice. Are you sure Windows 2003 fixes this?

Not 100% because I don't have a P4 with Win2k3 but 99%

Is there a cheaper alternative? I'm building a computer this weekend that has HT, but I wasn't aware of the XP problem. Looking at prices, a new copy of Win2k3 server will cost more than what I paid for the processor+motherboard.

JohnMK, did you actually measure the time some DivX encode takes with and without those idle-threads running in the background? IIRC the values Windows' task manager shows for processor load can be pretty much useless on HyperThreading systems.

Wrong. WinXP is HT-aware, though (according to Garf) it has bugs in implementation. Win2k, on the other hand, is not HT-aware.

Garf, do you have any links to bugs in XP's scheduler regarding HT? This is the first I'd heard about it.

That said, MS probably has hotfixes for the bugs if they're serious. One might have to call into support to get them though.

Or just go build a dual CPU system. No problems then.

Ending the idle FAH processes about doubles my DivX encoding performance, this tends to confirm Garf's opinion.

I don't need to measure, the trend is clear.

Well, OK then, looks like it really is a bug.
I can't believe buying a server version is the only solution to this, though. If Windows XP virtually ignores process priorities on HyperThreading systems, that would be a pretty serious flaw, which I guess even MS can't leave completely unfixed (the Pentium4 isn't a server processor after all and HT will presumably become standard on new desktop PCs in the near future).

Interesting. What are the main differences between WinXP and Win2003 generally speaking?

It's nothing new that Microsoft does'nt fix problems. Maybay if they don't self get problems from it, they ignore it. At least that was the case with MS Sidewinder FF Wheel Game-port version. No drivers for WinXP, so it was almost unusable after that...

New name, new biger price tag and now they can chek product key through windows update that i've heard...
Basicly W2k server moved to WinXP GUI with lots of fixes and some updates to server software.

Also make sure you have the latest XP kernel. At least one Hyperthreading-related problem was fixed since the SP1 kernel. The newest one i know of is linked here. Among others, it fixes this HT-related issue.

winxp comes with drivers for all game port sidewinder products

Since Win2K does see the two cpus, one wonders what MS means by 'HT-aware'.



I don't have any links. I do have dozens of logfiles showing the problem, it's especially....



...problematic on a dual since HT can significantly REDUCE performance in that case, with 2 processes getting scheduled on 1 physical CPU.



It seems more like they prefer people to have to pay for Win2k3 or Longhorn.

There is something, but i would not call them a drivers. Yes there is automatic sidewinder detection, but thats all about that. (hmmm... thist goes a bit off topic again) Force efects doesn't work most of the time, and there is nothing to adjust wheel settings like change buttons and pedals setups. So mostly it's un usable to me.

Are you sure XP (SP1) cannot do this? I had thought that it could.
edit: the product key checking of course, not the ability to increase its price...

surely any decent game will let you customize the controls in-game however it suits you. I've got their FF Pro joystick, and while the drivers certainly are nothing like the ones for win9x, they certainly do work properly. All the controls work and show up in any game I've tried, and the games let me assign whatever to whichever. Shame MS shut down the Sidewinder division, perhaps long before the actual shutdown.

Not all car driving games let me customize everything in my controller. Separated Pedals is one such thing. 90% of times i open some driving game force efects doesn't work. Is totaly random. Wheel it self is werry good shape after over four years of driving. Don't think that there is something that i have'nt done to get the wheel working... But the only aswer for me what i found was to by a Logitech MOMO wheel.

Audible!: I don't know about the WinXP key stuff, exept that the SP1 won't install with "wrong" key installed to WinXP. But SP1 is not the same thing as Windows update.

Mostly that it can tell the difference between the logical CPU and the physical one, and (tries to) make scheduling adjustments taking this into account. AFAIK, 2k Pro will not take advantage of the logical CPUs in a dual CPU HT system, whereas XP Pro will. (Remember that both 2k and XP Pro are only licensed for 2 CPUs.)

Admittedly, I haven't had a chance to try this yet. If I get a whole buncha spare time at work, maybe I can hijack our Dual P4 Xeon server for a while and see what XP Pro does on it.


Owie.


I doubt that MS would try to push workstation users to a server platform for this kind of bugfix. And with Longhorn not landing until '05 (at least!), workstation users ought to be calling in the pain.


2k Pro : 2k Srv :: XP Pro : 2k3 Srv

Roughly. There's no point in running 2k3 Srv on the vast majority of desktop systems -- even for most die-hard power users.

Thanks for the links CiTay!

Maybe you guys could take the game-controller topic elsewhere? Thanks.

Now back to the issue at hand . . .

Well obviously there is a point, if it has better HT handling than XP!?
I just installed the latest XP kernel linked by CiTay. I think I'll do some testing soon whether the HT handling got any better or not.

It does not fix the problem. Though WE obviously do not see it this way, I can actually come up with a pretty good argument as to why this is in fact Working As Designed.

For people afflicted by HT-related bugs in XP's scheduler that don't exist in 2k3's -- then there is a point. For other users who might have certain, specific needs, there is a point. These are very small segments of computer users though, even in fora like these, Ace's, or Ars. For damn near everyone else, there's no point, other than some sort of silly l33t bragging rights.

That said, with the way Intel is pimping HT, and eveyrone jumping onto the multi-core CPU bandwagon (which will require even more scheduler tuning), the most fixes will get into XP. Might take another SP or two, but they'll get there.

Pardon my ignorance... Can't you simply turn off HT on the bios or something? ^^!

But then you lose HT's benefits. And they are pretty substantial.