Thursday Feb 21, 2008

SPEC award recipients, Web Server

More SPECtacular awards given at SPEC's 2008 annual meeting in San Francisco, these to members of the Web Committee. As before, I won't post anyone's name without permission, but you know who you are and SPEC is grateful for your contributions.

It's a bit more complicated to tell you what these engineers did to earn this recognition. In 2007 SPEC released a maintenance version 1.20 to SPECweb2005 which included a lot of improvements. Most notable to me are new code that extends the level playing field of comparison, and tightened standards compliance rules that ensure real world applicability.

In order to focus on performance of the web server, to simplify the configurations tested and so reduce the cost of benchmarking, SPECweb2005 uses a simulated back-end database, BESIM, instead of the database server typically found with real world web applications. The great majority of the computing load is, by design, on the web server. Therefore the database component being synthetic code has not been an issue with fair comparison across platforms. However we discovered that when using an ISAPI implementation, two Ethernet packets were exchanged per HTTP response instead of the single packet with other implementations. No such results had been published, so there were no unfair comparisons - yet. But the problem needed to be fixed in order that ISAPI solutions could be measured on a level playing field. The solution was to adapt BESIM to use ISAPI v2.0 interface based on the Windows IIS implementation (with permission from Microsoft), so now different solutions behave alike, and they behave like real world web servers.

Another potential risk to fair comparison was in the Java Server Pages (JSP) which is typically used in benchmark results in preference to other slower scripting languages like PHP. We added a requirement to the run rules that software products prove their standards compliance by passing the relevant test suite. In the case of JSP that means that web servers which say they implement JSP must really implement JSP, not a subset needed to run the benchmark but omiting some slower features needed to implement real world web applications. Again, we don't know of any published results which did gain a performance advantage by cutting corners on standards conformance. Now that the rules are tightened we know there won't be any.

Thursday Oct 04, 2007

I'm unfair to HP

I measured electricity usage of our home computers using a cheap clamp-on ammeter from Radio Shack. I concluded that you need to make your own measurements because many of my preconceptions did not hold up to actual measurement. Since then I bought a more accurate and convenient Watts Up meter for $21 from amazon.com, and re-measured everything.

    To my colleagues at HP: I apologize for saying your PC used 300 watts.

100 watts is more like it. I likewise overstated the power usage of other PC's. I correct the measurements here, along with some discoveries about Windows and Ubuntu power management. You may be asking, why should I believe you now? Good question. You should not believe some random person you find on the net. You should get your own meter and measure your own equipment.

Why might you halfway believe me, enough to read a bit more? Well, Watts Up does seem to be a more reliable instrument, and unlike my earlier measurements it corrects for the power factor. It displays the power factor, which can vary greatly from one device to another. As a check on the new meter I measured a 3-way light bulb: the 60w setting measured 44w, the 100w setting measured 89w, and the 150w setting measured 135w. At least I'm assured it's not measuring high.

A colleague in the SPEC power committee laughed at my new meter, “You mean you trust a Watts Up meter?” For the upcoming SPEC power benchmarks that team of engineers from many companies, universities, and agencies has written elaborate requirements for the power meter and measurement protocol to be sure that results tested by one lab will be comparable to results tested by another lab. But for home, if you can get any information at all about the power your electronic gear uses, that's good.

First I looked at how many watts they used in normal home, light load operation. I switched the Linux distro on PC2002 to Ubuntu 7.04 Feisty Fawn, and its power management software seems to be the equal of Windows.


OS Computer Monitor Speakers Total
PC1999 Linux 39 26 1 66
PC1999 Windows 98 37 26 1 64
PC2002 Linux JDS 95 34 - 129
PC2002 Windows XP 61 34 - 95
PC2002 Linux Ubuntu 57 34 - 91
PC2006 Solaris 77 22 - 99
PC2006 Windows XP 67 22 - 89
 

Next I looked at how many watts they used under a heavy load. These weren't identical controlled loads, and workload really matters to how much power is used.


OS Computer Monitor Speakers Total
PC1999 Linux 46 26 1 73
PC1999 Windows 98 55 26 1 82
PC2002 Linux JDS 95 34 - 129
PC2002 Windows XP 100 34 - 134
PC2002 Linux Ubuntu 118 34 - 152
PC2006 Solaris ? 22 - ?
PC2006 Windows 115 22 - 137
 

How much of difference does workload make? Here is the power consumption of PC2002 running Ubuntu:

Operation

Watts (PC only)

download OS updates

70

system quiescent

57

100-Base-T file copies

80 - 100

Play Supertux arcade game

116

Scanning document with USB scanner and xsane

118

Afterwards [Note 1]

115

Killed mysqld

60

[Note 1] I don't know what set off the MySQL daemon, whether it was doing useful work, or had just run wild. But without the power meter, had I noticed anything amiss at all, I would have just been puzzled that the fan was running. When I stopped the daemon, power usage fell back down to normal levels.

Next I suspended each PC by pressing the sleep button as the system is configured out of the box.


OS Computer
PC1999 Linux 27
PC1999 Windows 27
PC2002 Windows 2
PC2002 Linux Ubuntu 2
PC2006 Windows 44
 

But wait, what's going on with that new HP PC taking so much more power suspended than the old HP PC? I looked at the Windows power management control panel and started reading the documentation. It turns out that the old PC was set by default for the sleep button to put it into what Windows calls “standby” state, while the new PC was set by default for the sleep button to put it into “away” mode. Away mode doesn't do much, and is intended for systems that might still be doing some processing without a person sitting there. So it's kind of like forcing it immediately into the quiescent state it would eventually go into if you just left it alone, and the monitor blanks. I changed it to go into suspend state, and then it used just 6 watts. Hibernate state used 4 watts. Michael Chu wrote a great description of how Windows XP power management works.

Here are some details of my configurations.


OS Computer Monitor Speakers
PC1999 Primary: Sun JDS3 Linux; Secondary Windows 98SE PC, Sony Vaio, Pentium III, 500MHz NEC, 14” LCD, 1024x768 Sony
M2000 Primary: Linspire Linux; Secondary: Windows 98SE Laptop, Sharp Actius, Mobile Pentium II, 300MHz integrated integrated
PC2002 Primary: Ubuntu 7.04 Linux; Secondary Windows XP Home PC, Compaq S6000Z, Athlon 2600XP, 1.9GHz Samsung 170MP, 17” LCD, 1280x1024 integrated
PC2006 Primary: Solaris Nevada; Secondary: Windows XP Media PC, HP a1540n, Athlon 64 x2 4200 dual core, 2.2GHz Sharp LL193A, 19” LCD, 1280x1024 integrated
M2007 Debian Linux Tablet, Nokia N800, TI OMAP 2420, 330MHz integrated integrated
 

About

I am a software engineer in San Diego, president of the Standard Performance Evaluation Corporation (spec.org), formerly a mathematician and a violist.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today