The real story behind the Unix 'survey'

Last month, there was a 'survey' on Unix server users' vendor preferences by a group named GCG. The result of the 'study' seemingly showed Sun slipping behind IBM and HP, and was widely picked up by media without going into facts per se. Later, Paul Murphy came out with the real facts the study obviously (or deliberately) ignored.

In short, the study used something called VPI to calculate a vendor's score. Out of 197 respondents selected for the survey, let's say Sun is primary vendor for 140 of them(as the largest Unix vendor). Most of the customers have more than one vendor. For any metric, if 120 out of total 197 vote for Sun as preferred vendor, it should be 120/197 = 61% ?

OTOH, if HP has 40 customers and for the same metric as above, if 44 vote for HP it should be 44/197 = 22%.

Now here is the interesting part. According to that 'survey' HP beats Sun in this metric. Why? They calculate something called Vendor Preference Index (VPI) which is even more interesting. They calculated it by dividing the number of votes a vendor gets by number who use the vendor's Unix.
Going by this, HP gets 44/40 = 1.1 VPI and Sun gets 120/140 = lowly 0.85!

So even though Sun has 61% of respondents giving it thumbs up as compared to HP's 22%, HP actually beats Sun if you go by VPI and the 'survey'!
The calculation goes on for many metrics and, as expected, results are heavily skewed against Sun.

Detailed study of the 'survey' can be read here from where all the above numbers are quoted, and as an example of media blindly reporting the results, read this.

Comments:

I am going to have to disagree with you here, Suren. I don't think that the study is skewed at all, it shows exactly what it purports to show. The problem is that people infer quite different facts about it.

The study is trying to measure brand loyalty, and does a fairly accurate job of it, at least mathematically. If I take a survey of political parties and ask 10,000 Republicans if they are happy with their party and 75% said yes, and then I ask 1000 Green party members if they are happy and 99% said yes, does the fact that 7500 Republicans said yes while only 990 Green party members did change the fact that the Green party shows greater party loyalty? That is what you are claiming is unfair with the survey, the normalization.

Posted by Brian Utterback on January 05, 2006 at 06:26 PM IST #

Brian, it is not as simple as that. Going by your reasoning, no vendor should get more than 100% of VPI. What goes on with this 'survey' is this: Taking the same example as in the post, let's say all the respondents are given a questionnaire of 10 q's. 140 of the respondents have Sun as their primary vendor and 40 have HP. Now, not many of those 140 who prefer Sun will fill all 10 q's in favour of Sun. This is a natural tendency to be seen as fair. Let's say 20 of the 140 write HP for q # 1 just to be fair while the remaining 120 tick Sun. Now if 22 of HP's 40 (who have it as primary vendor) tick HP for q # 1, it gets a total of 44 votes even though only 22 of them have HP as their primary vendor and 22/44 is worse than 120/140. It goes on to show that such studies are inherently skewed against the leader.

Posted by Suren on January 05, 2006 at 07:14 PM IST #

Post a Comment:
  • HTML Syntax: NOT allowed
About

surenssamant

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Bookmarks
Link

No bookmarks in folder

Blogroll

No bookmarks in folder