Scoring SCORE

In an article in their vehicles section, Forbes online published a list of the least safe 20 vehicles, ranked according to the Statisical Combination Of Risk Elements (SCORE) metric. The SCORE metric is published by an organization called Informed For Life (which I have not heard of before), and is an attempt to rate how safe a vehicle is by combining various statistics into one number.

I have some reservations about the usefulness of the SCORE metric, however, because it doesn't really correlate that well with actual driver death rates, as published by the IIHS.

Here is a plot published by IFL which claims to show the correlation between the SCORE an driver death rates per million registered years:


What you might notice right away is the large spread of SCORE values for any given driver death rate bin. For example, for the 50-60 bin, which is well below average, the SCORE varies from 80 to 180--from 20% better than average to 80% worse than average. I also notice the R^2 value of 0.48, which is not a stellar mean squared error. SCORE clearly has some correlation with driver death rates, but it is rough.

Here are some specific examples of vehicles that have high (bad) SCORE scores but have pretty good real-world safety records:

• Mid-sized SUV: 2004 Jeep Grand Cherokee
• SCORE = 181 (ave = 100)
  
• IIHS DDR = 57 (ave = 79)
  
• Small SUV: 2004 Ford Escape
• SCORE = 172
• IIHS DDR = 65
• Mid-sized car: 2004 Buick Regal
• SCORE = 123
• IIHS DDR = 57
  
• Compact car: 2004 Saturn Ion
• SCORE = 114
• IIHS DDR = 67
• Compact car: 2004 Pontiac Vibe
• SCORE = 115
• IIHS DDR = 62

All of these have higher-than average risk according to the SCORE metric, but in real life have lower than average fatality rates! It seems that SCORE puts too much emphasis on crash survivability, and not enough other factors like the ability of a vehicle to avoid an accident. It also seems to over-state the danger of rollovers. Something that is not widely publicized is that 70% of rollover fatalities occur when occupants are not wearing seatbelts (source: NHTSA). However, most people (~80%) are wearing their seatbelts nowadays, which partly accounts for why SUVs have a good safety record overall. The SCORE is really an estimate of how safe a car is in an assumed buy unspecified crash.

Aside; The IIHS driver death rate statistic also has some problems. IIHS does not account for driver behavior, which causes some vehicles such as sports cars to have much higher driver death rates than their crash test results would suggest. An interesting example of this is the DDR number for the Ford Mustang convertible (DDR 97) vs Ford Mustang hard-top (DDR 150). The two cars have nearly designs, from a crash point of view, but the Mustang hard-top has a much higher single vehicle death rate. It would appear that hard-top owners drive stupid more frequently than convertible owners.

The SCORE metric is an inconsistent predictor of vehicle safety in the real world. That doesn't mean it is worthless, but like anything, even the IIHS driver death rate statistic, it should be considered as part of a larger picture.