Microsoft: Building better security through statistics.

Just as it did last quarter, Microsoft’s own security report says Vista is more secure than other operating systems.

In order to pull this off, Microsoft had to redefine how to measure security. In their world it isn’t about actual attacks, but rather a game of statistical juggling.

You know the saying: There are lies, damned lies, and statistics.

Can we get back to reality? What matters are actual attacks in the wild. Was my system compromised (applications act funny, popups from nowhere, lost data, system degraded, crashes)? That’s what users care about. By this obvious measure Windows has always been a virus and malware magnet. There’s no denying this. It’s why antivirus and anti-malware software is a requirement on a Windows system (including Vista). No reputable party disputes this. No wonder Microsoft had to redefine security metrics!

And so began what I’ll call the ‘Microsoft Security Redefinition Campaign’ (MSRC). This was a two-step process:

  • STEP 1: Write off existing Windows viruses as only a symptom of being popular. As if someone with malicious intent seeking machines cares what OS you’re running. If he could hack your system easily, he would. Windows is low hanging fruit because it’s easily exploited. If it was only a numbers game then Mac OS X’s roughly 6% share in the US would translate to 6% of viruses. Mac OS X has nearly 0% in the wild. Microsoft invented this argument solely to wipe the slate clean for comparison to Linux, Mac OS X, and others. It’s like saying “we’ve been readily exploited for years but let’s start over”. Other platforms don’t have to pick starting points from which to attempt to look good.
  • STEP 2: Turn the process of frequently releasing security patches into a statistical calculation. All OS vendors patch their systems, but Microsoft tallies them in their own way, creates some statistics, and then releases it as a “report” that makes them look good by comparison.

Sadly, the MSRC has had some success. Even though Vista’s security is only on par with XP, Microsoft released a 90 day Vista vulnerability report last March. Oooh, a PDF file with graphs, and it says that Linux distros are not doing as well as Microsoft. Neither is Apple. Amazing. Countering everything known about UNIX security, as well as demonstrable, empirical evidence of precious few Linux/Mac users getting attacked in the wild, Microsoft implies that UNIX systems are somehow less secure. Nonsense!

But the press bought it, especially to use as a gloating response to Mac users allegedly claiming they don’t worry about security. (If I had a cent for every article that began “Mac users should worry about security after all…” I’d have more money than the multi-billion dollar antivirus industry Windows sustains.) Apple users do not claim they don’t worry about security; they do claim they don’t worry about the 110,000+ known viruses on Windows PCs, which is absolutely true. But since step one of the MSRC attempts to wipe those viruses off the board, Windows apologists pretend they’re not relevant.

The MSRC Bus Tour 2007 is again in full swing, and coming to a town near you. They just released a Vista six-month vulnerability report. More graphs and (surprise!) UNIX bashing.

Luckily, not everyone is easily duped. Microsoft Watch was not impressed with the 90-day report, saying this at the time:

“Last week, Jeff Jones, Microsoft’s security strategy director, released a rosy report about Windows Vista’s security progress. Counting Jones’ way, Vista has a pretty good 90-day track record compared with other operating systems. But counting another way, the vulnerability number is much higher.”

Now they’ve published an article denouncing the six-month report as well, saying:

“[Microsoft] is once again counting security bugs, and possibly to a fault.There are some things you count and compare, and some things you don’t. Security flaws should be in the don’t category, not that Microsoft seems to get it. For years, the company has used number of flaws as a measure for touting security improvements. Counting is a great security by PR approach, but little more.”

“The point: Don’t count on security flaw counting. The real flaw is the counting.”

I hope other sites in the tech community will call Microsoft on this as well. Let’s not forget it’s Windows’ many vulnerabilities that wrought the long list of viruses and malware for which third-party protection and assistance is mandatory. An entire industry is built around it! Did Vista plug all those holes? Too early to tell. Did Windows 98? 2000? XP? SP2? Internet Explorer 7.0? It would seem Windows security is still very much a work in progress.

We must measure security by results in the wild. What is the likelihood of attack in normal use? Not in a lab, or think tank, or theory, or contest. Those have a place in terms of security research, but are not substitutes for the real world interaction of a user browsing web sites, downloading files, getting email, etc. Did we suffer through an attack? That’s the question that matters to users. You can brag about changing our door locks every week (and even claim to have “better” locks) while our neighbor’s stays the same, but if our house is the one usually broken into which one do you think is considered more secure?

Screw Microsoft’s report. It’s a marketing piece. The real security metric is to measure actual attacks against users. This was obvious before we let the MSRC orchestrate a change when we weren’t looking. It’s time to pay attention and get back to reality.