Measuring the Right Metrics

***Originally Posted 5/4/07

As I indicated in my previous post, I've worked for a lot of companies in my 11 years in the
IT industry. I've spent most of my career as "the new guy", as is likely to happen when you
find a new home every 18 months or so. As the new guy, I've been in the best position to
recognize nonsensical processes, counterproductive policies, and disincentivizing metrics.

For example, on the surface, tracking how many calls a help desk technician can process in a
day might be an excellent way to demonstrate productivity. On the other hand, it's far
easier for that same person to answer the phone, take the name, guess at the problem, and
move on to the next call than it is to actually provide meaningful assistance. As the person
calling the help desk, would you prefer a quick phone call, or a resolution to your problem?
A more meaningful measurement is customer satisfaction, or time to resolution, or, another
common metric, average wait time after placing a call.

System availability is another spurious metric. Achieving 4 nines, or 5 nines (99.99 and
99.999% up time, respectively) receives a lot of press and, again, on the surface appears a
lofty goal. In a given year, there are 8,766 hours (24 x 365.25 days). If your systems are
available 99.999% of the time, that means they are down 5 minutes per year. With mirrored
redundancy, this is accomplishable, but is it necessary and cost effective? The average
worker only works 2,000 hours a year, 40 hours a week, 8 hours a day, Monday through Friday.
If a system is unavailable from 11 PM Friday until 3 AM Monday, does it really matter? A
more realistic measurement is uptime during work hours. For most applications, we can very
easily, and very cheaply, achieve 99.999% availability during work hours. In fact, many
applications I've managed have achieved 100% uptime during work hours.

Finally, after the debacle of the early 2000's, where IT spend increased rapidly and
business value decreased even faster, our industry began applying metrics to projects. Once
again, do we track the right metrics, and, more importantly, do we place the appropriate
emphasis on them. Three metrics any competent project manager will track during the course
of a project are cost, timing, and deliverables. Basically, is the project team delivering
the defined deliverables on time and on budget. But where do we track if the project
delivered to expectation? Where do we validate that, while technically functional, the
system meets the wants and needs of the user community? I can deliver an application that
has every field a user will ever enter on one page, and that will often meet the "technical"
requirements, but no one will want to use it. Placing the emphasis on the triple
constraints, exclusively, dilutes the value of the project by focusing effort on the wrong
areas.

Technology is at a cross-roads. As an industry, we are suffering the sins of our
forefathers: those well intentioned geeks who believed the myth that technology could solve
every problem, if given enough time and enough money. Their misguided belief has resulted in
leaders of finance, who view the world through "carcass value" lenses, controlling an
environment driven by innovation, risk, and creativity.

The true leaders of today's technology landscape recognize that we provide business value
not by taking orders, cutting costs, and keeping the lights on, but by understanding the
goals of the business, partnering with them, and introducing innovation to help achieve
those goals. Technology professionals earn their high salaries by solving problems the
business can't, not by automating their solutions.

We entered this profession to solve complex problems. Many of us thought those problems
would be life changing and world altering - and some of them are. But the more common
problems, the more abundant problems, the more important problems to our employers, are the problems they face every day that we can solve with two weeks of work. Where are the metrics
for that?