Recently I answered a question regarding why organizations use metrics, and if they [the organization] check to see if it already exists. The requester also wanted to know who measures the life-cycle of the metric. Who in the organization decides what is measured, and who controls the definition of the metric.

How does one go about measuring the effectiveness of metrics? Who watches the watcher?

You can read the original Answer request on LinkedIn.

I’d like to say that the stakeholder would typically define what’s being measured. The problem here is that the stakeholder may not understand or know what -is available- to be monitored. Perhaps as owners of the data points, defining and publishing a menu of available metrics could be beneficial? Hey! A la carte metrics!

John Fuller says in this chain: “some metrics are nice to know, but being actionable might be a different story.” His example was fuel consumption. Metrics may not only be provided “as requested” but can also be provided to modify behavior. Perhaps fuel consumption makes one a less aggressive driver, when that person sees their MPG drop below 20. Then again, perhaps not, if they don’t mind paying for more fuel to support their driving habits.

Metrics that are provided “as requested” by a stakeholder can include a broad range of purposeful uses. Typically I’ve seen that requested metrics are knee-jerk responses to applied historical lessons learned — meaning, the requestor has been burned by not knowing something. An example: the project went over budget: “give me a metric showing the budget utilization!” This is an effort to be proactive rather than reactive due to previous consequence. Who says a manager can’t learn from mistakes!

Using the dashboard of a vehicle as an example (continuing John F’s answer) shows that sometimes simplification of literally -millions- of data points down to a succinct few that matter — at a point in time when they are relevant. How much gas do I have left (operational)? Is my engine running hot (operational)? How fast am I going (legal compliance)? what gear am I in (procedural)? Am I consuming too much fuel at this rate (budgetary)? … along with -hidden- metrics we lovingly call “idiot lights” (whoops! oil pressure low?! out of gas?!) Some metrics (such as the out of gas example) apply to -other- metrics that we’ve ignored or let slip, (such as the fuel gauge) and serve as a warning indicator for those that may not have understood the importance of the metric (keeping the tank topped off.)

Too often we overload on metrics, it’s my opinion that we should attempt to simplify the information to that which is salient for a given consumer and make sure the consumer actually understands the data being presented. Data is pretty useless without context. (”Hey, what does that little genie-oil-lamp thingie on my dashboard mean? It’s been glowing red for a couple of weeks now!”)

Just a few passing thoughts on metrics

I know, I know… I shouldn’t use so many parentheses!! 80% of all metrics use parentheses! (Oh, did I mention that your metrics and statistics should be provable to survive scrutiny? Yeah, that too.)

[Edited to fix a stupid grammar error. Hey, nobody’s perfect!]

This entry was posted on Friday, August 15th, 2008 at 4:48 pm and is filed under IT Management, IT General, Articles, Q&A.