Performance means metrics, and metrics are worthless without clear as crystal goals to set them against. COBIT 5, ITIL, TOGAF and other IT management processes and policy ultimately derive their worth by improving performance in one or more IT silos. Performance can only be judged by goals and metrics are a measure of those goals.
Metrics (like human nature) can be both objective and subjective, so they need to be well defined in advance because you do not want to introduce biases that would substantiate any one individuals perceived view of value. Of course you may use metrics to substantiate our own conclusions too, but the reality is that if value is not felt by other stakeholders (esp. outside IT), the rest of the business will notice. I know to some of you this position sounds like performance management utopia, and maybe it is, but I’d argue transparency trumps vague 9 times out 10. Bottom line= If your goals are clear and achieved, than by proxy value has been delivered to your stakeholders, and you will have unbiased numbers to prove it.
In the EA world there are essentially three metrics that govern performance:
- Cost: meeting your budget goals
- Risk: meeting your security goals
- Scale: meeting your current and future operational goals
Different stake holders will have different agendas, and unfortunately IT is often placed in the middle of competing interests. Navigating this dichotomy ultimately starts with defining and setting metrics that speak to the overall financial goals your organization has set for itself:
- Fixed vs. Variable Cost Ratio: -or the difference between fixed billing vs. pay-go. Nearly two thirds of IT budgets are fixed costs, so variable costs mean scalability (i.e., a lower fixed-to-variable cost ratio because on-demand like AWS is highly scalable). Naturally outsourcing services can lower your fixed-to-variable cost ratio, but the benefits gained there must be balanced against other factors, such as security and compliance, integration requirements, and vendor lock in.
- CAPEX vs OPEX: the ability to move more into the operations bucket as opposed to the fixed upfront one, plays fairly well into the next metric. Some CxOs view these investments as assets, others see them as contributors to a higher fixed cost structure- as fixed asset depreciation is a fixed cost. Regardless, there is no magic number for the right CAPEX vs OPEX; it is purely business subjective.
- Unit Costs vs. Benchmarks: Per-unit base costs for (mostly commodity) services. Common categories and examples include:
- Licensing: cost for ongoing (typically annual) software vendor upgrades and support.
- Computing: human support, routers, switches, desktops, laptops, databases, storage (in TB) and servers.
- Unified communications: business costs such as IP phones, mobile devices, voicemail (per user), contact center, and email (per user, including storage costs).
There are essentially two separate types of scale we need to understand. The traditional one that MBA’s are concerned with (what we call Scale1) and ours (Scale2). The traditional Scale1 business simply tries to produce goods and services by simultaneously trying to optimize:
- Repeat-ability &
When you put those two things together you get a competitive cost advantage also known as “economies of scale” i.e. a proportionate saving in costs gained by an increased level of production. This is why cloud services exist and why Elon Musk built Tesla’s Gigafactory. Then there is the other type of scale that we deal with: IT scale. Both are vital and have to work together to keep the business alive.
Some call IT Scale “Scale2” since it is Internet based which makes it (in theory anyway) unlimited. A business doing business on the Internet sees scale in terms of data, intellectual property, and talent. A Scale2 business is driven by application and network optimization in terms of speed and usability because end users will abandon even a great product if the front end interaction fails. By proxy, successful Scale2 optimization means you are doing your job.
Scale2 Metrics vary by industry, stake holder, and end user expectations. You can find very good overviews here and here. In my experience as a designer and the fixer of broken stuff they are CSL/SLA specific. Metrics questions you should be asking:
- Are SLAs being met?
- Are incidents being handled inside the SLA? If not, why not?
- What are your current and future application performance goals? Example- how long does it take for pages to load?
- What percentage of time does an application function properly?
- How often do you audit your customer service levels (CSLs) and internal SLAs?These commitments should be looked at, at least twice a year so you know that the things you are measuring are real world oriented and not some base silo important only to a check mark on your bosses spread sheet! See the scale section for more details.
This is both the hardest and easiest group of metrics to define. If you have been reading this blog or know me you know I am a COBIT 5 guy (good piece on it’s relationship with ITIL & BS7799 here). Problem is COBIT has 300+ KPIs. I don’t want to delve into all that, but you can find some very through overviews on the ISACA site here (sorry couldn’t find that one free).
On the one hand, up-time and security of your applications obviously speaks to how well you and your team are doing your jobs, but on the other hand there will be things associated with this group of metrics that most internal and external stake holders will not know anything about until the shit hits the fan. So making these things visible early helps you set the stage for later conversations when things go wrong (Murphy never sleeps). See the risk section for more details.
As you know, metrics are key to making sure your organization proactively addresses issues before they become real problems. As an IT leader you are in the business of crisis avoidance and being prepared with hard facts is always in your best interest.