MEASUREMENT MANAGEMENT

James A. Ward, PMP

Measurement is essential to any quality improvement effort. Unless measurements are adopted and used, improvement will not be sustainable. However, most organizations that have initiated measurement programs have been dissatisfied with the results. Why is this the case and what can Information Systems management do to ensure success?

M

easurements are quantified observations of some aspect or attribute of a process, product or project. Measurements enhance our ability to understand things not accessible to our native abilities and intelligence. Useful measures provide insight as a basis for action. They support effective analysis and decision making.

The only way to learn how to measure is to measure. You can’t control what you can’t measure. At the same time, the usefulness of measures must be balanced against the cost and effort of obtaining those measures.

As an Information Systems executive, you have almost certainly been asked to establish and fund a measurement program. A program may have already been established in your organization or you are considering setting one up. Possibly, you have initiated and subsequently dropped a measurement program.

This column will explore measurement from a management perspective. What are the essential elements? How do you ensure that you get value from such a program? For that matter, why measure at all?

All activities performed by your organization should be measured. This is the only way that improvement will be sustained. The most difficult measures, and those where the most benefit can be realized, are those processes, projects and products of information systems development and maintenance. As an organization, Motorola is further along on this process than most.

The Motorola experience was that the software development process must be nudged from an art form to a measurable process before it can be rigorously controlled. This “nudging” involves establishing stable, repeatable and visible processes for all work done within the information systems department. This ensures that work will always be performed in the same way, with the goal of reducing variability. Work processes can then be effectively measured and compared across projects as a basis for control and continuous process improvement.

If you are just getting started in Total Quality Management, effort is better spent on establishing stable processes than on setting up a measurement program. A process must be defined before it can be measured and controlled. For a complete discussion of this topic, see the Quality Management article on “Continuous Process Improvement” which appeared in the Spring 1994 issue of the journal of Information Systems Management.

We will not discuss measurement techniques or specific measures. That is for the practitioners. There is ample literature on such topics as software metrics, statistical process control, lines of code and function point analysis.

Ten Precepts of Measurement

I have identified ten attributes of a successful measurement program and will discuss each in some detail. These are the attributes an Information Systems executive should look for and insist upon when establishing a measurement program.

Measure what the customer cares about.

Measurements should have focus, based on goals and models. However, in most organizations there is little agreement or knowledge of what goals should be. Goals must relate to customer satisfaction. Ultimately, the customer is the only arbiter of success. Define quality elements and critical success factors in the customer’s terms and develop measures to evaluate them. See the Quality Management article on “Meeting Customer Requirements” in the Summer 1994 issue of the journal of Information Systems Management for guidance in identifying customers and defining their needs.

Customers don’t care about such measures as "lines of code" or "function points" unless they are related to quality. These and other second order measures must correlate directly with such things as maintainability, expandability, reliability, etc. There is little agreement within our industry as to how to achieve these attributes of information systems, let alone how to measure them.

Mature TQM organizations use such customer focused measures as errors per user transaction and mean time to customer failure. Information systems organizations have achieved “Six Sigma” quality levels for these measures, representing less than three errors per million transactions.

Customer satisfaction surveys are essential to any measurement program. The construction, administration and evaluation of surveys is the subject of books and training classes. Seek professional guidance. Doing it wrong is worse than not doing it all.

Measure perceptions - everyone’s! Include employees, management and suppliers as well as customers. Include former or potential customers, suppliers, employees and management. These individuals will provide extremely pertinent information about your organization. The disparity in perceptions of quality and productivity between these groups will be quite revealing.

Measure the process, not the person.

The goal of a measurement program must be improvement, never evaluation. Measurement must support continuous process improvement. Over eighty per cent of quality problems can be corrected by concentrating on the process. This is management’s purview.

When measurements are used primarily as objects of management control, they are bound to be resisted. This is probably the single greatest reason for dissatisfaction and failure of measurement programs. Measures are adjusted (fudged) to tell management what they want to hear. Employees actually taking the measures get no value from them, only negative feedback.

The first instance of using measures for evaluation (especially if the measures are faulty), can irreparably harm even the best planned and administered measurement program.

Set goals.

Know what the results of measures should be, before you measure. Know what you desire. Benchmark against your competitors and against “world class” organizations. Setting goals for quality measures ensures that the organization remains focused on the key items important from the customer’s point of view. Goals must be ambitious, such as Motorola’s Six Sigma standard or “Zero Defects.” Six Sigma is the Motorola performance standard which specifies that all processes must be controlled to the level of quality which will yield no more than 3.4 errors per million opportunities. The term six sigma comes from the mathematical derivation of six standard deviations from the mean for any given statistically measurable process.

Goals should always be quantified and expressed in actual numbers, not percentages. A good example of this comes from Federal Express, which moves 30,000 packages per day. Even a 99 per cent success rate means that 300 packages didn’t get to their destination as required, representing 600 dissatisfied customers daily. Even on dissatisfied customer is too many. Percentages always imply some acceptable level of failure.

Know what to do about the results you get.

Use measures to manage effort more successfully. Measures must be a basis for judgment or action. When measures are not achieving goals or are not moving in the right direction, know what intervention to take. You must also know how your intervention will affect measures, both short term and long term.

Recognize and plan for indirect effects your intervention may have. There may be unexpected or undesired side effects. For instance, one manager started tracking the time people showed up for work because he thought he had a tardiness problem. As a result, people who had stayed long after normal working hours started leaving promptly at quitting time. Another manager attacked the same problem by making work more interesting and challenging.

Anticipate the results of your intervention.

Get feedback, from all participants. Establish the effectiveness of your measurement program by measuring how measures are actually used. Are you getting the results you want? It is more important to know this if you are achieving positive results than if you are getting negative ones. How will you sustain progress if you don’t know for certain what is causing it?

The process itself should yield measures.

Don’t create a separate system to gather measures, they should be a natural byproduct of performing the process. Measurement should not be additional overhead or burden upon the people collecting the measures. Automate the collection of measures where possible. This will become easier as the organization becomes more sophisticated in stabilizing and automating its processes. An automated measure is better than a manual one, but a manual measure is better than none. As an example, an automated project control system can be used to measure the number of tasks completed on schedule. A manual project management system should also yield this measure with little effort.

Measures should be numerical and objective.

Measurement is not done well subjectively. As stated earlier, measurements enhance our ability to sense things not accessible to our native abilities and intelligence. Subjectivity contradicts this. It also makes measures open to interpretation, and no two persons will interpret them in the same way.

Measurement criteria should be well documented and understood. Subjectivity will cloud this understanding. Where measures are objective and numeric, all parties know exactly where they stand.

Publish and publicize measurement results.

Provide feedback to the source, both of measures to be taken and the results of those measurements. Make sure everyone involved understands the purpose of the measurement program. Measures must be beneficial to the persons collecting the data and performing the process measured.

Publishing the criteria is as important as publishing the results. This will provide opportunity for feedback and improvement of the measurement process as well as the process being measured.

An easy example is the “Lines of Code” measure. What constitutes a line of code? Do COMMENTS count? If not, how do you get programmers to comment their code? How are differences in programming languages reconciled? What are you trying to accomplish with this measure?

Publishing results lets everyone know how you are doing. Publicize improvements. This will help maintain momentum - everyone likes to win. While individuals should not be measured, be quick to recognize individuals who have contributed to an improvement in the measures or the measurement process.

Ensure comparability of measures.

This goes back to the introductory comment about time being better spent in establishing stable processes than in setting up a measurement program. Without stable processes and standards, measures are meaningless. In systems development work, individual performance has always been the largest variable. This work must be converted from an individual art form to a repeatable process before it can be effectively measured and controlled. Encourage the use of the best practices by all your staff, bring everyone up to the level or your best people.

Until several projects have been done using the same set of tasks producing the same outputs, there will be little comparability within your organization across projects. One of Deming’s best definitions of quality is the elimination of variability. Process stability is the way to achieve this result.

What you measure is what you get!

Finally, this is the most important measurement precept, even more so than relating measures to customers. If you measure Lines of Code, you will get Lines of Code. People respond to whatever they are asked to do. If you measure how many projects are completed on schedule, you are likely to alter estimating and reporting practices more than you are to improve performance.

Measures of “productivity” are inherently dangerous because they will drive behavior, even if it is counter productive. Often, these measures will lead to optimizing various steps in the process while lengthening the entire process. Measure quality and let productivity take care of itself, because it will. As Deming once said, “Measures of productivity do not lead to improvement in productivity.”

Measurement is only a means to an end - the continuous improvement of processes - and never an end in itself. n

ABOUT THE AUTHOR

James A. Ward is an independent consultant specializing in systems development project management and implementation of quality and process improvement initiatives in Information Technology organizations. He holds a PMP certification from PMI. He can be reached at (904) 273-8777 or soozward@earthlink.net . Visit his web site at www.JamesAWard.com .