Is your measurement system well balanced?

Brown, Mark Graham

As quality management advocates we tend to think measurement is the foundation of an effective management strategy. We like to spout off statements such as: "You get what you measure" or "Improvement is not possible without measurement." That's pretty much old news for anyone who's read anything about quality. Here's the big news for those of us who happened to miss it: measuring more things doesn't get you more quality or guarantee quality. And even better news: implementing a change effort to make your organization more customer focused, does not necessarily entail more measurement of more things.

Although measurement is critical to improved performance, organizations don't always get what they measure. If measurement by itself really had that much impact on behavior, anyone who had a scale would never be fat. Measurement only provides you with data. If the data are not used to make good business decisions and to drive improvement efforts, a good measurement system is of little value.

Selecting the vital few applies to metrics as well as to problems or strategies -

Organizations have spent a large portion of time measuring and reporting data long before the TQM movement ever came along. Those who have followed Baldrige Award criteria as their roadmap for improving their organization have learned that selecting the vital few metrics on which to collect data is what is important, rather than collecting data on everything.

Creating a balanced scorecard... It is also important to have what Robert Kaplan and David Norton refer to (Sept./Oct., 1993 Harvard Business Review) as the "balanced scorecard." Kaplan and Norton suggest that most organizations who do not have a balanced set of measures, concentrate almost exclusively on short-term financial measures and ignore longer-term more strategic measures such as customer satisfaction, employee satisfaction, and growth.

A balanced set of criteria boxes -

The Baldrige Award criteria (in Section 2.0) recommends a balanced set of metrics (suitable for most organizations) that should include roughly the same amount of data in each of the following data drums:

*Customer satisfaction

*Employee satisfaction

*Financial performance

*Operational performance (e.g., cycle time, productivity, et cetera)

*Product/service quality

*Supplier performance

*Safety/environmental/public responsibility

The usual or obvious 80/20 isn't enough work anymore... In most organizations, about 80 percent of their measure are for two drums: financial and operational. Once a year they do a customer satisfaction survey and once every three years they do an employee morale survey. They have a few statistics on product quality, safety, and inspect incoming supplier shipments. This is just what we mean by not having a balanced scorecard.

I have developed the following survey to overcome the short-sighted and unbalanced way of measuring an organization's performance. The survey will assist you in evaluating whether you currently have a balanced set of measures and how you are using current data to improve organizational performance.

By completing this self-assessment instrument you will learn the characteristics of an effective measurement system, and how well yours stands up to these standards and practices.

Taking measure of your measurement system

Questionnaire directions -- The questionnaire is divided into three sections, each addressing an aspect of your measurement system.

* Part I (questions 1-5) is about your overall approach to measurement.

* Part II (questions 6-41) questions ask about specific types of measures.

* Part III (questions 41-51) is about how you analyze and use the data to improve your organization.

Read each statement and check the appropriate box, depending on the extent to which you strongly agree (5) or strongly disagree (1)with the statement. Answer every question even if you have to guess.

The scope of the questionnaire should pertain to your entire organization, or at least a large enough portion of the company or organization that could be a stand-alone business/organization. For example, you could do a business unit rather than the whole company, or one hospital in a chain of hospitals. You could and should not use the questionnaire to apply to a single department such as radiology, or human resources.

Part I: Overall approach to measurement

1. Our organization has developed a specific set of criteria for screening out extraneous measures from our data base.

-- 5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

2. Our data base was built with a plan, rather than something that just evolved over time.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

3. Our CEO or President looks at no more than 20 measures every month to evaluate the overall organization's performance.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

4. Measures of performance are mostly consistent across our business units/locations.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

5. We have a well balanced set of measures, with about equal amounts of measures/data in each of the following categories: financial performance, operational performance, customer satisfaction, employee satisfaction, product/service quality, supplier performance, and safety/environmental performance.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

Part II: Specific types of measures on your scorecard

Customer related measures

6. Our data base includes good hard measures of customer satisfaction such as repeat/lost business, returns, etc.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

7. Our organization collects data on customer feelings/satisfaction levels using a variety of techniques such as telephone surveys, mail surveys, and focus groups.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

8. Our scales for measuring customer satisfaction focus on delighting customers rather than just satisfying them.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

9. What we ask customers in our satisfaction surveys or discussions is based upon thorough research to identify customers' most important requirements.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

10. We combine various hard and soft measures of customer satisfaction into an overall Customer Satisfaction Index

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

Employee related measures

11. We survey our employees at least once a year to determine their satisfaction levels with various aspects of how the organization is run.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

12. Employee surveys are anonymous and more than 75 percent are returned each year.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

13. Research is done to determine what is important to employees before putting together or buying a survey with standard questions.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

14. Our organization collects data on other metrics that relate to employee satisfaction such as voluntary turnover, absenteeism, hours worked per week, requests for transfers, et cetera.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

15. Individual measures of employee satisfaction are aggregated into an overall employee satisfaction index, similar to the customer satisfaction index.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

Financial measures

16. We have identified a few (e.g., 4-6) key measures of our overall financial performance.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

17. Financial measures are a good mix of short and long-term measures of financial success

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

18. Financial measures are consistent across different units/locations.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

19. We collect financial data on our major competitors to use in evaluating our own performance and in setting goals.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

20. The organization aggregates financial data into one or two summary statistics that reflects overall performance, such as economic value added (EVA) or return on assets (ROA).

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

Operational measures

21. The organization has developed a set of 4-6 common operational measures such as value-added per employee that are used in all locations/functions.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

22. Any process measures that are collected are directly related to key product/service characteristics that customers care about.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

23. Cycle time is used as a key operational measure throughout the organization.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1)Strongly disagree

24. Operational measures allow you to prevent problems rather than just identify them.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

25. The organization has established measurable standards for all key process measures.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

Supplier measures

26. The organization has a rating system for evaluating supplier performance.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

27. Our supplier rating system is a mix of hard data such as products returned/shipments rejected, and soft measures such as our satisfaction levels with suppliers' responsiveness.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

28. The quality of goods and services purchased from suppliers is measured on a regular basis.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

29. Our organization asks suppliers for process data and encourages self-inspection.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

30. Staying within our price guidelines is only one of many measures used to evaluate and select suppliers.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

Product/service quality measures

31. Characteristics of products/services that are measured are those that are most important to customers.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

32. If 100 percent of products/services are not checked, then large enough sample sizes are used to ensure that all meet standards.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

33. Automated measurement devices are used wherever possible to avoid errors caused by poor human judgment.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

34. Measures for services are related to accomplishments rather than behaviors (e.g., percent of correct orders filled, or percent of flights that take off on-time versus smiling when greeting customer).

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

35. Measures of product/service quality are expressed as actual number rather than percentages of defect-free products/services.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

Safety/environmental/Public responsibility measures

36. The organization collects data on safe and environmental performance at least once a month, using several different metrics.

--5)Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

37. Measures of safety are more behavioral and preventive in nature rather than the typical lost time accidents.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

38. Environmental measures go beyond those mandated by the EPA and other regulator agencies.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

39. The organization collects data on measures of public responsibility such as hours of community service or awards received from community/civic groups.

--5) Strongly agree 4)Agree 3) Somewhat 2) Disagree 1) Strongly disagree

40. The organization has developed a public responsibility index that is an aggregation of safety, environmental, and community service measures.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

Part III: Reporting and analyzing data

41. The organization reports data from all sections of its scorecard in a single report to all key managers.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

42. Data are presented graphically in an easy to read format that requires minimal analysis to identify trends and levels of performance.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

43. Data on customer satisfaction, employee satisfaction, and public responsibility are reviewed as often and by the same executives as data on financial, operational, product/service, and supplier performance.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

44. The organization has done research to identify correlations between customer satisfaction levels and financial performance.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

45. The organization understands the relationships between all the key measures in its overall scorecard.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

46. Performance data are analyzed and used to make key decisions about the organization's business.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

47. The key measures are consistent with the organization's mission, values, and long-term goals and strategies.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

48. The organization continuously evaluates and improves its measures and the methods used to collect and report performance data.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

49. Automated and human (e.g., surveys/checklists) measurement devices are calibrated on a regular basis to ensure accuracy and reliability.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

50. The measures in the organization's scorecard are the same ones on which annual and longer-term goals are set during the planning process.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

Calculating your score -- Questions 1-5 relate to your entire measurement system, so they are worth more than the rest of the questions. Add up the total for questions 1-5. A perfect score would be 25, if you answered Strongly agree for all five questions. Write the total for questions 1-5 in the space below.

Proceed by adding up the total for questions 6-40. Next, add up the total points for questions 41-50, and multiply this number by 2.

Add the three sub-totals to give yourself a grand total score. A perfect score on this assessment is 330, so if you ended up with more than that, go back and check your math.

Total questions 1-5 X 2 =

Total questions 6-40 =

Total questions 41-50 X 2 =

Grand total =

Interpreting your score

Scores of 276-330 -- If your score on this survey ended up in this top band, you truly have a world-class approach to measuring your organization's performance. You have narrowed down your database to a few key metrics and must have a well-balanced set of metrics. It also is evident that you actually use the data you collect to make decisions about improving organizational performance. Yours should be an organization that others benchmark for measurement.

Scores of 226-275 -- If your score ended up in this second band you have a systematic approach to measurement that approaches being well-balanced. Chances are, you are weak in measures of customer satisfaction and employee satisfaction, and may not do a good job of aggregating individual metrics into summary statistics, and analyzing the data to improve organizational performance. You have made a great deal of progress in improving your organization's approach to measurement. However, additional refinement is needed over time, and more research needs to be done to identify correlations between long-term measures such as customer satisfaction/employee satisfaction and shorter-term measures such as financial performance. Being in this band probably means that your measurement system is better than 75-80 percent of organizations in North America.

Scores of 176-225 -- A score in this range puts you in about the middle, which says that you are off to a good start in re-engineering your approach to measurement. You probably have a good set of measures for some of the seven boxes on an organization's score card. You also probably have some major weaknesses in some types of measures. You are probably strong in financial, operational, and product/service quality data, and weak in the other four areas. Chances are you still have too many measures, and have inconsistencies across the different units/locations in your organization. A score in this range says that you are making some refinements in your approach to measurement, but still need to do quite a bit of work to put together a good solid measurement approach.

Scores of 175 or less -- This puts you at the 50 percent or below level, which means that you are a long way from having a balanced score card. You're in good company at this level, however. In my experience, this is where most businesses are, and where almost all government and healthcare organizations are. Most business organizations are only just starting to measure customer satisfaction and employee satisfaction. Government and healthcare organizations are weak in these two areas, and also tend to be weak in product/service quality data, and measures of supplier performance. Organizations that score at less than 50 percent on this survey probably still have not convinced upper management that strategic longer-term measures are just as important as the traditional financial and operational metrics.

How to improve your measurement system

Your first step, if you haven't done so already is to identify your vision for the future for your organization, and identify goals, and critical success factors necessary for you to achieve your vision. The example of a paper company's vision statement, goals, and critical success factors is just one example for your reference.(1)

Once you have defined your vision, goals, and factors necessary for you to achieve your goals, it becomes fairly easy to begin to identify important measures. Some will be obvious, and come directly from your goals or critical success factors. For example, we know that revenues, overhead expense, product cost, and profit are some of the important financial measures. The goals and critical success factors should drive the identification of measures for the organization's score card. Re-defining your measures needs to start from the top, and cascade down to all levels in the organization. It may be that you end up with thousands of metrics, but each function and each employee needs to have a balanced set of measures covering the seven categories outlined in this article.

Building a simple and well-balanced score card for your organization will lay the foundation for your improvement efforts and ensure that you don't focus all of your efforts on today's success without thinking about the future.

Author's note: For additional information on how to put the concepts in this article to use, consult: "Putting the Balanced Scorecard to Work" by Kaplan and Norton, in the September/October, 1993 issue of the Harvard Business Review.

(1) Vision, goals and success factors for a paper manufacturing company...

Vision... Be the preferred lowest cost, highest quality supplier of newsprint for customers in major cities across North America, with at least 35 percent of the market in the U.S., and 25 percent of the market in Mexico and Canada.

Goals...

* Achieve preferred supplier status by 50 percent of our customers

* Grow market share by 20 percent per year in the U.S., and 10 percent in Canada and Mexico.

* Reduce product cost by 10 percent per year

* Become a major supplier to the LA Times, New York Times, and USA Today

* Revenue increases of 12 percent per year

* Maintain profits at 15 percent or higher of revenue

Critical success factors...

* Invest in new paper machines in three of our mills

* Reduce manufacturing cost through innovations and process improvements

* Increase pulp quality and reduce costs

* Obtain initial contracts with LA Times, New York Times, and USA Today

* Reduce overhead expense by 10 percent

* Reduce product defects by 50 percent from 1994 levels

* Increase customer satisfaction levels by 20 percent within all customer organizations to achieve an average rating of 85/100

* Implement a systematic approach to managing supplier quality

* Reduce inspection and focus more on controlling key process variables.

Mark Graham Brown has spent the last 16 years consulting with organizations on improving their quality and productivity. Some of his current clients include IBM, Pacific Bell, Air Products & Chemicals, and Cargill. He is the author of several best-selling books, including Baldrige Award Winning Quality, The Pocket Guide to the Baldridge Criteria, and Why TQM Fails and What to Do About it. (with Darcy Hitchcock and Marsha Willard). He is currently working on a new book called: Keeping Score: Re-Engineering Your Organization's Measurement System, which will be published in 1995.

Copyright Association for Quality and Participation Oct/Nov 1994
Provided by ProQuest Information and Learning Company. All rights Reserved