Measurement

In May and June of 2006, CUX undertook a benchmarking study of companies with environments that required distributing learning, often in a regulated environment, to associates, sales people and others, in branches, dealerships, sales centers, etc. The work was sponsored by HSBC Consumer Lending.

The full report deals with all aspects of distributed learning, from infrastructure to learning design and development, to delivery issues. The issue of measurement, particularly as it relates to business performance, is one that every L&D organization faces. This excerpt describes CUX's findings and recommendations in this critical arena. The report is available to CUX members in the Collaboratory under Learning Strategies.

MEETING BUSINESS REQUIREMENTS

Determining how customers do or don't value an organization's product and services has great relevance for L&D. To measure customer satisfaction, the organization analyzes product sales, market penetration and other such metrics to evaluate product success. L&D organizations also must employ methods to judge the value of the solutions they provide.

In the past, L&D, like most business functions, was a victim of industrial-age thinking and counted its outputs as a measure of success. The number of courses produced, the number of seat-days delivered in the classroom, the percentage of high ratings in student class evaluations have all been measures of training success. These metrics illustrate performance in tactical training operations and are valid in discussions related to throughput, efficiency, and cost. They are the factors that relate to the notion of "running training like a business."

But a much more important discussion is beginning to take shape, as L&D shifts its emphasis to "running training for the business." In this approach, the priority is to monitor outcomes rather than outputs. As L&D recognizes its higher purpose to impact business outcomes, new measures of success emerge. The first consideration for monitoring outcomes is to determine if L&D products and services are meeting business requirements that were defined prior to solution development. Organizations are employing numerous techniques to determine if their products hit the mark.

Measurement has always been difficult for L&D. Many factors contribute to good or bad performance and it often can be difficult to isolate the specific impact of training. However, there is tremendous new focus on this as L&D groups recognize the need to illustrate the value of what they do.

To that end, survey respondents said they use all of the following methods to decide if their programs meet the business requirement and are delivering desired results:

• Feedback from senior managers on success of strategic initiatives
• Monitoring key business metrics
• Business unit success in achieving objectives
• Supervisor's assessment of improvements by low and average performers
• Asking individual to assess their own level of improvement

The technique employed most frequently is monitoring business metrics, and yet only 27% of respondents are using this method. Unfortunately, L&D groups are not often doing a good job getting agreement with managers upfront before training occurs as to the optimal measure of success. This also is slowly changing as more groups begin to build this agreement on measurement into the design stage of the learning program.

MEASURING IMPACT ON THE BUSINESS

Learning organization goals have to be set based on the organization's business strategy, and not on some arbitrary list of impact measures. For that reason this table and the discussion of business impact measures below should be used, not as any kind of validation or benchmark that describes what an organization should be measuring, but rather taken as an idea of what could be measured and what other organizations are looking at.

Another survey question probed the measurements that relate directly to the business. These included:

• Reducing time to market for new products/services
• Bringing in new business
• Improving sales effectiveness
• Developing innovative products/services
• Entering new markets
• Increasing market share
• Improving customer service
• Increasing revenues
• Increasing profits
• Improving product/service quality
• Improving safety
• Reducing operating costs
• Increasing new patents

The ones of least importance to this benchmarking group were increasing new patents, improving safety and reducing operating costs. Not surprising for a group of companies in service businesses. Shown below are some of the measures deemed most important by this group.

The caveat, as stated earlier, is that the impact of the learning has to be measured according to what moves the company toward its strategic goals, and so this list, and even what others are measuring, is just an example.

Most L&D organizations are now following business performance measures in an attempt to gauge their impact on the business. Not surprisingly, the most frequently observed measure is customer satisfaction because that's one that most businesses track and because L&D can trace its impact somewhat directly through the impact on customer-facing employee populations like sales, services and customer support. Even though it ranked highest among measures, only 15 percent of survey respondents say they are tracking it. About 10 percent of the population track the following:

• Revenue
• Cost
• Employee turnover
• Quality
• Productivity
• Profitability

An even smaller percentage track risk and cycle time.

ROI

There is much discussion about organizations attempting to develop a Return On Investment (ROI) for their training programs. Most of training's ROI analysis is done after training has already occurred, whereas most business investments try to project ROI before making an investment decision. L&D should work with their business partners to forecast ROI and then to track the metrics used in the forecasting process to validate that results are delivering the expected return. Estimating ROI after the fact may provide some evidence of a training benefit but many business leaders will discount the validity of those postcourse estimates and consider them anecdotal support for the value of training rather than hard numbers. And some L&D organizations are merely responding to pressure from the business to provide some evidence of their value.

In the survey population, 47 percent said they do ROI analysis on 10 percent of their courses or less. Another 33 percent say they don't do any ROI analysis. Two survey respondents say they conduct ROI on up to 25 percent of their programs, and one person said they conduct this analysis on anywhere from 51 to 75 percent of their courses.

The survey prompted participants for more information on the context of their ROI analysis. In agreement with the earlier discussion, 46 percent said they don't conduct ROI analysis because they don't believe they can accurately isolate the impact of training on performance. Nearly the same number of people say management doesn't pressure them for ROIs because they believe in the value of L&D without hard dollar justification.