Measuring Transfer for Results and Glory

DSA would like to thank Dr. Jeanne Farrington for contributing this month’s tip: Measuring Transfer for Results and Glory.

How are your courses doing? More to the point, are your learners using what they learned? In other words, what kind of Level III (transfer) results are you getting?

When we ask if conducting transfer evaluations is important, most training professionals say “Yes!” Training professionals and management teams generally want to know whether their leaders, managers, employees, customers, and partners are using what they learned.

However, when we ask what training professionals are doing to measure transfer, they often say, “Well, we’d love to do that, but even if we were sure how to do it, where would we find the time?”

Here’s an easy six-step method for not only measuring transfer easily and quickly, but also for using the results to market your courses to potential learners (and to justify your existence to an upper management team that is always looking for ways to trim the budget).

1. Decide who will work on the evaluation. You can use a training coordinator to set things up (1 day or less), but you should use an independent instructional designer (one without a vested interest in the outcome) to write the questions, conduct interviews, and write up the results (2 or 3 days).
    
2. Make some decisions about which learners to include in your evaluation.
   1. If you tested the learners at the end of the course to see who mastered the objectives, choose only those learners who passed the test.
   2. Include learners who took the course recently enough to remember it and long enough ago to have had time to use what they learned.
   3. Decide how many learners to interview. For a recent award-winning project, we randomly selected 10 learners who met the criteria for selection.
3. Choose your learners and request their participation.
   1. Generate a list of everyone who took the course during the selected time period and who also passed the test (assuming there was one)
   2. Number the learners and select a pool of perhaps 50 learners using a random number table. (Search for one on Google if you don’t have one handy. Often they appear with instructions about how to use them.)
   3. Write an email message or a telephone script and use it to invite your selected learners to participate. Tell them that you want to provide your learners with the best quality courses and that they can help. Ask if they’ve had the opportunity to [overall course topic here] since they took the course. If the answer is “yes,” then ask them if they are willing to participate in a 15-minute telephone interview.
   4. Schedule the interviews with the first 10 (if that’s your chosen number of participants) who say “yes.”
4. Write your interview questions. Ask everyone the same questions. Take excellent notes and/or record the calls. (You can use a survey, but it’s harder to get good quotes.)
   1. Ask a general question about whether they’ve had a chance to [overall course topic here] since they took the course. You’ll find out if the course topic is something they do regularly or once in awhile.
   2. Ask questions based on the course objectives. Ask what tools, concepts, or skills they’ve used to do [Objective 1, 2, 3, etc.] since they took the course. Use the key objectives. Ask about them one at a time. You might also include an enabling objective or two here, if it’s important for you to note their use of a particular skill.
   3. Include a “catch-all” question at the end. Something like, “Is there anything I haven’t asked that I should have asked about what you’ve used on the job from the course?”
5. Write up your results. Create a document that describes your study. Include a summary of the participant’s answers.
   1. Overall, are they using what they learned on the job?
   2. Objective by objective, how many are using, not using, or perhaps planning to use what they learned for each one?
   3. For each objective, include helpful quotes from the participants about how they are using what they learned on the job.
6. Do something useful with the information you have collected. For example:
   1. Use positive results and quotes in announcements or marketing materials for your course.
   2. Create a summary of positive results and report them to your management team so that they will realize what great work your department is doing.
   3. If there’s some part of the course that no one is using, figure out why and decide whether to discontinue it, modify it, or try to change something in the environment so that people will use it.

Benefits

Conducting transfer evaluations can be done fairly quickly. The benefits include:

• Improving results by making changes based on what does not transfer
• Marketing your courses by sharing positive results and quotes with potential learners
• Impressing your management team with the wisdom of continuing to fund your department

Not bad for a few days’ work.