FAQs
Although most practitioners recognise the ROI ProcessTM as an important addition to measurement and evaluation, they often struggle with how to address the issue. Many professionals see the ROI methodology as a ticket to increased funding and prosperity for HRD. They believe that without it, they may be lost in the shuffle, and with it, they may gain the respect they need to continue moving the function forward. Regardless of their motivation for pursuing ROI, the key questions are:
- Is the ROI ProcessTM a feasible process?
- Can it be implemented with reasonable resources?
- Will it provide the benefits necessary to make it a useful, routine tool?
Answers to these questions may lead to debates, even controversy. The controversy surrounding ROI stems from misunderstandings about what the process can and cannot do, and how it can or should be implemented in an organisation. As a conclusion these misunderstandings are summarised as 15 myths about the ROI methodology:
1. ROI is too complex for most users:
This issue has been a problem because of a few highly complex models that have been presented publicly. Unfortunately, these models have done little to help users and have caused confusion about ROI. The ROI methodology is a basic financial formula for accountability that is simple and understandable: earnings are divided by investment; earnings equate to net benefits from the training program, and the investment equals the actual cost of the program. Straying from this basic formula can add confusion and create tremendous misunderstanding. The ROI model is simplified with a step-by-step, systematic process. Each step is taken separately and issues addressed for a particular topic; the decisions are made incrementally all the way through the process. This helps to reduce a complex process to a more simplified and manageable efforts.
2. ROI is expensive, consuming too many critical resources:
The ROI Process can become expensive if it is not carefully organized, controlled, and properly implemented. While the cost of an external ROI impact study can be significant, there are many actions that can be taken to keep costs down.
3. If senior management does not require ROI, there is no need to pursue it:
This myth captures the most innocent bystanders. It is easy to be lulled into providing evaluation and measurement that simply meets the status quo, believing that no pressure or requests means no requirement. The truth is that if senior executives have only seen Level 1 reaction data, they may not be asking for higher level data because they think it is not available. In some cases, training and development leaders have convinced top management that programs cannot be evaluated at the ROI level or that the specific impact of a program cannot be determined. Given these conditions, it comes as no surprise that some top managers are not asking for Level 5 (ROI) data. There is another problem with this thinking. Paradigms are shifting - not only within the HRD context but within senior management teams as well. Senior managers are beginning to request this type of data. Changes in corporate leadership sometimes initiate important paradigm shifts. New leadership often requires proof of accountability. The process of integrating ROI into an organisation takes time - about 12 to 18 months for many organisations. It is not a quick fix, and when senior executives suddenly ask the corporate university to produce this kind of data, they may expect the results to be produced quickly. Because of this, training and performance improvement should initiate the ROI process and develop ROI impact studies long before senior management begins asking for ROI data.
4. ROI is a passing fad:
Unfortunately, this comment does apply to many of the processes being introduced to organisations today. Accountability for expenditures will always be present, and the ROI provides the ultimate level of accountability. As a tool, ROI has been used for years. Previously, ROI has been used to measure the investment of equipment and new plants. Now it is being used in many other areas, including training, education, and learning solutions. With its rich history, ROI will continue to be used as an important tool in measurement and evaluation.
5. ROI is only one type of data:
This is a common misunderstanding. The ROI calculation represents one type of data that shows the costs versus benefit for the program. However, six types of data are generated, representing both qualitative and quantitative data and often involves data from different sources, making the ROI methodology a rich source for a variety of data.
6. ROI is not future-oriented; it only reflects past performance
Unfortunately, many evaluation processes are past-oriented and reflect only what has happened with a program. This is the only way to have an accurate assessment of impact. However, the ROI methodology can easily be adapted to forecast the ROI.
7. ROI is rarely used by organisations:
This myth is easily dispelled when the evidence is fully examined. More than 1,000 organisations use the ROI methodology, and there are at least 100 case studies published about the ROI methodology. Leading organisations throughout the world, including businesses of all sizes and sectors, use the ROI methodology to increase accountability and improve programs. This process is also being used in the non-profit, educational, and government sectors. There is no doubt that it is a widely used process that is growing in use.
8. The ROI methodology cannot be easily replicated:
This is an understandable concern. In theory, any process worthy of implementation is one that can be replicated from one study to another. For example, if two different people conducted an ROI impact study on the same program, would they obtain the same results? Fortunately, the ROI methodology is a systematic process with certain standards and guiding principles. The likelihood of two different evaluators obtaining the same results is quite high. Because it is a process that involves step-by-step procedures, it can also be replicated from one program to another.
9. ROI is not a credible process; it is too subjective:
This myth has evolved because some ROI Studies involving estimates have been publicized and promoted in literature and conferences. Many ROI Studies have been conducted without the use of estimates. The problem with estimates often surfaces when attempting to isolate the effects of other factors. Using estimates from the participants is only one of several techniques used to isolate the effects of a program. Other techniques involve analytical approaches such as use of control groups and trend-line analysis. Sometimes estimating is used in other steps of the process such as converting data to monetary values or estimating output in the data collection phase. In each of these situations, other options are often available, but for convenience or economics, estimation is often used. While estimations often represent the worst-case scenario in ROI, they can be extremely reliable when they are obtained carefully, adjusted for error, and reported appropriately. The accounting, engineering, and technology fields routinely require the use of estimates - often without question or concern.
10. ROI is not possible for soft skills programs, only for production and sales:
ROI often is most effective in soft skills programs. Soft skills training and learning solutions often drive hard data items such as output, quality, cost, or time. Case after case shows successful application of the ROI methodology to programs such as team building, leadership, communications, and empowerment. Additional examples of successful ROI application can be found in compliance programs such as diversity, sexual harassment prevention, and policy implementation. Any type of program or process can be evaluated at the ROI level. The issue surfaces when ROI is used for programs that should not be evaluated at this level. The ROI methodology should be reserved for programs that are expensive, address operational problems and issues related to strategic objectives, or attract the interest of management in terms of increased accountability.
11. ROI is for manufacturing and service organisations only:
Although initial studies appeared in the manufacturing sector, the service sector quickly picked up the process as a useful tool. Then, it migrated to the non-profit sector as hospitals and healthcare firms began endorsing and using the process. Next, ROI moved through government sectors around the world and, now, educational institutions are beginning to use the ROI methodology. Several educational institutions use ROI to measure the impact of their formal degree programs and less-structured continuing education programs.
12. It is not always possible to isolate the influence of other factors:
Isolating the effects of other factors is always achieved when using the ROI methodology. There are at least nine ways to isolate the influence of other factors, and at least one method will work in any given situation. The challenge is to select an appropriate isolation method for the resources and accuracy needed in a particular situation. This myth probably stems from an unsuccessful attempt at using a control group arrangement - a classic way of isolating the effect of a process, program, or initiative. In practice, a control group does not work in a majority of situations, causing some researchers to abandon the issue of isolating other factors. In reality, many other techniques provide accurate, reliable, and valid methods for isolating the effects.
13. Since there is no control over what happens after participants leave training, a process based on measuring on-the-job improvements should not be used:
This myth is fading as organisations face the reality of implementing workplace learning solutions and realize the importance of measuring results. Although the HRD staff does not have direct control of what happens in the workplace, it does have influence on the process. A training or performance improvement program must be considered within the context of the workplace - the program is owned by the organisation. Many individuals and groups are involved in training with objectives that push expectations beyond the classroom or keyboard. Objectives focus on application and impact data used in the ROI analysis. Also, the partnership often needed between key managers produces objectives that drive the program. In effect, HRD is a process with partnerships and a common framework to drive the results - not just classroom activity.
14. ROI is appropriate only for large organisations:
While it is true that large organisations with enormous training budgets have the most interest in ROI, smaller organisations can also use the process, particularly when it is simplified and built into programs. Organisations with as few as 50 employees have successfully applied the ROI methodology, using it as a tool to bring increased accountability and involvement to training and development.
15. There are no standards for the ROI methodology:
An important problem facing measurement and evaluation is a lack of standardization or consistency. These questions are often asked: "What is a good ROI?" or, "What should be included in the cost so I can compare my data with other data?" or, "When should specific data be included in the ROI value instead of as an intangible?" While these questions are not easily answered, some help is on the way. Standards for the ROI methodology, using the guiding principles as a starting point, are under development. Also under development is a database that will share thousands of studies so that best practices, patterns, trends, and standards are readily available.