IT Evaluation Methods for
Intangible Benefits
Back to IT Investments Page
It is difficult and perhaps inappropriate to try to translate the benefits of IT usage into quantifiable productivity measures of output. Intangibles such as better responsiveness to customers and increased coordination with suppliers do not always increase the amount or even the intrinsic quality of output. Just as managers look beyond ‘productivity’ for some of the benefits of the IT, so are many researchers prepared to look beyond conventional productivity measurement techniques. Following are the widely used techniques to measure intangible benefits of IT investments.

1.
Multi-objective, multi-criteria (MOMC):
This method attempts to develop a general measure of utility where utility is defined as the satisfaction of individual’s preferences. The method is based on the belief that the people’s behavior is determined to some extent by their feeling that their preferences are recognized. People appraise the relative usefulness of different desired outcomes in terms of their preferences and they rank goals by applying a preference weight to each goal. Where there are many stakeholders, the best IT investment is that which will deliver the highest aggregate utility or which provides the highest overall measures of satisfied preferences.
The MOMC method is probably the best applied to complex projects which attempt to meet the needs of many different users and where the benefits are intangible. The method is able to explore the value of a set of system proposals in terms of relative preferences for different system features. Also, consensus on the most desired system attributes is achieved by means of a through exploration of alternatives and preferences. However the MOMC method does not provide any data for the ROI calculation, which can be used to compare an investment justified by the use of the standard cost benefit analysis. Development of MOMC based methods is still in their fancy, but they have already generated a great deal of interests and discussions among researchers.

2.
Value Analysis (VA):
This method emphasizes values rather than cost. The method is based on the following three assumptions.
# Innovation is value driven and not cost driven
# Intangibles can be identified and subjectively assessed but rarely measured accurately, as surrogate measures are often used to satisfy the requirement for most inputs
# An inevitable clash exists between the persons driven by the cost and those driven by effectiveness.
The analysis begins with the observation that most successful innovations are based on enhancing value added rather than on cost savings. A multi-stage interactive process begins with a prototype system. Rather than developing existing specifications, the analysis provides simple models that can be expanded and modified until all complex aspects of the problem are included. Users are asked to provide the analysts with the feedback on the values and limitations of the solution obtained from the prototype. The main difference between the other IT evaluation methods and VA is that the former methods directly aim at a final solution, while the later uses an evolutionary process to get to a satisfying solution, which can be further improved.

3.
Critical Success Factors (CSF):
The method is used to explore the potential value of information systems. It invites the analyst to explore with executives the factors, which are in their opinion, critical to the success of the business in particular, the factors that are important for the functions and the activities for which the executives are responsible. The issues can be ranked by the executives into levels of importance.
The implementation of methods based on CSF involves comprehensive interviews of key managers to obtain their views about business mission, objectives and current problems. After the interviews the managers opines are cross tabulated, compared and agreement about system investment priorities is expected to emerge.
The advantage of the method is that it provides a focus on the issues, which are regarded as important by the respondents. It also heavily relies on prototyping and pilot installations before proceeding. In this regard, the technique takes advantage of lessons from the disciplines of organizational learning, adaptive planning and personal development. The CSF method may be considered to provide the right antidote against some of the worst excesses for conceiving accurate, but meaningless, payback numbers obtained from accounting oriented cost benefit analysis.

4.
The User Satisfaction Method:
This method measures the extent to which users believe the information system available to them meets their information requirement.  It is a tool for the evaluation of information systems where surrogate measures of effectiveness are used. The decision to install an information system necessitates a choice of mechanisms to determine whether an information system is needed, and once implemented, whether it is functioning properly. It serves as a substitute for objective determinants of the information system effectiveness, which are frequently not available.

Theoretically, the determination of information system value is a matter of economics.
Net value of the system =  Actual benefits - The cost of system operation
and development

The applicable definition of satisfaction is the sum of the user’s weighted reactions to a set of factors. These factors could be the completeness, accuracy, validity, timeliness, security, consistency, relevance, links to other data etc.

Si =  S Rij Wij

Where
Si  is the measurement of satisfaction
Rij  is the reaction to factor j by individual i
Wij  is the importance of factor j to individual i

Operationalisation requires two steps:
a.  Identifying a set of ‘factors’ that are related to satisfaction
b. Developing a survey instrument that incorporates a technique for scaling an individual’s reactions to each of the factors. Part of the survey instrument development should include tests for reliability and validity.

5. 
Critical Success Factors (CSF):
The method is used to explore the potential value of information systems. It invites the analyst to explore with executives the factors, which are in their opinion, critical to the success of the business in particular, the factors that are important for the functions and the activities for which the executives are responsible. The issues can be ranked by the executives into levels of importance.
The implementation of methods based on CSF involves comprehensive interviews of key managers to obtain their views about business mission, objectives and current problems. After the interviews the managers opines are cross tabulated, compared and agreement about system investment priorities is expected to emerge.
The advantage of the method is that it provides a focus on the issues, which are regarded as important by the respondents. It also heavily relies on prototyping and pilot installations before proceeding. In this regard, the technique takes advantage of lessons from the disciplines of organisational learning, adaptive planning and personal development. The CSF method may be considered to provide the right antidote against some of the worst excesses for conceiving accurate, but meaningless, payback numbers obtained from accounting oriented cost benefit analysis.
Back to IT Investments Page