Reducing perceptual differences in end users: potential and possibilities for continuing training programs.

Glandon, Terry Ann ; Glandon, Sid ; Boyd, Michael W. 等

INTRODUCTION

This research compares the perceptions of different groups of end users on two dimensions of information technology (IT) support. (1) One dimension is the level of importance of certain features (response time, follow through, etc.) associated with IT support. The second dimension is the users' reported level of satisfaction associated with these IT support features. If a particular feature is very important to the user it is expected that such user will not be satisfied unless his or her expectations are met.

Various aspects of end user satisfaction have been studied in prior research. (2) However, these issues continue to be important to researchers and to organizations. Indeed, Lee, Kim and Lee (1995) call for increased research into the personal and organizational impact of end user computing in an organizational context because of the use of end user computing as a "competitive weapon." To achieve competitive advantage organizations must ensure that IT support helps end users maximize use of information technology. The first step in designing effective IT support is to obtain an understanding of what end users believe are important services. Focusing resources on these IT support features can improve efficiency and satisfaction among end users. If IT support is able to meet end user needs in the initial contact, waiting time and callbacks can be reduced or eliminated.

Nevertheless, it is unlikely that all users have identical perceptions of IT support. Even within one organization, users are rarely homogeneous--they have different skills, use different software and hardware, and may have different expectations of what computers should be able to do. Shaw, Partidge and Ang (2003) referred to these differences as the user's technological frame of reference. They found that users reporting overall satisfaction with IT support were self-directed learners, viewed the computer's role as a "task completer" and typically used more complex applications. On the other hand, dissatisfied users were not self-directed learners, viewed the computer's role as a "task enhancer," and usually worked with less complex applications.

In the Shaw et al. (2003) study, IT personnel preferred to solve non-routine problems, displaying "little patience" with repetitive requests, and responding more readily to technical issue requests. This would imply that IT personnel are more responsive to users who use more complex applications and are more technologically sophisticated. Working with such users may provide greater professional job satisfaction for IT staff; however, it probably does not adequately address the overall needs of the organization.

To address the gap between individual and organizational needs, a separate function of IT support could be created to provide continuous training for users. In most organizations the skills and knowledge of users varies widely depending on the education, training and job function of the users. Improving the technological competency of all users could reduce the need for routine requests, allowing IT support personnel to focus on the more complex issues. This would improve end user satisfaction with IT support, job satisfaction for IT personnel, as well as benefit the organization.

HYPOTHESIS DEVELOPMENT

A user's technological frame of reference may be molded by a number of personal and environmental factors. Because of the technical nature of their education and the early use of technology in their professional careers, accountants may be more familiar with information technology and more comfortable using a variety of software packages. In terms of functional responsibilities accountants utilize information technology to capture, record, and report financial information that is used by management and other stakeholders. Jiang et al. (2000) suggested that end users employed in various functional areas of organizations may have different perceptions of the importance of, and performance of, certain information technology support. We postulate that because of their education, training and functional responsibilities, accountants may have unique perceptions of the importance of certain IT services. Hypothesis H1 compares the importance accountant end users place on IT services with those of other end users. It is stated in the alternative form below:

H1: There is a difference in perceived importance of information technology services between accountant end users and other end users.

In their development of a technological frame of reference, Shaw et al. (2003) found that users who expected less from the computer were more easily satisfied, and users with higher expectations were more easily dissatisfied. It could be argued that a similar situation exists for users who place low (high) importance on IT support: those who place low importance on IT support would be easier to satisfy, while users who place high importance would be less satisfied with IT support. This is depicted graphically in Figure 1.

[FIGURE 1 OMITTED]

Following the results of Shaw et al. (2003), we expect that users who place high importance on IT support will be less satisfied than users who place low importance on IT support. Based on our previous argument that accountants will have a different perception of importance of IT support, we anticipate their satisfaction level will be different than the non-accountant group. Stated in the alternative form, Hypothesis 2 tests this proposal as follows:

H2: There is a difference in perceived satisfaction of information technology services between accountant end users and other end users.

Survey Instrument

A questionnaire was used to measure two dimensions of IT support. Users were requested to rate the importance of 18 items on a 5-point Likert-type scale, from not important (1) to very important (5). The second portion of the survey contained identical items, but asked users how satisfied they were with support they received. A similar 5-point scale was used, ranging from not satisfied to very satisfied.

Comparing the importance and satisfaction on identical measurement items can help assess satisfaction (Davis, Misra and Van Auken 2002, p. 219). For example:

1. It has the potential to provide more insight than a one-dimensional satisfaction survey that simply asks users whether they are satisfied with IT support. If they are satisfied with one aspect of IT support that is not very important to them, but not satisfied with another aspect that is important, one might infer that users would be less satisfied with IT support, in general.

2. It can be used for a baseline for continuous improvement for the IT function. By identifying initial gaps between importance and satisfaction, management can direct resources to issues that warrant improvement.

3. It allows for development of longitudinal trends in IT support. End users' requirements change with technological advances and organizational innovation. By administering the importance/satisfaction survey on a periodic (annual, bi-annual) basis, the organization can track the changes and alter IT support accordingly.

4. Measuring two dimensions using the same questions provides a quantitative basis for analysis, a scientific approach that can be validated.

Measurement Items

The survey consisted of two sections: the first section contained questions designed to measure the importance of quality, interpersonal skills, dependability, teamwork or leadership, and responsiveness. The second section asked users whether they were satisfied with IT support on those same items. Asking questions that measure different perceptions on identical issues allows analysis of a positive (negative) gap; if the difference is positive, the user's satisfaction rating is greater than his or her importance rating. Conversely, if the difference is negative, the user is suggesting that important IT issues are not being adequately addressed. For example, if users rated "follow through" as very important, but indicated they were only somewhat satisfied with IT's performance, a negative gap would result. A complete list of the measurement items is provided in the Results section.

RESEARCH METHODOLOGY

Two hundred twenty surveys were hand delivered to local business enterprises in the southern region of the U.S. The surveys were accompanied by a cover letter and a return envelope. The cover letter requested that management distribute the survey forms to accountant or non-accountant end users and assured the respondents of the confidentiality of their answers. No personal identifying information was requested on the questionnaire. The accountant end users were employed in a variety of accounting positions in the organizations. The non-accountant end users were employed in positions such as personnel, marketing and management. Approximately two weeks after initially distributing the surveys, phone calls were made to management to assure that the designated employees had completed the survey. One hundred ten accountant end users and 93 other end users completed and returned the survey.

A simple comparative ranking of the survey questions was performed by listing each group's mean score for all questions and rearranging one column (accountants) from 1 (highest mean) to 18 (lowest mean). This ranking allows visual comparison of the groups of the most important IT support features. Although this is not a statistical test, it provides important information regarding differences (and similarities) between the two groups.

Mean scores of Importance and of Satisfaction were created using these same 18 items. T-tests were used to compare the means between the groups to test Hypotheses 1 and 2. Item rankings and results of hypothesis testing are discussed in the following section.

RESULTS

Demographics of the respondents are presented in Table 1. As shown, more women than men are represented in the accounting profession (male=48 or 44%, female=57 or 51%, not reported=5 or 5%), and more men are employed in the other-user category (male=56 or 60%, female=37 or 40%). The average of participants' ages was 43 for accountants and 38 for the other end users. The industry breakdown also is presented, indicating that most respondents were employed in manufacturing, with service and the public sector following closely.

The survey was modified from one developed and validated by Jiang, et al. (2000) to compare satisfaction perceptions of end users and information systems personnel. To ensure data from the current study remain appropriate for the model, we conducted a confirmatory factor analysis and found that items from two of the constructs loaded on the same factor. It is understandable that "follow through" (dependability construct) is quite similar to "apply preventative/permanent solutions" or "stick with user's problem until resolved" (responsiveness construct). The new factor was revised to include the measurement items from both constructs and was renamed Dependability and Responsiveness.

Tests for internal reliability and goodness-of-fit were also conducted. Internal reliability, which refers to the extent to which the survey instrument is free from measurement error, was verified using Cronbach's alpha. As presented in Table 2, Cronbach's alpha values range from .70 to .89, considered to be acceptable levels of internal reliability. Goodness-of-fit statistics include the Goodness of Fit Index (GFI), Root Mean Squared Residual (RMR), Bentler's Comparative Fit Index, and Bollen's Non-normed Index. They are presented in Table 2, along with normally accepted values.

Simple item rankings by each group are presented in Table 3. The left column depicts accountant users that have been ranked from 1-18. The right column shows the comparative rankings of the other end users. It is easy to see that, for the most part, the rankings were relatively consistent between the groups. For example, a timely response was ranked very important by both groups, #1 for accountants and #2 for other users. Only three questions were rated substantially different: "sticking with the problem" was ranked #1 by non-accountant users, although it only ranked #8 for the accountants. It is possible that other users felt somewhat abandoned prior to satisfactory resolution of their issues, while it was not so for the accountants. "Follow through" was ranked much higher for other users (#5) than accountants (#11). This seems related to "sticking with the problem," which was ranked higher by other users, so this result is not unexpected. Interestingly, another difference is the importance for IT personnel to "understand and follow procedures and instructions." This was ranked #2 for accountants, but #8 for other users. Perhaps this is because the accounting function usually includes a series of steps or established procedures--accountants might expect IT personnel to work in a similar fashion.

Gap Analysis

Gap analysis has been used in prior studies to compare the individual's perception of importance of an activity with his or her satisfaction with the completion of that activity (c.f. Davis, et al. 2002). As described earlier, a positive gap exists when the satisfaction score exceeds the score for importance on the same measurement item. In the current study, only one construct resulted in a positive gap: the other user group rated satisfaction of teamwork and leadership higher than importance. Teamwork and leadership also had the lowest mean of any of the constructs. For both groups, importance of all other constructs was ranked higher than satisfaction, resulting in negative gaps. Table 4 presents the importance and satisfaction scores, along with the gaps for each item.

Tests of hypotheses

An independent-samples t-test was used to determine whether there were reliable differences of mean importance scores between the two groups. As presented in Table 5, the accountants placed greater importance on the IT support features (Mean of 4.46, S.D. of .42) than the other end users (Mean of 4.23, S.D. of .49). The data support Hypothesis 1 at p < 0.001.

To assess the possible impact of these differences, the standardized effect size was calculated by dividing the mean difference by the average standard deviations. In the current study, the standardized effect size is .50, considered to be a medium effect according to Cohen's (1988) guidelines (.2 for a small effect, .5 as medium, and .8 as large). (3) A medium effect can be relatively easy to visualize when describing tangible evidence, such as an algae bloom in an otherwise crystal clear mountain lake. It becomes more difficult when applied to differences in perceptions of importance of IT support. Nevertheless, that does not mean the effect is inconsequential or nonexistent. Such perceptual differences may have an overall impact on organizational resources. It was expected that users who placed higher importance on attributes of IT support would be less satisfied than those who seemed less concerned (by placing less importance on the measurement items). Hypothesis 2 was tested using an independent-samples t-test. The results indicate that although the accountant users were less satisfied, there was only a very small difference, which was not statistically significant, as shown in Table 5. This leads to the revised illustration shown in Figure 2, which suggests that regardless of the importance attributed to IT support, users were almost equally satisfied.

[FIGURE 2 OMITTED]

DISCUSSION

As predicted, the accountants' perspective on the importance on information technology services was distinctly different from other end users. Our research included responses from a large cross section of business entities; therefore, organizational climate is not a factor in explaining these differences. It is more likely explained by the psychological and subunit (departmental) climate in which accountants work (Glick 1985). Accountants begin their formal education using technology to solve problems and assemble financial information. Upon graduation, they become professionals who operate in an environment where technology is utilized extensively and they are likely to be technologically proficient. As sophisticated end users, it is expected they would place greater importance on the delivery of information technology services.

Nonetheless, it is unclear why they are not less satisfied than the other user group that did not place as much importance on technology support. It is possible that accountants are more aware of the difficulties encountered in installing and supporting software. Increased experience makes one sensitive to the possibilities of software programming deficiencies, equipment incompatibilities, and other possible problems involved in getting an application to run properly. Because of their experience, accountants may have more realistic expectations as to the ability of IT personnel to solve problems.

In addition to the psychological climate that accountants bring to the work place, personality traits may also play a role in explaining the differences between accountants and non-accountants. Day and Silverman (1989) found significant correlations between some personality traits of accountants and certain job performance dimensions. Using a well-established personality test, the study found that work orientation, degree of ascendancy, and degree and quality of interpersonal orientation were significantly correlated with the job performance dimensions of 1) potential for success, 2) technical ability, 3) client relations, and 4) cooperation with other personnel. (4) Although cognitive ability explains much of the reasoning in how individuals select a career in accounting, personality traits appear to be a key factor as well.

It is important to examine how the differences between the two groups potentially impact the organization. There is much research indicating that user involvement in information systems design improves user satisfaction with the system (c.f., Hunton and Beeler 1997; Baroudi, Olson, Ives and Blake 1986, and others). This same principle should apply to information technology support. When users and systems personnel work together during installation and support of new software it would seem likely that they would have a greater understanding of the benefits of the software and the deficiencies that must be overcome. Satisfaction with performance should improve when perceptions of importance are more closely aligned between IT personnel and end users.

Technology skill levels among end users in most organizations are uneven, at best. To a large extent, the end user's technological frame of reference influences the perception of the value of IT support services and how they can best be utilized. End users who have minimal technological experience are likely to expect IT support to solve all problems and make the system work. Experienced end users will be more realistic as to the ability of IT personnel to solve problems and will be better prepared to maximize available support services.

This research supports the notion of equalizing users' technological frames of reference. Certain end users might benefit from a structured educational/support function as described in Huang (2002). (5) General technology training could be developed for those who possess minimal technical skills. Once they have achieved the minimum level of general application skills, additional education could be introduced for specific business applications. Over time, all end users could be brought to an optimum level of competency, whereby the IT support function would provide maximum benefit to the end users and the organization. Such an educational support system could redeploy IT support to deal with more complex issues and increase the efficiency of the entire organization.

LIMITATIONS AND FUTURE RESEARCH

The use of gap analysis in examining the differences between importance and satisfaction has been criticized, because there is not necessarily a linear relationship between the two constructs (c.f. Anderson and Fornell 1994; Babakus and Boller 1992; and others). Although this is a valid criticism, the argument relates more to those relationships that are positive or zero. In the current study, all of the relationships are negative with the exception of the teamwork/leadership construct.

To develop this line of research on end user satisfaction with information technology support, future studies might incorporate an assessment of personality traits of different groups of users. The Jackson Personality Research Form is a well-recognized personality trait assessment instrument that would be suitable for this purpose.

REFERENCES

Anderson, E. & Fornell, C. (1994). A customer satisfaction research prospectus. In R. T. Rust & R. L. Oliver (eds.), Service Quality: New directions in theory and practice. 241-268. Thousand Oaks, CA. Sage Publishing.

Babakus, E. & Boller, G. W. (1992). An empirical assessment of the SERVQUAL scale. Journal of Business Research. 24, 253-268.

Baroudi, J., Olson, M., Ives, B. & Davis, G. (1986) An empirical study of the impact of user involvement on system usage and information satisfaction. Communications of the ACM, 29(3), 232-239.

Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences, 2nd edition. Hillsdale, NJ: Erlbaum.

Davis, R., Misra, S. & Van Auken, S. (2002), A gap analysis to marketing curriculum assessment: A study of skills and knowledge. Journal of Marketing Education, 24(3), 218-224.

Day, D. V. & Silverman, S. B. (1989). Personality and job performance: Evidence of incremental validity. Personnel Psychology. 42, 25-36.

Glick, W. H. (1985). Conceptualizing and measuring organizational and psychological climate: Pitfalls in multilevel research. Academy of Management Review. 10(3), 601-616.

Huang, A. H. (2002). A three-tier technology training strategy in a dynamic business environment. Journal of End User Computing, 14(2), 30-39.

Hunton, J.& Beeler, J. (1997). Effects of user participation in systems development. MIS Quarterly, 21(4): 359-388.

Jiang, J., Sobol, M. & Klein, G. (2000). Performance ratings and importance of performance measures for IS staff: The different perceptions of IS users and IS staff. IEEE Transactions on Engineering Management, 47(4), 424-434.

Lee, S., Kim, Y. & Lee, J. (1995). An empirical study of the relationships among end-user information systems acceptance, training, and effectiveness. Journal of Management Information Systems. 12(2), 189-202.

Mahmood, M., Burn, J., Leopoldo, A. & Jacquez, C. (2000). Variables affecting information technology end-user satisfaction: A meta-analysis of the empirical literature. International Journal of Human-Computer Studies, 52, 751-771.

Shaw, N., Lee-Partidge, J. & Ang, J.S.K. (2003), Understanding the hidden dissatisfaction of users toward end-user computing. Journal of End User Computing, 15(2), 1-22.

TerryAnn Glandon,University of Texas at El Paso

Sid Glandon, University of Texas at El Paso

Michael W. Boyd, Western Carolina University

ENDNOTES

(1) IT support activities include, but are not limited to: diagnosing and solving problems; designing and developing software; installing, debugging, testing, modifying, correcting, and maintaining hardware or software; repairing hardware; training users; answering questions; and keeping users informed

(2) For a meta-analysis, see Mahmood, Burn, Leopoldo and Jacquez (2000).

(3) (4.46-4.23)/((.42 + .49)/2) = .50

(4) The Jackson Personality Research Form--Form E was used. Work orientation was significantly correlated with Technical and Client Relations; Ascendancy was correlated with Potential, Technical, and Cooperation; Interpersonal was correlated with Potential, Technical, Client Relations, and Cooperation. Other significant correlations, not of interest in the current study, are also reported (Day and Silverman 1989, p. 31).

(5) Huang (2002) describes a three-tier IT training strategy that includes initial formal training in general technology to help establish a general level of technology proficiency, followed by specific training on software applications, and "just-in-time" training to help employees build on their fluency.

Table 1. Descriptive Statistics

Gender: Accountant Users Other Users

M 48 44% 56 60%
F 57 51% 37 40%
Not reported 5 5% -- --
 Totals 110 100% 93 100%

Age: Accountants Others

Max 63 61
Min 22 20
Average 43 38
Not reported 7 2

Industry:

Manufacturing 60
Service 54
Public Sector 45
Retail & Whls 24
Banking 16
Other 4
 Total 203

Table 2. Construct Reliability and Goodness-of Fit Indices

 Construct Cronbach Alpha

Quality 0.70
Interpersonal Skills 0.81
Dependability and Responsiveness 0.89
Teamwork and Leadership 0.82

 Goodness of Fit Indices Current Study Acceptable Values

Goodness of Fit Index (GFI) 0.92 >= 0.90
Root Mean Square Residual (RMR) 0.03 <= 0.05
Bentler's Comparative Fit Index 0.95 >= 0.90
Bollen's Non-normed Index 0.96 >= 0.90
n = 203

Table 3. Simple Rankings of Measurement Items

 Item Accountant Users Other Users

Respond in timely fashion 1 2
Understand and follow applicable
 procedures and instructions 2 8
Implement changes without errors
 or rework 3 3
Make ideas understood 4 4
Make an effort to listen to and
 understand the users 5 6
Apply preventative or permanent
 solutions to problems 6 7
Stick with the problem until it
 is resolved 7 1
Meet commitments 8 9
Show respect; build cooperative
 relationships; facilitate
 dialog 9 10
Recommend ways to be more
 effective and efficient 10 11
Follow through 11 5
Use tools and standards properly
 and consistently 12 12
Anticipate user's needs; giving
 high priority to user
 satisfaction 13 13
Willingness/ability to accept
 new assignments 14 14
Lead a team toward its stated
 objectives 15 15
Coach, instruct, and/or support
 other IT personnel 16 16
Contribute actively to project
 and non-project related
 efforts 17 18
Keep users informed about
 technologies related to user's
 job (hardware, software,
 books) 18 17

Table 4. Gap Analysis

 Interpersonal Dependability/ Teamwork/
 Quality Skills Responsiveness Leadership
Accountants:
 Importance 4.57 4.59 4.45 4.23
 Satisfaction 4.06 4.20 4.03 3.86
 Gap -0.51 -0.39 -0.42 -0.37
Other Users:
 Importance 4.32 4.39 4.27 3.87
 Satisfaction 4.15 4.13 4.18 4.09
 Gap -0.17 -0.26 -0.09 0.22
n = 203

Table 5. Results of Hypothesis Tests

 Accountant Users Other Users p

H1: Means S.D. Means S.D.
 Importance 4.46 0.42 4.23 0.49 < 0.001
H2:
 Satisfaction 4.06 0.65 4.14 0.75 n/s
n = 203