The Nature of Technology Use in Classrooms:
The Development and Validation of an Instrument to Measure Teachers’ Perceptions

Executive Summary

Kristine Y. Hogarty, Jeffrey D. Kromrey
University of South Florida

Purpose

The goal of this research was to develop and validate an instrument that would provide data to foster a better understanding of how educators and students use technology in the classroom. In addition, we felt that it was important to explore related areas such as teachers’ level of comfort and experience with computers as well as general attitudes toward computer use.

Method Instrumentation

The first step of survey development involved the conceptualization of the relevant domains. Exploration of extant literature on the use of technology in schools revealed a wide range of interest and investigation. An examination of the indicators of successful integration of technology and computers in the classroom was made in concert with the development of the major domains. The domains of primary interest for this study fell within four broad categories: integration; support; preparation, confidence and comfort; and attitude toward computer use. Once these domains were established, survey items were constructed based upon existing, validated instruments related to these areas. Each of the requisite domains was examined for comprehensiveness, and when necessary, additional items were drafted to ensure adequate coverage. The survey was designed and reviewed by content experts and was pilot tested with a sample of graduate students, many of whom are in-service teachers. Psychometric information based on the pilot responses and participants’ comments about item clarity and importance were used to guide minor revisions to the survey content and item wording. The revised set of 83 items grouped within eight sections, were ordered according to section length and content and assembled into a four-page booklet format.

The front page of the instrument was used to collect demographic information (e.g., school name, gender, and ethnicity). Page two consisted of three sections: teacher preparation for computer use (8 items), confidence and comfort using computers (9 items), and general school support (7 items). Responses to the preparation items were provided on a 5-point frequency scale (ranging from Not at all to Entirely). Responses to the confidence and comfort and general school support times were provided on a 5-point Likert scale (ranging from Strongly Disagree to Strongly Agre(e). The third page was comprised of two sections: types of software used to complete school related activities (14 items) and the integration of computers into the classroom (12 items). Responses to all of these items were provided on a 5-point frequency scale (ranging from Not at All to Every Day). The last page contained three sections: personal use of computers (5 items), technical support (8 items) and attitudes toward computer use (20 items). Teachers’ personal use was reported on a 5-point frequency scale (ranging from Not at All to Every Day), with technical support and attitude items reported on a 5-point Likert scale (ranging from Strongly Disagree to Strongly Agre(e). For all items concerned with the frequency of use (i.e., software use, integration, and personal us(e), an option of not applicable (N(a) was also provided.

Sample

The instrument was sent to all of the teachers in Pinellas County Schools. Eighty percent of the schools received the survey in paper form; twenty percent were administered an online web-based version. Both individual and school-based incentives were offered to encourage participation and yield a better response rate. The respondent sample of 1890 teachers represents an overall response rate of 39%, and was 17% male and 83% female. A broad range of teaching experience was represented in the sample. While a number of respondents reported that this was their first year teaching, there were others who had been teaching for over 40 years. The average amount of teaching experience for the survey respondents was approximately 15 years. Teachers reported an average number of students ranging from one to sixty with an average class size of 22. Additional information provided regarding grade level and subject area taught, suggested the representation of a wide variety of disciplines and a diversity of teaching experience. Only responses to the paper administration of the survey were used in these analyses.

Data Analysis

The analyses consisted of non-respondent bias analyses, exploratory factor analyses, correlations between factor score estimates, and known groups analyses. Exploratory factor analyses were conducted to provide initial evidence of the construct validity of the instrument. Because the survey was divided into sections, each designed to capture a different aspect of teachers’ perceptions, factor analyses were conducted separately for each section. Further evidence for construct validation was sought by examining the relationships between subscale scores and other variables external to the scores. Profile analyses were conducted for respondents classified by gender, and school level (i.e., elementary, middle and high).

Results

Non-respondent bias analyses conducted to reveal potential difference between teachers who responded to the survey and the population of teachers in the school district revealed no practically significant differences. As the results of the exploratory factor analyses supported, in large part, the anticipated structure of the instrument, 11 composite variables were created based on the proposed classification scheme: Confidence and Comfort, General School Support, Teacher Application Software Use, Teacher Instructional Software Use, Student Application Software Use, Student Instructional Software Use, Integration of Technology into Teaching, Personal Use, Technical Support, Technological Affinity and Technological Aversion. The reliability estimates, using Cronbach’s alpha, ranged from .91 to .73 for the set of 11 subscales, indicating an acceptable level of internal consistency. Subscales scores were first used to explore the relationship between teachers’ perceptions of computers and technology and integration in the classroom. Moderately positive relationships were observed for the integration of computers in the classroom and the use of software in the classroom for both teachers and students with correlations ranging from .63 to .31. Positive relationships were also witnessed for teachers’ confidence and comfort and application software use (r = .54). Finally, profile analyses were conducted for respondents classified by gender and school level and differences in responses on the subscales were investigated using ANOVAs. The results of these tests revealed that the mean responses for females and males differed significantly from each other on two of the 11 subscales. Female respondents evidenced significantly higher means on the integration and students’ instructional software use subscales. When differences across school level were examined, mean responses differed by a significant amount on six of the 11 subscales (General School Support, Student Application Software Use, Student Instructional Software Use, Integration of Computers, Technical Support and Technological Aversion). With the exception of the technical support subscale, elementary school teachers consistently evidence higher mean responses across the various subscales. Because teachers’ gender was not independent of school level (i.e., elementary educators have a larger proportion of females in their ranks than do secondary educators), we tested for gender differences in responses after statistically controlling for school level. These analyses revealed that no statistically significant gender differences remained after controlling for school level, but school level differences remained even after controlling for gender differences.

In summary, the evidence gathered to date support the validity of scores derived from the survey. We hope that the instrument will prove useful in future efforts to measure teachers’ attitudes toward computer use and provide insight into the extent to which teachers are integrating technology in the classroom. Finally, it is hoped that the results of this study will serve to provide educators, researchers, and administrators with information to guide and monitor efforts toward improving the use of technology in our schools. The current survey can provide a potentially rich array of variables for analyses addressing substantive questions about the use of computers in schools. In addition to such substantive cross-sectional research, trends across time in computer use can be readily obtained in longitudinal studies, and the instrument may be useful to provide one measure of the effectiveness of teacher preparation for technology use in classrooms. Finally, parallel instruments that may be used to gather information from students, principals, and support personnel may provide a foundation for a comprehensive technology use assessment system.

Authors Note: This research was supported, in part, by the University of South Florida and the Technology Literacy Challenge Fund for 1999-2000, under Grant No. # 520-12101-OCT22. The opinions expressed are those of the authors and do not reflect the views of the Florida Department of Education or the University of South Florida.

Further details on this study are reported in Hogarty, K. Y. and Kromrey, J. D. (2000, November). The Nature of Technology Use in Classrooms: The Development and Validation of an Instrument to Measure Teachers’ Perceptions. Paper presented at the Annual Meeting of the Florida Educational Research Association.