home > essays > essay 2
Simulation-Based Tools In
Self-Assessment of Teachers
John W. Jacobs, Applied Simulation Corporation

Introduction
Simulations to Enhance Teaching
Development of the Skills Profiler™
Current Users
Strategic Partnerships
Grants and Research
Summary
Corporate Contact Information
Author Note and Disclaimer

Introduction

There is apparently a growing shortage of qualified teachers, leading many individuals to think about entering the profession or transitioning into the profession from their current jobs. However, to be a successful K-12 teacher requires a variety of skills and abilities. Through various life experiences, some individuals can acquire many of the skills needed to manage the complexities of a classroom learning environment, thus making them good candidates for entering the teaching profession. The question is how best to identify which individuals possess the needed skills. Interest and personality inventories and multiple aptitude tests have been used, with mixed success, to direct qualified individuals into the teaching profession. New technologies may offer new means to solve the problem.

Simulation-based assessment has historically been shown to be an accurate and cost-effective way to assess complex skill sets related to a variety of jobs. Older simulation-based systems included role-playing or in-basket techniques. Psychologists and educators have achieved dramatic advances in the past few years in developing simulation-based assessment and training systems. Most of these systems were initially applied to problems of training "critical path" workers such as nuclear power plant operators, aircrew members, air traffic controllers, physicians, and so on. However, over the past decade, simulation-based training with videotape and computers has extended to "people" professions such as business managers, psychotherapists, and teachers. The present essay describes one simulation-based assessment of teacher skills that was developed using rigorous research and that has already achieved success in the field. Although this is not the only effort now underway to create simulation-based teacher assessments, it is certainly one of the furthest along in development and actual use.

Simulations to Enhance Teaching

Applied Simulation Corporation (ASC) provides educational consulting services and simulation-based assessment tools for the K-12 education market. The ATTP- Skills Profiler™ (formerly called the Teaching Skills Enhancement System or TSES) is ASC's premier teaching skills assessment system. The system allows teachers or teacher candidates to conduct a self-assessment in order to obtain confidential feedback related to the following five skills the research literature indicates as being key factors for improving student learning, motivation and achievement:

To complete the self-assessment, teachers view and respond to 16 realistic video situations that use a forced choice answer format. That is, each situation includes four answer choices, each of which are presented in writing and acted out. The self-assessment lasts 90 minutes, which can be separated into two sessions if needed.

Because the system uses a standard VHS video presentation, 15 or more teachers can complete the self-assessment process in a single session. The answer sheets are sent to ASC's home office in Winter Park, FL (a suburb of Orlando) where a central scoring computer scores it and produces an individualized feedback report summarizing the participant's strengths and weaknesses, and providing suggestions on how to improve weaker skill areas. This report is typically 8-12 pages in length and is an exceptional tool for determining an accurate skill profile and identifying where to focus professional development activities in order to obtain better results with students. For new teachers, the feedback report can be shared with a peer mentor in order to help focus the mentoring relationship.

For administrators, the ATTP- Skills Profiler offers Trend Analysis Reports (TARs), which summarize performance strengths and weaknesses across specified teacher groups. For example, a TAR can be provided for all teachers within a given school, grade level, or who teach a specific subject content (e.g., all math teachers). A TAR can also combine results across school, district or state boundaries. Using one or more TARs, a district can determine how best to allocate their professional development budget to focus resources on areas of true need. The system can also be used to conduct pre- to post-training evaluations, thereby providing administrators a way to demonstrate performance gains and program accountability.

Development of the Skills Profiler

The Skills Profiler system was developed using a rigorous Organizational Psychology methodology. The initial step in the development process involved conducting a job analysis of the teacher position. The job analysis identified teaching activities and skills required for successful classroom performance. As part of this analysis experienced teachers were asked to review and rate the importance and frequency of occurrence of the five skill areas upon which the system is based. Next, a survey of teacher training practices was conducted. Out of a total 600 surveys that were mailed, 200 district superintendents or district-level professional development leads responded. These results were added to the rating information. Taken together the results strongly support the job-relevance and critical nature of the five skill areas. Next, scripts were written and a focus group was conducted to review the scripts. The results of the focus group showed that the situations were realistic and targeted the appropriate skill component they were written to address. The focus group also provided input about the scaling of answer choices from most to least effective.

The final step in the validity process involved conducting a concurrent validation study. This involved having a large number of teachers view and respond to the video situations. Performance on the video simulation assessment instrument was then compared to on-the-job performance based on supervisory ratings of classroom teaching performance and whether the teacher reported receiving an award for teaching excellence. The results of the validation study demonstrated that the Skills Profiler is reliable and is equally effective for all teacher groups regardless of age, ethnicity, gender or grade level. The validation results also demonstrate that performance on the Skills Profiler is positively and significantly related to teaching effectiveness measures. Overall validity coefficients ranged from .38 to .43.

Current Users

State of Florida. The Florida Department of Education recently approved the use of ASC's video simulation assessment system within the state funded Teaching Skills Assessment Program (TSAP), developed by Eckerd College, located in St. Petersburg, Florida. The TSAP uses a multi-assessment methodology to assess 12 Educator Accomplished Practices, which are the state-defined teaching competencies, as part of the alternative teacher certification process. As a key element of the TSAP, the ASC video simulation system assesses five of the 12 Accomplished Practices. The TSAP is poised to become the assessment program of choice within Florida and is also being reviewed by the Departments of Education within the states of Kentucky and Wisconsin.

District Implementations. ASC's Skills Profiler has been purchased and implemented by four districts to-date, including ones in Florida, Georgia, and Illinois. In Florida, Highlands County and DeSoto County have implemented. In Illinois, East Moline School District recently conducted their initial implementation. In Georgia, Henry County is implementing this June, and will assess both new teachers and teacher mentors. Implementation plans are pending for several large districts in Florida, Maryland, and Texas.

For each implementation, ASC conducted a detailed analysis demonstrating a robust match between the state-defined teaching competencies and the five process teaching skills which are the basis for the Skills Profiler. A similar analysis was conducted using the newly developed National Teaching Standards and the results were also positive and robust.

Strategic Partnerships

Performance Learning Systems (PLS) is known nationally for providing high quality teacher professional development resources, including multi-day seminars and online instruction. PLS has been in business for over 10 years and has numerous clients nationally, with yearly revenues of $10MM. They are using the Skills Profiler as a value-added feature they can present to their clients. To date, PLS has sold the Skills Profiler to Henry County, GA and is poised to complete a sale to a large school district in Maryland. For both counties, the implementation uses a pre- and post-assessment methodology to help determine performance gains due to the PLS instructional program. The PLS home page can be found at: WWW.PLSWEB.COM.

Grants and Research

ASC welcomes individuals and organizations interested in conducting research in the area of teacher professional development. To date, the Skills Profiler has been included in three teacher professional development grants, with one receiving funding and two others pending. Of particular interest is the use of the Skills Profiler system to provide "focused" professional development, to include enhancing the teacher-mentor relationship. The Skills Profiler is also being used as part of doctoral prospectus involving the analysis of the psychometric properties associated with simulation-based assessment of complex skills.

Summary

ASC's Skills Profiler has been developed and validated using a traditional Organizational Psychology methods, thereby ensuring its utility for teachers of all grade levels and subject content. The system is easy to implement within a career counseling center allowing candidates to conduct a self-assessment as their schedule permits. In this way, the center staff remain free to focus on other work activities. At $63 per assessment, the cost of the system is also very reasonable, and may be an effective way to help direct qualified individuals into the currently understaffed teaching profession.

2.8 Corporate Contact Information

Applied Simulation Corporation Assessment
Tools for Teaching Proficiency - Skills Profiler™
PO Box 5117 Winter Park, FL 32793
800-752-1430 / 407-645-1481 (fax) www.appliedsim.com

2.9 Author Note and Disclaimer John W. Jacobs is CEO of Applied Simulation Corporation and provided the product description for the Skills Profiler that served as the basis for this essay. Skills Profiler and ATTP-Skills Profiler are tradmarks claimed by the Applied Simulation Corporation. Neither Andrew D. Carson nor VocationalPsychology.com specifically endorses the Skills Profiler, nor did they receive any compensation or other consideration in the publication of this essay.