Evaluation Research Group Office at TRI
Since its inception, ERGO has contracted with clients across a broad geographic and professional spectrum providing rigorous, client responsive evaluations and needs assessments generating information and evidence on which to make decisions. Key to this is intensive work with clients to clarify, focus, and finalize what it is they want to know and/or be able to do with findings.
"Evaluators are guilty until proven innocent."
- Michael Quinn Patton
"Being an evaluator is not easy. Telling someone you're an evaluator is like telling them that you're a cross between a proctologist and an IRS auditor."
- David Chavis
With these sentiments toward evaluation and evaluators, why would anyone willingly choose to be an evaluator? The reasons are many. Certainly the work has been infinitely varied and interesting, taking us from the smallest of rural Oregon towns to major urban areas, from early childhood education through adult education, and a host of different clients.
Clearly, observing the state of education and lives of educators and students at all levels, and the contexts in which they work and live makes for a fascinating professional life. From the humor, joy and interesting behaviors (ducking thrown chairs) seen through hundreds of observations in Head Start classrooms, to the anger and hostility seen in large urban focus groups (with grown adults standing on tables and shouting), to the dedication and professionalism seen in hundreds of observations and interviews of public school teachers, we have seen and learned much. From the long days of driving to places like Clatskanie, Astoria, Huntington, Nyssa, Tygh Valley, Charleston, Coquille, Madras, Medford, Ontario, Roseburg, Portland, and many places in between, we become attuned to the many cultures of Oregon and how to effectively work with people from every corner of the state.
On a deeper level, however, is the belief and knowledge that you can make a positive difference in the world by bringing an unbiased, critical lens to programs, practices, and policies for the purpose of improvement, and, when deemed prudent, to bring an end to wasteful, ineffective efforts. Evaluation and evaluators make a difference.
While educational program evaluation has long been associated with TRI, the current ERGO has its genesis in 1998 when a group of TRI faculty, under the encouragement of the late Dr. Vic Baldwin (director of TR at that time) successfully competed for a contract to conduct a three year evaluation of the Structure of Intellect pilot program funded by the Oregon Legislature. The program made claims regarding school-wide impacts that were to be evaluated for efficacy. An intense multi-method quasi-experimental evaluation design was employed by a team led by Dr. Andrew McConney, and included current ERGO members Robert Ayres, Laurel Cuthbertson, and Mark Schalock. The evaluation results revealed that the program did not live up to the claims made for it and a recommendation to discontinue funding was made, saving the state a potentially significant sum of money.
During the past 14 years, ERGO has had others come and go, including, Dr. Andy Rudd, Dr. Joe Hansen, Dr. Ella Taylor, and has had a number of super WOU student workers to provide valuable support. Today, ERGO is made up of three of the original members - Dr. Robert Ayres, Mark Schalock, Laurel Cuthbertson - and Dr. Christina Reagle, our newest member. ERGO faculty not only provide evaluation services to external clients, but also as internal project evaluators on other TRI funded projects.
Brief descriptions of several recent or current projects are provided here.
Evaluation of the Business Education Compact
Across Oregon many schools have embraced proficiency-based teaching and learning (PBTL) as a strategy to help all students have equal access to learning success. In 2010 the Business Education Compact (BEC) contracted with ERGO to create and implement an evaluation plan to capture, analyze, and report on data collected from the Partnership Sites. BEC and TRI established key performance indicators to assess program success over time as well as to determine the methodologies required to gather information about district, school, and classroom implementation. BEC and TRI remained in close collaboration throughout the project. As with most evaluation plans, some adjustments were needed as instruments were developed, data collection procedures began, and the normal every day life in public schools occurred.
ERGO was hired as a third party evaluation team to investigate whether PBTL made a positive difference in student learning. The initial quantitative and qualitative data collected from four school districts that participated in a pilot study during 2010-2011 coordinated by the Business Education Compact (BEC) revealed several promising indicators. Despite the fact that districts were at different points of implementation and that data was collected for a single year, 6,121 were impacted by proficiency because their teachers attended professional development and coaching sessions. In a quick look at the data points these beginning results are highlighted:
Schools are looking at PBTL to assist in meeting the Oregon State Legislature HB 2220 which requires school districts to report annually to parents about student proficiency in grade level standards.
Needs Assessment for the Oregon Forest Resources Institute
ERGO has established a long history of evaluation services for the Oregon Forest Resource Institute, beginning in 2000. ERGO was approached by OFRI Executive Director and director of education programs to develop “actionable” recommendations around the redesign of their K-12 education efforts. One of the Oregon Forest Resource Institute’s (OFRI) primary strategies for public education about forests and forest management is to train K-12 educators in the use of forestry and natural resources as a vehicle to teach Oregon standards. ERGO was asked to design and implement a study to ascertain the kinds of professional development opportunities K-12 teachers in Oregon would find attractive and to use that information to update OFRI’s professional development offerings, instructional materials, and website.
A mix of focus groups and the review of extant studies were carried out. Six focus group sessions were held. The groups were composed of selected Oregon K-12 teachers representing both urban and rural segments of the state, grade levels, and current levels of familiarity or use of forestry topics and concepts in their instruction. Reviews of extant literature, and a brief pen and paper questionnaire were conducted to validate focus group findings.
A number of findings were generated that lead to six primary recommendations to OFRI. These included:
Evaluation of Project SELS – Science and Engineering in the Lives of Students
ERGO has a sub-contract with Oregon State University to provide a third party evaluation of an NSF grant. The project, Science and Engineering in the Lives of Students (SELS), uses video case studies highlighting the role of science in construction problems as the core of a professional development program with video, print, and web components. Three content areas have been addressed: thermodynamics, electricity and materials and mechanics. The program utilizes an adaptation of a successful instructional model and problems familiar to the construction industry generated in previous ATE/NSF and Department of Education projects showing how to use construction problems for teaching science. The materials are organized into a complete professional development program and are being piloted with middle and high school science teachers and community college science and technology faculty. Expert community college faculty and high school teachers have served as a Content Review Panel in the development and testing of these materials.
ERGO has conducted a mixed-method study addressing broad questions related to quality and utility of the products and materials developed through the project, use and impact on teacher knowledge and practice, and resulting impact on student knowledge. A small cadre of middle school and high school teachers have tested the professional development materials and associated videos and other products, taking part in face to face summer training sessions, reviewing various versions of videos, implementing content in their classrooms and assessment student knowledge in each area. In-depth interviews and classroom observations have taken place to determine changes in the classroom. Evaluation of the website as a professional development vehicle is yet to take place.
Over the years ERGO has worked with a range of other public, private and quasi-public organizations, including:
Teaching Research Institute (503)838-8391 | or e-mail: firstname.lastname@example.org
Western Oregon University | 345 N. Monmouth Ave. | Monmouth OR 97361