Partnership to Reform Inquiry Science
in Montana (PRISM)
Why the PRISM Project?
The Partnership to Reform Inquiry Science in Montana (PRISM) grant seeks to develop and strengthen the pedagogical and content understandings of teachers associated with science in all partner schools within the Region III Southern Montana Alliance for Resources and Training (SMART). The restructuring of schools relies on the development of highly skilled, effective teachers. As they progress in their careers, teachers need to be encouraged to remain knowledgeable of new findings in content, pedagogical research, and innovative teaching strategies like inquiry. Findings such as those in the Third International Mathematics and Science Study (TIMSS) report, the National Assessment of Educational Progress (NAEP) and subsequent national and international research calls for a different way to look at the nature of science teaching and learning in American schools. From its inception, the National Science Education Standards (NSES) have called for inquiry as the predominant mode of delivery of science concepts in K-16 classrooms. Subsequent publications, i.e. Inquiry and the National Science Education Standards (NRC, 2000), have successfully defined inquiry for practitioners. However, it remains painfully clear, when observing science teaching in our schools, that little of these published efforts have resulted in significant change in classroom delivery. The PRISM grant seeks to work with LEA partners needing to improve student achievement in science by providing grades 3-6 science teachers with immediate understandings and support for research-based applications in inquiry methodology. To accomplish this, STEM faculty, Science Education faculty, LEA faculty and administrators, and Region III SMART schools will partner in the TRIAD model of professional development to achieve a better understanding of the delivery of science content through inquiry-based procedures.
Guskey Reflection on Professional Development
- Evidence of research-based characteristics of professional development.
Research on professional development (Guskey and Yoon 2009) suggests that workshops can be among the least effective methods of professional development because they often offer little opportunity for active-learning, lack follow-up, and do not provide teachers with an opportunity to adapt practices to their classrooms. Taking this research into account the PRISM Project utilized a blended model of professional development in which teachers actively engaged in inquiry activities presented by both content and pedagogical experts and their peers during several face-to-face workshops and then engaged in online discussions and science content modules that enabled them to implement inquiry-based science strategies in their classrooms.
- Outside Experts
Recognizing that most elementary schools and rural school districts lack the resources and expertise to implement quality, research-based professional development in inquiry science, the PRISM Project uses both content area specialists (STEM faculty) and pedagogical experts (education faculty) to provide training during workshops and to facilitate online discussions of content and pedagogy. To encourage the development of teacher leaders and to draw from the "real-world" classroom experience of project teachers, the PRISM Project also uses science coaches who successfully participated in the project for two years to act as peer coaches, facilitating online discussions and presenting at workshops.
The amount of time teachers are engaged in professional development appears to be critically important for the effectiveness of the professional development. But equally as important is the organization and focus of that time (Birman, et al 2000; Garet, et al 2001). The PRISM Project has provided teachers with 24 hours of face-to-face workshop time each year and has required much more time on-site. Workshop evaluations suggest the PRISM Project has demanded a significant amount of teachers' time. Completion of the SciPack online content modules, for example, can take several hours to complete. All time required by teachers has been carefully structured by project staff to align with project goals and to help teachers meet project outcomes.
A strength of the PRISM Project is the degree to which teachers are engaged in the project throughout the year through the use of monthly online discussions. Rather than expecting teachers to take knowledge and skills gained through face-to-face workshops and implement it in their classrooms, teachers are supported by content and pedagogical experts and peer coaches. Qualitative analysis of the online discussions suggests that teachers are actively engaged in meaningful discussions of science content and inquiry methods. Teachers are also required to complete at least three inquiry lessons in their classrooms, reflecting and documenting each to include in a Scoop Notebook. Evidence from Scoop Notebooks suggests that all teachers are implementing elements of inquiry science in their classrooms.
Rather than having teachers passively receive information during workshops, which is a common complaint of current professional development, PRISM teachers were actively engaged in inquiry science activities during face to face workshops. Teachers participated in lessons on buoyancy, mass and volume, and the properties of dry ice, for example. By experiencing inquiry lessons it is believed that the teachers gained a better understanding of how to implement them in their own classrooms. Evidence collected from teachers suggests that this is the case. Requiring teachers to implement their own inquiry science lessons also provided opportunities for them to be actively engaged in their learning.
To support teachers' ability to effectively implement inquiry science the PRISM Project provided them with a solid foundation of content knowledge on key topics in science through the use of NSTA SciPacks. These online content modules introduced teachers to concepts such as Force and Motion, Electricity, and the Solar System in an engaging manner. Results from pre and post teacher of teacher science content knowledge gained from the SciPacks were highly significant statistically.
Birman, B., Desimone, L., Porter, A. and Garet, M. (2000). Designing professional development that works. Educational Leadership, 57, 28-33.
Garet, M., Porter, A., Desimone, L., Birman, B., and Yoon, K. (2001). What makes professional development effective?: Results from a national sample of teachers. American Educational Research Journal, 38, 915-945.
Guskey, T. and Yoon, K. (2009). What works in professional development? Phi Delta Kappan, 90(7), 495-500.
PRISM Project Blended Learning Model...
Research on blended learning models for professional development suggests that such an approach can clearly enhance the content and pedagogical knowledge of teachers. As it is typically defined, a blended learning environment combines "some face-to-face instruction where learners are co-located, with web-based instruction where they are not in the same location" (Oweton, Sinclair, & Wideman, 2008, ¶11). The purpose of blended learning models is to provide teachers the opportunity to interact with peers over significant periods of time to develop knowledge and skills while remaining in their classrooms. This model will allow for STEM faculty to work with teachers online to develop science content and for the science education faculty to develop inquiry-based teaching strategies over longer periods of time. Thus, this method will enable university faculty to monitor learning and provide feedback online to teachers in the field.
The PRISM project has integrated multiple blended learning strategies that combine face-to-face workshops and school visits with online learning. Partner teachers meet three times a year for science inquiry workshops that give teachers opportunities to personalize their understanding of inquiry by exploring and performing science inquiry activities, learning questioning strategies, and creating inquiry-based lesson plans. Face-to-face workshops focus on building a community of learners and understanding the New Science Standards and the Science Inquiry Continuum.
In addition, training of "how to" work in an online environment was a needed component in a face-to-face format. Teachers were systematically walked through the process of signing on and using the online programs we have made available that include an online threaded discussion course as well as the NSTA SciPacks and website.
The Project Evaluators have begun routine visits to cohort teachers' schools. These initial visits have witnessed interesting discussions regarding science content and program responsibilities. Each school visit is scheduled for one hour and addresses concerns from both the teacher participants, principals and the project faculty. Discussion points include the inquiry continuum, the NSTA SciPacks, the online course, and related implementation issues.
Complementing the face-to-face experiences, the PRISM project offers many online learning opportunities. First there is the main web page of the project found at http://www.msubillings.edu/prism From this site participants can access the National Science Teachers Association home page and are linked to the NSTA SciPacks, SciGuides, and other content information provided by NSTA with a full membership status. The NSTA website is rich with professional development opportunities for participants. Participants can access teaching guides and curriculum materials through this site or research current understandings for teaching science in the classroom.
Second, the teachers are registered to participate in the D2L online threaded discussion course called Integrating Inquiry into the Curriculum. This online format has been very successful and encourages participants to asynchronously interact with the STEM faculty, Evaluators, Project Coordinator and Science Education faculty regarding all the happenings in the project. The partners are interacting on the SciPacks and other difficult concepts. The STEM faculty is explaining concepts beyond the content of the SciPacks helping teachers to begin to apply the newly learned information into their classroom. The Science Education faculty is requiring the participants to interact and post lessons regarding their use of Montana's Inquiry Continuum. In this blended learning format, the partners are learning about the lessons they are currently teaching and how these lessons fit on the Montana Inquiry Continuum.
The research design is based on Thomas Guskey's model for evaluation of professional development:
- Participant reactions will assess satisfaction with each workshop using survey information collected at the end of each workshop.
- Participant learning will be assessed by measuring their learning in science from the SciPacks (test scores), and from their online discussions about the content.
- Organizational support and change will be assessed through participant surveys, onsite interviews of project staff with participating teachers and administrators, and records of discussions at project meetings.
- Participant use of new knowledge and skills will be assessed through examination of Scoop notebooks, observation of participants' classrooms, records of online discussions, and their reflections on the success of the implementation.
- Student learning outcomes will be assessed though an analysis of student test scores (grade 4 only). Students will also be surveyed about their reaction to this type of learning.
Methods of Data Collection...
Multiple types of data were collected at various times during the year, both at workshops and onsite. Data was analyzed by the project's evaluators Michael Scarlett and David Davison. A quasi-experimental design was used to compare the participants' learning of science content and their level of comfort with teaching inquiry science with a comparison group of approximately 30 teachers. A summary of the type of data collected for each level of the evaluation is included in Table 1.
|Guskey Level||Source 1||Source 2||Source 3||Instruments used|
|1.||Participants' Reactions||Workshop evaluations||Partnership Survey||D2L participation rates/other qualitative D2L data||Workshop Evaluations
Relevant Partnership Survey Questions
|2.||Participants' Learning||SciPack pre/post tests||Threaded discussions on D2L||Teacher Empowerment Post Scale||Teacher Empowerment Scale|
|3.||Organizational Support and Change||Partnership Survey||Interview with STEM Faculty||Workshop evaluations||Partnership Survey
|4.||Participants' Use of New Knowledge and Skills||Scoop Lesson Ratings and pre/post lesson reflections||RTOP||Threaded discussions on D2L||Scoop Rating Scales
RTOP Rating Scales
|5.||Student Learning Outcomes||MontCAS data for 4th grade students||MontCAS Science Test|
Methods of Data Collection:
- Workshop surveys: At the end of each workshop, participants completed a survey indicating their level of satisfaction with the workshop and what they learned from it, and were learning from the project.
- For the first two of the six SciPacks completed, participants took a pre-test and a post-test to assess their learning from that SciPack. (The same pre- and post-tests were administered to a comparison group.)
- Online discussions focused on questions about science content and its application to participants' classrooms. These discussions were evaluated.
- All participants (along with many of their administrators) were visited informally in their school settings by an evaluator to assess their satisfaction and comfort with project goals and activities, and whether there were issues that needed to be addressed.
- Scoop notebooks were completed during the second year of a participant's involvement in the project. Examination of these notebooks by both the participants and the project evaluator indicated how well the participants were implementing new knowledge and skills in their lesson planning.
- A random sample of participants were observed using RTOP to assess how well they were using inquiry-based strategies in their science instruction.
- The Survey of Enacted Curriculum and the Teacher Empowerment Survey were administered as pre- and post-tests to evaluate teachers' beliefs about the teaching process.
Read teacher testimonies...
3rd Grade Teacher
No matter how long you teach there is always something new that will turn your teaching around; PRISM has done that for me. Inquiry teaching has been out there, but the question has been, how do we go about it? PRISM put this into focus and helped to enhance my curriculum. The program challenged my background knowledge and then gave me tools to improve these areas with an interactive online study. These SciPack programs are truly designed to help everyone succeed with their vocabulary building, knowledge sharing, quizzes and amazing experiments to do online and other experiments to do with simple objects in my room. My students loved these experiments and wanted to do them over and over, even in their spare time. We also worked with the MSU-Billings college professors through provocative questioning that required research and soul searching. We had help when we needed it and had the other teachers to share with through the on-line chat format.
My favorite part of this program was the interaction with other teachers in our grade level. In a rural school system we are often the only one in our school at our particular grade level. PRISM gave us a vehicle to bounce thoughts and procedures off of and a way to find out how someone else handles different topics. Inquiry, in general, takes the lesson out of the teachers' hands and gives it to the students to experience the excitement and wonder. It goes from "This is what will happen, this is how you will find out," to "What do you think will happen, how will you find out?" The science recipe book method is put back onto the shelf, allowing students to think and create ideas, to solve problems put before them. Stepping back and letting the students decide what they need for an experiment, ponder over their observations and put together their results is not an easy thing for the teacher to do. But, when the kids share their results with others and explain what happened and why, the teacher can see what they truly understand. It has made my science teaching more fun for me and my students. Third graders love science! With the inquiry method, this love for science will stay alive as they work their way through the grades.
5th Grade Teacher
Big Timber Grade School
I've always felt that I never have taught any two years the same in 38 years of teaching science to my fifth graders. I've always been active in trying new ways to teach. The PRISM program I'm involved in at MSU-Billings has provided me with tools I can use to teach science through inquiry.
A valued and often used tool is the inquiry continuum. Using the continuum helps me reflect on my methods and decide if there may be a more effective way to help students wrap their minds around a certain concept.
Another helpful tool is the Montana Content Standards framework. The content standards are written for each grade level and include the Essential Learning Expectations. Each content standard includes inquiry.
PRISM provides me access to NSTA's SciPacks. SciPacks provide an opportunity to increase/improve my content knowledge. At first, I thought they would be a waste of my time because I considered myself fairly knowledgeable. However, I was pleasantly surprised and challenged by the material. I did learn plenty by doing the SciPacks. I've even got some of my students learning from them.
My participation in PRISM is exciting, fun, challenging, and helpful. It has even caused me to step out of my comfort zone. I've developed totally new investigations and inquiry labs. After teaching as long as I have, I didn't think I'd need any new and different labs. Yet, Dr. Ken, Dr. Dave, Dr. Stuart, and Dr. John have inspired me to make my science lessons more inquiry based.
Science Teacher, Grades 6-8
I've spent three years in the Partnership to Reform Inquiry Science in Montana (PRISM) grant. Every year has helped me become a better teacher. All teachers know that if a student is engaged, real learning will take place, but with the current structures and time constraints it makes it difficult to build lessons that relate to, and engage students. We tend to squelch curiosity in order to "get through" the necessary material. We have been losing student interest and inhibiting the true explorers of the future. PRISM has been a mentor to me in several ways. It is a constant reminder that time is needed for student investigation – making me take time in my classes. PRISM has also given me great resource including, but not limited to, the National Science Teachers Association (NSTA) website. Through this connection I've become more "content" aware so I can feel ease in scientific areas I was inhibited in before. I only wish that it could be an ongoing learning center. Science is huge and so necessary. I'd like to see more interactions with math and technology in a continuance of this grant.
5th Grade Teacher
How PRISM has changed my science teaching:
Children are born curious. All go through the questioning stage early in life. Parents are asked why and how nearly a million times. A child will look at his or her world and wonder about all the workings. As a child grows, these questions still occur. As parents, we tend to give immediate answers of "because I said so" or "the stork brings them." Children start to expect answers and ask fewer questions as they grow.
PRISM has reminded me that children are curious and interested beings. Just like when they were young, fifth-grade students still want to know the how and the why of the world. Each lesson I teach in science, I try to remember to allow my students to be inquisitive. I also try to step back and allow them to formulate their own answers. There is a place for teaching and a place for stepping back.
I love to watch my students as they lead their own learning more and more. This would have never happened if not for PRISM. I used to think I was a teaching ace if I did a hands-on "let's make goo" demonstration. Now, I realize that hands-on and inquiry are two very different things. Often inquiry is hands-on, but not all hands-on is inquiry. I take time to create lessons that are stimulating for both mind and body. We all learn more this way!
PRISM has been beneficial to me in more than just my teaching. I love having a community of other professionals to discuss science and teaching with. PRISM has given me an opportunity to learn so much from my peers. Stuart, Michael, and Ken are a great help with any question or concerns. I wish we, as teachers, had more opportunities to learn from such amazing men. They are truly experts in their fields and have helped me to become a better teacher.
The PRISM project seeks to develop and strengthen the pedagogical and content understandings in science of Grade 3-6 teachers. It provides two cohorts of teachers with professional development that has significant and meaningful science content that will enhance the understanding and implementation of inquiry science techniques into the classroom. The PRISM project provides instructional strategies of inquiry that enable teachers to teach in a manner that will improve student achievement in science and further develop the ability of teachers to understand and use the Montana Standards for Science.
The PRISM project is a blended learning model that combines face-to-face workshops with online learning. Partner teachers meet three times a year for science inquiry workshops facilitated by Dr. Miller and his project team that includes STEM faculty and evaluators. Teachers are given opportunities to personalize their understanding of inquiry by exploring and performing science inquiry activities, learning questioning strategies, and creating innovative lesson plans (See Scoop link). Workshops provide a format for teachers to learn from each other and to develop a professional learning community.
Online learning is also a major component of the PRISM project. Teachers are empowered to expand their content learning by participating in online threaded discussions and completing SciPacks through the National Science Teachers Association Learning Center. The online discussions provide teachers the opportunity to reflect on science inquiry and content questions and interact with peers. Instructors monitor learning and provide feedback to teachers in the field. A continuation of the professional learning community established in the workshop setting is a huge benefit of the online discussions.
The other component of online learning is SciPacks. Teachers complete three SciPacks a year. SciPacks are online science content learning experiences that employ an inquiry based approach with engaging simulations, embedded questions, and pedagogical implications. An example of the SciPacks is the Force and Motion SciPack with the learning objectives focusing on understanding and explaining the concepts of speed, velocity, and acceleration. A comprehension of mass and inertia, the nature of forces, and the meaning and significance of Newton's Laws of Motion and Newton's Law of Gravity must also be demonstrated. SciPacks help teachers better understand the science content they teach and are a useful resource and tool for teachers to utilize in their classrooms.
The PRISM project strives to empower teachers to find new innovative ways to implement science inquiry lessons into their classrooms. It is an exciting project with potential for amazing science inquiry learning not only for teachers but also for students!
- Participants have enjoyed and learned much from the on-campus workshops, especially in terms of the modeling of inquiry-based lessons in science.
- The SciPacks proved to be most effective in increasing the teachers pedagogical content knowledge in elementary science.
- Teachers' lesson plans showed evidence of an increasing use of inquiry-based methods in science instruction.
- Desire to Learn was effective as a means of getting teachers to interact with each other.
- In a context of limited financial resources, blended learning (partly live, partly online) is shown to be an effective way to provide professional development for teachers in a large geographical area.
- Particularly in rural schools, teachers are called upon to wear many hats, and there appears to be an increasing number of demands on their time. A number of teachers pulled out of the program because they were unable to balance all of these demands.
- More time at the workshops should have been devoted to introducing the teachers to implementing the online aspects of the project. Some frustrations that occurred would have been eliminated had that been done.
- The two surveys based on teachers' self-perceptions of their beliefs (Survey of Enacted Curriculum and Teacher Empowerment Survey) are based on teachers reflecting on their own beliefs and behaviors. Some results were counter-intuitive. We judge this to be the result of teachers' baseline for self-perceptions changing as they gained more information, rather than a failure to change in the direction anticipated.
- While it might have been desirable to observe more of the teachers' science lessons, in a large geographical region (and without appropriate funding), this was not feasible.
Science Content Tests...
Cohort teachers completed two SciPack pre- and post-tests: Force and Motion and Energy. The test score gains are submitted as evidence that the participating teachers have successfully gained content knowledge appropriate for the teaching of elementary science.
Force and Motion Test
Mean gain score: 16.38
t = 4.10 (df=23);
p = 0.0004. This result is extremely statistically significant
Because SDs so different, Sign Test used to compare means: n+ = 19, n- = 3
P= 0.0009, statistically very significant
Thus Cohort 1 teachers showed significant improvement in science content knowledge
Force and Motion Test
Mean gain score: 23.78
t = 6.38 (df=27)
p = 0.0001. This result is extremely statistically significant
Mean gain score: 15.81
t = 5.18 (df=25)
p = 0.0001. This result is extremely statistically significant
Based on this analysis, we judge that these teachers have shown a significant improvement in science content knowledge based on the results of these two SciPacks.
Comparison Group Teachers
Comparison group teachers have completed the same two SciPack pre- and post-tests. The test scores are submitted as evidence that these teachers remained relatively unchanged in their content knowledge of these two areas of elementary science.
Force and Motion Test
Mean gain score: 3.92
t = 1.01 (df=25)
p = 0.324. This result is not statistically significant
The t-test calculations are reported here but the disparity of the standard deviations makes the use of this test inappropriate (not random samples from equivalent populations). [Mean gain score: -1.64, t = 0.56 (df=24), p= 0.582]
A preferable statistical comparison is made from using the Wilcoxon matched-pairs, signed-ranks test, a non-parametric test. According to this test, W+ = 130.5, W- = 100.5, N= 21, leading to p = 0.614, which is statistically not significant.
Based on this analysis, we judge that these teachers have shown no significant difference in science content knowledge based on the results of these two SciPacks.
In order to understand the degree to which the goals and objectives of the PRISM Project were reflected in teachers' practices, participants collected artifacts from three of their science lessons to create a "Scoop Notebook." Using a procedure developed by researchers from the University of Colorado, Boulder and the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) (Borko, H., Stecher, B., & Kuffner, K.. (2007). Using artifacts to characterize reform-oriented instruction: The Scoop Notebook and rating guide. National Center for Research on Evaluation Technical Report 707), teachers collected artifacts of classroom practice and made observations in the forms of pre and post-reflections similar to the way a scientist might take a sample from nature to analyze back in a laboratory. For each scooped lesson teachers were instructed to include a lesson plan, instructional materials (handouts, worksheets), examples of student work (high, medium, and low quality), tests, quizzes, rubrics, and pictures of the classroom set-up, including instructions on the whiteboard, and instructional tools/materials.
Scoop Notebooks were evaluated using rating scales that included 11 Dimensions of Reform Science developed by CRESST and the five Essential Features of Classroom Inquiry (Montana K-12 Content Standards and Performance Descriptors for Science-Inquiry Continuum, 2008. Office of Public Instruction). Teachers were also asked to rate their own lessons using the five Essential Features of Classroom Inquiry. Although the teachers evaluated each individual "scooped" lesson, the evaluator gave his rating based on an overall evaluation of the Scoop Notebook. A total of 16 Scoop Notebooks were collected, a turn-in rate of 64%. Forty-eight science lessons total were included in the Scoop Notebooks.
Overall, it was clear from the Scoop Notebooks that teachers were using inquiry-based science instruction in their classrooms and the Scoop Notebook was a useful tool for understanding what science instruction looks like in teachers' classrooms. The majority of the lessons incorporated elements of inquiry-based instruction either somewhat or to a great extent, according to both teachers and the evaluator. Of the 48 lessons evaluated, none were rated a 1 (not at all) for degree to which the lesson address an aspect of the inquiry continuum. Interestingly, one area in which there was disagreement between the teachers' rating and the evaluator's rating was the degree to which the lesson allowed learners to formulate explanations from evidence. This may be due to the difference between how the lesson actually taught and the evidence that was provided in the Scoop Notebook.
The ratings based on the 11 Dimensions of Reform Science also suggest that teachers are meeting the goals of the professional development by incorporating aspects of inquiry-based science into their teaching. Specifically, it was very clear from the lessons that teachers are engaging students in hands-on activities, creatively using available scientific resources. It was also clear that teachers are adept at grouping students to work cooperatively on scientific problems. Similar to the rating scale based on the Inquiry Continuum, it appears that teachers need to require their students to provide explanations/justification for activities completed during science lessons.
Survey of Enacted Curriculum...
The attached chart shows that Cohort 1 showed improvements in the following areas of Instructional Practices: Communicate Understanding, Analyze Information, Make Connections, and Active Learning. Until we have access to the individual teacher data the significance of these improvements cannot be assessed. However, Miller's Teacher Empowerment Survey did show significant improvement in the area of Instructional Strategies for Cohort 1 over the Comparison Group. This would confirm the belief that this is an area where the project has contributed to growth in the performance of the teacher participants. (It should be noted that with both instruments the teachers are recording a self-perception. There are instances in both instruments where the teachers' self-ratings deteriorated between the two test administrations: one explanation is that after participating in the project for a year, the teachers' self-perceptions were more realistic.)
The following factors appear to be worthy of more in-depth analysis based on statistical examination of individual teacher scores.
- Content ready: scores appear to show growth
- Active Engagement of Teachers: scores appear to show growth
- Communicate Understanding: scores may show aspects of growth
- Equity Readiness: scores show some median shifts
- Influence of standards: scores show limited shift
- PD Data Focus: scores may show some shift (probably not significant)
- PD Student Learning: two items (135, 138) appear to show shift in scores
The following items in the survey show a shift in the median score from 2009 to 2010:
|PD Standards Focus:||128, 129|
|PD Data Focus:||135|
|PD Content:||128, 129|
|PD Coherency:||119, 121|
|Content Ready:||82, 84, 86|
|Equity Readiness:||87, 89, 90|
|Communication Understanding:||28, 50|
|Analyze Information:||54, 55, 61|
|Active Learning:||31, 34, 44|
|PD Active Engagement Teachers:||112, 114, 115, 117 (negative)|
|Influence of Standards:||129|
The project agreed to use RTOP as the means of observing the extent of the implementation of inquiry methods by the project teachers in their classrooms. Because such observations are time-consuming, a random sample of eight of the 24 Cohort 2 teachers was selected for observation. These observations were originally intended to be conducted during Fall 2009 while Cohort 2 teachers were still being inducted into the project; scheduling problems caused a delay until Winter/Spring 2010. As can be noted from the observation scores (60, 55, 49, 45, 37, 29, 27, 24), several of the teachers were already using inquiry strategies in their science teaching.
The use of direct classroom observation was not part of the original project design. While it would, no doubt, have been interesting to make before and after comparisons of the teachers' science instruction, we expect to learn as much or more from the evaluation of the teachers' SCOOP notebooks. (Further, four of these eight teachers have since withdrawn from participating in the project for various reasons.) Therefore, this data serves merely to reinforce information gathered from other sources.
This instrument was developed by the Principal Investigator as part of his doctoral research at the University of Wyoming. (Miller, K. W., 1993. Restructuring the school through a partnership of teacher, administrator, and professional development leader. Unpublished doctoral dissertation, Laramie, WY: University of Wyoming.)
In this study, the pre-test was administered in October 1991 and the post-test in May 1992. A chief finding of this study was that teachers typically did not realize how unempowered they were until they worked on this project.
Based on a factor analysis of the scale items, five major factors emerged in Miller's study:
- Self-reflection (Items 3, 5, 8, 19)
- Educational Influence (Items 1, 2, 4, 6, 7, 22, 27)
- Collegial Relationships (Items 11, 13, 14, 16)
- Confidence: (Items 9, 15, 17, 18, 20, 25)
- Instructional Strategies: (Items 10, 12, 21, 23, 24, 26)
These subtests of factors are used in this investigation to assess teacher empowerment within this project.
Cohort 1 teachers took the pre-test in August 2008 and the post-test in April 2010, while the Comparison Group took the pre-test in November 2009 and the post-test in May 2010. This evaluator was concerned about the capacity of Cohort 1 teachers to respond to this survey in a consistent manner over such an extended period. What was particularly interesting was that both groups indicated a significant change of rating for Factor B, while there was no statistical difference between the two groups. One wonders what might have happened to cause the Comparison Group to feel more empowered relative to Factor B in a period of six months. Nevertheless, the data has been analyzed as though it is valid.
For both groups of teachers the data being analyzed is the change in rank of the teacher's self-perception on a given item. The changes from pre-test to post-test have been analyzed using the Wilcoxon Matched-Pairs Signed-Ranks test, an appropriate test for analyzing small groups of data, especially when the underlying distribution is not known.
Based on the array of comparisons, this evaluator concludes that there is a real difference in the ratings of the Cohort 1 teachers and the Comparison Group on Factor E, Instructional Strategies. In terms of the focus of the project and the emphasis in the workshops and the online instruction, we judge that this difference is to be expected.
Cohort 1 Results
W+ = 25.5, W- = 2.5, N = 7, p ≤ 0.047: significant
W+ = 21, W- = 0, N = 6, p ≤0.03: significant
W+ = 26.5, W- = 1.5, N = 7, p ≤ 0.03: significant
Comparing Results for Both Groups
W+ = 12, W- = 9, N = 6, p ≤ 0.84: not significant
W+ = 20, W- = 1, N = 6, p ≤ 0.6: approaches significance
W+ = 21, W- = 0, N = 6, p ≤0.03: significant
Analysis of Participants' Written Comments
At each workshop to date participant satisfaction responses have been uniformly positive. That was generally true in this workshop also. Participants enjoyed the activities, and the more reflective of them welcomed having essential features of inquiry science modeled for them through these activities. One comment that has been oft repeated is that a number of the participants are looking for a repertoire of inquiry-based activities to take away with them from the workshop. This is not a surprising finding–the evaluator has encountered this type of response in a wide range of professional development activities for teachers. It does make good sense for teachers to go away from a workshop with a lot of ready-made ideas to put into practice. The danger is when we cross the line providing teachers with access to valuable resources to "spoon-feeding" them. What is disturbing is that one goal of inquiry-based learning is to have teachers become more reflective in their selection of learning materials. Some of the D2L learning assignments set out to do just that, especially in terms of having participants convert a textbook based activity into an inquiry-based activity, and some of the responses suggested that this was too time-consuming. As we interviewed many of these teachers, typically outside of regular school hours, we noted that their school commitments engaged them for many hours and left them little time for reflection or for finding new resources. In that sense, if the goals of the project are to be attained in a sustainable manner, teachers need to learn to share access to valuable resources. We judge that this is one reason why these workshops have been so well appreciated. But they are also a costly portion of the project and, in a region as large as that embraced by this project, distance learning/conferencing becomes the only cost-effective way of promoting such collegiality.
The area of participant responses that was not always so positive dealt with their views towards D2L. There were some technical issues, and participant concerns about them were valid. However, in a society that has "sold out" to Facebook, Twitter and YouTube, we expect teachers to be technologically literate. Perhaps there should have been more before-the-fact discussion on how to use D2L effectively, and making sure that the staff and the participants were all on the same page. One possibility might have been for the staff and the participants to discuss together what should be expected using D2L. From personal interactions with the participants, as well as the reading of their workshop comments, we judge that the participants do want to use D2L effectively, but they want its use to be smooth.
Notwithstanding the mildly positive/neutral comments about D2L, participants continue to be quite satisfied with the workshops. As the project continues (and after it is completed), attention must be directed to finding ways of accomplishing the goals of the workshop in a distance learning format.
Analysis of Ratings
- I am making substantial use of the knowledge gained from the SciPacks in my teaching of science.
The median response for Cohort 1 was Strongly Agree and for Cohort 2 it was Agree. This difference might be explained by the Cohort 1 participants having had an additional year to implement this knowledge.) In any event, viewed along with the scores on the SciPack tests for both cohorts, it is clear that the participants have learned a great deal from the SciPacks and value what they have learned.
- The Inquiry Continuum is improving my ability to teach using the methods of inquiry science.
The median response for Cohort 1 was Agree and for Cohort 2 it was Strongly Agree. This result is again very positive for both groups, but we might have expected that it would have been stronger for Cohort 1. There is no ready explanation for this situation, especially because Cohort 1 teachers have had an additional year to become accustomed to using the continuum.
- The teacher workshops are improving my ability to teach using the methods of inquiry science.
The median response for both cohorts was Agree (leaning towards Strongly Agree). This positive response supports the written comments the participants made about the workshops.
- The teacher workshops have increased my knowledge of content needed to teach science in my classroom.
The median response for both cohorts was Agree. While one might initially expect the rating to be closer to Strongly Agree, we must remember that the workshops have focused on instructional strategies, not content. Of course, participant content knowledge would have been strengthened by the science activities in which they were engaged. The surprising rating is the teacher who was ambivalent about responding to this and the previous statement. (From personal interviews with each teacher, this evaluator was unaware of any such ambivalence.)
- My school's administration has supported my involvement in the PRISM Project.
The median response for both cohorts was Agree. Sometimes the response to a question of this type may be a matter of perception, but it is disconcerting to note that administrative endorsement was a precondition of participation in the project. At least one respondent validated this dissonance in the narrative comments. We note that administrative interest in the workshops has been modest at best, in spite of their receiving invitations to attend. This must raise questions about the capacity of the goals of the project to be spread to other teachers who might potentially be impacted by project teachers. It will be interesting to note the effectiveness of those teachers from Cohort 1 who will engage in Year 3 of the project, essentially to mentor their colleagues in inquiry science.
- The PRISM Project Staff has supported my involvement in the PRISM Project
The median response for both cohorts was Strongly Agree. It is pleasing that the participants are satisfied with their contacts with project staff.
- The D2L online course has been effective in improving my ability to teach using the methods of inquiry science.
The median response for Cohort 1 was Neutral and for Cohort 2 it was Agree. This response is not surprising in view of the extensive comments about D2L in the participants' additional comments. Some of the issues were technical in nature, but others raised questions about using the online instruction more effectively.
- The pedagogical discussions have been successful in improving my ability to teach using the methods of inquiry science.
The median response for Cohort 1 was Neutral and for Cohort 2 it was Agree. It is not unexpected that this item's ratings should be consistent with those of Item 7. In addition to the comments made in response to this survey, this evaluator gathered many comments of dissatisfaction about D2L from the personal interviews with Cohort 2 participants.
- The pedagogical discussions have been successful in increasing my knowledge of content needed to teach science in my classroom.
The median response for Cohort 1 was Neutral and for Cohort 2 it was Agree. This response is consistent with those of the two previous items and the remarks made earlier are valid here.
- The content discussions have been successful in increasing my knowledge of content needed to teach science in my classroom.
The median response for Cohort 1 was Neutral and for Cohort 2 it was Agree. Again, the responses to all of the items dealing with D2L remain consistent and point to the need to address how D2L is implemented.
- The content discussions have been successful in improving my ability to teach using the methods of inquiry science.
The median response for Cohort 1 was Neutral and for Cohort 2 it was Agree. Again, the responses to all of the items dealing with D2L remain consistent and point to the need to address how D2L is implemented. It needs to be remembered that one goal of this project is to make inquiry science accessible to remote area teachers, and online learning and discussions are integral to achieving this goal. It is critical, therefore, that problems (real or perceived) in making D2L effective must be addressed.
- Creating a Scoop Notebook has been successful in improving my ability to teach using the methods of inquiry science.
This item was answered only by Cohort 1 teachers since Cohort 2 teachers have not yet worked on their SCOOP notebooks. The median rating was Agree, indicating that the teachers view it positively. More discussion of this issue will be found in the analysis of the SCOOP notebooks.
These ratings taken as a whole support the participants' overall satisfaction with the project. The one issue that caused concern deals with the implementation of D2L. These ratings results are all consistent with the written comments that the participants made at the same time.