What Influence Does Regularly Using Einstein Project Materials Have on State Standardized Fourth Grade Science Test Scores?

This report is based on a research study conducted by
Scott Ashmann
Assistant Professor, Science Education
University of Wisconsin-Green Bay
April, 2007

Introduction

Since the time of Sputnik in the late 1950's, hands-on science has been strongly promoted as a method of science instruction. However, no consensus has been reached on the relationship of hands-on science to student achievement, though this topic has been researched using various methods since the turn of the 20th century. The importance of hands-on science has come under debate for practical, political, and theoretical reasons. Given the attention paid to hands-on science as an approach to science teaching and state standardized test scores as a measure of student achievement, this study set out to compare fourth grade science test scores between school districts that regularly used hands-on science activities via the Einstein Project units and

• State of Wisconsin average test scores, and
• a randomly selected sample of Wisconsin school districts that do not use Einstein Project materials.

The hypothesis was that school districts regularly using the Einstein Project units would have a higher percentage of students who scored in the "Advanced" and "Proficient" levels of the Wisconsin Knowledge and Concepts Exam.

Background of the Einstein Project and Wisconsin State Testing

In 1991, a core group of educators and business people joined together to engage in long-term efforts to affect lasting change in Northeastern Wisconsin's elementary and middle school science classrooms. Plans were made to expand the use of a hands-on science curriculum and to enhance classroom teachers' science knowledge. The Einstein Project was established, has since expanded into other parts of Wisconsin, and has become a national model for systemic change in science education.

The Einstein Project's hands-on science units and accompanying materials were developed by the National Science Foundation, the Smithsonian Institution, and the National Academy of Sciences. The eight-week units are packaged in a kit that contains enough materials for an entire class of 30 students. As kits are returned to the Einstein Project's science resource center from the school districts that lease them, they are restocked and refined through a teacher-student evaluation process. Currently, 34 units are available for grades K-6, in addition to seven middle school units. The resource center stocks over 1100 kits.

The Wisconsin Knowledge and Concepts Examination (WKCE) is a statewide standardized exam given each year to measure student achievement in five subject areas: reading, language arts, mathematics, science, and social studies. Wisconsin's model academic standards in English language arts, mathematics, science, and social studies determine the scope of the state exams. The score a student earns on each WKCE test determines his or her level of proficiency in that subject matter. Since 1997-98, four proficiency levels have been used: advanced, proficient, basic, and minimal performance. Achieving at the advanced or proficient level is the long term educational goal for all students. The currently used cut-scores dividing the proficiency levels were established at a state-level workshop held in February, 2003. The definitions of each proficiency level were determined as:

Advanced:	demonstrates in depth understanding of academic knowledge and skills tested on the WKCE
Proficient:	demonstrates competency in the academic knowledge and skills tested on the WKCE
Basic:	demonstrates some academic knowledge and skills tested on the WKCE
Minimal Performance:	demonstrates very limited academic knowledge and skills tested on the WKCE

Methodology

The Einstein Project's records were used to determine the number of kindergarten through third grade (K-3) units that were leased by each school district during the academic years from 1997-2005. The K-3 units have the potential to impact student learning before students take the WKCE in November of their fourth-grade year. Twenty-one school districts were determined to be regular users of the Einstein Project's K-3 units.

WKCE data were available through the Wisconsin Department of Public Instruction's (DPI) website (http://www.dpi.state.wi.us/). For the past five years, the DPI has made available detailed information about the performance of state schools through an extensive data resource called the Wisconsin Information Network for Successful Schools (WINSS). This network is seen as an electronic report card and provides information on what children should know and be able to do, how successful they are in acquiring this knowledge, and what work needs to be done to improve the performance of both schools and students. Relevant data to this study included the percentage of students in each district who were categorized as advanced or proficient in each academic year. This percentage is referred to as the A+P score and is the number that is widely reported each year. Maximizing this percentage is a goal of school districts.

Unfortunately, WKCE proficiency data for February, 2002 and earlier are not comparable to proficiency data for November, 2002 and later years. Cut-scores were originally determined in 1997-98, but were changed in 2002-03 after WKCE content was altered to improve alignment with state academic standards, the No Child Left Behind Act was implemented, and state testing dates were changed from February to November. The newer cut scores are based on what students should know and be able to do at the beginning of an academic year. Thus in this study, A+P scores were used for 2002-05. November, 2006 scores were not yet reported when the analysis occurred. It should be noted that not all data are reported in WINSS to protect student privacy. Care is taken to avoid disclosure of information about small groups of students. A group is considered small if the number of students in the group is five or fewer

Findings

This research study was organized around four questions that compared the percentage of students who were categorized as advanced or proficient on the WKCE (A+P score) for each year from regular users of the Einstein Project materials (hereafter referred to as Einstein Districts) with State A+P scores and the A+P scores from a randomly selected set of school districts that did not use the Einstein Project materials (hereafter referred to as Non-Einstein Districts).

1. How did the A+P scores for fourth grade science from Einstein Districts compare to State A+P scores?

A common question asked by many school personnel and parents is: How well is my school district doing compared to State averages? For each year of this study, the A+P score for each Einstein District and the State were downloaded from WINSS. Simple averages were calculated and are noted in the following table. For each year and overall, the Einstein Districts had a higher A+P score than the State.

Years	State A+P Scores	Einstein Districts' A+P Scores
2002	77.0	82.5
2003	80.8	88.5
2004	77.5	85.1
2005	77.5	84.3
Average of 2002-05	78.2	85.2

In addition, from 2002-05, 88.2% of the individual Einstein Districts had a higher A+P score than the State average. In other words, almost nine out of ten Einstein Districts exceeded the State A+P score. These two pieces of evidence point toward students from Einstein Districts outperforming the State averages.

2. How do the A+P scores for fourth grade science from Einstein Districts compare to the A+P scores from Non-Einstein Districts?

Included in the State A+P scores is the school district of Milwaukee. For a variety of reasons, the performance of Milwaukee Public Schools' students lowers the State A+P score. So what would happen if Milwaukee was removed from the comparison, and the Einstein Districts were compared to a randomly selected set of Wisconsin school districts? (By the luck of the draw when this random set was generated, Milwaukee was not selected.) Data in the following table are similar to the previous table, except the State scores have been replaced by the Non-Einstein Districts' scores. Once again, the Einstein Districts' A+P scores are higher every year and overall.

Years	Non-Einstein Districts' A+P Scores	Einstein Districts' A+P Scores
2002	80.4	82.5
2003	84.9	88.5
2004	82.8	85.1
2005	82.7	84.3
Average of 2002-05	82.7	85.2

3. Is the achievement gap in fourth grade science A+P scores smaller for Einstein Districts when compared to (a) the State and (b) Non-Einstein Districts?

The achievement gap in science between privileged students (primarily white males from affluent backgrounds) and underrepresented populations (traditionally, these groups include females, minority students, students with disabilities, economically disadvantaged students, and students who have limited English skills) has drawn much attention from researchers, the media, and school personnel. A common question has been: How can more students from these populations be successful in the sciences? One measure of success is academic achievement as measured by standardized tests, so a couple of comparisons were made in this study. For the comparison between the Einstein Districts and the State, it was found that the achievement gap was smaller in Einstein Districts every year and overall for students with disabilities, economically disadvantaged students, students with limited proficiency in English, and minority students. Even though the overall achievement gap for female students was slightly smaller for State A+P scores than Einstein District A+P scores, for two of the four years represented in this study the Einstein Districts achievement gap for female students was smaller than the State gap. In fact, in 2005, female students had a higher A+P score than male students in Einstein Districts, which is a rare occurrence for female students.

For the comparison between the Einstein Districts and the Non-Einstein Districts, the overall achievement gap was smaller in Einstein Districts for female students, students with disabilities, students with limited proficiency in English, and minority students. The difference in the achievement gap for students with limited proficiency in English was so large that it was statistically significant. However, the achievement gap for economically disadvantaged students was smaller for Non-Einstein Districts than Einstein Districts.

4. How do the fourth grade science A+P scores for Einstein Districts compare to (a) the State and (b) Non-Einstein districts with respect to:

• Females
• Students with disabilities
• Economically disadvantaged students
• Limited English proficient students
• Minority students?

The final comparison in this study looked at how underrepresented populations (e.g., female students) from Einstein Districts compared to their counterparts (e.g., other female students) in the State and from Non-Einstein Districts, whereas in the previous question the underrepresented population (e.g., females) were compared to their privileged counterparts (e.g., male students). For the first comparison, Einstein Districts outperformed the State each year and overall for female students, students with disabilities, economically disadvantaged students, students with limited proficiency in English, African American students, Hispanic students, and combined minority students. The Einstein District A+P scores for Asian students were greater each year and overall, except for 2002. However, the State A+P scores for Native American students were greater each year and overall.

For the second comparison in this question, Einstein Districts outperformed the Non-Einstein Districts each year and overall for female students, students with disabilities, students with limited proficiency in English, and Asian students. All of these differences are so large that each is statistically significant. Results were mixed for economically disadvantaged students, Native Americans, African Americans, Hispanics, and combined minority students, with some years the Einstein Districts outperforming the Non-Einstein Districts while other years the opposite occurred.

Conclusions

There are three overall conclusions that are solidly substantiated by the data gathered in this study. All strengthen the case for the use of a hands-on science curriculum in elementary classrooms and show signs of academic success for underrepresented student populations in the sciences:

1. The findings from this study strongly support a positive influence of the use of Einstein Project materials on state standardized 4th grade science test scores, based on comparisons between Einstein Districts and State scores and Einstein Districts and randomly selected Non-Einstein Districts.
2. Furthermore, the findings show that the use of Einstein Project materials helps to close the achievement gap for students with disabilities, students with limited English proficiency, and minority students.

In addition, female students, students with disabilities, and students with limited English proficiency and Asian students from Einstein Districts were found to significantly outperform their counterparts from Non-Einstein Districts.