Innovation Fund Applied Learning Grants Showing Early Success

February 5, 2013

By Sam Rauschenberg

In fall 2012, the Governor’s Office of Student Achievement (GOSA) contracted with The Findings Group to develop the Applied Learning Student Questionnaire (ALSQ) for the Race to the Top Innovation Fund Round 1 and 2 applied learning grantees. The ALSQ measures changes in student intrinsic motivation, self-management, and intent to persist in Science, Technology, Engineering, and Mathematics (STEM) studies during the fall semester. The survey also asks students about problem solving and implementation activities in each program. This education update highlights findings from the December 2012 survey administration, and the full report is attached at the bottom of this page.

Race to the Top Innovation Fund
The Innovation Fund is a $19.4 million fund created under Georgia’s Race to the Top (RT3) plan. It provides competitive grants to support partnerships between Georgia school districts or charter schools, institutions of higher education, businesses and non-profit organizations that develop or implement innovative and high-impact programs, particularly those related to STEM. The State intends to use the Innovation Fund to determine best practices in innovative programming related to STEM, applied learning, and teacher/leader recruitment and development to influence future education policy efforts. GOSA is charged with the statewide evaluation of RT3, including Innovation Fund projects.
ALSQ Survey

The ALSQ uses a pre-post retrospective design where students provide two answers for each item. One answer represents their opinion or understanding before participating in the program, and the other answer represents their opinion or understanding at the time of survey administration. As a result, students gauge growth on each item in one rather than two survey administrations. This design prevents their frame of reference from changing between administrations.

The ALSQ included 36 items split into five constructs: Intrinsic Motivation, Self-Management/Self-Regulation, Intent to Persist, Problem Solving, and Implementation Activities. Each construct included at least five items. Students responded to items on a five-point Likert Scale where 1 equals “Strongly Disagree,” 2 equals “Agree,” 3 equals “Neutral,” 4 equals “Agree,” and 5 equals “Strongly Agree.”

In December 2012, six grantees administered the survey to gauge mid-year progress of their students. The following grantees participated:

  • 21st Century STEM Collaborations: Applications of the Direct to Discovery ModelA collaboration between Barrow County Schools and Georgia Institute of Technology to integrate Direct-to-Discovery, a program using technology to link world-class scientists with K-12 classrooms, into the requirements of the Georgia Performance Standards.
  • Drew Charter School Partners of Innovation: A partnership between Georgia State University, Georgia Tech, and Drew Charter School to create one of the state’s first STEAM (Science, Technology, Engineering, Arts and Mathematics) schools.
  • Murray County STEM Academy A partnership between Murray County Schools, Georgia Northwestern Technical College, the Chatsworth-Murray County Chamber of Commerce and others, to open a program focused on remediating 8th grade students and developing their interest in STEM careers.
  • STEM for Life Program: A partnership between Carroll County Schools and Southwire Company to expand and replicate the existing 12 for Life Program, which supplements classroom learning with real-world experience in advanced manufacturing.
  • The STEM Targeted Education Program (STEP) Academy: A collaboration between Gwinnett County Public Schools, Gwinnett Technical College, and the Gwinnett Chamber of Commerce to create a Biotechnology Research and Development career pathway program with accelerated coursework and mentoring for at-risk, overage eighth grade students.
  • Tift County Mechatronics Partnership:  A partnership between Tift County Schools, Moultrie Technical College, ConAgra Foods, Heatcraft Manufacturing and others to develop a career pathway focused on Mechatronics, an interdisciplinary field of study involving control, electrical, mechanical, and computer engineering systems that equips students to work in a variety of industrial, manufacturing, and health sciences settings. 
Why evaluate non-cognitive skills?

In addition to increasing innovative, STEM-related applied learning opportunities for students, a goal of the Innovation Fund’s Applied Learning priority is to improve student self-management. This goal recognizes the growing body of research—led by University of Chicago economist James Heckman and popularized in the recent Paul Tough book, How Children Succeed—that stresses the importance of developing students’ non-cognitive skills as well as students’ cognitive skills for success in school and life.[1] These non-cognitive skills include curiosity, motivation, the ability to focus and follow through on tasks, the ability to learn from failure, and self-regulation. The ALSQ gauges whether students believe they have improved in these areas over the course of the program.

Results

In total, 860 students across the six programs completed the survey with a 95% response rate. Each construct was internally consistent with a Cronbach’s alpha greater than 0.70.[2] Overall, students in all six programs had statistically significant gains in perceived intrinsic motivation, self-management/self-regulation, and intent to persist in STEM.

Source: ALSQ Omnibus Report

Intrinsic Motivation

The percentage of students who were intrinsically motivated to study STEM increased from 55% to 76%.

This construct includes questions about whether students prefer challenging work, learn from mistakes, and are interested in learning about STEM through the program. As a whole, students’ perceptions of their skills grew the most in this construct relative to other constructs. More specifically, the percentage of students who were intrinsically motivated to study STEM increased from 55% to 76%. Most notably, that percentage increased from 41% to 79% for students in STEM for Life, and from 49% to 78% for students in STEP Academy. Drilling down in all programs, the percentage of students who try to learn from mistakes when they perform poorly on a test increased from 73% to 90%, and the percentage of students who prefer challenging work to learn new things increased from 34% to 62%. As a result, the programs appear to be increasing students’ interest in STEM and their drive to learn.

Self-Management/Self-Regulation

This construct includes items measuring students’ perceptions of their diligence to complete assignments, work hard, and attend class regularly and on time. During the fall semester, the percentage of students who have confidence in their self-management skills increased from 42% to 53%. This gain included a 15 percentage point increase in the number of students who agreed with the statement, “I finish whatever I begin.” While these gains show that students believe their self-management skills are improving, 29% of students still lack confidence in these skills. As such, there is room for programs to improve in this construct.

Intent to Persist

In STEM for Life, the percentage of students intending to graduate from high school increased from 78% to 100%.

This construct measures whether a student intends to graduate from high school and pursue a degree and/or career in a STEM field. Since each program focuses on applied STEM learning, programs expect that more students will have STEM aspirations as they experience real world applications with the field. Thus far, the programs seem to have been successful at this goal. The percentage of students with STEM aspirations increased from 47% to 57%. Additionally, the percentage of students who intend to graduate high school increased slightly from 94% to 96%. As with the Intrinsic Motivation construct, STEM for Life experienced the greatest gains among programs in this construct. The percentage of students considering a career in STEM increased from 7% to 32%, and the percentage of students intending to graduate from high school increased from 78% to 100%.

Problem Solving

This construct measures whether programs have an inquiry-based learning environment providing higher-order cognitive tasks and real-world applications. Overall, 71% of students believed that their program focused on developing problem solving skills. Tift County Mechatronics earned the highest ratings of any program with 81% of students believing the program focuses on developing these skills. At a more granular level, more than three-quarters of students believed that the classroom environment encouraged students to think and ask questions, and teachers provided feedback on how to improve student work. However, programs seem to be falling short in allowing students the freedom to choose and plan topics for projects because only 50% of students reported this freedom.

Implementation Activities

This construct captures whether the program includes hands-on activities designed to increase exposure to STEM topics. Overall, programs seem to be doing a good job of showing students how what they are learning in the program is related to what they already knew prior to the program. However, only 50% of students reported interacting with STEM professionals, and 55% of students believed they were learning what STEM professionals do. Since a goal of these programs is to increase the number of students interested in STEM careers, the results indicate that programs could increase the time devoted to relating program activities to STEM careers.

Educational Plans

The percentage of students who plan to attend some form of postsecondary option increased from 87% to 91%. At STEM for Life, the percentage of students who plan to pursue postsecondary education increased from 54% to 84%. Across all programs, the percent of students who plan to attend either two- or four-year colleges also decreased. However, this shift is likely a result of more students planning to attend graduate or professional school, which increased from 43% to 62%. In sum, the programs seem to be broadening the educational expectations of students who participate.

Overall Program Ratings

Tift County Mechatronics had the highest program rating with 97% of students rated the program as “Good” or “Excellent.”

The survey also asked students to rate the program on a 1 to 5 scale, where 1 equals Very Poor and 5 equals Excellent. The average score across programs was 4.12, meaning that students generally believed the programs were high quality. Tift County Mechatronics had the highest rating of 4.77, where 97% of students rated the program as “Good” or “Excellent.” Two other programs, Drew Charter School and 21st Century STEM Collaborations, earned a rating above 4.0. Conversely, the Murray STEM Academy earned the lowest rating of 2.70. More than 70% of students rated the program as either average or below average. Relative to the other programs, Murray STEM Academy also experienced the smallest gains in the three constructs and the lowest rating on Problem Solving and Implementation activities. Thus, leaders in this program should use survey findings to improve programming this spring.

Conclusion

Overall, the grantees have improved student perceptions of their intrinsic motivation, self-management skills, intent to persist in STEM, and desire to pursue postsecondary education. Going forward, the programs should increase student interactions with STEM professionals and allow students more freedom to plan and develop projects. However, if the general positive nature of student perceptions continue throughout the grant, these programs will not only increase the number of students who want to enter STEM fields but also equip them to be successful in both school and life.

What’s Next
In late spring 2013, programs will administer surveys again to gauge progress throughout the school year. Future survey administrations will also include grantees from Round 3 of the Innovation Fund as well as programs that focus exclusively on summer programming. In addition to these surveys, each program submitted mid-year program reports to GOSA by January 31st. GOSA will analyze these reports and publish summaries in March 2013.

 

 

[1] For more reading on this topic, see Heckman, James J. (2008). “Schools, Skills and Synapses,” Economic Inquiry, 46(3): 289-324 or Tough, Paul (2012), How Children Succeed.

[2] Cronbach’s alpha measures the internal consistency of items within each construct. The value ranges from 0.00 to 1.00. Higher values indicate greater construct reliability.