Innovation Fund Applied Learning Grants Showing Early Success

February 5, 2013

By Sam Rauschenberg

In fall 2012, the Governor’s Office of Student Achievement (GOSA) contracted with The Findings Group to develop the Applied Learning Student Questionnaire (ALSQ) for the Race to the Top Innovation Fund Round 1 and 2 applied learning grantees. The ALSQ measures changes in student intrinsic motivation, self-management, and intent to persist in Science, Technology, Engineering, and Mathematics (STEM) studies during the fall semester. The survey also asks students about problem solving and implementation activities in each program. This education update highlights findings from the December 2012 survey administration, and the full report is attached at the bottom of this page.

ALSQ Survey

The ALSQ included 36 items split into five constructs: Intrinsic Motivation, Self-Management/Self-Regulation, Intent to Persist, Problem Solving, and Implementation Activities. Each construct included at least five items. Students responded to items on a five-point Likert Scale where 1 equals “Strongly Disagree,” 2 equals “Agree,” 3 equals “Neutral,” 4 equals “Agree,” and 5 equals “Strongly Agree.”

In December 2012, six grantees administered the survey to gauge mid-year progress of their students. The following grantees participated:

• 21st Century STEM Collaborations: Applications of the Direct to Discovery Model : A collaboration between Barrow County Schools and Georgia Institute of Technology to integrate Direct-to-Discovery, a program using technology to link world-class scientists with K-12 classrooms, into the requirements of the Georgia Performance Standards.
• Drew Charter School Partners of Innovation : A partnership between Georgia State University, Georgia Tech, and Drew Charter School to create one of the state’s first STEAM (Science, Technology, Engineering, Arts and Mathematics) schools.
• Murray County STEM Academy :  A partnership between Murray County Schools, Georgia Northwestern Technical College, the Chatsworth-Murray County Chamber of Commerce and others, to open a program focused on remediating 8th grade students and developing their interest in STEM careers.
• STEM for Life Program : A partnership between Carroll County Schools and Southwire Company to expand and replicate the existing 12 for Life Program, which supplements classroom learning with real-world experience in advanced manufacturing.
• The STEM Targeted Education Program (STEP) Academy : A collaboration between Gwinnett County Public Schools, Gwinnett Technical College, and the Gwinnett Chamber of Commerce to create a Biotechnology Research and Development career pathway program with accelerated coursework and mentoring for at-risk, overage eighth grade students.
• Tift County Mechatronics Partnership :  A partnership between Tift County Schools, Moultrie Technical College, ConAgra Foods, Heatcraft Manufacturing and others to develop a career pathway focused on Mechatronics, an interdisciplinary field of study involving control, electrical, mechanical, and computer engineering systems that equips students to work in a variety of industrial, manufacturing, and health sciences settings. 

Why evaluate non-cognitive skills?

In addition to increasing innovative, STEM-related applied learning opportunities for students, a goal of the Innovation Fund’s Applied Learning priority is to improve student self-management. This goal recognizes the growing body of research—led by University of Chicago economist James Heckman and popularized in the recent Paul Tough book, How Children Succeed—that stresses the importance of developing students’ non-cognitive skills as well as students’ cognitive skills for success in school and life.[1] These non-cognitive skills include curiosity, motivation, the ability to focus and follow through on tasks, the ability to learn from failure, and self-regulation. The ALSQ gauges whether students believe they have improved in these areas over the course of the program.

Results

In total, 860 students across the six programs completed the survey with a 95% response rate. Each construct was internally consistent with a Cronbach’s alpha greater than 0.70.[2] Overall, students in all six programs had statistically significant gains in perceived intrinsic motivation, self-management/self-regulation, and intent to persist in STEM.

Intrinsic Motivation

This construct includes questions about whether students prefer challenging work, learn from mistakes, and are interested in learning about STEM through the program. As a whole, students’ perceptions of their skills grew the most in this construct relative to other constructs. More specifically, the percentage of students who were intrinsically motivated to study STEM increased from 55% to 76%. Most notably, that percentage increased from 41% to 79% for students in STEM for Life, and from 49% to 78% for students in STEP Academy. Drilling down in all programs, the percentage of students who try to learn from mistakes when they perform poorly on a test increased from 73% to 90%, and the percentage of students who prefer challenging work to learn new things increased from 34% to 62%. As a result, the programs appear to be increasing students’ interest in STEM and their drive to learn.

Self-Management/Self-Regulation

Intent to Persist

This construct measures whether a student intends to graduate from high school and pursue a degree and/or career in a STEM field. Since each program focuses on applied STEM learning, programs expect that more students will have STEM aspirations as they experience real world applications with the field. Thus far, the programs seem to have been successful at this goal. The percentage of students with STEM aspirations increased from 47% to 57%. Additionally, the percentage of students who intend to graduate high school increased slightly from 94% to 96%. As with the Intrinsic Motivation construct, STEM for Life experienced the greatest gains among programs in this construct. The percentage of students considering a career in STEM increased from 7% to 32%, and the percentage of students intending to graduate from high school increased from 78% to 100%.

Problem Solving

This construct measures whether programs have an inquiry-based learning environment providing higher-order cognitive tasks and real-world applications. Overall, 71% of students believed that their program focused on developing problem solving skills. Tift County Mechatronics earned the highest ratings of any program with 81% of students believing the program focuses on developing these skills. At a more granular level, more than three-quarters of students believed that the classroom environment encouraged students to think and ask questions, and teachers provided feedback on how to improve student work. However, programs seem to be falling short in allowing students the freedom to choose and plan topics for projects because only 50% of students reported this freedom.

Implementation Activities

This construct captures whether the program includes hands-on activities designed to increase exposure to STEM topics. Overall, programs seem to be doing a good job of showing students how what they are learning in the program is related to what they already knew prior to the program. However, only 50% of students reported interacting with STEM professionals, and 55% of students believed they were learning what STEM professionals do. Since a goal of these programs is to increase the number of students interested in STEM careers, the results indicate that programs could increase the time devoted to relating program activities to STEM careers.

Educational Plans

The percentage of students who plan to attend some form of postsecondary option increased from 87% to 91%. At STEM for Life, the percentage of students who plan to pursue postsecondary education increased from 54% to 84%. Across all programs, the percent of students who plan to attend either two- or four-year colleges also decreased. However, this shift is likely a result of more students planning to attend graduate or professional school, which increased from 43% to 62%. In sum, the programs seem to be broadening the educational expectations of students who participate.

Overall Program Ratings

The survey also asked students to rate the program on a 1 to 5 scale, where 1 equals Very Poor and 5 equals Excellent. The average score across programs was 4.12, meaning that students generally believed the programs were high quality. Tift County Mechatronics had the highest rating of 4.77, where 97% of students rated the program as “Good” or “Excellent.” Two other programs, Drew Charter School and 21st Century STEM Collaborations, earned a rating above 4.0. Conversely, the Murray STEM Academy earned the lowest rating of 2.70. More than 70% of students rated the program as either average or below average. Relative to the other programs, Murray STEM Academy also experienced the smallest gains in the three constructs and the lowest rating on Problem Solving and Implementation activities. Thus, leaders in this program should use survey findings to improve programming this spring.

Conclusion

Overall, the grantees have improved student perceptions of their intrinsic motivation, self-management skills, intent to persist in STEM, and desire to pursue postsecondary education. Going forward, the programs should increase student interactions with STEM professionals and allow students more freedom to plan and develop projects. However, if the general positive nature of student perceptions continue throughout the grant, these programs will not only increase the number of students who want to enter STEM fields but also equip them to be successful in both school and life.

[1] For more reading on this topic, see Heckman, James J. (2008). “Schools, Skills and Synapses,” Economic Inquiry, 46(3): 289-324 or Tough, Paul (2012), How Children Succeed.

[2] Cronbach’s alpha measures the internal consistency of items within each construct. The value ranges from 0.00 to 1.00. Higher values indicate greater construct reliability.