An Overview of GOSA’s Academic Auditing Program

February 17, 2016

By Dave Greenstein

In 2008, the Governor’s Office of Student Achievement (GOSA) launched its academic auditing program as an extension of its statutory role to report student progress and school completion in Georgia’s public schools.[1]  Through an academic audit, GOSA reviews student assessment data and other school records to maximize the integrity of the data that schools and districts report to the Georgia Department of Education (GaDOE) and ensure that related policies and procedures are followed with fidelity.  Valid data are crucial for improvement planning, education policy development, school funding, and determining the performance status of schools.  This education update provides a history and overview of GOSA’s two areas of academic auditing: State Assessment Administration Auditing (Erasure Analysis) and Data Collections Auditing.

State Assessment Administration Auditing

In 2009, GOSA took a leading role in the erasure analysis of the Criterion-Referenced Competency Tests (CRCT) that flagged public elementary and middle schools in Georgia with high rates of answers being changed from wrong to right.  Wrong-to-right (WTR) answer changes may be due to student action (e.g., getting off track, misbubbling, or simply changing answers) or adult intervention (e.g., hints, changing answer documents, etc.).  It is important to note that the results of the erasure analysis are used as an initial flag to spur further investigation of a number of additional indicators to determine if it appears likely that cheating may have occurred.  The erasure analysis results do not indicate that cheating necessarily occurred, but rather the need for further investigation and review.

Since then, GOSA has conducted an annual erasure analysis of state assessments and has reported the findings each year to the State Board of Education along with recommendations for interventions in flagged schools, which can include further inquiry, monitoring of assessment administration, teacher rotation during testing, and on-site audits.[2]  In 2012, GOSA added erasure analysis of the high school End of Course Test (EOCT), ensuring all Georgia public schools are monitored annually via the erasure analysis processes.

Georgia has seen a significant decrease in the number of elementary and middle schools of concern since 2009.  As a whole, Georgia districts and schools have significantly increased the use of best practices in test administration procedures and security.  The chart below illustrates the percentage of schools, by year, requiring further inquiry for the CRCT and the Georgia Milestones End of Grade (EOG) tests.  The reduction in 2015 is partially due to the transition to online testing.  In 2015, 70% of EOG tests were administered using paper and pencil, and 30% were administered online.  The erasure analysis currently includes only paper and pencil tests.  

         (Source:  February 2016 Presentation to the State Board of Education)

For the 2014-2015 school year, Georgia introduced the new Georgia Milestones tests with the eventual goal of fully transitioning from paper and pencil to online administration.[3]  The transition will occur gradually over the course of five years, with the goal of 100% of Georgia students testing online by 2019.  This transition provides new opportunities for the auditing program.  While wrong-to-right answer changes will still be tracked in the online analysis, the online platform will eliminate some student errors often seen on paper tests, including getting off track, misbubbling, or marking answers in the wrong test section. 

Online testing also allows for new methods of analysis.  In addition to the traditional answer change analysis, GOSA is working with the state’s testing vendor, Data Recognition Corporation, to explore the development of new potential analyses, such as timed response and pattern response.  Timed response analysis tracks how quickly a student answers a question and compares this against the average time it takes all students throughout the entire state to answer the same question.  Pattern response compares a student’s answers against the answers of other students in the class to see if multiple students have the same response patterns of right and wrong answers.  Coupled with a seating chart, pattern response analysis could aid in identifying potential sharing of answers among a group.  

Data Collections Auditing

Beginning in fall 2015, GOSA expanded its auditing program to assist in verifying data that districts report to the GaDOE for accountability and funding purposes. This effort began with a focus on the accuracy of enrollment reporting and will expand in 2016 with an examination of student withdrawal reporting.

As it relates to student enrollment reporting, the GaDOE collects enrollment data from all LEAs through three main collections: October Full-Time Equivalent (FTE), March FTE, and Student Record (collected in June).  FTE enrollment counts measure how many students are enrolled on the count date and is used to determine state education funding allocations. Student Record includes enrollment information for all students enrolled in a school throughout the current year (July 1 to June 30).  Therefore, all students included in either FTE count should also be included in Student Record.  When students appear in an FTE count but do not appear in the Student Record count, the GaDOE data collection system flags the error and notifies the LEA.  LEAs are required to enter a comment explaining a reason for errors that are not able to be corrected in the Student Record application.  Reasons for such errors may include improperly coded classes, dual enrollment of students, and special classifications of students (e.g., special education and remedial education). 

GOSA’s enrollment verification audit seeks to gain additional information and documentation (when required) to explain such discrepancies. More specifically, the audit has the following goals:

• Verify accuracy of data reported by school systems to the GaDOE;
• Ensure school systems follow GaDOE requirements for data training, reporting, and storage;
• Guard against improper data reporting practices and/or potential fraud; and
• Provide additional support to school systems and schools in refining and improving data reporting and storage practices.

Because FTE and Student Record data are utilized to calculate funding for LEAs and schools, the accuracy of FTE and Student Record data are essential to ensure schools receive the correct funding from the state.[4]

Last fall, GOSA conducted this type of audit for the first time and identified nine schools in seven LEAs for further inquiry. The LEAs conducted internal analyses to determine the reason(s) for the reporting errors, as well as to identify and implement the appropriate steps to be taken to address the concerns and reduce the likelihood of future errors.  Once these schools submitted the inquiry response form, two were completely cleared of concern and seven require no further immediate action but will be monitored in school year 2015-2016 to ensure that protocols put into place has been effective.

Withdrawal Code Verification Inquiry

In 2016, GOSA will also begin auditing student withdrawal codes that all Georgia LEAs report to the GaDOE.  Errors in withdrawal code reporting may result in inaccurate calculation of graduation rates, which affect the school’s score on the state’s accountability measure, the College and Career Ready Performance Index (CCRPI).  For example, in high schools, certain withdrawal codes, particularly those used for students who transfer to other schools, remove the student from the data set used to calculate a school’s graduation rate.  As such, it is important to verify, to the extent possible, that students coded as a transfer actually enroll in another school or that the school has other documentation supporting this designation.

The process for this audit is still in development, and the process document will be posted on GOSA’s web site later in 2016.

Summary

Moving forward, GOSA will continue its focus on maximizing the integrity of the data that schools and districts report to the GaDOE and ensuring that related policies and procedures are followed with fidelity.  The expansion to additional online analysis, such as response time and response similarity, as well as data collections audits will strengthen these efforts and ensure that reliable data is used for policy decisions, funding allotments, and accountability.

[1] GOSA has the authority to conduct audits, in cooperation with GaDOE, pursuant to its duties as set forth in O.C.G.A. § 20-14-26 (a)(2). GOSA conducts all audits and investigations in accordance with guidelines established by the Office of the U.S. Comptroller General and the Office of the U.S. Inspector General.

[2] To learn more details about the Erasure Analysis, the process document is available here.

[3] For more information on Georgia Milestones, see GOSA’s education update available here.  

[4] To learn more details about the Enrollment Verification Audit, the process document is available here.