Atlanta Public Schools Sets New Security Procedures to Improve CRCT Integrity

March 4, 2013

By Stephanie Sidney

This education update highlights the work that Atlanta Public Schools (APS) has done to improve its auditing and security procedures for administering the Criterion-Referenced Competency Test (CRCT).  These practices have improved the integrity of assessment data and may serve as a model for other school districts around the state.

What is the CRCT?
The CRCT is a standardized assessment given to elementary and middle school students each spring in Georgia.  The test is designed to measure the acquisition of specific content knowledge and skills on the Georgia curriculum standards. Additionally, the CRCT results serve as the criteria by which some student promotion and retention decisions are made.  Other important decisions for individual students and for schools are based on CRCT data, making it critical that reported scores are an accurate representation of students' knowledge and skills.

Under the leadership of Superintendent Erroll B. Davis, Jr., APS has implemented measures to increase the security of its assessment data.  According to Gregory Cizek, President of the National Council on Measurement in Education, school districts must establish policies and procedures to ensure data integrity within each school.  APS has done that by putting in place a security system that includes the development a CRCT Test Administration Audit Plan as well as creating more stringent testing procedures for the CRCT and CRCT-modified for grades 3-8.  According to Dr. Rubye Sullivan, Director of Research and Evaluation for School Improvement at APS, “There is an important relationship between audit findings and the continuous improvement of the security process related to assessment.”  Dr. Sullivan also adds that APS audit findings are fed back into the Testing and Assessment Team as a part of the district’s continuous improvement process.

“We have developed a comprehensive ethics program which mandates that every employee participate in annual ethics training as a condition of employment. We have strengthened test security measures, including locked safe rooms, tighter chains-of-custody, and clearer test protocols, to prevent improprieties and tampering. Finally, we set trigger points that can result in automatic investigations of schools whose test scores show larger-than-normal, year-over-year changes. We are determined to ensure that our assessments serve children, not adults.”

 --Erroll B. Davis, Jr, Superintendent,  Atlanta Public Schools

Actions Taken Since 2010

APS has taken steps to ensure that testing materials are secure before, during, and after a test is administered to students.  Video surveillance constantly monitors the inventory of testing materials. APS has also created a more stringent process for distribution of materials, testing administration, and collection and shipment of completed tests.  In addition, APS has taken the following actions since 2010:

  • Dismissal of employees identified in the 2009 Special Investigator’s probe of classrooms having an unusually high number of wrong-to-right erasures when compared to the state average
  • Security monitoring at each building, led by school principals, during test administration
  • Creation of 2011 Trigger Report
    • APS identified schools with mean scale score gains that were more than three standard deviations from the state mean scale score change.
    • Members of the APS research team visited the identified schools and conducted interviews to determine potential causes of the gains.
    • The research team matched reported class-level erasure data with the gains data to identify classrooms with both unusual gains and unusual erasures.
    • The resulting Trigger Report was then delivered to APS Office of Internal Compliance (OIC), and became a part of APS’s erasure investigation.
APS’ New Test Security Procedure

To protect the integrity of its testing system and to ensure that no breach of security occurs during the dissemination of tests, APS has established a clear and concise chain of custody plan requiring that only authorized officials can account for and handle testing materials.  The handling of testing materials, pre- and post-assessment, now largely remains in the hands of a school’s principal and the Lead School Testing Coordinator (LSTC) who must sign legally binding verification forms that they are responsible for what happens to testing material while it is on school property.  Receipt and shipping of testing materials is also confirmed by binding signature of the Principal and LSTC.  Upon receipt of testing materials, the Principal and LSCT must sign and fax an acknowledgement form to the APS District Warehouse (Brewer Test Center) by close of business on day of delivery.  Materials must then be inventoried and certified by signature of the principal and LSTC within 24 hours.  Logistics Services drivers may only release testing materials to the custody of the Principal and LSTC.  If these individuals are not available, testing materials are returned to the Brewer Test Center, and an alternate day is scheduled for delivery at a time when the Principal and LSTC are available. 

The most significant difference from past procedures is that immediately following each testing session, answer documents are sealed in an envelope by a certified system administrator, assigned a serial number, and entered into a verification log. The Principal and LSTC are responsible for return of all completed test materials to the Brewer Test Center.

APS’ Office of Internal Compliance

APS’ Office of Internal Compliance (OIC) conducts a self-imposed audit of its testing practices and policies to affirm the integrity of assessment results. When any testing irregularities occur, the OIC reviews these irregularities to determine which cases require further review and possible investigation. This review process happens continually during all test administration windows.

Summary of New Test Security Measures

  • Test booklets are stored pre- and post-testing at a secure location and monitored to prevent overnight access. Shipment schedules are designed to minimize the time that testing materials remain on school property.

  • Access to secured room containing testing materials is restricted to the Principal and LSTC.

  • Post-testing materials are scanned and organized by testing location and kept in secure location.

  • A legal signature is required from school personnel when assessments are distributed and collected.

Conclusion

These test security measures help restore a standard of ethics and integrity to APS and confidence in test scores for the benefit of its students.  Such initiatives could also serve as a best practice for school districts throughout the state, as they improve security and integrity not only for the CRCT but for all K-12 student assessments.  These measures also ensure that accurate data are submitted to state and federal program officials as a representation of student test scores when allocating grant funds.

GOSA's Role in Academic Auditing
GOSA has provided an academic auditing program for local school districts since 2008.  Through this auditing program, GOSA reviews student assessment data and other school records reported to the Georgia Department of Education to confirm accuracy, examine assessment processes, and analyze the effectiveness of local school initiatives in improving achievement.