Measuring What Matters: Validity and Reliability in Assessment

It's no secret that assessment remains an essential part of the teaching and learning cycle. However, the conversation surrounding building practical assessments, interpreting data, and building positive assessment cultures is ongoing. 

In partnership with Hanover Research, we surveyed educators about the role of assessment today. In 2023, 70% of educators reported evaluating their assessment at least once a year. In 2022, only 38% of educators reported evaluating their assessments yearly. With interest in assessment evaluation rising, a natural question emerges: What characteristics are these educators looking for as they assess their assessments?

Quality and Reliability (still) Matter Most in Assessment

According to our survey, the overwhelming majority of educators want assessments to have these characteristics:

• Reliable (91%)
• Valid (91%)
• High-quality (90%)

Similarly, 87% of educators ranked 'alignment with district scope and sequence' at the top of their list.

Each of these characteristics points to a common theme: trust. Educators want to trust that the assessments they deliver will be worth their time and help every student reach mastery. High-quality assessments give them an accurate picture of where their students are in their learning. 

What is An Example of a Reliable and Valid Assessment?

At first glance, terms like reliable and valid may seem interchangeable. While their definitions are similar, defining what each means is essential. 

Reliability of an assessment: the extent to which it consistently and accurately measures learning. (Are the results from the assessment repeatable?) 

Some critical factors for determining an assessment's reliability:

• The suitability and phrasing of the questions or tasks for the students – Are the questions phrased similarly to how the material was taught? Students shouldn't have to overcome extra hurdles like deciphering an assessment's terminology in addition to showcasing what they know.
• The consistency in test administration, such as the length of time given for the assessment. 
• Students' readiness for the assessment. Have students had adequate time to prepare for the assessment? Also, consider other external factors that impact students' lives at school. A hot afternoon in a classroom with no air conditioning or immediately after physical activity might not be the best time for students to be assessed. 

Validity of an assessment: the extent to which it measures what it was designed to measure. 

There are several different types of assessment validity. Here are some of the most important ones to remember while designing assessments. 

• Face validity - Are the assessment items appropriate?
• Content validity - Does the assessment content cover what you want to assess?
• Criterion-related validity - How well does the test measure what you want it to?
  
  Some examples of invalid assessments would be a math assessment that tests students' science knowledge or a spelling test that prompts student's social studies skills.

Best Practices for Building More Reliable, Valid Assessments

Creating high-quality, reliable, valid assessments is essential for every classroom. However, building these assessments is challenging and time-consuming. In our survey, most educators reported that the current assessment model is disruptive. How can educators create more reliable, valid assessments in the time they dedicate to lesson planning?

For creating reliable, valid assessments, here are some steps that teachers and educators can take:

• Plan, plan, plan 
  • While teachers want to avoid teaching to the test, thinking with the end skills in mind is essential. As teachers plan their unit, it can be helpful to identify the formative, benchmark, and summative assessments they'll deliver along the way and use familiar language throughout instruction and assessment.
• Establish the assessment purpose.
  • Outline the why. How does a formative assessment–such as a bellringer when students walk in daily–prepare them for mastery? What will students be able to do as a result of the end-of-unit summative assessment? 
• Go back to Bloom's Taxonomy 
  • As educators analyze assessments, they can ensure that what they ask students to produce in an assessment aligns with Bloom's taxonomy. Are they given the opportunity to show they remember a concept before they're asked to apply it?
• Take advantage of technology.
  • Assessment management systems can help organize, streamline teacher collaboration, and ensure alignment with state and district standards.
• Create the assessment items (& evaluate their effectiveness)
  • Teachers should have the flexibility to create their assessment content. Building assessment literacy districtwide can help ensure assessment in every classroom is high-quality. Additionally, districtwide assessment content can help scale trustworthy assessments districtwide.

One School's Story: Benchmark-Ready at Fairfield County

For many schools, it takes time to develop a comprehensive assessment strategy. The best strategies continually evolve to student needs and are supported by technology designed with educators in mind. 

“And with Mastery Item Bank, teachers can create high-quality formative assessments. So, the data is valid and reliable, and it helps teachers to inform their instructional decisions again and again. So that they can focus on targeted areas of improvement.” - Amy Coker, Testing Coordinator at Fairfield County in South Carolina.

Don't miss the rest of how Fairfield County became benchmark-ready with Mastery Connect, Item Bank, and Mastery View Predictive Assessments. 

Preparing for The Future of Assessment Today

“Assessment will continue to play an important role in the future to ensure that the decisions we make and the actions we take are sustainable and fair, and can help assess student learning outcomes and the quality of education.” - Administrator Respondent in 2023 State of Assessment.

As we progress, assessments–like many other aspects of education–need to grow in their reliability and personalization to student needs. This ongoing evolution encourages us to continue looking to educators for what they need out of their assessment technology and creating solutions informed by their experience. Dig into more of the latest assessment trends and findings in the State of Assessment.