We Can Measure What Matters — If We Stop Measuring What’s Easy

Rethinking Assessment in the AI Era

There’s a line I keep coming back to: we can measure what matters—if we stop measuring what’s easy.

For decades, we’ve used tests as our shorthand for progress. They gave us numbers, charts, and accountability. They made data visible. But they also made learning small—flattened into bubbles and benchmarks. The irony is that we’ve built a system that rewards efficiency, not understanding. It measures compliance and timing, not curiosity and persistence. The result? We end up teaching to what’s easy to count, not what’s worth knowing.  And now, standing at the edge of the AI era, we have both the technology and the moral responsibility to do better. 

We can finally stop rewarding the bubbles and clock—and start assessing what truly matters: reasoning, communication, creativity, growth. The constraints that once made standardized tests the “least bad” option are loosening. It’s time to take the leap.

The Problem with “Easy to Measure”

High-stakes tests were never meant to be the whole picture. They were designed for

comparability, not complexity. But over time, they became the scoreboard—and once that happens, the game changes.

Students learn to perform, not to explore. Teachers plan for what’s tested, not what’s lasting. Leaders spend more time defending scores than describing growth.

And now, with AI able to write an essay or solve a problem faster than most students, we’re left asking: What exactly are we measuring?

If we’re honest, much of what our tests capture—recall, speed, format—is no longer a proxy for real understanding.  High stakes standardized tests are increasingly misaligned with what we value.  They are a relic of a pre-AI world.

In the AI world AI makes it possible to move beyond one-size-fits-all assessment and toward personalized, authentic measures of comprehension. Instead of static snapshots, we can gather dynamic evidence of growth: revision patterns, self-reflection, collaboration, and the application of skills across contexts.

AI doesn’t just automate grading—it gives us mirrors into the learning process. We can see thinking unfold, trace how ideas evolve, and make the invisible parts of learning visible. It’s not replacing judgment; it’s revealing it.

When we use AI this way, assessment becomes an act of knowing—not sorting.

A Better Path: Competency and Connection

Competency-based assessment (CBE) isn’t new, but the conditions to make it work at scale finally are. At its heart, CBE asks a simple question: Can a learner demonstrate what they know and can do?  It replaces seat time with mastery, pacing with progress, and test dates with evidence of learning.  AI finally allows us to scale that vision with fidelity. In classrooms, adaptive tools can pinpoint misconceptions, track growth over time, and provide educators with actionable insights rather than arbitrary scores.

Imagine a student’s digital portfolio traveling with them—a “competency passport” that captures the full arc of learning. A middle school research project, a high school capstone, a college-level internship reflection—all connected as evidence of mastery, creativity, and resilience.

When learning is documented this way, assessment stops being a gatekeeper and becomes a gateway. It tells a story of growth that belongs to the learner, not the system.

In a classroom, that might look like:

• A student building a model instead of filling in a worksheet.

• A team defending their design to a panel instead of turning in an essay.

• A learner revising an AI draft into something thoughtful and personal—showing discernment, not just output.

  
  

In competency models learning outcomes (competencies) are clearly defined in observable, performance-based terms. Students advance when they demonstrate mastery, not when they accumulate hours or “cover the curriculum.” There are multiple pathways, multiple demonstrations of competence are baked in (projects, portfolios, simulations, peer review). And feedback is ongoing and formative; assessment is integrated, not tacked on as a final exam.

These are not hypotheticals. They’re happening in K–12 classrooms and increasingly in higher education programs built around micro credentials, portfolios, and demonstrations of skill. 

This rethinking isn’t limited to K–12. Higher education has its own reckoning to face. As more colleges and universities embrace micro-credentials, digital badges, and skills-based transcripts, AI can help validate and verify competencies in authentic, low-stakes ways. Rather than measuring “seat time,” we can recognize learning wherever it happens—on campus, on the job, or online.

The old boundaries between “school” and “college” are softening as both systems begin to value what students can do, not just what they’ve completed.

The shift from course completion to demonstrated competence builds a bridge between education and the workforce. It signals that learning is not a series of checkboxes, but a continuous progression of ability and understanding.

This shift from coverage to demonstration changes everything.

Why AI Changes Everything

AI doesn’t make assessment irrelevant—it makes traditional assessment obsolete.  It gives us the tools to observe, personalize, and validate learning in ways we never could before. AI is not just a disruptor—it’s an enabler. Rather than viewing AI as a threat to integrity, we should see it as catalytic for reimagining assessment.

• Personalized feedback: Smart scaffolding & adaptive feedback loops.  AI can analyze patterns in a student’s work, giving teachers insight into misconceptions before they harden.  It can monitor student work in real time (drafts, decisions, revisions), diagnose misconceptions, and provide micro-nudges. In a CBE environment, that means fewer surprises and more support along a student’s mastery journey. (schoolai.com)

• Authentic performance tasks: It can generate scenarios, simulations, and case studies that let students apply skills in context.

• Equity and insight: AI analytics can help us see who’s being served and who’s being left behind.  AI can help surface where learners struggle, which competencies are bottlenecks, and whether assessment tasks inadvertently advantage one group over another. AI-enabled dashboards can help leaders intervene early and equitably. (U.S. Department of Education)

• Bridging worlds: It can recognize evidence of learning wherever it occurs—school, internship, community project, or work site. Generative models can help teachers and assessment designers prototype complex, multimodal tasks—case simulations, decision trees, scenario dilemmas, adaptive branching prompts—that would be prohibitively labor-intensive at scale. (Digital Education Council)

• Recognition of prior and informal learning AI can analyze portfolios, artifacts, workplace outputs, and even unstructured evidence of learning to validate competencies. This helps bridge K–12, higher ed, and career pathways.

  
  

AI doesn’t replace judgment; it amplifies visibility. It lets teachers spend less time scoring and more time understanding. It allows institutions to connect the dots—from a student’s first coding project in middle school to their senior capstone in college. AI lowers the transactional cost of high-fidelity, individualized assessment. It turns the dream of “authentic, competency-based assessment at scale” from fantasy into something within reach.

The New Assessment Landscape

If we stop chasing the test and start building toward demonstration, the landscape begins to change. Imagine a system that values:

• Portfolios that follow a learner across years and systems.

• Oral defenses that reveal reasoning and communication.

• Performance tasks that simulate the complexity of real work.

• Micro credentials that stack into degrees and careers.

• Self- and peer-assessment that cultivate reflection, not compliance.

  
  

AI helps us design, scale, and calibrate these assessments—but the goal remains profoundly human: to understand how someone thinks, creates, and applies knowledge.

Assessment Strategies That Work in the AI Era

Here are concrete alternatives to traditional high-stakes tests—purposed, resilient, and aligned with deeper learning goals.

Strategy	What It Measures	Why It Works (AI-era strength)	Challenges & Mitigations
Performance tasks / project-based assessments	Application, integration, synthesis, iteration	Harder for AI to “do” everything—students must make decisions and document trade-offs	Rubric clarity is paramount; scaffolding needed for novices
Oral defenses, viva voce, Socratic Q&A	Depth of understanding, reasoning under pressure	AI-generated text can be probed or questioned; shows student thinking	Requires faculty time and training
Portfolios + evidence dossiers	Growth, process, reflexivity, metacognition	AI can augment review (e.g. flag revisions, anomalies), but judgment stays human	Requires rubric development, assessor calibration
Simulations, games, scenario judgments	Transfer, domain thinking, decision-making under uncertainty	AI can power adaptive branching in simulations	Complex to design, but modular scaffolds help
Badges / microcredentials / capstones	Discrete competencies or transdisciplinary skills	Modular and stackable—they allow learners to “assemble” credentials	Alignment, articulation, recognition across institutions
Peer review + self-assessment	Metacognition, evaluation, argumentation	AI can moderate, suggest prompts, scaffold meta-feedback	Need strong norms, training, structured rubrics

Additionally:

• “Edit to Competency” assignments (e.g. students revise an AI‒generated draft toward mastery) put students in the driver’s seat of critique, not ghostwriting. (LinkedIn)

• Embedded assessments or stealth assessment (within games, simulations, daily activities) capture competence organically without high-stakes pressure. (eLearning Industry)

• Adaptive, modular “mini-checks” (competency quizzes that adjust to student responses) let learners “test as they learn” rather than one big summative push.

• Recognition of Prior Learning (RPL) / prior-learning assessment across K–12, higher ed, and adult learners: use artifact-based validation of what students already know. (Wikipedia)

K–12 and Higher Ed: The Continuum of Competency

For too long, we’ve treated K–12 and higher education as two disconnected systems, each with its own rules of readiness. But imagine if every transcript—from high school to college—told a story of competence instead of a list of courses.

If K–12 systems adopt competency-based transcripts (versus GPA/test-score-based), colleges and universities can read student readiness in richer ways (not just SAT or AP).Higher ed institutions can offer stackable credentials or micro credentials that map to workplace competencies, enabling lifelong learners to bridge in and out. Recognition of prior learning (RPL) can validate real-world experience (internships, work, civic projects) as satisfying college-level competencies—reducing redundancy.AI systems can help flag skill gaps in the transition from K–12 to higher ed (e.g. writing, reasoning, digital literacy) and generate targeted bridge modules. Assessment data can flow across institutions (with privacy safeguards) to support a learner’s “competency passport”—allowing movement across schools while preserving demonstrated mastery. Such ecosystems reshape not only assessment but pathway design, transfer policy, and curriculum alignment.

•  A high schooler’s project in environmental science could satisfy a university’s entry-level research competency.

• A college course could recognize prior learning gained through community service or work experience.

• AI could help validate the throughline—tracking growth, highlighting mastery, and suggesting next-step competencies.

In higher ed, many faculty are already rethinking assessments in the GenAI era. A recent study proposes the “Against, Avoid, Adopt, Explore” framework for redesigning assignments—from prohibitive to generative uses of technology. (MDPI) Another paper on AI-resistant assessments highlights how higher-ed instructors are pushing tasks toward critical thinking, creativity, and collaboration—less dependent on recall. (Frontiers)

The Leadership Challenge

Systemic change doesn’t happen because technology makes it possible; it happens because leaders make it purposeful.

This work requires courage.  Rethinking assessment means rethinking accountability, policy, and sometimes even funding. It means saying out loud that the data we’ve relied on may be incomplete—and that we can do better for students and educators alike.

Leaders in districts and state offices have a crucial role here. By piloting competency-based models, investing in AI literacy, and advocating for transparency and equity, they can help reshape how learning is recognized and rewarded.

AI is not the threat to authentic learning—it’s the opportunity to restore it. But only if we design with intention.

So here’s the charge for curriculum leaders, policymakers, and assessment coordinators:

1. Redefine accountability—move from point-in-time proficiency to progress over time. Build accountability around progression, competencies attained, growth, and portfolio quality—not just proficiency thresholds on standardized tests.  Yeah, I get that this is not easy, but should it be?

2. Pilot authentically—fund districts and universities to co-design AI-supported, competency-based assessments.  Fund district–university partnerships to prototype AI-infused, competency-based assessments in real classrooms. Share rubrics, case studies, and lessons learned.

3. Build capacity—train educators to use AI as an assessment ally, not an adversary.  Teachers, assessment coordinators, and higher ed faculty need support in rubric design, AI-aware assessment frameworks (e.g. AIAS), bias mitigation, and calibration. (arXiv)

4. Invest in portfolios and interoperability—Invest in systems that can capture richer student artifacts, track modular credentials and support AI analytics and protect equity/privacy. 

5. Protect equity and privacy—use data to reveal barriers, not reinforce them.

6. Cross-sector alignment -Engage higher ed, workforce boards, employers, and K–12 systems collaboratively to declare common competencies and validation practices.

Closing Thought

We have reached a moment where our tools have outgrown our traditions. We have a rare opening.  We can break free from legacy assessments, focused on the recall and narrow coverage. If we design courageously, we can use AI not to police students, but to empower them—to nudge, support, personalize, and elevate.

To curriculum leaders, policy makers, and assessment innovators: let’s stop asking, “How can we test faster?” and start asking, “How can we assess deeper?

 AI gives us precision; competency-based learning gives us purpose. Together, they let us see students as more than test scores—they become thinkers, creators, problem-solvers, and citizens in motion.

Assessment should not be a moment of judgment. It should be a record of becoming.

If we get this right, the next generation of assessment won’t be defined by algorithms or score reports—it will be defined by trust, evidence, and human potential seen clearly at last.

The question isn’t can we measure what matters.  It’s will we choose to.

And that’s the part that requires leadership, not algorithms.

Winnie O'Leary October 2025

Winnie O’Leary has spent over 30 years in education as a classroom teacher, school board member, a family advocate, special education teacher, curriculum writer, National Intervention Specialist and Educational Initiative Manager. Winnie earned a BA from Mary Washington College and an MEd from the University of New Orleans. Her experiences have allowed her to work with districts all over the country where she finds something new and exciting every day.