Personalizing the learning path
Every learner was getting the same next question.
Adaptive Practice Engine · Edfora · EdTech · Senior Product Manager · Jul 2023 – Jul 2026
A fixed practice sequence was too hard for some learners and too easy for others. Three ways to fix difficulty fit; the one chosen was a 3PL IRT engine.
The adaptive practice loop · as the PRD specifies it
3PL Item Response Theory · statistical model · not an LLM
Inputs · Raw student data · 3PL question parameters · Content IDs
01 Ability
θ per concept, estimated from historical performance
Edge case · Initial ability estimation — method not recorded
02 Score
P(θ) for each candidate question, from difficulty (b), discrimination (a) and guessing (c)
Edge case · Questions missing 3PL parameters
03 Select
A question near current ability; when candidates are comparable, the more discriminating one first
Edge case · Several questions with the same median P(θ)
04 Respond
The learner answers
- Correct → harder
- Incorrect → easier
05 Update
θ updated after every response; back to Score until the practice ends
Assignment completionObserved
18%static path45%after the adaptive system
Before / after · 2-year academic-cycle dataset
- Observed across the academic cycles before and after the adaptive system; not attributed to the engine alone.
- How completion was counted isn't recorded.
- Learning mastery was not captured.
How it was measured: Assignment completion
Before/after · 2-year academic-cycle dataset
Definition not recorded: how completion was counted isn't captured in the analysis.
The static learning path before, the adaptive system after; no holdout. Concurrent product changes aren't on record. Completion was the measured outcome; learning mastery was not captured in this analysis.
Scope: Core Practice & Learning Experience. Context: Edfora's products reach 100K+ learners overall; that figure is not specific to this engine.
01Context
Core practice for an EdTech product.
I owned the roadmap and prioritization, the problem analysis, the PRD and adaptive product logic, and post-launch tracking. I worked with engineering on the build, design on the experience, and academic leads on the learning requirements.
The roadmap was RICE-based, fed by regular interviews and usability tests with students and faculty.
- Role
- Senior PM, Core Practice & Learning Experience
- Team
- 1 PM (me), 1 APM, 1 designer, 5–7 engineers, 2–3 academic leads
- Period
- Jul 2023 – Jul 2026
- Edfora context
- 100K+ learners across Edfora's products overall; not this engine's reach
02Objective
Find each learner's gaps, and keep them practising.
- Product
- Identify each learner's learning gaps, and give them a personalized difficulty trajectory
- Business
- More engagement and time spent in practice, fewer remedial drop-offs and, in the end, support for renewal and subscription conversion
- Not claimed
- No engagement, drop-off, renewal or conversion result is on record for this engine
03Problem
One sequence fails in two directions.
A fixed sequence couldn't adapt to different learner ability levels: some questions were too hard, others too easy.
Too hard
Frustration, then drop-off
Right fit
The learner keeps going
Too easy
Little to learn, then drift
A fixed sequence puts every learner at the same difficulty, so some land on each side of the fit.
04Signal
Completion stood at 18% on the static path.
Both failure modes look identical in completion data: an assignment that doesn't get finished. So completion alone couldn't say what to build; it had to be read against how each learner was performing.
05Hypothesis
Product reasoningMatch the question to the learner, not the learner to the sequence.
- If
- each learner gets questions matched to their current ability instead of a fixed sequence
- Then
- fewer hit a wall or coast, and more finish the assignment
- Measured by
- Assignment completion (primary), with practice drop-off as the second signal
06Options
Product reasoningThree ways to fix difficulty fit.
- Option A
Let learners choose their difficulty
Works becauseSimple to build, and gives learners control.
Falls short becauseThe learners who most need an easier path are the least able to judge it.
- Option B
Move learners between bands with rules
Works becauseEasy to build and to explain to teachers.
Falls short becauseTreats every question as equally informative: a lucky guess counts as mastery.
- Option C
Estimate ability and match calibrated questions
Chosen
Works becauseWeighs each answer by how much it reveals, and discounts guesses.
CostsNeeds calibrated question parameters; harder to explain than a sequence.
07Decision
A 3PL Item Response Theory engine.
A statistical model, not an LLM: it estimates each learner's ability and selects question difficulty to match.
- Where we were
- 18% completion on one fixed sequence for every learner
- Where we aimed
- Each learner's next question matched to their current ability
- Chose
- Ability estimation with calibrated questions (3PL IRT)
- Didn't choose
- Learner-chosen difficulty; rule-based difficulty bands
- Constraint
- Every question needs calibrated parameters
- Risk
- Harder to explain than a sequence; a new learner's first questions are the least certain
- Success measure
- Assignment completion, with practice drop-off as the second signal
08System
Learner ability on one side, question parameters on the other.
The PRD specifies the loop end to end: initialization, ability calculation, concept selection, question assignment, response evaluation, ability update and difficulty adjustment, until the practice ends. Its inputs are raw student data, each question's 3PL parameters and content IDs.
09Cold start
A new learner starts with no estimate.
The PRD names initial ability estimation as an edge case. With no history, the first questions are the least informed and the estimate sharpens as answers come in, which is where a fixed sequence and an adaptive one behave most alike. The method the PRD specifies isn't in the evidence available here, so it isn't described.
10Rollout
Measured as a before/after, not a controlled test.
Assignment completion was compared across a 2-year academic-cycle dataset: the static learning path before, the adaptive system after.
11Result
Completion: 18% → 45%.
18% on the static path, 45% after the adaptive system, across a 2-year academic-cycle dataset. The figure at the top of this page carries the full evidence: an observed change, not attributed to the engine alone.
12Attribution
What this number can and can't prove.
13Learning
When completion drops, check the fit before adding content.
Personalize only where it clearly improves the job; everywhere else, a stable default wins. Ability-based matching is harder to explain, depends on well-calibrated questions, and a new learner's first questions carry the most uncertainty.
14Differently
Product reasoningWhat I'd do differently: calibrate and hold out earlier.
I would establish dynamic difficulty calibration and an explicit holdout strategy earlier, so the adaptive system could be evaluated not only on completion but also on learning effectiveness and incremental impact. This is a product reflection, not a measured finding.
15Next
Product reasoningWhat I would do next
- 01
Hold out a share of learners on the static path, to isolate the engine's effect from everything else that changed.
- 02
Read completion by ability band, starting with new learners, whose first questions carry the most uncertainty.