Skip to content
Nitesh Tiwari
Back to Work

Personalizing the learning path

Every learner was getting the same next question.

Adaptive Practice Engine · Edfora · EdTech · Senior Product Manager · Jul 2023 – Jul 2026

A fixed practice sequence was too hard for some learners and too easy for others. Three ways to fix difficulty fit; the one chosen was a 3PL IRT engine.

The adaptive practice loop · as the PRD specifies it

3PL Item Response Theory · statistical model · not an LLM

Inputs · Raw student data · 3PL question parameters · Content IDs

  1. 01  Ability

    θ per concept, estimated from historical performance

    Edge case · Initial ability estimation — method not recorded

  2. 02  Score

    P(θ) for each candidate question, from difficulty (b), discrimination (a) and guessing (c)

    Edge case · Questions missing 3PL parameters

  3. 03  Select

    A question near current ability; when candidates are comparable, the more discriminating one first

    Edge case · Several questions with the same median P(θ)

  4. 04  Respond

    The learner answers

    • Correct → harder
    • Incorrect → easier
  5. 05  Update

    θ updated after every response; back to Score until the practice ends

The loop repeats from Score after every answer. How the edge cases were handled isn’t in the evidence available, so it isn’t described.

Assignment completionObserved

18%static path45%after the adaptive system

Before / after · 2-year academic-cycle dataset

Assignment completion rose from 18% to 45% across the academic cycles before and after the adaptive system launched; the change is observed, not attributed to the engine alone.
  • Observed across the academic cycles before and after the adaptive system; not attributed to the engine alone.
  • How completion was counted isn't recorded.
  • Learning mastery was not captured.
How it was measured: Assignment completion

Before/after · 2-year academic-cycle dataset

Definition not recorded: how completion was counted isn't captured in the analysis.

The static learning path before, the adaptive system after; no holdout. Concurrent product changes aren't on record. Completion was the measured outcome; learning mastery was not captured in this analysis.

Scope: Core Practice & Learning Experience. Context: Edfora's products reach 100K+ learners overall; that figure is not specific to this engine.

01Context

Core practice for an EdTech product.

I owned the roadmap and prioritization, the problem analysis, the PRD and adaptive product logic, and post-launch tracking. I worked with engineering on the build, design on the experience, and academic leads on the learning requirements.

The roadmap was RICE-based, fed by regular interviews and usability tests with students and faculty.

Role
Senior PM, Core Practice & Learning Experience
Team
1 PM (me), 1 APM, 1 designer, 5–7 engineers, 2–3 academic leads
Period
Jul 2023 – Jul 2026
Edfora context
100K+ learners across Edfora's products overall; not this engine's reach

02Objective

Find each learner's gaps, and keep them practising.

Product
Identify each learner's learning gaps, and give them a personalized difficulty trajectory
Business
More engagement and time spent in practice, fewer remedial drop-offs and, in the end, support for renewal and subscription conversion
Not claimed
No engagement, drop-off, renewal or conversion result is on record for this engine

03Problem

One sequence fails in two directions.

A fixed sequence couldn't adapt to different learner ability levels: some questions were too hard, others too easy.

Task difficulty vs. learner readinessIllustrative model
  1. Too hard

    Frustration, then drop-off

  2. Right fit

    The learner keeps going

  3. Too easy

    Little to learn, then drift

A fixed sequence puts every learner at the same difficulty, so some land on each side of the fit.

04Signal

Completion stood at 18% on the static path.

Both failure modes look identical in completion data: an assignment that doesn't get finished. So completion alone couldn't say what to build; it had to be read against how each learner was performing.

05Hypothesis

Product reasoning

Match the question to the learner, not the learner to the sequence.

If
each learner gets questions matched to their current ability instead of a fixed sequence
Then
fewer hit a wall or coast, and more finish the assignment
Measured by
Assignment completion (primary), with practice drop-off as the second signal

06Options

Product reasoning

Three ways to fix difficulty fit.

  1. Option A

    Let learners choose their difficulty

    Works because

    Simple to build, and gives learners control.

    Falls short because

    The learners who most need an easier path are the least able to judge it.

  2. Option B

    Move learners between bands with rules

    Works because

    Easy to build and to explain to teachers.

    Falls short because

    Treats every question as equally informative: a lucky guess counts as mastery.

  3. Option C

    Estimate ability and match calibrated questions

    Chosen

    Works because

    Weighs each answer by how much it reveals, and discounts guesses.

    Costs

    Needs calibrated question parameters; harder to explain than a sequence.

07Decision

A 3PL Item Response Theory engine.

A statistical model, not an LLM: it estimates each learner's ability and selects question difficulty to match.

Where we were
18% completion on one fixed sequence for every learner
Where we aimed
Each learner's next question matched to their current ability
Chose
Ability estimation with calibrated questions (3PL IRT)
Didn't choose
Learner-chosen difficulty; rule-based difficulty bands
Constraint
Every question needs calibrated parameters
Risk
Harder to explain than a sequence; a new learner's first questions are the least certain
Success measure
Assignment completion, with practice drop-off as the second signal

08System

Learner ability on one side, question parameters on the other.

The PRD specifies the loop end to end: initialization, ability calculation, concept selection, question assignment, response evaluation, ability update and difficulty adjustment, until the practice ends. Its inputs are raw student data, each question's 3PL parameters and content IDs.

09Cold start

A new learner starts with no estimate.

The PRD names initial ability estimation as an edge case. With no history, the first questions are the least informed and the estimate sharpens as answers come in, which is where a fixed sequence and an adaptive one behave most alike. The method the PRD specifies isn't in the evidence available here, so it isn't described.

10Rollout

Measured as a before/after, not a controlled test.

Assignment completion was compared across a 2-year academic-cycle dataset: the static learning path before, the adaptive system after.

11Result

Completion: 18% → 45%.

18% on the static path, 45% after the adaptive system, across a 2-year academic-cycle dataset. The figure at the top of this page carries the full evidence: an observed change, not attributed to the engine alone.

12Attribution

What this number can and can't prove.

13Learning

When completion drops, check the fit before adding content.

Personalize only where it clearly improves the job; everywhere else, a stable default wins. Ability-based matching is harder to explain, depends on well-calibrated questions, and a new learner's first questions carry the most uncertainty.

14Differently

Product reasoning

What I'd do differently: calibrate and hold out earlier.

I would establish dynamic difficulty calibration and an explicit holdout strategy earlier, so the adaptive system could be evaluated not only on completion but also on learning effectiveness and incremental impact. This is a product reflection, not a measured finding.

15Next

Product reasoning

What I would do next

  1. 01

    Hold out a share of learners on the static path, to isolate the engine's effect from everything else that changed.

  2. 02

    Read completion by ability band, starting with new learners, whose first questions carry the most uncertainty.