Skip to content
Nitesh Tiwari
Back to Work

Faster answers without scaling support

Reducing Doubt Turn-Around Time (TAT) from 24 Hours to <15 Minutes via a Hybrid Peer & Guided-Hint Architecture

Doubt Resolution · Edfora · EdTech · myPAT · Glorifire · Senior Product Manager · Jul 2023 – Jul 2026

During peak JEE exam preparation, doubt resolution on myPAT and Glorifire took about 24 hours, and students studying late at night got blocked. The dilemma: reduce resolution friction without scaling human support linearly, while preserving academic accuracy and trust.

  1. Evaluated

    An AI auto-resolver, as a strategic option

  2. Selected

    A hybrid: guided hints, verified peer solutions, SME escalation

  3. Quality gate

    The hybrid, behind a 90% circuit-breaker; >90% accuracy maintained

  4. Not shipped first

    The AI auto-resolver, while answer-quality risk outweighed its scale

Senior PM ownership, end to end: from discovery to phased launch. ~24 h was a reported peak bottleneck, not a median; <15 min is the median TAT measured in the 2-week, 10,000-student pilot and stable after the live rollout, for the ~70% repetitive-doubt pool.

01Signal

At peak JEE exam preparation, doubt resolution took about 24 hours.

During peak JEE exam preparation, myPAT and Glorifire faced a doubt-resolution bottleneck of about 24 hours. A doubt is a learner stuck on a question; SMEs and faculty resolved them, so a student studying late at night could stay blocked until morning. Approximately 70% of logged tickets were repetitive/pattern-based questions.

  • ~70%

    of logged tickets were repetitive/pattern-based questions. Reported

02Ownership

I owned it end to end, from discovery to phased launch.

As Senior PM, I personally owned and executed each of these.

Discover
Discovery · user research
Decide
AI vs. tutor vs. hybrid evaluation · RICE scoring · unit economics modeling
Define
PRD · UX flows
Align
Cross-functional engineering coordination · academic team alignment
Launch
Cohort pilot design · telemetry implementation · phased launch

03Tension

Faster answers, without linear human support or lost accuracy.

Engineering
AI-first resolver: scale, and lower recurring dependence on faculty
Faculty
Human resolution: accuracy and academic integrity
Product & growth
A hybrid: high-frequency, lower-complexity doubts handled at scale; complex ones escalated to people

04Options

Three strategies, not three features.

  1. A. AI auto-resolver

    Gains

    Fast and highly scalable, with low variable faculty cost.

    Risk

    Higher risk of incorrect answers damaging academic trust.

  2. B. 1-on-1 live tutor marketplace

    Gains

    High accuracy.

    Risk

    Poor scalability and unit economics for off-hours demand, and significant operational overhead.

  3. C. Hybrid P2P community + guided hints◆ Chosen

    Gains

    Instantly unblocks repetitive, pattern-based doubts, and escalates complex cases to SME and faculty support.

    Costs

    Needs verification and quality guardrails so unverified answers don't get through.

05Basis

I evaluated the alternatives on RICE and unit economics.

I evaluated the alternatives using RICE prioritization and unit economics. A tutor-first model scales human resolution effort with doubt volume, and about 70% of logged tickets were repetitive or pattern-based: volume a guided-hint and peer system can unblock at once.

Decision frame
RICE prioritization and unit economics
Not shown
RICE scores and numerical unit-economics inputs (costs, prices, staffing)

06Trade-off

Product reasoning

Scale against academic integrity.

  1. AI-first

    Scale
    Highest, in theory
    Accuracy
    Highest risk
    Human role
    Not by default

  2. Human-first

    Scale
    Linear cost
    Accuracy
    Highest confidence
    Human role
    Always

  3. Hybrid (chosen)

    Scale
    Repetitive volume at scale
    Accuracy
    Gated at 90%, with rollback
    Human role
    On escalation
Product reasoning from the documented options and stakeholder positions; the cells are qualitative, not scores.

07Decision

Selected: a hybrid, with people where it matters.

Chose
Step-wise guided hint system · verified peer solutions · SME and faculty escalation
Guardrails
Step-wise hints instead of direct answer dumps; verified mentor and peer credibility signals; SME and faculty escalation for complex cases
Not chosen
The 1-on-1 live tutor marketplace
Not shipped first
The AI auto-resolver, evaluated as a strategic option

08Quality gate

A circuit-breaker quality gate, and accuracy held above 90%.

The quality bar was a number before launch: a circuit-breaker at 90% accuracy, with an agreed rollback.

Gate
Circuit-breaker at 90% accuracy, with an agreed rollback
Audited by
SME sampling of resolved doubts
Monitored with
Post-resolution student satisfaction ratings
Result
>90% resolution accuracy maintained

09Pilot

10,000 JEE students for two weeks, against a holdout, then a phased launch.

Cohort
A 10,000-student JEE pilot, cohort-gated to limit the blast radius
Duration
2 weeks (14 days)
Compared
Guided-hint access vs. the standard response queue, with a holdout
Pilot read
Resolution speed, SLA performance and initial CSAT
Then
A phased launch to live rollout
Not in the record
The split, holdout size and statistical significance

10Outcome

Faster answers, more returning learners, accuracy held.

median doubt-resolution TAT, from a ~24 h reported peak
<15 min
median doubt-resolution TAT, from a ~24 h reported peak. Measured
Median in the 2-week, 10,000-student JEE pilot, stable after the live rollout, for the ~70% repetitive-doubt pool; doubt created → first qualifying resolution. ~24 h was a peak bottleneck, not a median.
relative uplift in D14 retention
+18%
relative uplift in D14 retention. Measured
Pilot holdout / A-B cohort against the standard queue
resolution accuracy maintained on the hybrid
>90%
resolution accuracy maintained on the hybrid. Measured
Quality guardrail, audited through SME sampling and post-resolution satisfaction ratings
support and operational cost reduction
~60%
support and operational cost reduction. Derived
Calculated from avoided SME and faculty headcount scaling against ticket growth
The ~60% is derived from avoided headcount scaling, not a directly observed financial saving.

11AI judgment

Product reasoning

AI was evaluated, and not shipped as the first solution.

AI was evaluated as a strategic option but was not shipped as the first solution because academic trust and answer-quality risk outweighed its scalability advantage at that stage.

  1. Evaluated

    An AI auto-resolver: fast, highly scalable, low variable faculty cost

  2. Weighed against

    Academic trust and answer-quality risk, against its scale

  3. Not shipped first

    The hybrid shipped instead, behind a 90% accuracy circuit-breaker
No AI resolver, model or LLM went into production in this decision.

12Learning

Product reasoning

Problem first. Model second.

The question was never whether AI could answer doubts. It was how to cut the wait without trading away accuracy.

  1. 01

    Scale is not the only optimization target: accuracy and integrity set the limits it had to work within.

  2. 02

    A quality gate decides whether an AI path is viable. Written as a number with a rollback, it turned a debate into something a pilot could settle.

  3. 03

    A hybrid can beat a binary AI-versus-human choice: automate the repetitive volume, keep people where judgment matters.

13Differently

Product reasoning

What I'd do differently: progressive disclosure from day one.

I would have designed the guided-hint experience around progressive disclosure from day one: starting with the smallest useful hint, then escalating toward peer or SME support only when needed. This would preserve the student's opportunity to reason before seeing a full solution. This is a product reflection, not a measured finding.