Building a Real-Time AI Interviewer for Technical Hiring

The first version worked.

It could listen to a candidate, generate a response, ask the next question, and keep the interview moving.

But it did not yet feel like an interview.

That difference became the core engineering problem.

A technical interview is highly sensitive to delay. If the interviewer pauses too long after every answer, the conversation feels artificial. If the system cannot understand what the candidate is coding in real time, it loses the ability to ask meaningful follow-ups. And if the final recommendation cannot be traced back to evidence from the conversation, the result is difficult for a recruiter to trust.

The challenge was therefore not simply to connect speech recognition, an LLM, and text-to-speech.

It was to make the entire interaction feel responsive, context-aware, and defensible.

Latency became the first product constraint

The early implementation used a conventional sequential voice flow.

Speech was transcribed, interpreted, sent through the language model, converted back to speech, and then played to the candidate.

Functionally, it worked.

Conversationally, it felt slow.

Each additional stage added a little delay, and those delays accumulated quickly enough to break the rhythm of the interview.

This changed how we evaluated the system.

We were no longer asking:

Does every component work?

We started asking:

Does the candidate experience the interaction as one continuous conversation?

That shift influenced several later decisions, including how voice models were evaluated and how the realtime loop was orchestrated.

Voice quality was only one part of the TTS decision

A technically impressive voice model is not automatically the right model for an interview product.

We evaluated speech generation across three competing dimensions:

• perceived voice quality
• time to first audio
• cost per interview minute

The best standalone voice was not necessarily the best system choice.

A model that sounds marginally better but adds noticeable response delay can make the overall interview feel worse.

Likewise, a very fast model can become expensive once it is running across long technical interviews at scale.

The useful optimization target was therefore not raw model quality.

It was the best balance of latency, naturalness, and cost inside the complete interview experience.

Generic questions were not enough

A good technical interview should not feel like a shuffled question bank.

If a candidate has several years of backend experience, the interviewer should be able to ask about that experience. If the role requires distributed systems, the interview should lean into those requirements. If the candidate performs strongly on one question, the next question should be able to adapt.

To support that, the product used contextual retrieval over both sides of the interview:

This allowed the interviewer to generate questions that were grounded in the candidate's background while still remaining aligned with the job requirements.

The result was a more specific interview rather than a generic sequence of prompts.

Coding changed the interaction completely

Technical interviews are not just conversations.

Candidates need to solve problems, write code, run it, debug it, and explain what they are doing.

That meant the interview experience had to understand both the spoken conversation and the coding session.

A sandboxed execution environment was introduced so candidates could write and run code during the interview while the AI interviewer continued the discussion.

This created a much richer signal.

Instead of only asking:

What is the time complexity?

the interviewer could react to the actual approach the candidate had implemented and ask about trade-offs, edge cases, or a failing test.

That moved the experience closer to a real technical interview rather than a voice-based quiz.

Evaluation had to be explainable

Generating questions was only half the system.

The harder problem was turning a long interview into a hiring signal that someone could actually use.

A recruiter does not want another forty-page transcript.

They want to know:

• how the candidate performed
• where the strongest signals appeared
• which skills were weak
• how the candidate compared with others
• why the system reached its recommendation

The evaluation experience therefore focused on structured evidence rather than a single opaque score.

Hard-skill signals could come from coding performance, problem-solving, and technical knowledge. Communication and other softer signals could be derived from the conversation itself.

The final output was designed to support both candidate-level feedback and employer-side decision-making.

The scoring problem was not just accuracy

A human interviewer naturally carries context from the entire conversation.

An AI system has to reproduce that consistency intentionally.

A candidate may struggle early, recover later, change their approach after feedback, or explain a technically correct solution poorly.

A useful evaluation cannot treat every response as an isolated event.

That made consistency across the complete interview more important than individual question scoring.

The product therefore evolved toward a broader evaluation view: skill-level performance, evidence from the interview, code execution results, and final recommendation all needed to agree with each other.

For hiring workflows, this consistency matters as much as raw model capability.

Four workstreams, one product

The engagement eventually grew across four major areas:

The total implementation represented roughly 150+ engineer-days of scoped engineering work.

A significant share of that effort went into the realtime interviewer itself.

That was intentional.

If the conversation layer feels broken, every downstream feature becomes irrelevant.

Question generation, evaluation, and dashboards only become valuable once candidates can complete an interview that feels natural enough to finish.

Why we built the slow version first

One of the most useful engineering decisions was building a working sequential voice pipeline before optimizing it.

On paper, that can look like throwaway work.

In practice, it created a measurable baseline.

The team could observe where latency appeared, test the interview flow with something real, compare voice providers under actual conditions, and show the product early rather than waiting for the entire system to be optimized.

That prototype answered an important question:

Is the interview concept itself useful?

Once that was proven, the engineering effort could focus on making it fast enough for production.

This sequencing reduced the risk of optimizing the wrong problem.

Production feedback mattered more than perfect planning

Realtime voice products fail in ways that are difficult to reproduce in a clean development environment.

• Candidates have different microphones.
• Networks fluctuate.
• People interrupt the interviewer.
• They pause mid-sentence.
• They restart answers.
• They switch between speaking and coding.

These edge cases are not secondary details.

They are the product.

That is why the build included meaningful room for iteration after the first working versions.

The goal was not to freeze the system once each feature existed.

It was to improve the interaction based on real interviews and real failure modes.

What made the system difficult

The complexity came from combining several product constraints that pull in different directions.

Lower latency can increase infrastructure or model cost.

More candidate context can improve personalization but also increases reasoning complexity.

More aggressive evaluation can create richer reports but risks producing signals that recruiters cannot verify.

A highly adaptive interviewer can feel intelligent, but it also needs to remain consistent across candidates.

The system therefore had to balance:

Optimizing any one of those in isolation would have produced a weaker product.

What we learned

The first lesson was that latency is part of intelligence. A system can generate excellent answers and still feel unintelligent if every response arrives too slowly.

The second was that contextual interviewing is much more valuable than generic question generation. Candidate and role context changes the quality of the conversation far more than simply adding more questions.

The third was that live coding creates stronger interview signal than conversation alone. Seeing the candidate's actual implementation allows the interviewer to ask better follow-ups and makes technical evaluation more grounded.

Finally, explainability is not optional in hiring. A score without evidence is difficult to trust. A recommendation tied back to the interview, code execution, and skill-level performance is much more useful to a recruiter.