Same Interview, New Primitives

I’ve been on both sides of the coding interview many times, and it’s finally starting to change. Felix Rieseberg, an engineering manager at Anthropic, wrote in Hunger Beats Tenure  that he’s dropped the coding interview, because Claude is now better at small coding challenges than the candidates are. Steve Yegge’s The Last Technical Interview  calls the whole process “a big game of darts.” Plenty of companies are rebuilding their loops to let candidates use AI.

Letting candidates use AI is the easy part. The harder question is what you measure once they do. I think the answer is something we already know how to do.

The question hasn’t changed

Interviews were always testing judgment over the resources available. For most of my career those resources were data structures, algorithms, memory, and time complexity. Picking a hash map over a list was a judgment call, and interviewers watched for it.

Now the resources are models, context, tokens, and sessions. Planning with a strong model and handing the implementation to a cheaper one is the same kind of decision as the hash map. So is knowing how much context to give a model, and when to throw a session away and start clean. The question hasn’t changed. The primitives have.

That means you don’t need to invent a new kind of interview. You can run the old one over the new primitives: watch someone build software with an LLM the way you used to watch them write code, and judge how they use the model, the context, and the time.

It also answers the worry that you’d be testing tool trivia that goes stale by next quarter. The tools churn every release. Judgment about where to spend a strong model doesn’t. That’s the layer above the model, and it’s the part that compounds.

The awkward hour

I’ve been sure about this for a while. Back when tab completion in Cursor was the most exciting thing in my editor, before I went fully agentic, I was interviewing candidates in a traditional coding format. Even then, watching someone write code by hand was one of the most awkward things I did at work. When a question came up mid-interview, I’d ask ChatGPT. I told every candidate at the start that they could use whatever tools they’d normally use. A few reached for ChatGPT, clumsily. Most didn’t.

Today everything I do goes through an LLM: code, docs, Slack, the browser. I barely read code. Looking back, those interviews were measuring a way of working that was already on its way out.

The format has always bugged me. Many years ago, long before tab completion, I interviewed for a front-end role and was asked to implement a JavaScript Promise. I understand what the question was after, which was whether I understood async JavaScript. But the Promise already exists, and nobody will ever ask me to write one at work.

Make the problem harder

If candidates get a model, the problems have to get much harder. Nobody should be fixing five unit tests or implementing a Promise on a whiteboard. Give them a whole codebase and the kind of work the job actually involves: build a new feature, make a performance improvement, write the documentation, dig into how people use the product. That’s the job now, with an LLM doing the typing.

A hard enough problem also tests what people worry gets lost when you stop watching someone write code. You can’t finish it by accepting whatever the model hands you. You have to read what it did, notice where it’s wrong, and steer it back.

Anthropic is already doing a version of this. Tristan Hume wrote about their performance engineering take-home , which explicitly allows AI. They’ve redesigned it three times because each new Claude model caught up with the candidates. What counts as hard enough keeps moving.

Read the session

What they built is the result. The session is the evidence. Every interview done with an LLM leaves a complete record of how the candidate worked, which is more than a whiteboard ever gave anyone. Read it like a pull request, except what you’re reviewing is the process.

Planning. Did they ask for a plan, read it, and push back on it? Or did they start prompting and hope?

Model selection. Did they use the right model for each job? Plan and explore with a strong model, then delegate implementation to a cheaper one? Give candidates a menu of models with visible costs, or you can’t see this at all.

Context. Did they give the model the right context once, or explain the same thing five times? Token usage is one indicator. Treat it as evidence, not a score. Two dead ends on the way to a working solution can be good judgment, and if candidates know tokens are scored, they’ll optimize for that instead of the problem.

Verification. Did they have the model write tests before changing anything? Did they read the diffs? When the model wrote something slow, did they notice and ask for better? Models are very good at fixing inefficient code once someone asks.

Recovery. When the model went the wrong way, how quickly did they notice? Did they correct course or start a clean session?

Shipping. Did they know when to stop? Part of the job is saying “this is good enough,” shipping it, and writing up the next five improvements as tickets.

Audience. Did they produce documentation for people and documentation for agents, and know which is which? A team update reads differently from a runbook written for an LLM that’s helping an on-call engineer at 3am. How much of the writing is slop, and how much did they clearly edit?

The loop. What does their iteration loop look like? Which MCP servers did they reach for? Do they type to the model, or talk to it? I do most of my work by voice now, and it’s much faster.

Notice how much of this list is the old interview. Do you break the problem down well? Do you test? Do you care about performance? Can you explain what you did? Same questions, new primitives.

One more thing I’d ask about, even though you won’t see it in an hour: what runs after you ship? I have scheduled jobs that clean up dead code in a couple of repos with a lot of tech debt, and others that review my own agent sessions to improve my harness. Someone who has set up that kind of maintenance is telling you how they think about the work.

If the session is the unit of agentic work, it’s also the right unit to evaluate.

Close the laptop

If one part of the interview watches how a candidate works with a model, another part should watch how they work with people. Close the laptop and talk. How clearly can they explain an idea without a model helping them find the words? Can they walk you through their process, unassisted, in a way you can follow?

System design sits somewhere in between. It’s technical, but it’s mostly a conversation: holding a system in your head at a level above syntax, and explaining it to someone else. It’s as abstract as it ever was, and it’s where the technical interview and the behavioral one meet.

This part matters more than it used to. For a long time the picture of a great engineer was someone so deep in the code that they needed a manager to talk to everyone for them, because any interruption cost hours of rebuilding the codebase in their head. Those engineers will still do well. But if the work is explaining what you want, to a model and to a team, then explaining what you want is the job.

Trust the vibes

Here’s the part that might get me in trouble as an interviewer: even in a technical interview, a lot of what I’m evaluating is the vibes. Junior or senior, much of my decision has come from the conversation. Do they seem curious? Are they enthusiastic? Did they think about what the company does before they showed up? That sounds basic, and a surprising number of candidates miss it.

If you can talk with me for an hour about how you build software with LLMs, what you tried, what broke, and what you changed, I learn most of what I need. And if someone rips through the code but is an entitled jerk, I won’t hire them. I have to work with this person.

“Vibes” gets a bad rap, the same way vibe coding does. But vibes are what interviews have undervalued for a long time, and they count for a lot more now. Felix calls it hunger, and I think we’re describing the same thing.

Part of it now is readiness to adopt the new tools. There’s still a debate about whether AI is overvalued as a business. That’s a different question. Whether models are good at writing code isn’t up for debate anymore. A candidate who interviews brilliantly and then shows up and writes everything by hand isn’t going to be your most effective engineer.

You still need a scorecard. “How were the vibes?” doesn’t hold up when someone asks how you hire. So write down what your gut is reacting to: curiosity, enthusiasm, whether they did their homework. That’s a scorecard too.

Let’s stop pretending

If a company asks me to write code by hand in an interview today, I read it as a sign their process is out of date, and probably more than just the interview. Nobody needs a new philosophy of hiring to fix that. We’ve always interviewed for judgment. Only the primitives changed. Give candidates the tools the job uses, make the problems harder, and keep the rest of the interview.

Stop watching people type. Watch them work.