# Assessing AI-assisted developers

How to assess AI-assisted developers by verification, eval design, context control and ownership instead of banning tools.

Source: https://www.wepair.works/library/assessing-ai-assisted-development
Published: 2026-09-13
Updated: 2026-09-13

Assessment guidance reviewed September 2026. Assessing AI-assisted developers means observing how they choose tasks for AI, constrain context, verify generated work and remain accountable for production result. Tool ban hides exactly behavior team needs to evaluate.

## Banning AI in interviews selects for the wrong thing

Banning AI in interviews selects for candidates who practiced performing without tools, not necessarily engineers who produce safest work with modern workflow. If role permits AI after start, interview should inspect its use under observation. Policy mismatch creates false signal.

A candidate can be excellent without AI, and some work should remain manual. Assessment question is not whether tool appears. It is whether person decides appropriately, understands output and verifies consequences. Blind use fails; ideological refusal can also waste leverage.

Hidden use is worse than allowed use. Ban encourages candidates to conceal assistance or memorize generated patterns. Assessor sees final code but cannot judge prompt, context, review or correction. Explicit permission makes process discussable and comparable.

Interview should state boundaries: which tools are allowed, whether candidate may authenticate personal account, what data must not leave environment and how output is treated. Provide safe option when possible. Do not ask person to paste proprietary code into public service.

Security policy matters. Candidate should recognize secrets, personal data and client code boundaries before sharing context. A technically correct completion produced through unsafe disclosure is failed result. Assessment environment must avoid real confidential data too.

Role distinction matters. Product engineer may use AI for test scaffolding, refactor or documentation. AI engineer may design retrieval, evaluation and model integration. Both need verification, but depth differs. [AI-engineer assessment](/hire/ai-engineer) should test systems around model, not only ability to call API.

## What good AI-assisted work looks like

Good AI-assisted work starts with engineer-defined problem and evidence. Person knows desired behavior, relevant constraints and way to prove change before asking tool to produce code. Prompt is not substitute for model; it is one interface into implementation.

Task selection is deliberate. Repetitive transformations, unfamiliar API lookup, test-case generation and draft documentation can benefit. Core invariants, security decisions and ambiguous product behavior need stronger human control. Candidate should explain why this subtask is suitable.

Context is scoped. Strong engineer gives tool enough code, types and constraints without dumping whole repository. They identify authoritative files and remove irrelevant noise. When output conflicts with source, source wins. More tokens do not guarantee better result.

Generated change stays small enough to review. Candidate asks for one function, migration draft or test set, then reads diff. Huge generated patch creates verification debt. Speed is useful only while reviewer can still explain every line.

Verification matches risk. UI change gets exercised through surface. Data migration gets transaction, rollback and representative volume. Retrieval change gets eval set and failure categories. Auth change gets negative cases. Candidate should choose evidence before declaring completion.

Good workflow also records provenance when policy requires it. Commit message need not narrate every completion, but team should know where licences, generated assets or external data create obligations. Candidate follows project rules rather than assuming AI output is ownerless.

## Signals of someone who cannot verify output

Someone who cannot verify output treats compilation as proof. They accept plausible imports, fabricated APIs or tests that only echo implementation. When challenged, they ask model again instead of inspecting source, runtime or docs. Each loop increases code without increasing certainty.

One signal is explanation gap. Candidate can describe prompt but not resulting control flow, failure mode or type. Ask them to predict behavior before run and explain why check is sufficient. If answer remains “tool said,” ownership is absent.

Another signal is test theatre. Generated tests assert mock calls, copied fields or non-empty output while missing consumer behavior. Candidate celebrates coverage number without asking which plausible bug would fail. Good engineer deletes tautological test and writes observable scenario.

Unbounded context is warning. Candidate pastes entire repo, secrets or customer data into tool without checking policy. Even if code works, process is unsafe. Strong person minimizes disclosure and knows when approved enterprise tooling changes rules.

Generated abstraction is common failure. Model invents helper, adapter and config for one branch. Candidate accepts because code looks organized. Ask what can be deleted. Senior engineer reduces patch to smallest design fitting existing convention.

Repeated prompting without instrumentation is another failure. When output breaks, candidate changes words randomly rather than reading error, reproducing behavior or isolating input. AI can generate hypotheses, but debugger, logs and failing example decide among them.

Confidence should follow evidence. Candidate who says “done” after one happy path underestimates model error and software uncertainty. Better answer names what ran, what remains unverified and which risk would require deeper check.

## How we score it

We score AI-assisted development across task choice, context hygiene, verification, comprehension, correction and ownership. Tool brand and typing speed do not score. Candidate can use no AI and still pass if decision is reasoned; candidate can produce much code and fail if cannot defend it.

| Axis            | Strong evidence                       | Failure evidence                    |
| --------------- | ------------------------------------- | ----------------------------------- |
| Task choice     | Uses AI where result is reviewable    | Delegates ambiguous core decision   |
| Context hygiene | Shares minimum safe source            | Exposes secrets or irrelevant repo  |
| Verification    | Runs behavior-level proof             | Stops at compile or mock echo       |
| Comprehension   | Explains every changed boundary       | Cannot trace generated control flow |
| Correction      | Diagnoses and narrows bad output      | Re-prompts blindly                  |
| Ownership       | Accepts responsibility and names risk | Attributes decision to tool         |

Assessment task should contain a trap model can produce plausibly. Retrieval pipeline with no eval set, migration missing transaction, retry loop duplicating side effects or React optimization without measurement all work. We are not testing whether model fails; models will. We test whether engineer notices.

Assessor observes sequence. Did candidate define expected behavior? Did they inspect relevant code? What did they delegate? Which output did they reject? What evidence changed mind? Notes capture process rather than only final branch.

Challenge one generated choice. Ask why library exists, what error path does or which assumption test proves. Strong candidate either defends with source or changes it. Weak candidate protects output because changing it feels like admitting failure.

Score relative to role. AI engineer should create eval categories, understand model uncertainty, inspect cost and latency, and design fallback. Application engineer integrating API should validate schema, errors and user experience. Everyone must protect data and own code.

Written verdict names tool use honestly. “Used Claude for migration, caught missing transaction and rewrote backfill” is useful evidence. “AI-assisted” alone says nothing. Client should know both leverage and correction.

## What we still fail people for

We still fail candidates for unsafe output even when tool caused it. Engineer owns change. Generated SQL that can corrupt data, auth code that trusts client input or dependency that does not exist is candidate's responsibility once accepted.

We fail inability to explain. Production incidents will not accept “AI wrote it.” Person must trace state, reproduce failure and choose rollback. Fluency in prompting does not replace system model.

We fail hidden data risk. Copying secrets, customer data or proprietary source outside approved boundary shows judgment gap. Interview environment should state policy, but senior engineer should ask when unclear.

We fail fake verification. A passing generated test that asserts mock returned mock provides no confidence. Candidate should connect proof to consumer-visible behavior, invariant or real failure. Tests earn place by catching plausible bug.

We fail abstraction inflation. If candidate keeps hundreds of generated lines where twenty fit existing pattern, maintenance cost outweighs speed. Seniority includes deleting tempting output.

We fail evasive ownership. Good candidate can say tool suggestion was wrong and explain correction. Someone who blames prompt, model or missing instruction for every mistake will do same in production. Uncertainty is acceptable; refusal to own it is not.

wepair.works evaluates AI use inside live pairing rather than banning it. Founder sends client written evidence, including what candidate got wrong. Decision stays grounded in observed behavior: safe use, verification and accountability. AI skill is not quantity generated. It is quality of judgment retained.