Skip to content

Add example drawings to coding interviews - #262

Open
Chuyutseng wants to merge 1 commit into
sysprog21:mainfrom
Chuyutseng:add-example-drawing-area
Open

Chuyutseng wants to merge 1 commit into
sysprog21:mainfrom
Chuyutseng:add-example-drawing-area

Conversation

@Chuyutseng

@Chuyutseng Chuyutseng commented Oct 8, 2026 •

Copy link
Copy Markdown

Coding candidates need a way to draw examples while keeping their code visible. This adds an optional, collapsible Example drawing area below the editor, beside the candidate test cases. It is disabled by default in the lobby. The lower panel stays compact when collapsed, and candidates can drag the divider to resize it.

The first version includes pen, eraser, undo and clear, with color selection. The buttons use icons with accessible labels and tooltips. Shapes will be added in a later change.

Drawings are sent to Gemini Live only when requested. The browser keeps the latest JPEG snapshot on the server, but edits, drawing handovers and reconnects do not send it to Gemini. When the candidate refers to the drawing, Jim is instructed to call read_board. The server then sends the latest image through realtimeInput.video before returning the tool response, at most once per tool batch. A drawing that is not referenced adds no image inputs. Once read, the image remains in the Live context and incurs the existing per-turn context cost.

Drawing activity delays idle reminders, and stopping drawing can hand the turn back to Jim without automatically sending an image. Code and drawings are preserved in recordings and replay, and the final drawing is shown separately in the report. Drawings do not add or deduct coding or communication points and are not sent as images to the report model.

Validation:

  • ./scripts/test.sh exited with status 0, with all 1118 browser tests passing.
  • Browser tests cover the toolbar, panel resizing, pen strokes, undo, clear, drawing turns and reconnecting.
  • Backend tests confirm that coding drawings are sent only on an explicit read and that Whiteboard keeps its existing behavior. Restoring automatic image sends causes the new regression test to fail.

Closes #255


Summary by cubic

Adds an opt-in Example drawing area to coding interviews so candidates can sketch examples without hiding their code. The lobby toggle is disabled by default and unavailable in Whiteboard, and the choice is signed into token metadata, so disabled or non-coding sessions refuse drawing streams and read_board calls. Drawings stay out of assessment, and the contract bundle is bumped to 35.

  • After the first drawing arrives, coding stops forwarding camera frames so Gemini sees the drawing, not the camera; images are sent only when Jim calls read_board, once per turn.
  • Drawing activity resets the idle-window reminder without adding evidence or requesting a model reply; a drawing handover counts as a turn, cannot restart a stale countdown, and a failed resend no longer stops the room loop.
  • Excess strokes are rejected so Gemini sees the same drawing as the candidate, and recordings replay the drawing alongside code.
  • The final drawing appears in the report with a note that it does not affect scores; report prompts and validation forbid picture-only credit but allow discussing candidate-described images, keeping the original rubric.
  • Provides pen, eraser, undo and clear with color selection; strokes use the existing whiteboard wire format. The lower panel collapses compactly and the divider drags to resize it.

Closes #255.

Written for commit 7213b57. Summary will update on new commits.

View guided diff

cubic-dev-ai[bot]

This comment was marked as resolved.

@Chuyutseng
Chuyutseng force-pushed the add-example-drawing-area branch from 46fe4ca to ede0d50 Compare October 8, 2026 08:31
cubic-dev-ai[bot]

This comment was marked as resolved.

@Chuyutseng
Chuyutseng force-pushed the add-example-drawing-area branch 2 times, most recently from afc217e to 9828618 Compare October 8, 2026 14:07
Comment thread src/livekit.rs Outdated
Comment thread src/livekit/board.rs
Comment thread web/interview.js Outdated
Comment thread web/replay.js
@jserv

jserv commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

After a reconnect in a coding interview the agent can be left holding code older than what the candidate typed, and if the interview ended while the drawing resend failed, the queued end_interview is never sent and no report is produced. Details are on the reconnect hunk in web/interview.js.

cubic-dev-ai[bot]

This comment was marked as resolved.

Comment thread src/agent/events.rs
Comment thread src/agent/report.rs Outdated
Comment thread src/livekit/board.rs Outdated
Comment thread web/interview.js Outdated
Comment thread web/interview.js Outdated
Comment thread web/replay.html
jserv

This comment was marked as duplicate.

cubic-dev-ai[bot]

This comment was marked as resolved.

@Chuyutseng
Chuyutseng force-pushed the add-example-drawing-area branch from 9d89db0 to d6cea65 Compare October 10, 2026 17:18
jserv

This comment was marked as outdated.

@Chuyutseng
Chuyutseng force-pushed the add-example-drawing-area branch from d6cea65 to fa47c83 Compare October 10, 2026 17:50
@jserv
jserv requested review from ColtenOuO and alanhc October 10, 2026 18:19
Comment thread src/agent/prompts.rs Outdated
Comment thread src/agent/report.rs
Comment thread src/livekit/board.rs Outdated
Comment thread src/livekit/board.rs Outdated
Comment thread src/livekit/board.rs Outdated
Comment thread src/livekit/board.rs Outdated
Comment thread web/interview.html Outdated
Comment thread web/example-board.js
Comment thread web/example-board.js
@ColtenOuO

Copy link
Copy Markdown
Collaborator

Could the example drawing area be something the candidate turns on in the lobby, off by default, rather than always present in a coding interview?

CodeTrial aims to put the candidate in a situation as close to a real interview as possible, not to make the session as convenient as possible. A typical online coding interview gives the candidate a shared editor and nothing else, and explaining an example in words or as comments in the code is part of what is being practised. A drawing panel that is always there changes that setting by default, and it also changes what Jim sees (camera frames stop once a drawing arrives).

The lobby already lets the candidate shape the interview (loop, Editor or Whiteboard), so an opt-in toggle there would fit. Someone who wants the aid can have it, and the default session stays as close to the real thing as possible.

@Chuyutseng
Chuyutseng force-pushed the add-example-drawing-area branch 3 times, most recently from 90aa7cf to 9bd5995 Compare October 12, 2026 11:14
@jserv
jserv requested a review from ColtenOuO October 12, 2026 19:04
jserv

This comment was marked as outdated.

jserv

This comment was marked as outdated.

@Chuyutseng
Chuyutseng force-pushed the add-example-drawing-area branch from 930a5b2 to 3a8c9f0 Compare October 13, 2026 16:48
Comment thread web/example-board.js Outdated
Comment thread src/agent/prompts.rs Outdated
Comment thread src/agent/report.rs
.iter()
.any(|phrase| normalized.contains(&format!(" {phrase} ")));
if visual_reference || candidate_image {
errors.push(format!("{path}: example drawings are outside coding and communication assessment; remove the visual reference and any credit or deduction based on it. Use only code, tests or the candidate's actual conversation under the original rubric, never the interviewer's interpretation of a picture"));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check can cost the whole report. Once drawing_received is set, normal coding advice like "Draw a diagram of the pointer moves before coding" in improvementPlan matches through recommendation, and "your sketch" matches through owned. After MAX_REPORT_REPAIRS, finish returns the error, because sanitize_report_candidate doesn't remove these strings and so there is never a salvage held. Make the drawing reference a salvage case instead: drop the offending item or sentence and accept the rest, rather than failing the report on wording.

@jserv

jserv commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Pen strokes in the example drawing carry a color object instead of a hex string, so every recorded stroke is rejected on replay and the color picker paints black. The detail is in the comment on web/example-board.js.

@Chuyutseng
Chuyutseng force-pushed the add-example-drawing-area branch from 3a8c9f0 to 4948e7f Compare October 13, 2026 17:41
@jserv

jserv commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

@Chuyutseng Five mutants shards on 4948e7f fail on surviving mutants in code this branch adds, so the new tests do not yet pin it down: 11 in sanitize_example_drawing_candidate (src/agent/report.rs:662 to :795, mostly its counters and the ==/> checks), 3 in apply_control (src/agent/events.rs:659, :707, :708), and the example_drawing_enabled field in initial_runtime_state (src/livekit.rs:3136). Each MISSED line in the job log names the change that went undetected; add a test that fails under it, or say on the thread why a mutant is equivalent. This is separate from the stroke color fix asked for on web/example-board.js.

@Chuyutseng
Chuyutseng force-pushed the add-example-drawing-area branch from 4948e7f to 79cbfa3 Compare October 14, 2026 05:29

@ColtenOuO ColtenOuO left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The choice is not remembered. rememberSettings() and restoreSettings() in web/app.js save only the duration, the loop and the mode, so a candidate who turns drawing on finds the box unchecked again after a reload or on their next visit. The editor options on the same page are remembered through saveEditorOptions, so this one checkbox behaves differently from the others next to it.

Could you add it to the stored settings (written on change and restored in restoreSettings(), still unchecked when nothing is stored), and add a case to tests/browser/lobby.test.js that checks the box, reloads and expects it to stay checked?

jserv

This comment was marked as outdated.

@Chuyutseng
Chuyutseng force-pushed the add-example-drawing-area branch from 79cbfa3 to ca4e67d Compare October 14, 2026 07:25

@jserv jserv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase latest main branch and resolve conflicts.

@Chuyutseng
Chuyutseng force-pushed the add-example-drawing-area branch from ca4e67d to 8c67c64 Compare October 14, 2026 07:30
Let candidates explain examples without hiding their code.
Offer pen, eraser, undo and clear in an optional drawing area.
Send example images only when Jim requests read_board.
Preserve the camera policy and keep drawings outside assessment.
Save drawings in reports and replay as session artifacts.

Closes sysprog21#255
@Chuyutseng
Chuyutseng force-pushed the add-example-drawing-area branch from 8c67c64 to 7213b57 Compare October 14, 2026 10:08
@jserv
jserv requested a review from ColtenOuO October 14, 2026 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an example drawing area to coding interviews

3 participants