Coding interview: How would you implement Unicode source-code spoof diagnostics?

Prompt and context

A repository permits non-ASCII identifiers and multilingual comments. A change contains bidi controls or visually confusable identifiers, yet a plain text viewer renders a misleading order. Design a scanner that finds the risk without rejecting legitimate Arabic, Hebrew, or other multilingual comments and strings.

What the interviewer evaluates

• Whether lexing, bidirectional rendering, and security diagnostics are separate stages.
• Whether you understand UTS #55 lexical source-code atoms and its HL4 display semantics.
• Whether you use UTS #39 confusable and restriction rules instead of treating every non-ASCII character as hostile.
• Whether you handle Unicode data versions, incremental scans, explainable findings, and false-positive policy.

Clarifying questions to ask

First confirm the target language, compiler version, allowed identifier scripts, and comment languages; identify whether the scanner runs in an editor, CI, or code-hosting platform. Then clarify whether findings warn, block submission, or require review, and whether old Unicode data or unparseable code must remain supported.

30-second answer

I would use the target language lexer to obtain tokens, identifiers, strings, and comments, then record original code points, directional controls, and scripts for each atom. Source display follows UTS #55’s lexical structure rather than treating the file as one ordinary paragraph. Diagnostics combine UTS #39 restriction and confusable data, reporting positions, rules, versions, and rendered differences. Findings can warn by default and block by repository policy; the original text is retained so valid RTL text is not mistaken for an attack.

Step-by-step deep dive

1. Establish lexical boundaries first

Do not apply ordinary paragraph bidi processing character by character. Numeric literals, identifiers, single-line strings, and comment content form atoms; nested languages need boundaries from the outer grammar. Reuse a compiler or mature parser’s token ranges, and explicitly downgrade to conservative diagnostics when parsing fails.

2. Mark bidi controls and default-ignorable characters

For each atom, record controls such as RLO, LRO, PDF, RLI, LRI, FSI, and PDI with their scopes. Reports should show escaped code points and original positions, preventing a terminal or Web page from applying the controls again. Do not blindly delete them because legitimate layout can need directional controls.

3. Organize source display with UTS #55

UTS #55 recommends applying higher-level protocol HL4 to lexical structure: token order follows language syntax, while strings, identifiers, and comments are indivisible atoms rendered with their directional properties. This preserves readable RTL text and auditable code structure at the same time.

4. Apply identifier and confusable rules

Apply the language’s allowed normalization, case, and UAX #31 profile to identifiers, then use UTS #39 for mixed-script and confusable diagnostics. Do not make a skeleton the identifier itself; it is an intermediate comparison value for one Unicode data version and is useful for finding potential collisions.

5. Write the smallest scanning pipeline

The pseudocode below shows stage boundaries only; production code still needs the target-language lexer and Unicode data files:

type Atom = { kind: string; start: number; end: number; text: string };
type Finding = { code: string; start: number; end: number; detail: string };

function diagnose(source: string, atoms: Atom[], unicodeVersion: string): Finding[] {
  const findings: Finding[] = [];
  for (const atom of atoms) {
    findings.push(...scanBidiControls(atom));
    if (atom.kind === "identifier") {
      findings.push(...scanIdentifierConfusables(atom, unicodeVersion));
    }
  }
  return findings;
}

scanBidiControls should preserve control code points and scopes; scanIdentifierConfusables should return mixed-script, restriction-level, or existing-identifier collision findings. The diagnostic stage should not rewrite source code.

6. Handle versions, caches, and incremental scans

Write the Unicode data version, language lexer version, and rule configuration into each report. Cache token results by file content and lexer version; an incremental build rescans affected tokens and identifiers in the same scope. After a Unicode data upgrade, recompute skeletons and collision sets instead of reusing old cache entries.

7. Separate findings from policy

Each finding should include file, line and column, rule code, original code points, visible rendering, and a remediation hint. A policy layer chooses warning, blocking, review, or allow; the same finding can have different policies in an editor, CI, or code-hosting UI. Logs should never render unescaped controls directly.

High-quality sample answer

I would split the implementation into lexer, Unicode analysis, render preview, and policy reporting. The lexer returns exact ranges for tokens, identifiers, strings, and comments; an unparseable region is marked uncertain rather than assigned character boundaries. Unicode analysis records Bidi_Control, script sets, normalization profile, and UTS #39 confusable results for each atom. The preview keeps token syntax order using the UTS #55 HL4 approach, applies bidi rules inside each atom, and shows escaped code points alongside the rendering. Reports carry Unicode data version, lexer version, rule code, and location; policy chooses warning or blocking by field risk. A same-version skeleton can help find identifier collisions, but it is not a username or stable hash. Cache keys include file content, language version, and Unicode version, and upgrades trigger a batch recomputation. Tests cover RTL comments, controls inside strings, nested languages, mixed-script identifiers, parse failures, incremental edits, and differences across terminal and Web renderers.

Common mistakes and failed approaches

• Rendering every character left to right, making valid RTL comments and strings unreadable.
• Blocking every non-ASCII character and breaking legitimate languages and international identifiers.
• Replacing a lexer with regular expressions, losing whether a control is in a string, comment, or identifier.
• Keeping only cleaned text and losing original code points and audit positions.
• Comparing skeletons produced by different Unicode versions without a migration plan.

Follow-up questions and responses

Why not delete every Bidi_Control character?

Some legitimate text needs directional controls, and deletion changes what users see. Block or escape them in source code according to language policy; report context and retain the original in comments and strings.

What should the scanner do when lexing fails?

Return an explicit uncertain range, run conservative control and code-point diagnostics, and do not claim that the region is safe. CI can require review until a parser supports the language precisely.

How do you prove that multilingual code is not falsely flagged?

Use valid RTL comments, strings, and allowed identifiers as positive cases, and RLO/LRO, mixed-script, and confusable collisions as negative cases. Compare logical order, visible order, and report locations across multiple editors, terminals, and Web renderers.