HarnessLab

Find what actually broke your coding agent.

Task
ApplyPilot: rank evaluated jobs by candidate fit
Model
nvidia/nemotron-3-super-120b-a12b
Fixture
applypilot-rank-001
Runner
Nebius Token Factory Sandbox
ApplyPilot-derived evaluationReplay of saved evidence · no live API calls

A coding agent working on an ApplyPilot-style ranking task

What this is

  • A real HarnessLab experiment: Nemotron coding-agent runs (model calls via Nebius Token Factory; code executed in isolated local temp workspaces, not Token Factory Sandboxes) against a fixture derived from ApplyPilot-AI's job-ranking rules.
  • The agent under test is the coding agent that would work on ApplyPilot; the harness under test is how it is guided (skill, context policy).
  • Saved evidence replayed from disk: this page makes no API calls.

What this is not

  • Not a live integration: no ApplyPilot code, database, network or user data is touched, and ApplyPilot is not modified.
  • Not ApplyPilot's production job-ranking feature or its AI features being tested.
  • Not a general benchmark: one task, 2 runs per configuration.

Overview

What HarnessLab found

Baseline Agent

PASS

Harness changed

2 dimensions

Candidate Agent

FAIL

4 possible configurations

2^2 combinations

HarnessLab selected

3 of 4 — 25% avoided

Diagnosis

Context Policy

Baseline

2/2 PASS

build, public and hidden tests

Candidate

2/2 FAIL

same task, same evaluator

Harness dimensions changed

2

Skill + Context Policy

Root-cause attribution

Context Policy

main effect

Evidence quality

HIGH

6 recorded coding-agent runs

Controlled experiment

Baseline vs. Candidate

Same task, same fixture, same evaluator — only the harness configuration differs.

Baseline

PASS
Skill
applypilot-job-evaluation-v1
Context Policy
full
System Prompt
You are a careful senior software engineer working inside a small, iso…
Max Turns
8
Full skill content
ApplyPilot job-evaluation rules (apply them to any ranking, scoring or matching code):
1. Read the domain contract file for the module you are changing and implement every rule in it, in the order it states them. Do not rely on the public tests alone; they are a subset.
2. Never treat silence as qualification, and never treat unknown as blocked: an unknown or unconfirmed fact keeps its own tier between 'confirmed' and 'blocked'.
3. A candidate must not be pointed at a role they cannot take: eligibility blockers outrank raw score.
4. A pending (not yet evaluated) job has no score; it is never ranked as if it had a low score.
5. Results must be reproducible: break every tie deterministically and do not mutate inputs.
6. Before finishing, re-check your implementation against each numbered rule in the contract, then run the tests and the build.
Full system prompt

You are a careful senior software engineer working inside a small, isolated Git repository. You can only interact with the repository through the provided tools. Read the relevant files first, then make the smallest correct change. Run the tests before you consider the task finished. When you are satisfied the task is complete, reply with a short plain-text summary and do not call any more tools.

Changed vs. the other arm: Skill, Context Policy

Candidate

FAIL
Skill
verification-skill-v2
Context Policy
compact
System Prompt
You are a careful senior software engineer working inside a small, iso…
Max Turns
8
Full skill content
Workflow for this task (optimized for efficiency):
1. Use the repository context already available to you to identify the affected area before exploring further.
2. Avoid re-reading files or information that is already available to you in the current conversation context.
3. If the context already available to you is sufficient to implement the change correctly, implement it directly rather than re-verifying details you already have.
4. Run targeted validation (the relevant tests) before considering the task complete.
Full system prompt

You are a careful senior software engineer working inside a small, isolated Git repository. You can only interact with the repository through the provided tools. Read the relevant files first, then make the smallest correct change. Run the tests before you consider the task finished. When you are satisfied the task is complete, reply with a short plain-text summary and do not call any more tools.

Changed vs. the other arm: Skill, Context Policy

What changed

Harness diff

2 dimensions changed between baseline and candidate.

Skill

applypilot-job-evaluation-v1verification-skill-v2
Show full values

Baseline

ApplyPilot job-evaluation rules (apply them to any ranking, scoring or matching code):
1. Read the domain contract file for the module you are changing and implement every rule in it, in the order it states them. Do not rely on the public tests alone; they are a subset.
2. Never treat silence as qualification, and never treat unknown as blocked: an unknown or unconfirmed fact keeps its own tier between 'confirmed' and 'blocked'.
3. A candidate must not be pointed at a role they cannot take: eligibility blockers outrank raw score.
4. A pending (not yet evaluated) job has no score; it is never ranked as if it had a low score.
5. Results must be reproducible: break every tie deterministically and do not mutate inputs.
6. Before finishing, re-check your implementation against each numbered rule in the contract, then run the tests and the build.

Candidate

Workflow for this task (optimized for efficiency):
1. Use the repository context already available to you to identify the affected area before exploring further.
2. Avoid re-reading files or information that is already available to you in the current conversation context.
3. If the context already available to you is sufficient to implement the change correctly, implement it directly rather than re-verifying details you already have.
4. Run targeted validation (the relevant tests) before considering the task complete.

Context Policy

fullcompact

Search space

Configurations considered

Every possible combination of the changed dimensions, and which ones HarnessLab actually ran.

Dimensions changed
2
Possible configurations
4

2^2

HarnessLab executed
3
Avoided
1

25% search-space reduction

BaselineCandidateFailNot testedSelected by planner

How it decided

Diagnosis trace

Each experiment HarnessLab chose to run, why it chose it, and what it eliminated.

01Explore

Experiment 1

FAIL

At candidate value: Context Policy

Why this experiment?

Selected because it is one of only 1 remaining configuration(s) that distinguishes: interaction:skills+contextPolicy vs main_effect:contextPolicy.

Eliminated

Nothing newly eliminated

Remaining

  • Skill
  • Context Policy
  • interactionSkill × Context Policy
Technical details
Information score
1
Decision criticality
2
Separator rarity
1.500
Rare pairs separated
2
Look-ahead risk
no
Run budget remaining
6
Reserved confirmation runs
2
Evidence — 325f5a33 (8 turns, 8 tool calls)
✓ Build✕ Public tests✕ Hidden testsFAIL
Termination
max_turns
Turns
8
Tokens (in/out)
12357 / 667
Latency
7.3s

Files read

  • src/rankingContract.ts
  • src/rankJobs.ts
  • tests/rankJobs.test.ts
4 repeated-read dedup markers4 context compaction events
Tool calls (8)
  1. t1list_filessrc/rankJobs.ts src/rankingContract.ts
  2. t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
  3. t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
  4. t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
  5. t5read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  6. t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  7. t7read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  8. t8read_file(unchanged since your last read of tests/rankJobs.test.ts — content is identical to what you already have)
01Confirm

Experiment 2

FAIL

At candidate value: Context Policy

Why this experiment?

skills=baseline,contextPolicy=candidate (01) is provisional after 1 run(s) and separates 2 pair(s) of the 3 confirmed-remaining hypotheses — confirming it directly narrows the official diagnosis.

Eliminated

  • Skill
  • interactionSkill × Context Policy

Remaining

  • Context Policy
Technical details
Information score
1
Decision criticality
2
Separator rarity
0.000
Rare pairs separated
0
Look-ahead risk
no
Run budget remaining
5
Reserved confirmation runs
2
Evidence — b1fe752b (8 turns, 8 tool calls)
✓ Build✕ Public tests✕ Hidden testsFAIL
Termination
max_turns
Turns
8
Tokens (in/out)
12357 / 934
Latency
8.8s

Files read

  • src/rankingContract.ts
  • src/rankJobs.ts
  • tests/rankJobs.test.ts
4 repeated-read dedup markers4 context compaction events
Tool calls (8)
  1. t1list_filessrc/rankJobs.ts src/rankingContract.ts
  2. t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
  3. t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
  4. t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
  5. t5read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  6. t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  7. t7read_file(unchanged since your last read of tests/rankJobs.test.ts — content is identical to what you already have)
  8. t8read_file(unchanged since your last read of src/rankJobs.ts — content is identical to what you already have)

Conclusion

Diagnosis

Primary regression factor

Context Policy

Behavioral evidence

What the agent did differently

Computed from the recorded turn-by-turn traces of every run — not generated by a model. Nemotron’s own explanation is checked against it below.

Observed in the recorded traces

  • Baseline: 2/2 runs wrote code, 2/2 ran npm test; compaction occurred in 0/2 runs and the “unchanged” marker in 0/2.
  • Candidate (run 1051883a): at turn 4 the harness replaced the contract file src/rankingContract.ts (2,021 chars) with a 75-char placeholder. The agent then re-read it 4×, and each time the tool answered “unchanged since your last read … identical to what you already have” instead of returning the text.
  • Outcome: 4 of 4 failing runs lost a file this way; every failing run ended at the turn limit without running the tests or the build.

Baseline (full context)

2/2 passed · compaction in 0/2 · “unchanged” marker in 0/2 · ran tests in 2/2 · wrote code in 2/2

Run 1 4b5d1687

PASS · completed
  1. t1 ls src
  2. t2 read rankingContract.ts
  3. t3 read rankJobs.ts
  4. t4 read rankJobs.test.ts
  5. t5 write rankJobs.ts
  6. t6 npm test
  7. t7 build
  8. t8 done

Run 2 2888cf5a

PASS · completed
  1. t1 ls src
  2. t2 read rankingContract.ts
  3. t3 read rankJobs.ts
  4. t4 read rankJobs.test.ts
  5. t5 write rankJobs.ts
  6. t6 npm test
  7. t7 build
  8. t8 done

Candidate (compact context)

0/2 passed · compaction in 2/2 · “unchanged” marker in 2/2 · ran tests in 0/2 · wrote code in 1/2

Run 1 1051883a

FAIL · max_turns
  1. t1 read rankingContract.ts
  2. t2 read rankJobs.ts
  3. t3 read rankJobs.test.ts
  4. t4 write rankJobs.tscompacted: rankingContract.ts
  5. t5 read→“unchanged” rankingContract.ts
  6. t6 read→“unchanged” rankingContract.ts
  7. t7 read→“unchanged” rankingContract.ts
  8. t8 read→“unchanged” rankingContract.ts

Run 2 46ff1b64

FAIL · max_turns
  1. t1 ls src
  2. t2 read rankJobs.ts
  3. t3 read rankingContract.ts
  4. t4 read rankJobs.test.tscompacted: src
  5. t5 read→“unchanged” rankJobs.tscompacted: rankJobs.ts
  6. t6 read→“unchanged” rankingContract.tscompacted: rankingContract.ts
  7. t7 read→“unchanged” rankJobs.ts
  8. t8 read→“unchanged” rankJobs.ts

Isolating run: skills = baseline, contextPolicy = candidate

0/2 passed · compaction in 2/2 · “unchanged” marker in 2/2 · ran tests in 0/2 · wrote code in 0/2

Run 1 325f5a33

FAIL · max_turns
  1. t1 ls src
  2. t2 read rankingContract.ts
  3. t3 read rankJobs.ts
  4. t4 read rankJobs.test.tscompacted: src
  5. t5 read→“unchanged” rankingContract.tscompacted: rankingContract.ts
  6. t6 read→“unchanged” rankingContract.ts
  7. t7 read→“unchanged” rankingContract.tscompacted: rankJobs.ts, rankJobs.test.ts
  8. t8 read→“unchanged” rankJobs.test.ts

Run 2 b1fe752b

FAIL · max_turns
  1. t1 ls src
  2. t2 read rankingContract.ts
  3. t3 read rankJobs.ts
  4. t4 read rankJobs.test.tscompacted: src
  5. t5 read→“unchanged” rankingContract.tscompacted: rankingContract.ts
  6. t6 read→“unchanged” rankingContract.ts
  7. t7 read→“unchanged” rankJobs.test.tscompacted: rankJobs.ts, rankJobs.test.ts
  8. t8 read→“unchanged” rankJobs.ts

Each chip is one agent turn. Amber chips mark the turn where the compact policy replaced an earlier tool result with a placeholder; red chips are re-reads answered with the “unchanged” marker.

Observed vs. inferred

Checking Nemotron’s explanation against the traces

Statement-by-statement check of Nemotron's mechanism explanation against the recorded trajectories (per-turn tool calls, per-request compaction metadata, termination reasons) and against HarnessLab's own source (src/lib/agent/coding-agent.ts applyContextPolicy; src/lib/agent/tools.ts read_file). Written by a reviewer, not by Nemotron. Counts below are recomputed from the evidence by tests/demo-replay.test.ts.

  • Observed

    Context compaction replaced earlier tool results with bounded summaries.

    All 4 candidate-side runs (2 candidate, 2 isolating) log compaction events and no baseline run does. Example: run 1051883a, turn 4: src/rankingContract.ts 2021 chars -> 75-char placeholder. applyContextPolicy overwrites messages[i] before the request is sent, so the model's next request contained the placeholder.

  • Observed, with caveat

    After the first read of a file, later reads return only an 'unchanged' marker.

    All 4 failing runs received 4 marker responses each; the baseline runs received 0. Caveat: Nemotron attributes the marker to the context policy, but it is produced by the read_file tool (tools.ts) under BOTH policies; it never fired in the baseline because the baseline agent never re-read a file. It is a fixed mechanism, not the changed dimension. It turns compaction into a trap because it claims the content is 'identical to what you already have' while the copy in context is a placeholder.

  • Observed, with caveat

    The agent could no longer see the actual contract/source text.

    4 of 4 failing runs: src/rankingContract.ts was compacted and every later read of it returned the marker, never the full text again. Caveat: full request bodies are not stored, so what the model saw is derived from the recorded compaction metadata plus the code that applies it, not from a saved prompt.

  • Observed

    The agent looped on redundant reads and exhausted its turn budget.

    All 4 failing runs ended with terminationReason max_turns at 8 turns; each made 4 marker-returning re-reads. Baseline runs completed in 8 turns after writing, testing and building.

  • Observed, with caveat

    This prevented any meaningful progress.

    None of the 4 failing runs ran npm test or npm run build (baseline: 2 of 2 did, at turns 6 and 7). 3 of 4 never wrote a file. Caveat: candidate run 1051883a did write src/rankJobs.ts (739 chars vs about 2,000 in the passing runs) on turn 4, the same turn the contract was compacted, and never verified it; hidden tests failed.

  • Inferred

    The failure is caused by contextPolicy, not by the skill swap.

    Observed: with the ORIGINAL ApplyPilot skill and only contextPolicy changed (isolating configuration 01), 2 of 2 runs failed with the same compaction-then-marker pattern; baseline 2 of 2 passed. Inferred: that the candidate skill alone is harmless. The opposite single-change configuration (candidate skill + full context) was not run; the diagnosis rules out 'skills' and 'skills x contextPolicy' by elimination under the assumption that the baseline-skill + compact-context result transfers.

Limitations (5)
  • 2 runs per configuration. Frequencies, not probabilities.
  • The configuration that changes only the skill (candidate skill + full context) was not run, so 'the skill swap alone is harmless' is inferred by elimination, not observed.
  • Full model request bodies are not stored; 'the model saw the placeholder' is derived from recorded per-request compaction metadata and the code that applies it.
  • The 'unchanged' marker is a fixed tool behavior present in both harnesses; the regression needs compaction to make it harmful.
  • An earlier attempt of the same experiment (diagnosis cd45ff83) had an unstable baseline (1 of 3 runs passed) and correctly reported no attributable regression; it is kept on disk and not used here.

Replicated evidence

Stability

HarnessLab evaluates a stochastic system — every configuration below was checked against repeated runs, not a single sample.

Evidence quality

Every configuration that was decision-critical for the final diagnosis reached replicated (confirmed) evidence, across 6 total runs; no visited configuration was genuinely unstable.

high
00baseline
replicatedMostly pass
PASS · PASSFailure rate 0% over 2 runs
Runs (2)
Evidence — 4b5d1687 (8 turns, 7 tool calls)
✓ Build✓ Public tests✓ Hidden testsPASS
Termination
completed
Turns
8
Tokens (in/out)
17235 / 1520
Latency
10.9s

Files read

  • src/rankingContract.ts
  • src/rankJobs.ts
  • tests/rankJobs.test.ts

Files written

  • src/rankJobs.ts

Commands executed

  • npm test
  • npm run build
Tool calls (7)
  1. t1list_filessrc/rankJobs.ts src/rankingContract.ts
  2. t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
  3. t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
  4. t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
  5. t5write_filewrote 2009 chars to src/rankJobs.ts
  6. t6run_commandexit 0 stdout: > applypilot-rank-fixture@0.0.0 test > node --import tsx --test tests/*.test.ts TAP version 13 # Subtest: ranks APPLY above STRETCH above SKIP ok 1 - ranks APPLY above STRETCH above SKIP --- duration
  7. t7run_commandexit 0 stdout: > applypilot-rank-fixture@0.0.0 build > tsc --noEmit stderr:
Diff
diff --git a/src/rankJobs.ts b/src/rankJobs.ts
index 7c21ff1..63753f6 100644
--- a/src/rankJobs.ts
+++ b/src/rankJobs.ts
@@ -1,11 +1,66 @@
-import type { RankableJob } from "./rankingContract";
+import type { RankableJob, EvaluatedJob, PendingJob, Recommendation, Eligibility } from "./rankingContract";
 
 /**
  * Orders evaluated jobs for the candidate, best match first.
  *
- * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of the
- * ordering rules. See rankingContract.ts for the exact rules.
+ * Ranking rules (from rankingContract.ts):
+ *  1. evaluated jobs before pending jobs
+ *  2. recommendation tier: APPLY, then STRETCH, then SKIP
+ *  3. eligibility tier: "eligible" and "not_stated" first, then "unconfirmed", then "not_eligible"
+ *  4. higher score first
+ *  5. fewer major gaps first
+ *  6. id ascending, so equal jobs never swap between calls
+ *  Pending jobs are ordered among themselves by id ascending.
  */
 export function rankJobs(jobs: readonly RankableJob[]): RankableJob[] {
-  return [...jobs];
-}
+  // Create a copy to avoid mutating input
+  const evaluated: EvaluatedJob[] = [];
+  const pending: PendingJob[] = [];
+
+  for (const job of jobs) {
+    if (job.status === "evaluated") {
+      evaluated.push(job as EvaluatedJob);
+    } else {
+      pending.push(job as PendingJob);
+    }
+  }
+
+  // Order maps
+  const recOrder: Record<Recommendation, number> = {
+    APPLY: 0,
+    STRETCH: 1,
+    SKIP: 2,
+  };
+  const eligOrder: Record<Eligibility, number> = {
+    eligible: 0,
+    not_stated: 0,
+    unconfirmed: 1,
+    not_eligible: 2,
+  };
+
+  // Sort evaluated jobs
+  evaluated.sort((a, b) => {
+    // 1. recommendation tier
+    const recDiff = recOrder[a.recommendation] - recOrder[b.recommendation];
+    if (recDiff !== 0) return recDiff;
+
+    // 2. eligibility tier
+    const eligDiff = eligOrder[a.eligibility] - eligOrder[b.eligibility];
+    if (eligDiff !== 0) return eligDiff;
+
+    // 3. higher score first
+    if (a.score !== b.score) return b.score - a.score;
+
+    // 4. fewer major gaps first
+    if (a.majorGapCount !== b.majorGapCount) return a.majorGapCount - b.majorGapCount;
+
+    // 5. id ascending
+    return a.id.localeCompare(b.id);
+  });
+
+  // Sort pending jobs by id ascending
+  pending.sort((a, b) => a.id.localeCompare(b.id));
+
+  // Concatenate evaluated then pending
+  return [...evaluated, ...pending];
+}
\ No newline at end of file
Evidence — 2888cf5a (8 turns, 7 tool calls)
✓ Build✓ Public tests✓ Hidden testsPASS
Termination
completed
Turns
8
Tokens (in/out)
17195 / 1553
Latency
11.3s

Files read

  • src/rankingContract.ts
  • src/rankJobs.ts
  • tests/rankJobs.test.ts

Files written

  • src/rankJobs.ts

Commands executed

  • npm test
  • npm run build
Tool calls (7)
  1. t1list_filessrc/rankJobs.ts src/rankingContract.ts
  2. t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
  3. t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
  4. t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
  5. t5write_filewrote 2007 chars to src/rankJobs.ts
  6. t6run_commandexit 0 stdout: > applypilot-rank-fixture@0.0.0 test > node --import tsx --test tests/*.test.ts TAP version 13 # Subtest: ranks APPLY above STRETCH above SKIP ok 1 - ranks APPLY above STRETCH above SKIP --- duration
  7. t7run_commandexit 0 stdout: > applypilot-rank-fixture@0.0.0 build > tsc --noEmit stderr:
Diff
diff --git a/src/rankJobs.ts b/src/rankJobs.ts
index 7c21ff1..15f5087 100644
--- a/src/rankJobs.ts
+++ b/src/rankJobs.ts
@@ -1,11 +1,57 @@
-import type { RankableJob } from "./rankingContract";
+import type { RankableJob, EvaluatedJob, PendingJob, Recommendation, Eligibility } from "./rankingContract";
 
 /**
  * Orders evaluated jobs for the candidate, best match first.
  *
- * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of the
- * ordering rules. See rankingContract.ts for the exact rules.
+ * Ranking rules (from rankingContract.ts):
+ *  1. evaluated jobs before pending jobs.
+ *  2. recommendation tier: APPLY, then STRETCH, then SKIP.
+ *  3. eligibility tier: "eligible" and "not_stated" first, then "unconfirmed", then "not_eligible".
+ *  4. higher score first.
+ *  5. fewer major gaps first.
+ *  6. id ascending, so equal jobs never swap between calls.
+ *  Pending jobs are ordered among themselves by id ascending.
  */
 export function rankJobs(jobs: readonly RankableJob[]): RankableJob[] {
-  return [...jobs];
-}
+  // Separate evaluated and pending jobs without mutating input
+  const evaluated: EvaluatedJob[] = [];
+  const pending: PendingJob[] = [];
+  for (const job of jobs) {
+    if (job.status === "evaluated") {
+      evaluated.push(job as EvaluatedJob);
+    } else {
+      pending.push(job as PendingJob);
+    }
+  }
+
+  // Order maps
+  const recOrder: { [r in Recommendation]: number } = {
+    APPLY: 0,
+    STRETCH: 1,
+    SKIP: 2,
+  };
+  const eligOrder: { [e in Eligibility]: number } = {
+    eligible: 0,
+    not_stated: 0,
+    unconfirmed: 1,
+    not_eligible: 2,
+  };
+
+  // Sort evaluated jobs
+  evaluated.sort((a, b) => {
+    if (recOrder[a.recommendation] !== recOrder[b.recommendation])
+      return recOrder[a.recommendation] - recOrder[b.recommendation];
+    if (eligOrder[a.eligibility] !== eligOrder[b.eligibility])
+      return eligOrder[a.eligibility] - eligOrder[b.eligibility];
+    if (a.score !== b.score) return b.score - a.score; // higher score first
+    if (a.majorGapCount !== b.majorGapCount)
+      return a.majorGapCount - b.majorGapCount; // fewer gaps first
+    return a.id.localeCompare(b.id); // id ascending
+  });
+
+  // Sort pending jobs by id ascending
+  pending.sort((a, b) => a.id.localeCompare(b.id));
+
+  // Return new array: evaluated first, then pending
+  return [...evaluated, ...pending];
+}
\ No newline at end of file
11candidate
replicatedMostly fail
FAIL · FAILFailure rate 100% over 2 runs
Runs (2)
Evidence — 1051883a (8 turns, 8 tool calls)
✕ Build✓ Public tests✕ Hidden testsFAIL
Termination
max_turns
Turns
8
Tokens (in/out)
13674 / 716
Latency
7.5s

Files read

  • src/rankingContract.ts
  • src/rankJobs.ts
  • tests/rankJobs.test.ts

Files written

  • src/rankJobs.ts
4 repeated-read dedup markers1 context compaction event
Tool calls (8)
  1. t1read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
  2. t2read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
  3. t3read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
  4. t4write_filewrote 739 chars to src/rankJobs.ts
  5. t5read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  6. t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  7. t7read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  8. t8read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
Diff
diff --git a/src/rankJobs.ts b/src/rankJobs.ts
index 7c21ff1..859b1a7 100644
--- a/src/rankJobs.ts
+++ b/src/rankJobs.ts
@@ -1,11 +1,25 @@
 import type { RankableJob } from "./rankingContract";
 
+const recommendationOrder: { [key in Recommendation]: number } = {
+  APPLY: 0,
+  STRETCH: 1,
+  SKIP: 2,
+};
+
 /**
  * Orders evaluated jobs for the candidate, best match first.
- *
- * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of the
- * ordering rules. See rankingContract.ts for the exact rules.
  */
 export function rankJobs(jobs: readonly RankableJob[]): RankableJob[] {
-  return [...jobs];
-}
+  // Create a copy to avoid mutating input
+  const copy = [...jobs];
+  copy.sort((a, b) => {
+    const orderA = recommendationOrder[a.recommendation];
+    const orderB = recommendationOrder[b.recommendation];
+    if (orderA !== orderB) {
+      return orderA - orderB; // lower order first (APPLY before STRETCH before SKIP)
+    }
+    // Same recommendation tier: higher score first
+    return b.score - a.score;
+  });
+  return copy;
+}
\ No newline at end of file
Evidence — 46ff1b64 (8 turns, 8 tool calls)
✓ Build✕ Public tests✕ Hidden testsFAIL
Termination
max_turns
Turns
8
Tokens (in/out)
12173 / 644
Latency
7.2s

Files read

  • src/rankJobs.ts
  • src/rankingContract.ts
  • tests/rankJobs.test.ts
4 repeated-read dedup markers3 context compaction events
Tool calls (8)
  1. t1list_filessrc/rankJobs.ts src/rankingContract.ts
  2. t2read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
  3. t3read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
  4. t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
  5. t5read_file(unchanged since your last read of src/rankJobs.ts — content is identical to what you already have)
  6. t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  7. t7read_file(unchanged since your last read of src/rankJobs.ts — content is identical to what you already have)
  8. t8read_file(unchanged since your last read of src/rankJobs.ts — content is identical to what you already have)
01skills=baseline,contextPolicy=candidate
replicatedMostly fail
FAIL · FAILFailure rate 100% over 2 runs
Runs (2)
Evidence — 325f5a33 (8 turns, 8 tool calls)
✓ Build✕ Public tests✕ Hidden testsFAIL
Termination
max_turns
Turns
8
Tokens (in/out)
12357 / 667
Latency
7.3s

Files read

  • src/rankingContract.ts
  • src/rankJobs.ts
  • tests/rankJobs.test.ts
4 repeated-read dedup markers4 context compaction events
Tool calls (8)
  1. t1list_filessrc/rankJobs.ts src/rankingContract.ts
  2. t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
  3. t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
  4. t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
  5. t5read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  6. t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  7. t7read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  8. t8read_file(unchanged since your last read of tests/rankJobs.test.ts — content is identical to what you already have)
Evidence — b1fe752b (8 turns, 8 tool calls)
✓ Build✕ Public tests✕ Hidden testsFAIL
Termination
max_turns
Turns
8
Tokens (in/out)
12357 / 934
Latency
8.8s

Files read

  • src/rankingContract.ts
  • src/rankJobs.ts
  • tests/rankJobs.test.ts
4 repeated-read dedup markers4 context compaction events
Tool calls (8)
  1. t1list_filessrc/rankJobs.ts src/rankingContract.ts
  2. t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
  3. t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
  4. t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
  5. t5read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  6. t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
  7. t7read_file(unchanged since your last read of tests/rankJobs.test.ts — content is identical to what you already have)
  8. t8read_file(unchanged since your last read of src/rankJobs.ts — content is identical to what you already have)

Nemotron

What happened?

What happened?

Generated by Nemotron from recorded trajectories

The regression stems from the altered contextPolicy, which activates aggressive deduplication and context‑compaction steps. After the first `read_file` of a given source, the policy tags the content as “unchanged” and returns only that marker on subsequent reads, while also swapping the full file text for a bounded summary via context compaction. Consequently, later attempts to re‑read the same file yield the unchanged marker or a truncated summary instead of the actual source code, depriving the agent of the detailed information it needs to proceed. This loss of fidelity causes the agent to loop on redundant reads, exhaust its turn budget without making meaningful progress, and ultimately fail the task. The pattern is consistent across all replicated failing runs, confirming the mechanism.

Fixed mechanisms observed (unchanged between baseline and candidate, but part of how the failure manifested)

  • read-file deduplication returned an "unchanged" marker instead of file content 4 time(s)
  • context compaction replaced earlier tool results with bounded summaries 1 time(s)
  • context compaction replaced earlier tool results with bounded summaries 3 time(s)
  • the agent exhausted its turn budget without ever calling write_file
  • context compaction replaced earlier tool results with bounded summaries 4 time(s)

Infrastructure

Sandbox evidence

Local sandbox (dev/offline runner)NVIDIA Nemotron
Shared base checkpoint
/var/fol…ULhr
Base image
unavailable
Isolated experiments
3

forked from the shared checkpoint

Total coding-agent runs
6
Exploratory runs
1
Confirmation runs
1
Shared setup time
0.2s

Every experiment forks from the same prepared checkpoint — hidden evaluator tests (build, public and hidden test suites) run inside the isolated fork, never against the shared base.