HarnessLab
Find what actually broke your coding agent.
- Task
- ApplyPilot: rank evaluated jobs by candidate fit
- Model
- nvidia/nemotron-3-super-120b-a12b
- Fixture
- applypilot-rank-001
- Runner
- Nebius Token Factory Sandbox
A coding agent working on an ApplyPilot-style ranking task
What this is
- A real HarnessLab experiment: Nemotron coding-agent runs (model calls via Nebius Token Factory; code executed in isolated local temp workspaces, not Token Factory Sandboxes) against a fixture derived from ApplyPilot-AI's job-ranking rules.
- The agent under test is the coding agent that would work on ApplyPilot; the harness under test is how it is guided (skill, context policy).
- Saved evidence replayed from disk: this page makes no API calls.
What this is not
- Not a live integration: no ApplyPilot code, database, network or user data is touched, and ApplyPilot is not modified.
- Not ApplyPilot's production job-ranking feature or its AI features being tested.
- Not a general benchmark: one task, 2 runs per configuration.
Overview
What HarnessLab found
Baseline Agent
PASSHarness changed
2 dimensions
Candidate Agent
FAIL4 possible configurations
2^2 combinations
HarnessLab selected
3 of 4 — 25% avoided
Diagnosis
Context Policy
Baseline
2/2 PASS
build, public and hidden tests
Candidate
2/2 FAIL
same task, same evaluator
Harness dimensions changed
2
Skill + Context Policy
Root-cause attribution
Context Policy
main effect
Evidence quality
HIGH
6 recorded coding-agent runs
Controlled experiment
Baseline vs. Candidate
Same task, same fixture, same evaluator — only the harness configuration differs.
Baseline
PASS- Skill
- applypilot-job-evaluation-v1
- Context Policy
- full
- System Prompt
- You are a careful senior software engineer working inside a small, iso…
- Max Turns
- 8
Full skill content
ApplyPilot job-evaluation rules (apply them to any ranking, scoring or matching code): 1. Read the domain contract file for the module you are changing and implement every rule in it, in the order it states them. Do not rely on the public tests alone; they are a subset. 2. Never treat silence as qualification, and never treat unknown as blocked: an unknown or unconfirmed fact keeps its own tier between 'confirmed' and 'blocked'. 3. A candidate must not be pointed at a role they cannot take: eligibility blockers outrank raw score. 4. A pending (not yet evaluated) job has no score; it is never ranked as if it had a low score. 5. Results must be reproducible: break every tie deterministically and do not mutate inputs. 6. Before finishing, re-check your implementation against each numbered rule in the contract, then run the tests and the build.
Full system prompt
You are a careful senior software engineer working inside a small, isolated Git repository. You can only interact with the repository through the provided tools. Read the relevant files first, then make the smallest correct change. Run the tests before you consider the task finished. When you are satisfied the task is complete, reply with a short plain-text summary and do not call any more tools.
Changed vs. the other arm: Skill, Context Policy
Candidate
FAIL- Skill
- verification-skill-v2
- Context Policy
- compact
- System Prompt
- You are a careful senior software engineer working inside a small, iso…
- Max Turns
- 8
Full skill content
Workflow for this task (optimized for efficiency): 1. Use the repository context already available to you to identify the affected area before exploring further. 2. Avoid re-reading files or information that is already available to you in the current conversation context. 3. If the context already available to you is sufficient to implement the change correctly, implement it directly rather than re-verifying details you already have. 4. Run targeted validation (the relevant tests) before considering the task complete.
Full system prompt
You are a careful senior software engineer working inside a small, isolated Git repository. You can only interact with the repository through the provided tools. Read the relevant files first, then make the smallest correct change. Run the tests before you consider the task finished. When you are satisfied the task is complete, reply with a short plain-text summary and do not call any more tools.
Changed vs. the other arm: Skill, Context Policy
What changed
Harness diff
2 dimensions changed between baseline and candidate.
Skill
Show full values
Baseline
ApplyPilot job-evaluation rules (apply them to any ranking, scoring or matching code): 1. Read the domain contract file for the module you are changing and implement every rule in it, in the order it states them. Do not rely on the public tests alone; they are a subset. 2. Never treat silence as qualification, and never treat unknown as blocked: an unknown or unconfirmed fact keeps its own tier between 'confirmed' and 'blocked'. 3. A candidate must not be pointed at a role they cannot take: eligibility blockers outrank raw score. 4. A pending (not yet evaluated) job has no score; it is never ranked as if it had a low score. 5. Results must be reproducible: break every tie deterministically and do not mutate inputs. 6. Before finishing, re-check your implementation against each numbered rule in the contract, then run the tests and the build.
Candidate
Workflow for this task (optimized for efficiency): 1. Use the repository context already available to you to identify the affected area before exploring further. 2. Avoid re-reading files or information that is already available to you in the current conversation context. 3. If the context already available to you is sufficient to implement the change correctly, implement it directly rather than re-verifying details you already have. 4. Run targeted validation (the relevant tests) before considering the task complete.
Context Policy
Search space
Configurations considered
Every possible combination of the changed dimensions, and which ones HarnessLab actually ran.
- Dimensions changed
- 2
- Possible configurations
- 4
- HarnessLab executed
- 3
- Avoided
- 1
2^2
25% search-space reduction
How it decided
Diagnosis trace
Each experiment HarnessLab chose to run, why it chose it, and what it eliminated.
Experiment 1
At candidate value: Context Policy
Why this experiment?
Selected because it is one of only 1 remaining configuration(s) that distinguishes: interaction:skills+contextPolicy vs main_effect:contextPolicy.
Eliminated
Nothing newly eliminated
Remaining
- Skill
- Context Policy
- interactionSkill × Context Policy
Technical details
- Information score
- 1
- Decision criticality
- 2
- Separator rarity
- 1.500
- Rare pairs separated
- 2
- Look-ahead risk
- no
- Run budget remaining
- 6
- Reserved confirmation runs
- 2
Evidence — 325f5a33 (8 turns, 8 tool calls)
- Termination
- max_turns
- Turns
- 8
- Tokens (in/out)
- 12357 / 667
- Latency
- 7.3s
Files read
- src/rankingContract.ts
- src/rankJobs.ts
- tests/rankJobs.test.ts
Tool calls (8)
- t1list_filessrc/rankJobs.ts src/rankingContract.ts
- t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
- t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
- t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
- t5read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t7read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t8read_file(unchanged since your last read of tests/rankJobs.test.ts — content is identical to what you already have)
Experiment 2
At candidate value: Context Policy
Why this experiment?
skills=baseline,contextPolicy=candidate (01) is provisional after 1 run(s) and separates 2 pair(s) of the 3 confirmed-remaining hypotheses — confirming it directly narrows the official diagnosis.
Eliminated
- Skill
- interactionSkill × Context Policy
Remaining
- Context Policy
Technical details
- Information score
- 1
- Decision criticality
- 2
- Separator rarity
- 0.000
- Rare pairs separated
- 0
- Look-ahead risk
- no
- Run budget remaining
- 5
- Reserved confirmation runs
- 2
Evidence — b1fe752b (8 turns, 8 tool calls)
- Termination
- max_turns
- Turns
- 8
- Tokens (in/out)
- 12357 / 934
- Latency
- 8.8s
Files read
- src/rankingContract.ts
- src/rankJobs.ts
- tests/rankJobs.test.ts
Tool calls (8)
- t1list_filessrc/rankJobs.ts src/rankingContract.ts
- t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
- t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
- t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
- t5read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t7read_file(unchanged since your last read of tests/rankJobs.test.ts — content is identical to what you already have)
- t8read_file(unchanged since your last read of src/rankJobs.ts — content is identical to what you already have)
Conclusion
Diagnosis
Primary regression factor
Context Policy
Behavioral evidence
What the agent did differently
Computed from the recorded turn-by-turn traces of every run — not generated by a model. Nemotron’s own explanation is checked against it below.
Observed in the recorded traces
- Baseline: 2/2 runs wrote code, 2/2 ran
npm test; compaction occurred in 0/2 runs and the “unchanged” marker in 0/2. - Candidate (run 1051883a): at turn 4 the harness replaced the contract file
src/rankingContract.ts(2,021 chars) with a 75-char placeholder. The agent then re-read it 4×, and each time the tool answered “unchanged since your last read … identical to what you already have” instead of returning the text. - Outcome: 4 of 4 failing runs lost a file this way; every failing run ended at the turn limit without running the tests or the build.
Baseline (full context)
2/2 passed · compaction in 0/2 · “unchanged” marker in 0/2 · ran tests in 2/2 · wrote code in 2/2
Run 1 4b5d1687
PASS · completed- t1 ls src
- t2 read rankingContract.ts
- t3 read rankJobs.ts
- t4 read rankJobs.test.ts
- t5 write rankJobs.ts
- t6 npm test
- t7 build
- t8 done
Run 2 2888cf5a
PASS · completed- t1 ls src
- t2 read rankingContract.ts
- t3 read rankJobs.ts
- t4 read rankJobs.test.ts
- t5 write rankJobs.ts
- t6 npm test
- t7 build
- t8 done
Candidate (compact context)
0/2 passed · compaction in 2/2 · “unchanged” marker in 2/2 · ran tests in 0/2 · wrote code in 1/2
Run 1 1051883a
FAIL · max_turns- t1 read rankingContract.ts
- t2 read rankJobs.ts
- t3 read rankJobs.test.ts
- t4 write rankJobs.tscompacted: rankingContract.ts
- t5 read→“unchanged” rankingContract.ts
- t6 read→“unchanged” rankingContract.ts
- t7 read→“unchanged” rankingContract.ts
- t8 read→“unchanged” rankingContract.ts
Run 2 46ff1b64
FAIL · max_turns- t1 ls src
- t2 read rankJobs.ts
- t3 read rankingContract.ts
- t4 read rankJobs.test.tscompacted: src
- t5 read→“unchanged” rankJobs.tscompacted: rankJobs.ts
- t6 read→“unchanged” rankingContract.tscompacted: rankingContract.ts
- t7 read→“unchanged” rankJobs.ts
- t8 read→“unchanged” rankJobs.ts
Isolating run: skills = baseline, contextPolicy = candidate
0/2 passed · compaction in 2/2 · “unchanged” marker in 2/2 · ran tests in 0/2 · wrote code in 0/2
Run 1 325f5a33
FAIL · max_turns- t1 ls src
- t2 read rankingContract.ts
- t3 read rankJobs.ts
- t4 read rankJobs.test.tscompacted: src
- t5 read→“unchanged” rankingContract.tscompacted: rankingContract.ts
- t6 read→“unchanged” rankingContract.ts
- t7 read→“unchanged” rankingContract.tscompacted: rankJobs.ts, rankJobs.test.ts
- t8 read→“unchanged” rankJobs.test.ts
Run 2 b1fe752b
FAIL · max_turns- t1 ls src
- t2 read rankingContract.ts
- t3 read rankJobs.ts
- t4 read rankJobs.test.tscompacted: src
- t5 read→“unchanged” rankingContract.tscompacted: rankingContract.ts
- t6 read→“unchanged” rankingContract.ts
- t7 read→“unchanged” rankJobs.test.tscompacted: rankJobs.ts, rankJobs.test.ts
- t8 read→“unchanged” rankJobs.ts
Each chip is one agent turn. Amber chips mark the turn where the compact policy replaced an earlier tool result with a placeholder; red chips are re-reads answered with the “unchanged” marker.
Observed vs. inferred
Checking Nemotron’s explanation against the traces
Statement-by-statement check of Nemotron's mechanism explanation against the recorded trajectories (per-turn tool calls, per-request compaction metadata, termination reasons) and against HarnessLab's own source (src/lib/agent/coding-agent.ts applyContextPolicy; src/lib/agent/tools.ts read_file). Written by a reviewer, not by Nemotron. Counts below are recomputed from the evidence by tests/demo-replay.test.ts.
- Observed
Context compaction replaced earlier tool results with bounded summaries.
All 4 candidate-side runs (2 candidate, 2 isolating) log compaction events and no baseline run does. Example: run 1051883a, turn 4: src/rankingContract.ts 2021 chars -> 75-char placeholder. applyContextPolicy overwrites messages[i] before the request is sent, so the model's next request contained the placeholder.
- Observed, with caveat
After the first read of a file, later reads return only an 'unchanged' marker.
All 4 failing runs received 4 marker responses each; the baseline runs received 0. Caveat: Nemotron attributes the marker to the context policy, but it is produced by the read_file tool (tools.ts) under BOTH policies; it never fired in the baseline because the baseline agent never re-read a file. It is a fixed mechanism, not the changed dimension. It turns compaction into a trap because it claims the content is 'identical to what you already have' while the copy in context is a placeholder.
- Observed, with caveat
The agent could no longer see the actual contract/source text.
4 of 4 failing runs: src/rankingContract.ts was compacted and every later read of it returned the marker, never the full text again. Caveat: full request bodies are not stored, so what the model saw is derived from the recorded compaction metadata plus the code that applies it, not from a saved prompt.
- Observed
The agent looped on redundant reads and exhausted its turn budget.
All 4 failing runs ended with terminationReason max_turns at 8 turns; each made 4 marker-returning re-reads. Baseline runs completed in 8 turns after writing, testing and building.
- Observed, with caveat
This prevented any meaningful progress.
None of the 4 failing runs ran npm test or npm run build (baseline: 2 of 2 did, at turns 6 and 7). 3 of 4 never wrote a file. Caveat: candidate run 1051883a did write src/rankJobs.ts (739 chars vs about 2,000 in the passing runs) on turn 4, the same turn the contract was compacted, and never verified it; hidden tests failed.
- Inferred
The failure is caused by contextPolicy, not by the skill swap.
Observed: with the ORIGINAL ApplyPilot skill and only contextPolicy changed (isolating configuration 01), 2 of 2 runs failed with the same compaction-then-marker pattern; baseline 2 of 2 passed. Inferred: that the candidate skill alone is harmless. The opposite single-change configuration (candidate skill + full context) was not run; the diagnosis rules out 'skills' and 'skills x contextPolicy' by elimination under the assumption that the baseline-skill + compact-context result transfers.
Limitations (5)
- 2 runs per configuration. Frequencies, not probabilities.
- The configuration that changes only the skill (candidate skill + full context) was not run, so 'the skill swap alone is harmless' is inferred by elimination, not observed.
- Full model request bodies are not stored; 'the model saw the placeholder' is derived from recorded per-request compaction metadata and the code that applies it.
- The 'unchanged' marker is a fixed tool behavior present in both harnesses; the regression needs compaction to make it harmful.
- An earlier attempt of the same experiment (diagnosis cd45ff83) had an unstable baseline (1 of 3 runs passed) and correctly reported no attributable regression; it is kept on disk and not used here.
Replicated evidence
Stability
HarnessLab evaluates a stochastic system — every configuration below was checked against repeated runs, not a single sample.
Evidence quality
Every configuration that was decision-critical for the final diagnosis reached replicated (confirmed) evidence, across 6 total runs; no visited configuration was genuinely unstable.
Runs (2)
Evidence — 4b5d1687 (8 turns, 7 tool calls)
- Termination
- completed
- Turns
- 8
- Tokens (in/out)
- 17235 / 1520
- Latency
- 10.9s
Files read
- src/rankingContract.ts
- src/rankJobs.ts
- tests/rankJobs.test.ts
Files written
- src/rankJobs.ts
Commands executed
- npm test
- npm run build
Tool calls (7)
- t1list_filessrc/rankJobs.ts src/rankingContract.ts
- t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
- t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
- t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
- t5write_filewrote 2009 chars to src/rankJobs.ts
- t6run_commandexit 0 stdout: > applypilot-rank-fixture@0.0.0 test > node --import tsx --test tests/*.test.ts TAP version 13 # Subtest: ranks APPLY above STRETCH above SKIP ok 1 - ranks APPLY above STRETCH above SKIP --- duration
- t7run_commandexit 0 stdout: > applypilot-rank-fixture@0.0.0 build > tsc --noEmit stderr:
Diff
diff --git a/src/rankJobs.ts b/src/rankJobs.ts
index 7c21ff1..63753f6 100644
--- a/src/rankJobs.ts
+++ b/src/rankJobs.ts
@@ -1,11 +1,66 @@
-import type { RankableJob } from "./rankingContract";
+import type { RankableJob, EvaluatedJob, PendingJob, Recommendation, Eligibility } from "./rankingContract";
/**
* Orders evaluated jobs for the candidate, best match first.
*
- * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of the
- * ordering rules. See rankingContract.ts for the exact rules.
+ * Ranking rules (from rankingContract.ts):
+ * 1. evaluated jobs before pending jobs
+ * 2. recommendation tier: APPLY, then STRETCH, then SKIP
+ * 3. eligibility tier: "eligible" and "not_stated" first, then "unconfirmed", then "not_eligible"
+ * 4. higher score first
+ * 5. fewer major gaps first
+ * 6. id ascending, so equal jobs never swap between calls
+ * Pending jobs are ordered among themselves by id ascending.
*/
export function rankJobs(jobs: readonly RankableJob[]): RankableJob[] {
- return [...jobs];
-}
+ // Create a copy to avoid mutating input
+ const evaluated: EvaluatedJob[] = [];
+ const pending: PendingJob[] = [];
+
+ for (const job of jobs) {
+ if (job.status === "evaluated") {
+ evaluated.push(job as EvaluatedJob);
+ } else {
+ pending.push(job as PendingJob);
+ }
+ }
+
+ // Order maps
+ const recOrder: Record<Recommendation, number> = {
+ APPLY: 0,
+ STRETCH: 1,
+ SKIP: 2,
+ };
+ const eligOrder: Record<Eligibility, number> = {
+ eligible: 0,
+ not_stated: 0,
+ unconfirmed: 1,
+ not_eligible: 2,
+ };
+
+ // Sort evaluated jobs
+ evaluated.sort((a, b) => {
+ // 1. recommendation tier
+ const recDiff = recOrder[a.recommendation] - recOrder[b.recommendation];
+ if (recDiff !== 0) return recDiff;
+
+ // 2. eligibility tier
+ const eligDiff = eligOrder[a.eligibility] - eligOrder[b.eligibility];
+ if (eligDiff !== 0) return eligDiff;
+
+ // 3. higher score first
+ if (a.score !== b.score) return b.score - a.score;
+
+ // 4. fewer major gaps first
+ if (a.majorGapCount !== b.majorGapCount) return a.majorGapCount - b.majorGapCount;
+
+ // 5. id ascending
+ return a.id.localeCompare(b.id);
+ });
+
+ // Sort pending jobs by id ascending
+ pending.sort((a, b) => a.id.localeCompare(b.id));
+
+ // Concatenate evaluated then pending
+ return [...evaluated, ...pending];
+}
\ No newline at end of file
Evidence — 2888cf5a (8 turns, 7 tool calls)
- Termination
- completed
- Turns
- 8
- Tokens (in/out)
- 17195 / 1553
- Latency
- 11.3s
Files read
- src/rankingContract.ts
- src/rankJobs.ts
- tests/rankJobs.test.ts
Files written
- src/rankJobs.ts
Commands executed
- npm test
- npm run build
Tool calls (7)
- t1list_filessrc/rankJobs.ts src/rankingContract.ts
- t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
- t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
- t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
- t5write_filewrote 2007 chars to src/rankJobs.ts
- t6run_commandexit 0 stdout: > applypilot-rank-fixture@0.0.0 test > node --import tsx --test tests/*.test.ts TAP version 13 # Subtest: ranks APPLY above STRETCH above SKIP ok 1 - ranks APPLY above STRETCH above SKIP --- duration
- t7run_commandexit 0 stdout: > applypilot-rank-fixture@0.0.0 build > tsc --noEmit stderr:
Diff
diff --git a/src/rankJobs.ts b/src/rankJobs.ts
index 7c21ff1..15f5087 100644
--- a/src/rankJobs.ts
+++ b/src/rankJobs.ts
@@ -1,11 +1,57 @@
-import type { RankableJob } from "./rankingContract";
+import type { RankableJob, EvaluatedJob, PendingJob, Recommendation, Eligibility } from "./rankingContract";
/**
* Orders evaluated jobs for the candidate, best match first.
*
- * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of the
- * ordering rules. See rankingContract.ts for the exact rules.
+ * Ranking rules (from rankingContract.ts):
+ * 1. evaluated jobs before pending jobs.
+ * 2. recommendation tier: APPLY, then STRETCH, then SKIP.
+ * 3. eligibility tier: "eligible" and "not_stated" first, then "unconfirmed", then "not_eligible".
+ * 4. higher score first.
+ * 5. fewer major gaps first.
+ * 6. id ascending, so equal jobs never swap between calls.
+ * Pending jobs are ordered among themselves by id ascending.
*/
export function rankJobs(jobs: readonly RankableJob[]): RankableJob[] {
- return [...jobs];
-}
+ // Separate evaluated and pending jobs without mutating input
+ const evaluated: EvaluatedJob[] = [];
+ const pending: PendingJob[] = [];
+ for (const job of jobs) {
+ if (job.status === "evaluated") {
+ evaluated.push(job as EvaluatedJob);
+ } else {
+ pending.push(job as PendingJob);
+ }
+ }
+
+ // Order maps
+ const recOrder: { [r in Recommendation]: number } = {
+ APPLY: 0,
+ STRETCH: 1,
+ SKIP: 2,
+ };
+ const eligOrder: { [e in Eligibility]: number } = {
+ eligible: 0,
+ not_stated: 0,
+ unconfirmed: 1,
+ not_eligible: 2,
+ };
+
+ // Sort evaluated jobs
+ evaluated.sort((a, b) => {
+ if (recOrder[a.recommendation] !== recOrder[b.recommendation])
+ return recOrder[a.recommendation] - recOrder[b.recommendation];
+ if (eligOrder[a.eligibility] !== eligOrder[b.eligibility])
+ return eligOrder[a.eligibility] - eligOrder[b.eligibility];
+ if (a.score !== b.score) return b.score - a.score; // higher score first
+ if (a.majorGapCount !== b.majorGapCount)
+ return a.majorGapCount - b.majorGapCount; // fewer gaps first
+ return a.id.localeCompare(b.id); // id ascending
+ });
+
+ // Sort pending jobs by id ascending
+ pending.sort((a, b) => a.id.localeCompare(b.id));
+
+ // Return new array: evaluated first, then pending
+ return [...evaluated, ...pending];
+}
\ No newline at end of file
Runs (2)
Evidence — 1051883a (8 turns, 8 tool calls)
- Termination
- max_turns
- Turns
- 8
- Tokens (in/out)
- 13674 / 716
- Latency
- 7.5s
Files read
- src/rankingContract.ts
- src/rankJobs.ts
- tests/rankJobs.test.ts
Files written
- src/rankJobs.ts
Tool calls (8)
- t1read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
- t2read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
- t3read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
- t4write_filewrote 739 chars to src/rankJobs.ts
- t5read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t7read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t8read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
Diff
diff --git a/src/rankJobs.ts b/src/rankJobs.ts
index 7c21ff1..859b1a7 100644
--- a/src/rankJobs.ts
+++ b/src/rankJobs.ts
@@ -1,11 +1,25 @@
import type { RankableJob } from "./rankingContract";
+const recommendationOrder: { [key in Recommendation]: number } = {
+ APPLY: 0,
+ STRETCH: 1,
+ SKIP: 2,
+};
+
/**
* Orders evaluated jobs for the candidate, best match first.
- *
- * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of the
- * ordering rules. See rankingContract.ts for the exact rules.
*/
export function rankJobs(jobs: readonly RankableJob[]): RankableJob[] {
- return [...jobs];
-}
+ // Create a copy to avoid mutating input
+ const copy = [...jobs];
+ copy.sort((a, b) => {
+ const orderA = recommendationOrder[a.recommendation];
+ const orderB = recommendationOrder[b.recommendation];
+ if (orderA !== orderB) {
+ return orderA - orderB; // lower order first (APPLY before STRETCH before SKIP)
+ }
+ // Same recommendation tier: higher score first
+ return b.score - a.score;
+ });
+ return copy;
+}
\ No newline at end of file
Evidence — 46ff1b64 (8 turns, 8 tool calls)
- Termination
- max_turns
- Turns
- 8
- Tokens (in/out)
- 12173 / 644
- Latency
- 7.2s
Files read
- src/rankJobs.ts
- src/rankingContract.ts
- tests/rankJobs.test.ts
Tool calls (8)
- t1list_filessrc/rankJobs.ts src/rankingContract.ts
- t2read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
- t3read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
- t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
- t5read_file(unchanged since your last read of src/rankJobs.ts — content is identical to what you already have)
- t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t7read_file(unchanged since your last read of src/rankJobs.ts — content is identical to what you already have)
- t8read_file(unchanged since your last read of src/rankJobs.ts — content is identical to what you already have)
Runs (2)
Evidence — 325f5a33 (8 turns, 8 tool calls)
- Termination
- max_turns
- Turns
- 8
- Tokens (in/out)
- 12357 / 667
- Latency
- 7.3s
Files read
- src/rankingContract.ts
- src/rankJobs.ts
- tests/rankJobs.test.ts
Tool calls (8)
- t1list_filessrc/rankJobs.ts src/rankingContract.ts
- t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
- t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
- t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
- t5read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t7read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t8read_file(unchanged since your last read of tests/rankJobs.test.ts — content is identical to what you already have)
Evidence — b1fe752b (8 turns, 8 tool calls)
- Termination
- max_turns
- Turns
- 8
- Tokens (in/out)
- 12357 / 934
- Latency
- 8.8s
Files read
- src/rankingContract.ts
- src/rankJobs.ts
- tests/rankJobs.test.ts
Tool calls (8)
- t1list_filessrc/rankJobs.ts src/rankingContract.ts
- t2read_file/** * Job ranking contract. * * Derived from ApplyPilot-AI's ranked-jobs ordering (services/job-dashboard/ranking.ts, `compareRank`), * reduced to the fields below. Every job here has already been evaluated by ApplyP
- t3read_fileimport type { RankableJob } from "./rankingContract"; /** * Orders evaluated jobs for the candidate, best match first. * * TODO(applypilot-rank-001): stub. Returns the jobs in their given order and implements none of
- t4read_fileimport assert from "node:assert/strict"; import { test } from "node:test"; import { rankJobs } from "../src/rankJobs"; import type { RankableJob, Recommendation } from "../src/rankingContract"; function job(id: string,
- t5read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t6read_file(unchanged since your last read of src/rankingContract.ts — content is identical to what you already have)
- t7read_file(unchanged since your last read of tests/rankJobs.test.ts — content is identical to what you already have)
- t8read_file(unchanged since your last read of src/rankJobs.ts — content is identical to what you already have)
Nemotron
What happened?
What happened?
Generated by Nemotron from recorded trajectoriesThe regression stems from the altered contextPolicy, which activates aggressive deduplication and context‑compaction steps. After the first `read_file` of a given source, the policy tags the content as “unchanged” and returns only that marker on subsequent reads, while also swapping the full file text for a bounded summary via context compaction. Consequently, later attempts to re‑read the same file yield the unchanged marker or a truncated summary instead of the actual source code, depriving the agent of the detailed information it needs to proceed. This loss of fidelity causes the agent to loop on redundant reads, exhaust its turn budget without making meaningful progress, and ultimately fail the task. The pattern is consistent across all replicated failing runs, confirming the mechanism.
Fixed mechanisms observed (unchanged between baseline and candidate, but part of how the failure manifested)
- read-file deduplication returned an "unchanged" marker instead of file content 4 time(s)
- context compaction replaced earlier tool results with bounded summaries 1 time(s)
- context compaction replaced earlier tool results with bounded summaries 3 time(s)
- the agent exhausted its turn budget without ever calling write_file
- context compaction replaced earlier tool results with bounded summaries 4 time(s)
Infrastructure
Sandbox evidence
- Shared base checkpoint
- /var/fol…ULhr
- Base image
- unavailable
- Isolated experiments
- 3
- Total coding-agent runs
- 6
forked from the shared checkpoint
- Exploratory runs
- 1
- Confirmation runs
- 1
- Shared setup time
- 0.2s
Every experiment forks from the same prepared checkpoint — hidden evaluator tests (build, public and hidden test suites) run inside the isolated fork, never against the shared base.