fix: Prevent lost task completions after server restart

Three bugs causing empty phase diffs when server restarts during agent
execution:

1. Startup ordering race: reconcileAfterRestart() emitted agent:stopped
   before orchestrator registered listeners — events lost. Moved
   reconciliation to after orchestrator.start().

2. Stuck in_progress tasks: recoverDispatchQueues() only re-queued
   pending tasks. Added recovery for in_progress tasks whose agents
   are dead (not running/waiting_for_input).

3. Branch force-reset destroys work: git branch -f wiped commits when
   a second agent was dispatched for the same task. Now checks if the
   branch has commits beyond baseBranch before resetting.

Also adds:
- agent:crashed handler with auto-retry (MAX_TASK_RETRIES=3)
- retryCount column on tasks table + migration
- retryCount reset on manual retryBlockedTask()
This commit is contained in:
Lukas May
2026-03-06 12:19:59 +01:00
parent a69527b7d6
commit eac03862e3
9 changed files with 94 additions and 13 deletions

View File

@@ -70,10 +70,21 @@ export class SimpleGitWorktreeManager implements WorktreeManager {
// Create worktree — reuse existing branch or create new one
const branchExists = await this.branchExists(branch);
if (branchExists) {
// Branch exists from a previous run — reset it to baseBranch and check it out.
// Only safe because branch !== baseBranch (checked above), so we're resetting
// an agent-scoped branch, not a shared branch like main or the initiative branch.
await this.git.raw(['branch', '-f', branch, baseBranch]);
// Branch exists from a previous run. Check if it has commits beyond baseBranch
// before resetting — a previous agent may have done real work on this branch.
try {
const aheadCount = await this.git.raw(['rev-list', '--count', `${baseBranch}..${branch}`]);
if (parseInt(aheadCount.trim(), 10) > 0) {
log.warn({ branch, baseBranch, aheadBy: aheadCount.trim() }, 'branch has commits beyond base, preserving');
} else {
await this.git.raw(['branch', '-f', branch, baseBranch]);
}
} catch {
// If rev-list fails (e.g. baseBranch doesn't exist yet), fall back to reset
await this.git.raw(['branch', '-f', branch, baseBranch]);
}
// Prune stale worktree references before adding new one
await this.git.raw(['worktree', 'prune']);
await this.git.raw(['worktree', 'add', worktreePath, branch]);
} else {
// git worktree add -b <branch> <path> <base-branch>