Act as an Autonomous App Builder Harness Specification — ChatGPT Prompt

The Prompt

# APP-BUILDER HARNESS ## Autonomous Coding Agent Build Specification — V1 > Purpose: Give this entire specification to a coding agent. > > The agent must implement the project sequentially from M1 through M12. > > **ZERO clarification questions. ZERO unverified completion claims.** --- # 0. OPERATING CONTRACT You are an autonomous senior Python engineer. Build the **App-Builder Harness** exactly according to this specification. You MUST: 1. Work sequentially from **M1 → M12**. 2. Never ask the user for clarification. 3. Resolve ambiguity using the defaults defined here. 4. Run real tests and commands before declaring PASS. 5. Stop milestone progression whenever its acceptance tests are red. 6. Repair failures before proceeding. 7. Record evidence for every completed requirement. 8. Keep the implementation inside the V1 scope. 9. Never claim success based solely on LLM output. 10. Finish only when the complete V1 Final Gate passes. --- # 1. PROJECT GOAL Build a reliability-first autonomous software-engineering harness that transforms a natural-language application request into a working, tested, reviewed project. Canonical pipeline: ```text User Request ↓ Spec Agent ↓ Spec Validation ↓ Planner ↓ Task DAG ↓ Sequential Builder ↓ Real Verification ↓ Failure Classification ↓ Repair Loop ↓ Reverification ↓ Traceability Matrix ↓ Requirement Review ↓ Runtime Verification ↓ Final Deliverable ``` Example input: ```text Build me a task management web app with authentication, projects, tasks, and a dashboard. ``` --- # 2. CENTRAL INVARIANT — EVIDENCE OVER CLAIMS Every requirement MUST be traceable through: ```text REQ-ID ↓ TASK-ID ↓ FILE(S) ↓ TEST / COMMAND ↓ EXECUTION RESULT ↓ PASS EVIDENCE ``` A requirement is **not complete** merely because: * an LLM generated code; * a file exists; * an agent says it is finished; * static inspection looks correct. A PASS requires real evidence. Minimum PASS evidence: ```text implementation file exists + verification command executed + exit_code == 0 + result persisted ``` Never output: ```text AI says finished ``` as completion evidence. --- # 3. V1 SCOPE LOCK ## Harness runtime * Python 3.11+ * Pydantic >= 2.0 * pytest >= 7.0 * Python standard library for infrastructure whenever possible Allowed stdlib examples: ```text argparse asyncio hashlib json logging pathlib shlex socket sqlite3 subprocess time urllib uuid ``` ## Harness architecture Repository: ```text app-builder/ └── src/harness/ ``` State storage: ```text SQLite / JSON files ``` Sandbox: ```text LocalSandbox ``` Execution model: ```text Sequential DAG only ``` LLM abstraction: ```text LLMProvider ├── MockProvider └── EnvOpenAICompatibleProvider ``` --- # 4. STRICT V1 NON-GOALS DO NOT implement: ```text PostgreSQL Docker sandbox production deployment browser automation Playwright Selenium visual canvas web management UI parallel builders parallel DAG execution file-conflict scheduler multi-user support long-term agent memory internet research distributed workers ``` A browser verification module may exist only as an explicit post-V1 stub. --- # 5. DEPENDENCY BOUNDARY The dependency restriction: ```text pydantic>=2 pytest>=7 ``` applies to the **App-Builder Harness itself**. Generated applications may contain their own: ```text requirements.txt package.json ``` according to their inferred stack. Do not silently add dependencies to the harness. Whenever a new harness dependency is intentionally introduced, all of the following MUST be updated together: ```text pyproject.toml requirements.txt tests documentation ``` --- # 6. OFFLINE BEHAVIOR "No API key" MUST never block the pipeline. If: ```text OPENAI_API_KEY ``` is unavailable: ```text EnvOpenAICompatibleProvider ↓ MockProvider ↓ deterministic fallback implementation ``` The harness must therefore remain testable without an LLM connection. "Offline" in this specification means: ```text No external LLM/web call is required for harness correctness. ``` The Final Gate execution environment must already contain any runtime packages required to execute its generated reference application. Do not make network availability a prerequisite for core harness unit tests. --- # 7. DEFAULTS When ambiguity exists, DO NOT ask. Use these defaults. ## Stack inference ```text "web app" → frontend=React → backend=FastAPI → db=SQLite "API" → backend=FastAPI → db=SQLite "todo" → FastAPI + SQLite + minimal HTML unspecified → FastAPI + SQLite ``` ## App defaults ```text app_type = "web" ``` Infer pages from explicit nouns such as: ```text dashboard login projects tasks settings ``` Infer data entities from domain nouns. Every explicit feature noun becomes a `must_have` candidate. Default non-goals: ```text deployment mobile-app browser-automation ``` unless explicitly requested. ## Planning defaults Small applications: ```text 5–12 tasks ``` Each task SHOULD: ```text touch <= 5 files have >= 1 requirement have >= 1 verification command ``` ## Timeouts ```text sandbox command: 120 seconds runtime startup: 15 seconds repair attempts: maximum 3 ``` --- # 8. IDENTIFIER CONTRACT Use deterministic ID formats. ```text REQ-001 REQ-002 TASK-001 TASK-002 TEST-001 TEST-002 run_XXXXXXXX ``` Requirement IDs and Task IDs must be sequential within a run. Never reuse an ID for a different object. --- # 9. RUN ARTIFACT CONTRACT Each generated run lives under: ```text runs/<run_id>/ ``` Required final layout: ```text runs/<run_id>/ ├── state.json ├── spec.json ├── tasks.json ├── verification.json ├── review.json ├── events.log ├── checkpoints/ │ ├── 001-spec_created.json │ ├── 002-plan_created.json │ └── ... └── workspace/ ├── source code └── tests ``` The canonical final deliverable for every run is: ```text workspace/ spec.json tasks.json verification.json review.json events.log state.json ``` --- # 10. EVENT CONTRACT All meaningful state transitions append one JSON object to: ```text runs/<run_id>/events.log ``` Required fields: ```text run_id trace_id stage agent task_id timestamp duration_ms model tokens tool status error ``` Optional values may be empty, but keys must exist. --- # 11. STATUS CONTRACT ## TaskStatus ```text PENDING READY RUNNING COMPLETED FAILED TIMED_OUT BLOCKED ``` `READY` may be computed by the DAG engine rather than persisted. A persisted ready-but-not-started task may remain: ```text PENDING ``` ## RunStatus ```text CREATED SPEC_DONE PLAN_DONE BUILDING VERIFYING REPAIRING REVIEW_DONE PASS FAIL ``` Preserve the distinction: ```text FAILED != TIMED_OUT ``` --- # 12. BUILD ORDER The exact milestone order is: ```text M1 Infrastructure M2 Specification M3 Planning + DAG M4 Building M5 Verification M6 Self-Repair M7 Traceability M8 Requirement Review M9 Recovery M10 Runtime Verification M11 CLI + Benchmarks M12 Final Gate ``` Do not reorder. Do not start milestone `M(N+1)` while `M(N)` has failing acceptance tests. Run: ```bash pytest -q ``` after every milestone. --- # M1 — INFRASTRUCTURE ## Goal Create an installable repository containing: * data models; * run state; * providers; * local sandbox; * event logging; * deterministic IDs. No agents yet. ## Required structure ```text app-builder/ ├── src/harness/__init__.py ├── src/harness/models/spec.py ├── src/harness/models/tasks.py ├── src/harness/models/results.py ├── src/harness/models/run.py ├── src/harness/state/store.py ├── src/harness/providers/base.py ├── src/harness/providers/mock.py ├── src/harness/providers/env.py ├── src/harness/sandbox/local.py ├── src/harness/utils/logging.py ├── src/harness/utils/ids.py ├── tests/ ├── runs/ ├── examples/ ├── pyproject.toml ├── requirements.txt ├── README.md └── .gitignore ``` --- ## Core models ### Stack ```python frontend: str = "FastAPI" backend: str = "FastAPI" db: str = "SQLite" ``` ### Requirement ```python id: str text: str must_have: bool = True acceptance: str = "" ``` ### AppSpec ```python app_type: str stack: Stack pages: list[Page] components: list[Component] data_model: list[DataEntity] requirements: list[Requirement] must_have: list[str] explicit_non_goals: list[str] acceptance_criteria: list[str] ``` The source specification does not fully define the fields of `Page`, `Component`, and `DataEntity`. Use minimal Pydantic models sufficient for the specified validation rules; do not add unrelated domain complexity. ### Task ```python id: str description: str depends_on: list[str] = [] files_touched: list[str] = [] requirements: list[str] = [] verification: list[str] = [] status: TaskStatus = PENDING ``` ### VerificationResult ```python task_id: str status: str command: str exit_code: int stdout: str stderr: str duration_ms: int failure_type: FailureType | None ``` ### FailureType ```text CODE_ERROR TEST_FAILURE TYPE_ERROR DEPENDENCY_ERROR CONFIG_ERROR ENVIRONMENT_ERROR TIMEOUT UNKNOWN ``` ### RequirementResult ```python requirement_id: str status: str evidence: list[str] = [] missing: list[str] = [] ``` ### RunState ```python run_id: str trace_id: str user_request: str workspace: str spec: AppSpec | None plan: list[Task] task_results: dict[str, Any] verification_results: list[VerificationResult] review: dict | None checkpoints: list[str] status: RunStatus ``` All models MUST use Pydantic v2 `BaseModel`. All enums MUST derive from: ```python str, Enum ``` --- ## LocalSandbox ```python LocalSandbox(root: Path) ``` Required methods: ```python create_workspace(run_id) -> Path write_file(rel, content) -> Path read_file(rel) -> str list_files(rel=".") -> list[str] delete_file(rel) execute(cmd, cwd=None, timeout_s=120) -> dict ``` Every path MUST be resolved against the configured sandbox root. Reject: ```text ../ absolute paths symlink escape resolved paths outside root ``` Prefer command execution as: ```python subprocess.run(list_args, shell=False) ``` Verification commands must be parsed with `shlex.split()`. Do not use `shell=True` in V1. Return: ```json { "exit_code": 0, "stdout": "", "stderr": "", "duration_ms": 0 } ``` --- ## RunStore ```python RunStore(base=Path("runs")) ``` Methods: ```python create_run(user_request) -> RunState save(state) load(run_id) -> RunState checkpoint(state, event_name, payload) list_runs() -> list[str] resume(run_id) -> RunState ``` --- ## LLMProvider Abstract interface: ```python generate(prompt: str, system: str = "") -> str generate_structured( prompt: str, schema: type[BaseModel] ) -> BaseModel stream(prompt: str) -> Iterator[str] ``` Agents depend ONLY on this abstraction. ### MockProvider Must be: ```text deterministic offline-safe test-friendly ``` ### EnvOpenAICompatibleProvider Read: ```text OPENAI_API_KEY OPENAI_BASE_URL ``` If API key is absent: ```text delegate to MockProvider ``` --- ## Acceptance Must pass: ```bash pip install -e . pytest -q ``` Tests must include: ```text AppSpec roundtrip Task roundtrip sandbox ../ rejection sandbox absolute-path rejection RunStore save/load MockProvider deterministic behavior ``` No network required. ## Forbidden Do not implement: ```text SpecAgent Planner Builder Verifier DAG execution ``` --- # M2 — SPECIFICATION AGENT + VALIDATOR ## Goal Transform: ```text user request ``` into: ```text validated spec.json ``` No application code generation yet. ## Files ```text src/harness/agents/spec.py src/harness/validation/spec_validator.py tests/test_spec.py examples/todo_spec.json ``` --- ## SpecAgent Interface: ```python SpecAgent(llm: LLMProvider) generate(user_request: str) -> AppSpec ``` System instruction: ```text Output ONLY JSON matching AppSpec. No prose. Infer stack, pages, data model, requirements, must-have features, explicit non-goals and acceptance criteria. ``` Post-processing MUST: ```text assign REQ-001..N ensure must_have is non-empty ensure acceptance_criteria is non-empty populate default explicit_non_goals when absent ``` If LLM output is invalid: ```text use deterministic rule-based fallback ``` Fallback must understand at minimum: ```text auth projects tasks dashboard todo ``` --- ## SpecValidator Return: ```python list[str] ``` Empty list means PASS. ### Structural checks Verify: ```text app_type non-empty stack fields non-empty requirements >= 1 acceptance_criteria >= 1 REQ IDs unique REQ IDs match ^REQ-\d{3}$ ``` ### Logical checks Verify: ```text page.entity references known data entity when entity is present requirement does not conflict with explicit non-goal backend is defined when requirement mentions API/backend requirement text is unique case-insensitively every must_have maps to >=1 requirement ``` --- ## Pipeline stage Implement: ```python spec_stage( store, run_id, llm, max_attempts=2 ) ``` Flow: ```text generate → validate → if invalid: regenerate with validation feedback → maximum 2 generation attempts → persist spec → checkpoint spec_created ``` Persist: ```text runs/<run_id>/spec.json ``` --- ## Acceptance Test: ```text valid Todo request → PASS conflicting login non-goal → FAIL missing acceptance → FAIL duplicate requirement → FAIL first generation invalid + second valid → PASS ``` Create: ```text examples/todo_spec.json ``` with four requirements. ## Forbidden No planning. No code generation. --- # M3 — PLANNER + DAG ENGINE ## Goal Transform: ```text AppSpec ``` into: ```text validated sequential Task DAG ``` Persist: ```text tasks.json ``` ## Files ```text src/harness/agents/planner.py src/harness/orchestration/dag.py src/harness/orchestration/runner.py tests/test_planner.py tests/test_dag.py ``` --- ## Planner rules Enforce these in code. Do NOT rely only on the LLM prompt. ### Ordering 1. DB/schema before dependent application logic. 2. Shared components before pages. 3. Auth infrastructure before authenticated routes. ### Coverage Every: ```text REQ ``` must map to at least one Task. Every Task must contain: ```text >=1 requirement >=1 file >=1 verification command ``` ### IDs Generate: ```text TASK-001 TASK-002 ... ``` Dependencies may reference only earlier tasks. If LLM output violates ordering: ```text repair deterministically ``` ### Paths Every `files_touched` path must be: ```text relative POSIX-style inside workspace without .. ``` --- ## Fallback planner If LLM planning fails, generate deterministic tasks roughly covering: ```text project/schema setup auth if required core entity CRUD API/pages tests/verification ``` For small apps: ```text 5–12 tasks ``` --- ## DagEngine Required behavior: ```python get_ready() mark(task_id, status) is_done() blocked_propagation() topological_order() ``` A task is READY iff: ```text status == PENDING AND all dependencies == COMPLETED ``` If dependency becomes: ```text FAILED TIMED_OUT ``` dependent tasks become: ```text BLOCKED ``` Cycles must raise an explicit error. --- ## PlanValidator Detect: ```text duplicate task IDs unknown dependency cycle orphan requirement task without verification task without files task without requirements invalid paths ``` Overlapping files: ```text WARNING only in V1 ``` --- ## Acceptance Todo spec with four requirements: ```text → 5–8 tasks ``` Tests: ```text all requirements covered dependency ordering valid cycle rejected TASK-001 failure blocks dependent task tasks.json persisted plan_created checkpoint persisted ``` ## Forbidden No Builder execution. No concurrency. --- # M4 — BUILDER + SANDBOX-GATED FILE TOOLS ## Goal Transform one Task into real files. ## Files ```text src/harness/agents/builder.py src/harness/tools/files.py tests/test_builder.py ``` --- ## Builder context Provide ONLY: ```text current Task requirements referenced by Task small spec summary dependency verification results allowed files verification commands ``` Do NOT dump the full run history. Example input: ```python task requirements spec_summary = { "app_type": ..., "stack": ..., "data_model": ... } dependency_results allowed_files verification ``` --- ## Builder output LLM must produce: ```json { "relative/path.py": "full file contents" } ``` No prose. No markdown fences. --- ## Write restrictions Builder may write ONLY: ```text task.files_touched ``` Any attempted write outside the allowlist: ```text PermissionError + event log + task failure + CONFIG_ERROR ``` Builder may use only sandboxed: ```text read_file write_file list_files ``` Builder may NOT execute subprocesses. --- ## Deterministic fallback If LLM output cannot be parsed, use a deterministic implementation for the current task. The fallback MUST obey the same `files_touched` allowlist. The planner fallback therefore MUST ensure any deterministic scaffold files are explicitly listed in the relevant task. Reference generated application may include: ```text src/main.py src/models.py requirements.txt tests/test_health.py ``` Do not create files outside the current Task's declared set. --- ## Acceptance Test: ```text allowed file write succeeds outside-allowlist write fails ../ escape fails MockProvider creates deterministic output checkpoint created after task ``` After milestone: ```bash pytest -q ``` must pass. ## Forbidden No verification execution inside Builder. No repair. --- # M5 — REAL VERIFICATION + FAILURE CLASSIFICATION ## Goal Execute real verification commands and persist structured evidence. ## Files ```text src/harness/verification/verifier.py src/harness/verification/classifier.py tests/test_verifier.py tests/test_classifier.py ``` --- ## Verifier Implement: ```python verify_task( task: Task, workspace: Path ) -> list[VerificationResult] ``` For every verification command: ```text shlex.split(command) → LocalSandbox.execute(...) ``` Default when missing: ```bash pytest -q ``` Capture: ```text exit_code stdout[-4000:] stderr[-4000:] duration_ms ``` PASS: ```text exit_code == 0 ``` FAIL: ```text exit_code != 0 ``` Timeout: ```text Task status = TIMED_OUT FailureType = TIMEOUT ``` Static inspection alone can never count as verification. --- ## Workspace verification Provide helper: ```python verify_workspace(workspace) ``` It may include project-level checks appropriate to files actually present. Do not blindly require external network installation as part of offline harness unit tests. --- ## Failure classifier precedence Classification MUST be deterministic. Use precedence: ```text 1. TIMEOUT 2. ENVIRONMENT_ERROR 3. DEPENDENCY_ERROR 4. TYPE_ERROR 5. CODE_ERROR 6. CONFIG_ERROR 7. TEST_FAILURE 8. UNKNOWN ``` The precedence matters. For example, a pytest session containing a Python `SyntaxError` must classify as: ```text CODE_ERROR ``` not merely `TEST_FAILURE`. ### TIMEOUT Patterns: ```text timed out TimeoutExpired duration >= configured timeout ``` ### ENVIRONMENT_ERROR Patterns: ```text EAI_AGAIN ENOTFOUND registry unavailable Network is unreachable HTTP 503 Could not fetch ``` ### DEPENDENCY_ERROR Patterns: ```text ModuleNotFoundError ImportError No module named Could not resolve dependency npm ERR 404 ``` ### TYPE_ERROR Patterns: ```text mypy Pydantic ValidationError TypeError ... expected TS2322 Property ... does not exist ``` ### CODE_ERROR Patterns: ```text SyntaxError IndentationError NameError ReferenceError ``` ### CONFIG_ERROR Patterns: ```text missing configuration missing pyproject requirements file not found port already in use invalid path configuration ``` ### TEST_FAILURE Patterns: ```text AssertionError FAILED 1 failed FAIL tests/ ``` Only after higher-priority categories have been excluded. --- ## Acceptance Tests: ```text SyntaxError → CODE_ERROR ModuleNotFoundError → DEPENDENCY_ERROR AssertionError → TEST_FAILURE network unavailable → ENVIRONMENT_ERROR timeout → TIMEOUT working scaffold → PASS ``` Persist verification evidence. ## Forbidden No automatic repair yet. --- # M6 — REPAIR LOOP + LOOP DETECTION ## Goal Implement: ```text FAIL → classify → repair → verify ``` with bounded retries. ## Files ```text src/harness/orchestration/repair.py src/harness/agents/repair_agent.py tests/test_repair.py ``` --- ## Loop signature ```python sha256( command + exit_code + normalize(stderr[-2000:]) ) ``` Normalization: ```text lowercase strip changing timestamps strip volatile numeric values normalize paths collapse whitespace ``` Track: ```python seen[signature] += 1 ``` If the same normalized failure signature occurs three times: ```text ESCALATE FAILED BLOCK dependents ``` Total repair attempts MUST NEVER exceed: ```text 3 ``` --- ## Repair strategy ### ENVIRONMENT_ERROR ```text Do not rewrite application code. Retry once. If still failing → FAILED. ``` ### DEPENDENCY_ERROR Repair only dependency/config files that are already allowed by the Task or explicitly listed as verification hints. ### TYPE_ERROR Patch relevant code only. ### CODE_ERROR Patch relevant code only. ### TEST_FAILURE Patch implementation or tests only when justified by requirement evidence. Do not simply weaken tests to obtain PASS. ### CONFIG_ERROR Patch configuration files only. ### TIMEOUT Allow one bounded adjustment/retry. Do not create an unbounded timeout. --- ## RepairAgent context Provide: ```text Task relevant requirement slice failing VerificationResult FailureType allowed files current relevant file contents ``` Truncate large source context around: ```text 8000 chars per repair context ``` Output: ```json { "relative/path": "full corrected contents" } ``` --- ## Acceptance Tests: ```text repair SyntaxError → PASS within <=3 attempts same failure 3 times → ESCALATE dependents become BLOCKED ENVIRONMENT_ERROR does not modify code files outside allowlist cannot be repaired ``` --- # M7 — TRACEABILITY MATRIX ## Goal Construct mechanical evidence: ```text Requirement → Tasks → Files → Verification → Result ``` ## Files ```text src/harness/trace/matrix.py tests/test_trace.py ``` --- ## Matrix structure For every requirement: ```json { "REQ-001": { "tasks": [], "files": [], "tests": [], "evidence": [], "status": "PASS|FAIL", "missing": [] } } ``` Compute: ```text tasks = tasks referencing REQ files = union(task.files_touched) tests = union(task.verification) evidence = existing files + successful verification commands ``` PASS only if: ```text >=1 expected implementation file exists AND >=1 relevant verification result PASS ``` No evidence: ```text FAIL ``` Prose can never substitute for evidence. --- ## Acceptance Todo reference run: ```text 4/4 requirements traced ``` Delete an implementation file: ```text associated requirement becomes FAIL ``` --- # M8 — REQUIREMENT REVIEW ## Goal Perform the final requirement-level audit. ## Files ```text src/harness/agents/reviewer.py tests/test_review.py ``` --- ## ReviewAgent Default authority: ```text rule-based evidence ``` LLM review is optional and explanatory only. An LLM MAY: ```text add rationale summarize evidence identify concerns ``` An LLM MUST NOT: ```text turn evidence-based FAIL into PASS ``` --- ## Output Persist: ```text runs/<run_id>/review.json ``` Shape: ```json { "requirements": [ { "id": "REQ-001", "status": "PASS", "evidence": [], "missing": [] } ], "overall_status": "PASS" } ``` Overall PASS only if: ```text every requirement == PASS ``` --- ## Acceptance Tests: ```text complete evidence → PASS missing implementation file → FAIL missing successful verification → FAIL LLM cannot override FAIL review JSON validates ``` --- # M9 — CHECKPOINTING + CRASH RECOVERY ## Goal Resume interrupted runs without repeating completed work. ## Files ```text src/harness/state/checkpoints.py src/harness/orchestration/pipeline.py tests/test_recovery.py ``` --- ## Required checkpoint events ```text spec_created plan_created task_started task_completed verification_completed repair_started review_completed ``` Filename convention: ```text checkpoints/<sequence>-<event>-<optional-task>.json ``` Example: ```text 001-spec_created.json 002-plan_created.json 003-task_started-TASK-001.json 004-task_completed-TASK-001.json ``` --- ## Resume algorithm ```text load state.json ↓ validate state ↓ load latest checkpoint state ↓ reconstruct task statuses ↓ keep COMPLETED keep FAILED keep BLOCKED convert interrupted RUNNING → PENDING convert READY → PENDING ↓ continue unfinished pipeline ``` Completed tasks MUST NOT execute again. Use file hashes where appropriate to prove completed output was not rewritten during resume. --- ## Corruption handling Corrupt: ```text state.json checkpoint JSON ``` must produce a clear explicit error. Never silently restart the run from scratch. --- ## Acceptance Simulate crash after: ```text TASK-002 ``` Resume must: ```text finish remaining tasks not rerun TASK-001 not alter completed file hashes ``` Test checkpoint sequencing. --- # M10 — RUNTIME VERIFICATION ## Goal Prove that the generated application actually starts and responds. Static tests alone are insufficient. ## Files ```text src/harness/verification/runtime.py src/harness/verification/browser.py tests/test_runtime.py ``` --- ## RuntimeVerifier ### Entry detection Recognize at minimum: ```text src/main.py:app app.py:app package.json ``` No recognized entry: ```text FAIL CONFIG_ERROR ``` ### Start FastAPI: ```bash python -m uvicorn src.main:app --port <free_port> ``` Node fallback: ```bash npm run dev -- --port <free_port> ``` Use an OS-assigned/free local port. Start process with: ```python subprocess.Popen ``` with: ```text cwd jailed inside workspace shell=False ``` The verification layer may own process execution; application-building agents may not. ### Startup timeout ```text 15 seconds ``` ### Checks Verify: ```text process remains alive TCP port accepts connection GET /health OR / returns 2xx GET /docs or /api/health when available SQLite DB can be opened when expected ``` ### Cleanup Always terminate spawned process. Use `finally` cleanup. Never leave orphan development servers. ### Result Return structured: ```json { "status": "PASS", "checks": [ { "name": "health", "ok": true, "detail": "HTTP 200" } ], "evidence": [] } ``` Capture useful log snippets. --- ## Browser V1 stub Implement only: ```python def verify_acceptance(...): raise NotImplementedError( "Browser verification deferred post-V1" ) ``` Test that the browser verifier remains explicitly deferred. Do NOT install Playwright or Selenium. --- ## Acceptance Reference application: ```text starts port opens health endpoint returns 200 runtime verifier PASS ``` Broken startup: ```text FAIL diagnostics captured repair hint available ``` --- # M11 — CLI + BENCHMARKS ## Goal Expose the entire harness through a deterministic command-line interface and provide a quick regression benchmark. ## Files ```text src/harness/cli.py src/harness/orchestration/pipeline.py src/harness/benchmarks.py tests/test_cli.py tests/test_benchmarks.py ``` Add console entrypoint: ```toml [project.scripts] builder = "harness.cli:main" ``` Use: ```text argparse only ``` No Click/Typer dependency. --- ## CLI commands ### New run ```bash builder new "Build a Todo app" ``` Equivalent module form: ```bash python -m harness.cli new "Build a Todo app" ``` It must execute the full pipeline. ### Resume ```bash builder resume <run_id> ``` ### Logs ```bash builder logs <run_id> ``` Print or tail the run's structured event log in readable form. ### Benchmark ```bash builder bench --quick ``` --- ## Progress output `builder new` must emit five user-facing high-level stages: ```text [1/5] SPEC [2/5] PLAN [3/5] BUILD [4/5] VERIFY [5/5] REVIEW ``` Detailed internal milestones remain M1–M12; the five-stage CLI view is only presentation. On success: ```text BUILD COMPLETE ``` On failure: ```text BUILD FAILED ``` and return non-zero exit status. --- ## Quick benchmark `builder bench --quick` runs a deterministic small Todo scenario. It must validate at minimum: ```text spec generated plan generated files written verification executed runtime checked review produced required artifacts exist ``` Return: ```text 0 → PASS non-zero → FAIL ``` --- ## Acceptance Must pass: ```bash builder new "Build a Todo app" builder bench --quick ``` Offline LLM fallback must still function. Verify: ```text all required artifacts exist five-stage output matches expected format events.log contains required keys quick benchmark passes ``` ## Forbidden No Web UI. No production Docker environment. --- # M12 — FINAL GATE ## Goal Prove V1 reliability before declaring completion. All gates are mandatory. --- ## Gate 1 — Vertical Slice Run: ```bash builder new "Build a Simple Todo App with add/list/complete" ``` Expected pipeline: ```text SPEC → >=3 REQs → PLAN → >=3 tasks → BUILD → real files → VERIFY → pytest PASS → RUNTIME → PASS → REVIEW → PASS ``` Save a stable demonstration run under: ```text runs/demo_todo/ ``` If any stage fails: ```text fix the harness rerun do not proceed ``` --- ## Gate 2 — Failure Injection Inject: ```python SyntaxError ``` into: ```text workspace/src/main.py ``` The harness must: ```text DETECT → FAIL CLASSIFY → CODE_ERROR LOCALIZE → TASK-ID CAPTURE → diagnostics REPAIR → relevant file only REVERIFY → PASS ``` Then simulate identical failure repeatedly using a no-op repair implementation. Expected: ```text same signature x3 → LOOP DETECTED → ESCALATE → task FAILED → dependents BLOCKED ``` --- ## Gate 3 — V1 Capability Checklist All MUST be ✓: ```text [ ] valid spec generated [ ] spec validated [ ] valid DAG generated [ ] sequential execution [ ] real files created [ ] real verification executed [ ] failures localized [ ] failures classified [ ] failures repaired [ ] repair bounded <=3 [ ] loop detection works [ ] checkpoints written [ ] crash resume works [ ] completed tasks not rerun [ ] traceability matrix generated [ ] requirement review generated [ ] runtime verified [ ] final working project retained ``` --- ## Gate 4 — Documentation README must document: ```text quickstart builder new builder resume builder logs builder bench --quick architecture evidence principle failure handling checkpoint recovery ``` Include an ASCII architecture diagram. Create: ```text examples/todo_run/ ├── spec.json ├── tasks.json ├── verification.json └── review.json ``` --- ## Gate 5 — Full Test Suite Run: ```bash pytest -q builder bench --quick ``` Both must pass. --- ## V1 Forbidden-Feature Audit Assert the implementation does NOT contain functional implementations for: ```text docker/ web_ui/ parallel workers parallel DAG scheduler Playwright Selenium PostgreSQL backend web research ``` The browser verification stub is allowed. --- # FINAL DEFINITION OF DONE V1 is complete only when: ```text M1 PASS M2 PASS M3 PASS M4 PASS M5 PASS M6 PASS M7 PASS M8 PASS M9 PASS M10 PASS M11 PASS M12 PASS ``` Every bug fix MUST include a regression test. Every new module MUST have tests. Keep source files focused. Prefer: ```text <400 lines per file ``` Split larger files when practical. Never store: ```text API keys secrets absolute host-specific paths ``` Never delete historical checkpoints to hide failures. --- # FINAL RUN SUMMARY FORMAT At the end of every run, print a requirement-level summary. Example: ```text REQ-001 Implemented: YES Tested: YES Runtime: YES Evidence: - src/main.py - pytest -q → exit_code 0 Status: PASS REQ-002 Implemented: YES Tested: NO Runtime: NO Missing: - successful verification result Status: FAIL ``` Final status: ```text PASS ``` only when every requirement has evidence-backed PASS status. Never use: ```text "AI says finished" ``` as evidence. --- # EXECUTION COMMAND Start now. Implement: ```text M1 ``` Run its tests. If green, continue to: ```text M2 ``` Continue sequentially until M12. Do not ask questions. Do not stop at intermediate milestones. Do not skip failed gates. Do not declare V1 complete until the Final Gate is green.