Benchmark report

Task verified

10m 14s elapsed

The saved attempt met the documented success criterion.

Experience score

96/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current AAInstallSAuthSExecuteSDocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
Docs promise 201 on POST /research but the API returned 200, aborting the first script run and forcing a status-check patch and a separate poller for the already-created task.

Where the time went

10:14 agent time
  • Starting1:54
  • Research1:05
  • Attempt5:08
  • Verify2:08
10m 14sTotal elapsed
1:32Time to first success
0:00Waited on the owner

Judge reason

Saved code (tavily-extract/extract_isolation_levels.py) sends the fixed URL https://www.postgresql.org/docs/17/transaction-iso.html plus the required JSON output_schema (levels array with name, dirtyRead, nonrepeatableRead, phantomRead, serializationAnomaly) to Tavily's documented POST /research operation with Bearer auth, polling GET /research/{request_id}. Fetched docs in the transcript confirm POST /extract has no JSON-schema parameter while /research documents output_schema and carries no beta/preview/gated marking, so the chosen operation matches the plan and the product's documentation. Two live runs are evidenced: request_id d63c2001... (evidence/research_full_response.json, http_status 200, status completed, response_time 8.06, credits 7) and the full end-to-end rerun 56b40992... (evidence/rerun/research_full_response.json, create_response HTTP 200 pending then GET 200 completed, 9.27s). Both responses cite sources=[https://www.postgresql.org/docs/17/transaction-iso.html] only, and content.levels has exactly four entries whose 16 cells match Table 13.1 verbatim: Read uncommitted 'Allowed, but not in PG'/Possible/Possible/Possible; Read committed Not possible/Possible/Possible/Possible; Repeatable read Not possible/Not possible/'Allowed, but not in PG'/Possible; Serializable Not possible x4. verify.py re-run output (evidence/verify_output.txt) asserts these independently and passes. No self-scraping, no own-LLM call, and no mocked or hard-coded data: the script only issues HTTP calls to api.tavily.com and dumps the raw response.

Attempt record

  1. 5:21Answered from saved key: Tavily API keyno wait
  2. —First success verified

Attempt artifacts saved.