Benchmark report

Task verified

5m 36s elapsed

The saved attempt met the documented success criterion.

Experience score

95/ 100

Agent-Ready
  • SInstall
  • SAuth
  • SExecute
  • ADocs
  • ACurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install S, Auth S, Execute S, Docs A, Current ASInstallSAuthSExecuteADocsACurrent
Agent profile · this run
Judge's note · what slowed the agent most
SDK/doc drift: with_raw_response.create rejected the documented kwargs and needed CreateInboxRequest, costing one failed run and an orphan inbox.

Where the time went

5:37 agent time
  • Research0:36
  • Attempt3:27
  • Verify1:32
5m 36sTotal elapsed
2:34Time to first success
0:00Waited on the owner

Judge reason

Saved runnable code (agentmail-task/code/send_email.py) calls the documented AgentMail send operation POST /v0/inboxes/{inbox_id}/messages/send through the official Python SDK 2.0.24 with a Bearer key read at runtime from a private file. AgentMail documents no test recipient, so the inbox fallback was used: evidence/01_create_inbox.json shows a live POST /v0/inboxes -> HTTP 200 creating dizzyarea469@agentmail.to, and 02/02b show GET /v0/inboxes and GET /v0/inboxes/{id} -> 200 confirming the account owns that sender inbox. evidence/03_send_message.json records HTTP 200 with response {message_id: <010001a1ad971cd0-a477d8ca-176b-4352-ad99-e8c58cfad840-000000@email.amazonses.com>, thread_id: 783d50ee-...}; the saved request body (to=dizzyarea469@agentmail.to, subject 'AgentMail transactional test - benchmark run', multi-line plain-text body) matches the SUBJECT/TEXT literals and send() call in the code exactly, and stdout 00_send_stdout.txt prints the same HTTP 200 and message_id. Not SMTP, not a dashboard. Cleanup is present: evidence/04_delete_inbox.json and 00_delete_stdout.txt show DELETE /v0/inboxes/{id} -> HTTP 202 empty body for dizzyarea469@agentmail.to (plus an orphan inbox from an earlier aborted run). No mocked or hard-coded responses: transcript shows two genuine failed runs (SDK raw create kwarg TypeError, then a pagination assertion) before the successful live run.

Attempt record

  1. 1:36Answered from saved key: AgentMail API key (AGENTMAIL_API_KEY)no wait
  2. —First success verified

Attempt artifacts saved.