Benchmark report

Task verified

3m 23s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Almost nothing: a guessed auth.md URL returned 404, and AgentMail documents no test recipient, so an inbox had to be created for the run first.

Where the time went

3:23 agent time
  • Research0:22
  • Attempt1:35
  • Verify1:26
3m 23sTotal elapsed
0:56Time to first success
0:00Waited on the owner

Judge reason

Saved code (agentmail-send/send_transactional.py) makes an authenticated POST to the documented send operation https://api.agentmail.to/v0/inboxes/{inbox_id}/messages/send with Authorization: Bearer, built from docs.agentmail.to/api-reference pages fetched in the transcript. AgentMail documents no public test recipient, so the script first created both inboxes through the product's own API (POST /v0/inboxes), and evidence/01 and 02 show live 200 responses returning bench-recipient-1793605598@agentmail.to and bench-sender-1793605598@agentmail.to (sender address discovered from the API's own 'email' field). evidence/03-send-message.json records the send: request body {to: bench-recipient-...@agentmail.to, subject 'Order AM-4815 confirmed', plain-text order-confirmation body} byte-identical to the payload constructed in the code, and response HTTP 200 with the documented shape {message_id, thread_id} = '<010001a19b398254-45e5fd78-4403-47d3-b7a4-a2448d79b0fc-000000@email.amazonses.com>' / '3a14d039-c82f-45fc-a72a-8166478f62c3', plus live server headers (Apigw-Requestid GM7rBhkOIAMESXQ=, X-Amz-Cf-Id, Date Mon, 02 Nov 2026 07:46:40 GMT) that no mock produced. run-stdout.txt and the transcript tool result at the run step print the same status and identifiers; transport is HTTPS REST via stdlib urllib, not SMTP or dashboard. Exactly one send call is made. No fabricated or hard-coded response: all values flow from the parsed live response.

Attempt record

  1. 0:32Answered from saved key: AgentMail API keyno wait
  2. —First success verified

Attempt artifacts saved.