Executor.sh comparison (executor-cmp-001)

Side-by-side token comparison against executor.sh on a GitHub PR list — the benchmark behind Both Sides of Context Compression.

Tokenizer: cl100k_base. focus=input (model output omitted from headline ratios).

Headline numbers (live, `vercel/next.js`)

Layer	Executor	ClawQL	Notes
Layer 1	1,044 (homepage) / 115 (live `execute`)	394 (codemode)	394 does not beat live 115
Layer 2	143,466 (`pulls.list`, full REST)	907 (`execute` + `fields`)	158× on tool result alone
Combined	144,510 (pub L1) / 143,581 (live L1)	1,301	111× / 110×

Layer 2 is 100% of Executor’s input and 70% of ClawQL’s on this task.

Live flamegraph (public)

Static snapshot — same HTML as mcp-api-adapter’s built-in demo:

clawql.com/mcp-ui/trace/compare/executor (canonical blog link)
docs.clawql.com mirror (same page)
JSON: compare.json

Regenerate after measurement changes: npm run generate:executor-cmp-static (from repo root).

Reproduce

# Fixture (CI shape)
npm run benchmark:executor-comparison

# Live GitHub + ClawQL
BENCHMARK_LIVE=1 CMP_GITHUB_REPO=vercel/next.js CMP_PER_PAGE=30 \
  npm run benchmark:executor-comparison

# Live both arms (Executor CLI + ClawQL)
BENCHMARK_LIVE=1 CMP_GITHUB_REPO=vercel/next.js CMP_PER_PAGE=30 \
  EXECUTOR_BIN=/path/to/executor EXECUTOR_CWD=/path/to/executor-cwd \
  EXECUTOR_GITHUB_PULLS_PATH=github.user.githubMain.pulls.list \
  npm run benchmark:executor-comparison

Multi-turn series: npm run benchmark:executor-comparison:multiturn and npm run benchmark:executor-comparison:uniform.

Raw artifacts (repo)

Methodology: docs/benchmarks/executor-comparison/README.md
Live single-action: executor-cmp-001.live.json
Uniform N=5: executor-cmp-002b.uniform-pulls.live.json
Mixed N=5: executor-cmp-002.multiturn.live.json
OpenBench task: benchmarks/openbench/tasks/executor-github-pr-filter/

What not to claim