[RESEARCH]: How do you measure AI Agent Intelligence?
I have been working on quantifying the agentic intelligence of open weight models, and I've built a rubric that scores a model purely from what its own tool-call logs and content output can prove.
The task: An agent plays a maze runner game through the Ollama chat API. It needs to call five tools to read its position, the map it has uncovered, and the outcome of its last move, then returns a batch of moves as JSON. Turns cost a fixed budget, so batching correctly is rewarded and wall collisions are penalized. Everything below is computed from the exported log - nothing is inferred from intent the log cannot show.
Each run was scored on 9 capabilities and 6 violations, every answer strictly yes or no. The two are scored separately and never cancel: a run can plan well and still bleed errors, and the profile says both. A "no" means not observed, never incapable. These metrics were extracted from the game played at level 1 of a maze runner game hosted at https://dmigwi.github.io/tapoo/ where most barely made it through.
gemma4 (Google) — 6/9 capabilities, 1/6 violations
The only run that finished a maze task, and the cleanest profile here. Never batched to depth, never recovered after a failure. It won by exhausting the maze, not outplaying it.qwen3.6(Alibaba) — 5/9 capabilities, 5/6 violations
The only run to plan at full depth, and the only one to trip five of six violations.nemotron-3-super(Nvidia) — 3/9 capabilities, 4/6 violations
Perfect form, blind play. The only run with flawless formatting - yet it never landed a batch, skipped its own outcome check, and resubmitted a move already proven invalid.minimax-m3(MiniMaxAI) — 0/9 capabilities, 2/6 violations
Never delivered a single move. It repeatedly called tools it invented (submit_moves, predict_moves) rather than returning the moves as content
Of the 4 sampled open weight models, Google's gemma4 was the best because it choose to be conservative but still inadequate because consistent wins require some batch optimization the model failed to apply. I will be testing higher models soon.
