BLOG / WHY IS ARC-AGI INTERESTING?
Why is ARC-AGI interesting?
Disclaimer: I am competing in both the ARC-AGI-2 and ARC-AGI-3 competitions on Kaggle, and I intend to do as well as I can. I am not affiliated with ARC Prize in any other way.
It’s Saturated Already?
ARC-AGI-1/2 have been saturated by frontier LLMs (see the caveat about benchmaxxing below).
The next interesting question is cost per task. DeepSeek-V4-Flash-0731, for example, is far ahead of the pack in its price range.
On the other hand, every widely shared ARC-AGI-3 frontier-model-plus-harness result I’ve seen used only the 25 public games. We don’t yet know whether those results generalize to the semi-private games.
We also can’t know for sure whether model labs were benchmaxxing — for example, by training on synthetic tasks similar to ARC-AGI-1/2/3 once the benchmarks’ design became known.
The Harness Question
Is a result only valid if the official benchmarking is used?
I don’t think so. I think a result is valid as long as it generalizes to the semi-private set, with cost per task tracked properly.
The official benchmarking is also a harness; just a baseline / simple one.
For ARC-AGI-3, I’m not sure what the baseline harness for playing games should be. To play a game, an agent (human or computer) needs to remember what happened and plan ahead. Keeping the last N frames in the LLM’s context is easy to implement, but is it too crude a memory mechanism? Should we give the LLM a bash or Python tool, since these models are good at writing code?
If a few extra priors, workflows, and code execution in the harness can help a model that much, it’s a pretty interesting result in itself, don’t you think?
Deep RL on ARC-AGI-3
One question that interests me is whether deep RL (e.g., DQN, PPO) can solve ARC-AGI-3 games, setting action efficiency aside for now. If these methods can solve Atari games tabula rasa, maybe they could crack ARC-AGI-3 games tabula rasa too.
It turns out that solving even just the first level or two is quite hard for all 25 public games:
- Level passing is the only reliable reward signal. It’s easy to pass level 1 of ls20 by accident, but very hard to get lucky enough to pass level 2.
- A 64x64 image frame is small, but a mouse-click action space of 64x64 is not. cd82 is a game that requires both arrow keys and clicking.
- The RESET action (in all games) and the UNDO action (in some games) are confusing. Leaving them disabled is probably best for RL.
Besides these model-free methods, I also briefly tried some model-based ones. I haven’t gotten good results.
So this is an interesting research question at least for me: Can we brute-force our way through the twists and puzzles of a sparse-reward environment like ARC-AGI-3 games? Or do we have to introduce human priors (by using an LLM, for example)?
The Kaggle Compute Hell
Joining Kaggle competitions is the only way to get a method tested on the private set. The private set differs from the semi-private set in that it isn’t used for testing with model APIs (so model labs can’t get the private test data). As with other Kaggle competitions, the private set score is only revealed at the end of the competition. The public leaderboard score is the semi-private set score.
In other words, it should be impossible to cheat on Kaggle by training on test data. A method that works on Kaggle either lets us learn something genuinely new and interesting, or exposes weaknesses in the benchmark. Results from the ARC-AGI-1 competition led to ARC-AGI-2, for example.
Now, here’s the compute problem with Kaggle: the compute provided for the scoring rerun severely limits the class of LLMs that can be used.
| Competition | GPU | # Tasks | Time Limit |
|---|---|---|---|
| ARC-AGI-2 | 4x L4 | 240 | 12 hours |
| ARC-AGI-3 | RTX 6000 Pro | 110 | 9 hours |
An L4 has 24GB of VRAM and a memory bandwidth of 300 GB/s (very low — a 4090’s is 1008 GB/s). If we use the 4 GPUs independently, we get 12 minutes (720 seconds) per task per GPU, ignoring initialization overhead. That might seem like a decent amount of time, but realistically we can only run 3B/4B models. Reasoning from off-the-shelf open LLMs (when enabled) can generate a lot of tokens, which blows up the time budget quickly.
Likely because of this constraint, the current winning approach — like NVARC — pre-trains a small specialized model on a lot of external and synthetic ARC-AGI data. That pre-training itself requires serious compute (e.g., 8x H100) to finish in a reasonable time.
If the competition provided more compute for the scoring rerun, the design space of solutions would be larger and the results more interesting.
Same story with ARC-AGI-3: the RTX 6000 Pro allows models like Qwen3.6-27B. This is the model used in the milestone 1 winning solution, duck-harness.
It’s a struggle, though. In the published notebook, the 110 games are divided into 4 waves of 28 games. In other words, up to 28 games try to use the locally served Qwen3.6-27B concurrently. You can imagine the slowness, contention, and request timeouts. Some games go unsolved in the scoring rerun simply because they can’t get enough turns with the LLM.
Again, the theme here is that the competition’s compute constraint limited the design space of a solution. Maybe low-resource results are interesting on their own?
Summary
The interesting questions posed by ARC-AGI-2/3 are:
- How to solve ARC-AGI-1/2-style puzzles well at very low cost?
- How to optimize an agent policy in a sparse-reward game where new twists are being introduced in each level?
- How to design an LLM harness or other AI systems to play games at human-level action efficiency, with or without inference compute constraints?