Verify it yourself

You don't need our source, and you don't have to take our word for any of it. Send puzzles we have never seen, mark the answers yourself, and see how the numbers hold up.

The short way

A zero-dependency client does every step below for you: it sends ARC's own puzzles from your clone, marks the answers on your machine, and diffs each one against our published record. Node 20 or later, no key.

git clone https://github.com/kiinara-intelligence/genesis-arc.git && cd genesis-arc
git clone https://github.com/fchollet/ARC-AGI.git
npx tsx arc-verify.ts --bench arc1

The source, the protocol and the per-task record are in genesis-arc(opens in a new tab). The steps below are the same thing by hand.

1 · Check what you are talking to

Before sending anything.

curl -s https://arc.kiinaraintelligence.com/api/v0.17/manifest

It states the hash, what the hash covers, and the method used to compute it. The same hash appears in the published census and in every response. It is our label for the build: the solver's source is private, so you cannot recompute it. What verifies the claim is step 4, your own marking, diffed against the published record.

6ab5dfa07bda8c5100618c04a3983409ab9ffb0e3679e1208dc58a23a95da283

2 · Send your own puzzles

No key. ARC's own JSON, unmodified.

curl -s -X POST https://arc.kiinaraintelligence.com/api/v0.17/solve \
  -H 'content-type: application/json' \
  -d '{"tasks":[{"id":"whatever-you-like","train":[…],"test":[{"input":[…]}]}]}'

Send 25-50 tasks per request. A full ARC-AGI-1 run is 8-16 requests. The open tier covers it with no key. A limit that stopped you reproducing the headline would be a strange thing for us to build.

3 · Read the reply

{
  "version": "v0.17",
  "solverHash": "…",
  "results": [
    { "id": "…", "status": "answered",
      "attempts":  [ [ [[…]] ] ],
      "families":  [ "object-program∘crop" ],
      "solveMs": 2.9 },
    { "status": "refused",
      "reason": "no_corroborated_rule",
      "attempts": [], "families": [] }
  ]
}

attempts is four levels deep, always: proposal, then test input, then row, then column. A proposal answers every test input of a task at once, and 83 of the 1,400 public training tasks have more than one. It does not collapse when a task happens to have one, so a client that assumed three levels would break on those tasks silently.

families is parallel to attempts. When two proposals both corroborate, picking between them would need the answer key, which this endpoint never sees. So it reports every member that corroborated and leaves the choosing to you.

Refusals are the product. Count the three reasons separately. A malformed task comes back as status: "error", never as a refusal, so your refusal counts stay clean.

4 · Score it on your own machine

We do not score, and we will not tell you whether you were right. The answer keys are public: github.com/fchollet/ARC-AGI(opens in a new tab) for ARC-AGI-1, and github.com/arcprize/ARC-AGI-2(opens in a new tab) for ARC-AGI-2. Compare, then report coverage, precision-when-answering, the wrong count, and the three refusal reasons.

That is deliberate. Marking your own homework is worth less than letting someone else mark it, so we'd rather you did.

Things you can check

The manifest declares these, each with the test that would falsify it. arc-verify.ts --properties runs the testable ones. If one doesn't hold, we'd want to know, and so would everyone else.

deterministic: The same task gives the same result every time. Send one task, or one batch, twice; results are identical.
stateless: No request affects another. Send the same batch in two orders; per-task results are identical.
no-cross-task-leakage: A task's result does not depend on what else is in its batch. Send a task alone and inside a full batch; the result is identical.
hash-reporting: Every response carries the solverHash this manifest states. Compare each response's solverHash with this manifest.
content-blind: The application logs method, path, status, latency and a task count — no body, no grid. Not testable from outside. A promise, stated so it can be held to.

What this does not claim

It is not third-party verification. Every figure is self-reported; nobody outside Kiinara has run them. Checking that is what this page is for.
It does not break the evaluation seal. Published figures are the public training halves.
It does not expose the model. The ARC adapter is separate and has never called it.