Blog

A binary-analysis agent has to show its evidence

On 72 binary-analysis tasks BinReX succeeded 83.3% of the time. Codex with IDA, on the same GPT-5, managed 43.1%. The difference comes down to looking up similar functions first, and backing every step with evidence.

Hand an analyst a stripped binary and the questions vary wildly. Does this driver have an exploitable bug? Is there a backdoor in this sshd? Is this code doing encryption, and is the key hard-coded? Most existing tools handle one of these. Change the question and you change the tool, or write IDAPython by hand.

An LLM agent looks like a good fit. It reads disassembly, writes scripts and takes any question you give it. When we tried it, it got stuck in two places.

Two problems

The first is finding the right code. A stripped binary has thousands of functions named sub_XXXX. The agent has no idea which one checks the password and which one handles network requests, so it reads them one by one and runs out of context.

The second is checking the answer. Model output is probabilistic, but binary analysis needs evidence someone can check: an address, a slice of control flow, a constant. Codex filled in about nine in ten fields of its reports, but only 43% of its evidence held up.

How BinReX works

BinReX does not ask the model to guess what each function does. It looks it up. We collected 57 million functions from 10,134 open-source projects and trained two encoders, one on what functions do and one on their names. During analysis it searches that collection for functions similar to the one in front of it, and close matches become hints for the agent: sub_40F20 is probably checking a password.

With hints in hand, a planning agent splits the user’s question into subtasks. Each subtask is written in a constrained intermediate form, then compiled by rules into an IDAPython script that runs in IDA Pro. What the script prints is the evidence: addresses, constants, call relations. A validation agent then checks whether that evidence supports the conclusion. If it does not, a fixing agent rewrites the script and runs it again.

One example from the paper shows why that last step matters. Analyzing a backdoored sshd, retrieval flagged sub_40F20 as the password check. The first script only inspected the left side of the comparison and found nothing. Had it stopped there, the agent would have reported no backdoor. The validator noticed that the hint and the empty result disagreed, the fixer added a check on the right side, and it found the hard-coded password at 0x40F58.

Results

83.3%task success; Codex with IDA: 43.1%

36.1 htotal time on 72 tasks; Codex with IDA: 87.9 h

0false positives among 395 real samples, after review

We built a 72-task benchmark, BinREval, across seven kinds of work: vulnerable signed drivers (BYOVD), backdoors, ransomware, CTF reversing, locating crypto functions, string obfuscation and extracting hard-coded secrets. There is no partial credit. A right conclusion with evidence that does not check out counts as a failure.

All three agents ran on GPT-5. Codex with command-line tools succeeded 27.8% of the time, Codex with IDA 43.1%, BinReX 83.3%. Take out any one of retrieval, planning, evidence extraction or fixing, and BinReX drops to between 57% and 61%.

At partner security vendors, BinReX found 295 malware samples abusing vulnerable drivers and 100 sshd backdoor variants. Three senior reverse engineers checked all 395 and found no false positives. A BYOVD driver took an expert about 15.5 hours. BinReX took 5.5 minutes.

What it can’t do yet

BinReX is only as good as IDA Pro’s decompiler. Packed binaries (UPX, say) and VM-protected code need unpacking first, and problems you can only see by running the program, like races and side channels, are out of reach. Control-flow flattening cuts its success on string-obfuscation tasks from 77.8% to 66.7%.

More and more, we think an agent is only as useful as its conclusions are checkable. BinReX is our first step in that direction.

Paper · USENIX Security 2026Towards Generality: Task-Adaptive Binary Analysis via Semantic Retrieval and Verifiable Reasoning

BLOG

More posts

Next step

AI that understands how code runs.

Contact