Muhammad Asjad

A Memorization Floor for LLM Refinement of Decompiled Code

The question

Compiling a program throws away its variable and function names. Decompilers such as Ghidra recover the logic but produce names like param_1 and iVar2. A popular fix is to have a large language model rewrite that output with meaningful names, and published results say it works. Those results are measured on code the models have probably seen during training. So is the model reading the decompiled code, or recalling names it memorized?

What I did

I designed a control I call the memorization floor. Each function is refined twice: once from normal decompiler output, and once from a copy in which the information the model should rely on has been destroyed. I tested three versions of that damage: scrambled dataflow, removed type prefixes, and permuted names. If naming quality stays the same, the model wasn't using that input. The comparison is made within each function, so corpus difficulty cancels out, and a full run costs about twenty API calls.

The main test set is a corpus of C functions written after the analysis plan was committed, so no released model could have trained on them. A second benchmark covers 25 functions at five compiler optimization levels (-O0 to -O3). I registered a second model from a different vendor in advance and gave it byte-identical inputs.

What I found

• The models do recover real names. Refined output scores +0.072 to +0.137 above a permutation null built from each model's own vocabulary.
• That recovery doesn't depend on the input I destroyed. Scrambling the dataflow changes naming quality by +0.001 (95% CI −0.026 to +0.026). Removing type prefixes or permuting names changes it no more.
• It replicates across vendors. Twelve contrasts over two models and two corpora gave twelve nulls.
• Readability stays at ceiling in every condition, so a human reading the output gets no warning.

The null is bounded. Effects below 0.056 would not show up, and the model could still be naming from the operations, which the ablation leaves intact. No registered hypothesis was confirmed. The paper reports the five instrument failures behind that, including a test harness biased against one arm and an equivalence checker that turned out not to discriminate on these inputs.

What came out of it

That equivalence checker was codealign (Dramko et al., TOSEM 2025). I reduced its failures to three reproducible issues, two of them a few lines of plain C, and filed them with the authors. The study is also the first output of my final-year project, IRES, an LLM-powered decompilation system supervised by Dr. Khurram Shehzad.

What Does Emotion Accuracy Measure? A Pre-Registered Study of Lexical Provenance in Emotion-Conditioned Image Captioning

The question

Emotion-conditioned captioning asks a model to describe a photo in a requested tone, such as happy, sad or angry. The standard score runs a classifier over the caption and checks whether it recovers the requested emotion. The training captions for this task are now often written by a large language model. I asked what that accuracy measures when they are.

What I did

I trained one fixed captioning architecture on eight caption corpora that differ only in where the captions came from, how they are structured and how many there are. One frozen classifier scores all eight. For each evaluation set I also computed a keyword anchor: the accuracy a bag of the top 25 words per emotion reaches with no model at all. The quantity I make claims about is the margin, accuracy minus anchor. Hypotheses were pre-registered, and three blind human raters judged tone.

This is the second version of the study. The first pilot failed, and its post-mortem shaped this design.

What I found

• Raw accuracy follows word choice, not tone. The most keyword-stereotyped corpus scores 10.6 points higher than a fairer one but loses to it by 2.5 points on the margin. Its captions have less than half the CIDEr-D, and human raters rank it last on tone. Both directions of the reversal survive Holm correction.
• A model that ignores the image can win. A baseline that outputs five fixed strings and never sees the photo reaches 0.80 accuracy. That beats five of the eight trained models, at a margin of −0.20.
• One registered prediction was confirmed cleanly. More training data improves conditioning by +13.6 margin points, against a pre-set 7-point criterion.
• One result went against my own expectation. At matched volume, removing register pairing improved conditioning by 5.3 margin points on five of five folds. It is reported as a post-hoc finding.

Why it matters

On this task, raw emotion accuracy rewards keyword stuffing. The paper recommends reporting accuracy next to a matched keyword anchor and making claims only on the difference.

Detecting Unseen Attacks in Network Traffic

The question

Intrusion-detection benchmarks measure how well a model recognizes attacks it has already seen. Real attackers don't stay inside the training set. When a model meets an attack family that was absent from its training data, does it still flag the traffic as an attack? And does that depend on the kind of model?

What I did

Eight model families, each tuned per task on validation data only and run over five seeds, on the canonical NSL-KDD train and test split. I withheld each attack category from training in turn. I also measured every category's ceiling with it present, so the reported number is the novelty cost (ceiling minus unseen recall) rather than raw recall. All models are compared at a matched 1% false-positive rate.

What I found

• Novelty costs every architecture about the same: 0.19 to 0.23 recall. Trees against neural nets: p = 0.58.
• Unseen attacks are still caught. Models keep 50 to 79% of their achievable recall, depending on the category.
• How you measure skill decides the answer. Robustness correlates with skill at +0.738 when skill is measured on the same task, and at −0.714 on a different one. Both are significant: same models, same run, opposite conclusions. An earlier draft of this project reported the wrong one.
• Tuning changed the story. Untuned, tree ensembles seemed to beat neural nets. Once every family was tuned, the MLPs led and the gap disappeared.

The follow-up

I then asked whether purpose-built open-set detectors beat a plain classifier's own confidence. I tested twelve methods, including energy, ODIN, OpenMax and Deep SVDD, on 23 held-out families across NSL-KDD, UNSW-NB15 and CICIDS2017. On the two modern benchmarks, none of the twelve beat the baseline, even after tuning. Those two benchmarks agree on the ranking (ρ = +0.825), while NSL-KDD agrees with neither. How far attacks sit from benign traffic explains the pattern. I pre-registered predictions for CICIDS2017 before training any detector on it. The ranking replicated, one of two numeric thresholds was falsified, and the paper reports that falsification as it happened.