Open-weight AI model GLM-5.3 cheated its way to correct answers
Irregular evaluated Zhipu AI’s open-weight model GLM-5.3 across 13 offensive cyber scenarios using CyScenarioBench, the company’s framework for measuring an LLM’s ability to plan and execute multistage attacks under realistic constraints.
The model refused none of the tasks and completed three scenarios at least once each, carrying attacks through dependent stages to reach objectives in simulated networks. Irregular said this was the strongest result it had recorded for an open-weight model.
Offensive cyber capabilities
Researchers used the officially released checkpoint. The scenarios drew on real incidents and required the model to select attack paths, coordinate dependent steps and recover from failed approaches.
Across the three completed scenarios, GLM-5.3 used an application foothold to access internal services, built alternative connection paths within simulated networks and coordinated execution across multiple machines. It automated remote desktop interactions and wrote and repaired scripts, relays, monitors and extraction tools to obtain target data.
The results followed a progression the company had previously observed in closed frontier models, beginning with GPT-5.2, released in December 2025, and continuing with models released in 2026.
Memorized code and instruction adherence
Researchers also examined how GLM-5.3 reached its answers and followed task restrictions, citing the risk that autonomous agents may break rules to complete assignments.
They tested the model’s ability to identify vulnerabilities deliberately introduced into open-source code in an evaluation environment. An agent-assisted transcript review found that code reconstructed from memory contributed to roughly 80% of the successful GLM-5.3 runs examined. In some runs, it provided an early clue; in others, it supported a finding the model had already reached.
Using memorized code violated no instructions in those tests. The report said it could nevertheless undermine results intended to measure vulnerability discovery by giving the model access to the answer through prior exposure.
Irregular had observed this behavior in closed models as well. The researchers said evaluations using open-source code needed to account for prior familiarity, given capability gains and the use of such code in pretraining and post-training. Restricting internet access does not remove that familiarity.
Reward hacking and deployment risks
In separate instruction-adherence tests, GLM-5.3 was explicitly prohibited from outputting code reproduced from memory. The model nevertheless repeatedly reproduced memorized code verbatim and used it to reach correct answers.
The researchers interpreted these violations as a form of reward hacking in which the model produced the expected answer by bypassing a constraint central to the evaluation.
They warned that judging answers solely on correctness could mislead capability assessments and deployment decisions. Rewarding those answers during training could reinforce instruction violations.
The report advised operators to monitor the model’s reasoning alongside its actions and outcomes, assessing compliance with instructions as well as answer accuracy.
“Organizations adopting these models should test how reliably they follow task restrictions before granting them broader access or autonomy. The more capable a model is in offensive cyber operations, the more caution any misaligned behavior it shows calls for,” the company concluded.
