Garak: Open-source LLM vulnerability scanner

LLMs can make mistakes, leak data, or be tricked into doing things they were not meant to do. Garak is a free, open-source tool designed to test these weaknesses. It checks for problems like hallucinations, prompt injections, jailbreaks, and toxic outputs. By running different tests, it helps developers understand where a model might fail and how to make it safer.

Garak works with a wide range of models and platforms. It supports Hugging Face Hub generative models, Replicate text models, OpenAI API chat and continuation models, LiteLLM, and most systems accessible through REST. It also handles GGUF models, such as llama.cpp version 1046 or newer, along with many other LLMs.

Garak creates several types of logs when it runs. The main log file, called garak.log, records debugging information from garak and its plugins and continues across multiple runs. Each run also generates a separate report in JSONL format, which includes details of every probing attempt. This report file is named at the start of the run and again at the end if the run is successful. Entries are added as results come in and when they are evaluated, with a status attribute showing the stage of the attempt. In addition, garak keeps a hit log that tracks attempts that revealed a vulnerability.

Garak is available for free on GitHub.

Must read:

• 35 open-source security tools to power your red team, SOC, and cloud security
• GitHub CISO on security strategy and collaborating with the open-source community

Subscribe to the Help Net Security ad-free monthly newsletter to stay informed on the essential open-source cybersecurity tools. Subscribe here!