GitHub adopts cheaper AI model to catch secrets developers leave in code
GitHub has trained a credential classifier to improve generic secret detection and reduce false alerts. The new pipeline cut median post-push scan execution time by 86% and model serving costs by more than 98% compared with its previous large language model (LLM) pipeline.
GitHub’s secret scanning covers two stages of development. Push protection checks changes before a push is accepted, helping developers prevent credential leaks. Post-push scanning examines repository content and creates alerts for potential secrets that need investigation and remediation. The classifier’s speed also enabled GitHub to extend generic secret detection to push protection.
Source: GitHub
How the scanner assesses potential secrets
Structured credentials often have a recognizable prefix, length, character set or checksum. Generic secrets can resemble build identifiers, asset hashes, sample values and other non-sensitive strings. Their use within a file helps determine whether they are credentials.
“These cases are hard negatives: strings that match a credential pattern but are not usable credentials. Adding more rules can suppress some of them, but each exception risks hiding a real secret. A learned classifier can instead use signals such as file path, secret type, variable names, and nearby code or text,” Batu Ozturkler and Shengyu Fu explained.
The pipeline begins with regular expressions that identify strings resembling passwords, access tokens and other credentials. A fine-tuned ModernBERT model then evaluates each candidate using its file path, credential type, variable names and surrounding code or text.
The classifier assigns a score estimating the likelihood that the candidate is a real secret. Candidates that meet the scanner’s threshold can trigger an alert during post-push scanning or a block during push protection. Likely false positives are filtered out.
The two stages combine pattern matching with contextual analysis. ModernBERT handles classification over short sections of code and text, supporting the throughput and response times needed for scanning at GitHub’s scale.
Training and validating the model
GitHub built its training and evaluation datasets from product feedback, repository examples identified through regex scanning, and synthetic examples covering cases with limited data or difficult distinctions.
The datasets included real secrets and false positives, with human validation and annotation supporting the labeling process. Synthetic examples covered documentation secrets, test fixtures, public keys, checksums and strings resembling credentials. These examples underwent deterministic validation and model-based quality review before inclusion.
Before splitting the data, the team grouped rows sharing a repository, candidate value or example to prevent near-duplicates from appearing in both training and evaluation.
The team used offline evaluation to select classification thresholds, then validated performance on traffic resembling production use before rollout. Its goal was to maintain or improve detection coverage and reduce false alerts and serving costs.
The trained model was exportd to ONNX format with 8-bit quantization to support faster inference.
Evaluation results and next steps
In an offline evaluation of post-push scanning on the same public traffic, the new pipeline achieved 13% higher precision and 19% higher recall than the previous LLM pipeline. These metrics measure the proportion of findings that are real secrets and the proportion of real secrets detected.
A customer A/B experiment recorded an 11.5% increase in repository owner-weighted alert acceptance. GitHub weights acceptance by repository owner to limit the influence of customers who resolve large batches of alerts at once.
“Higher precision means fewer distracting alerts. Higher recall means more real secrets are caught,” Ozturkler and Fu added.
GitHub plans to use production feedback to identify missed secrets and false positives, then incorporate those findings into training and evaluation data. The company is also investigating architectures that could combine secret detection and extraction in one step to simplify the pipeline.
