GitHub Swapped Its LLM for a Smaller Model

GitHub announced on October 13 that it had pulled the large language model out of its secret scanning pipeline and replaced it with a classifier built on ModernBERT. The job is checking repositories for credentials left in code. By GitHub's own measurements, the smaller model costs over 98% less to serve, runs considerably faster, and finds more while raising fewer false alarms.

What GitHub reports

The source of those figures deserves as much weight as the figures. These are GitHub's own numbers from GitHub's own traffic, the announcement does not say which model the old pipeline used, and nobody outside the company has reproduced them. Replacing a poorly optimised LLM setup with a well-tuned classifier would produce impressive figures too.

The direction is still credible, because it matches what the task looks like. Accepting the direction while holding the magnitude loosely is the right posture with any vendor-reported benchmark.

Why a classifier wins this one

Three properties make secret scanning the ideal case. The output is fixed: this string is a live credential, or it is a placeholder. The volume is enormous, since scanning covers the full history of every branch across every customer. And GitHub has an extraordinary quantity of labelled examples from years of running the service.

A general-purpose model asked that question brings reasoning capability, world knowledge and the ability to write an explanation, none of which the task needs, and you pay for all of it on every call. At a few thousand calls the overhead is invisible. At GitHub's volume it is the entire cost base.

The accuracy improvement is less surprising than it sounds. A model trained on one distinction, with millions of examples of that distinction, is simply better at it than a generalist reasoning about it from first principles.

The test for your own workloads

Three questions identify a candidate. Does the task have a fixed set of possible outputs rather than producing writing? Do you run it at high volume? Do you have a pile of historical examples that people already labelled by hand?

Yes to all three describes more business tasks than people expect. Routing support tickets to the right queue. Flagging invoices that need review. Sorting inbound email by type. Detecting duplicate records. Deciding whether a document is a contract, an invoice or a delivery note.

No to any of them, and a general model is the right choice. Writing a reply, handling an unusual case, anything needing knowledge beyond your own data. Forcing those into a classifier is the opposite mistake and it produces a system that fails silently on everything it was not trained for.

Scale decides whether it is worth it

A 98% reduction on a bill of forty dollars a month is not a project. Training, evaluating and maintaining a specialised model costs real engineering time, and that cost is fixed while the saving scales with volume.

Work out what the task costs you annually at current volumes and at the rate you expect to be paying, then compare that to a few weeks of work plus ongoing maintenance. Most small businesses will find the answer is no for now and yes for one specific high-volume workflow, which is the only one worth doing. Getting the cost per unit of work right first is the exercise in cash flow forecasting when AI costs are variable.

There is a middle option too. Moving a task from a frontier model to a cheaper general model is a configuration change rather than a project, and frequently captures a large share of the saving with none of the maintenance. That tiering approach is covered in the AI price war.

Read the credit model, not just the price

One practical detail from the same release matters for anyone budgeting. GitHub's AI-powered secret detection sits inside Secret Protection, priced per active committer, and the push-time checking now in limited preview consumes AI credits on every push, including when nothing is blocked.

A consumption line that accrues on normal activity rather than on findings is easy to miss at purchase and visible on the invoice, which is the pattern described in consumption pricing and unpredictable bills.

The useful generalisation

The last two years trained everyone to reach for the largest available model, because capability was the constraint and nothing else worked reliably. Capability stopped being the constraint on narrow tasks a while ago, and defaults are slower to change than capabilities.

The habit worth building is asking what the job actually requires before choosing what to run it on. For most businesses that is a question about a handful of high-volume workflows rather than a strategy, and it belongs in the same review as the security tooling you already pay for and the terms attached to it in what to check in AI terms of service.

Frequently Asked Questions