Blog/Engineering

Getting provider keys out of your services

3 June 20265 min readObstruo team
EngineeringAPI keysIntegration

Provider keys spread the way secrets always spread. The migration off them is smaller than teams expect.

Provider keys spread the way secrets always spread. One in a deployment config, one in a CI variable, two in a notebook, one in a service nobody owns. Rotation becomes a project, and revoking a key means finding out what breaks.

The fix is unglamorous. Keys move into a vault, applications hold one key for the layer in front of the providers, and the provider keys never leave that boundary again.

The sprawl is measurable, and AI keys are the fastest-growing part of it

This used to be a story engineers told each other. It is now a dataset.

GitGuardian's State of Secrets Sprawl 2026 counted 28.65 million new hardcoded secrets in public GitHub commits during 2025, a 34% year-on-year rise and the largest single-year jump they have recorded. Within that, credentials for AI services grew 81%, faster than any other category.

Two findings from that report describe the shape of the problem better than any anecdote.

Secrets do not sit in one place. Analysis of machines compromised in the Shai-Hulud 2 supply chain attack found that each live secret appeared, on average, in eight different locations on the same machine: environment files, shell history, IDE configuration, cached tokens, build artefacts. Your key is not in the config. It is in the config and seven other places you did not list.

And it is not mostly on laptops. 59% of the compromised machines were CI/CD runners rather than personal workstations, which turns an individual hygiene problem into shared build infrastructure exposure.

Then the finding that should end the argument: 64% of the secrets confirmed valid in 2022 had still not been revoked by 2026. Not undetected. Unrevoked. Detection is not remediation, and the reason teams give is consistently the same one: rotating a credential risks breaking something nobody can enumerate in advance.

The bill is the attack

A leaked database credential is a disclosure problem. A leaked provider key is a metered one, and the meter runs whether or not you are watching it.

Sysdig named this LLMjacking in May 2024 after documenting stolen cloud credentials being used to run inference on victims' accounts, modelling a worst case around $46,000 per day at the time. It has since industrialised. Operation Bizarre Bazaar, running from December 2025 into January 2026, logged more than 35,000 attack sessions, with daily costs to victims passing $100,000 where flagship models were reachable. The same reporting puts credential theft aimed specifically at AI services up 376% between the last quarter of 2025 and the first of 2026, with a resale market brokering access across dozens of providers.

The economics are the point. Attackers are not after your data here. They are after inference capacity that costs real money, which they resell while you receive the invoice. Keys committed to public repositories are found by automated scanners in minutes, and the OWASP LLM Top 10 gave the general failure a name in its 2025 edition: unbounded consumption.

This is why "we should tidy up our keys" is the wrong internal pitch. The right one is that an unbounded, unattributed, unrevocable credential to a metered service is sitting in eight places on a CI runner.

The migration is one base URL and one key

Because the endpoint is OpenAI-compatible, most services change two lines: the base URL and the key. No SDK swap, no request rewriting, no new client library to review. Logical model names resolve to real providers on the other side, so the application stops naming a vendor.

  • Point the SDK at your endpoint and keep the rest of the code as it is.
  • Use a logical model name so routing can change without a deploy.
  • Run one service through it first, compare the responses, then move the rest.
  • Revoke the direct provider keys last, and treat any breakage as an inventory finding.

What does not move in two lines

Worth being straight about this, because the failure mode of a migration sold as trivial is a team discovering the exception in production.

The two-line change holds for chat completions, which is what the overwhelming majority of application code uses. It holds less well at the edges:

  • Provider-specific request fields. Anything outside the common schema needs checking rather than assuming, including reasoning and thinking controls, safety settings, and provider-specific sampling parameters.
  • Newer endpoint shapes. Compatibility usually means the chat completions shape. Code written against a provider's newer stateful or response-oriented endpoints is a port, not a redirect.
  • Streaming event details. The stream works; the exact event sequence and usage accounting can differ enough to upset code that parses it closely.
  • Tool calling at the margins. Basic function calling maps cleanly. Parallel calls, strict schema enforcement, and provider-native tool types are worth an explicit test.
  • Files, batches, embeddings and fine-tuning. These live on separate surfaces and migrate separately, if at all.

The practical consequence is not "do not migrate". It is that the pilot service should be a representative one rather than the simplest one, and that response comparison should run long enough to cover the routes that use the unusual parameters. An afternoon per service is the right estimate for the common case, not for the one service that does something clever with streaming.

Revoking the direct keys is the migration

The last step is the one teams skip. Until the direct keys are revoked, you have a governed path and an ungoverned one, and traffic will find the ungoverned one.

It is worth being blunt about why this step gets deferred: revocation is the moment the migration stops being additive and starts being able to break things. That is also precisely what makes it valuable. Breakage after revocation is the only reliable inventory of who was calling providers directly, and it arrives on your schedule, during working hours, with someone watching. The alternative is discovering the same inventory when a key leaks.

Do it in a window, keep the old keys retrievable for a short rollback period, and write down every service that surfaced. That list is the real deliverable of the project.

But now there is one key to steal

The obvious objection, and it deserves a straight answer rather than a slogan.

Concentrating credentials does concentrate value. What changes is not the existence of a target but the properties of one:

  • Revocation is a single action rather than an archaeology project across repositories, CI systems and notebooks.
  • Application keys are scoped and disposable. A leaked application key reaches one project's routes and models, not your whole provider account, and replacing it is a config change rather than a coordinated release.
  • Spend is bounded. Per-project rate and budget limits turn the LLMjacking scenario from an open invoice into a capped one. This is the control that most directly answers the attack above, and it is impossible to apply consistently when every service holds its own provider key.
  • Abuse is visible. Unusual volume against one endpoint is a signal. The same volume spread across six services with six keys is six unremarkable graphs.

And the honest caveat: the layer itself is a dependency, and dependencies in this ecosystem have been attacked. The LiteLLM supply chain compromise harvested credentials from machines running it. Treat the gateway as infrastructure: pinned versions, vaulted provider credentials, restricted egress, and the same review you would give anything else holding production secrets. The argument is not that concentration is free. It is that one governed, monitored, budget-capped credential store beats an unknown number of ungoverned ones, which is the actual comparison.

What you get beyond tidiness

Rotation becomes a change in one place instead of a coordinated release. Swapping a provider stops being a code change. Spend becomes attributable per project instead of arriving as one invoice, which is also what makes a budget cap meaningful rather than theoretical.

And because every call now goes through one place, redaction, routing rules and the audit record apply to all of it rather than to whichever services remembered. The value of every other control in this stack is capped by the fraction of traffic that passes through it, and that fraction is decided by whether the direct keys still work.

None of that requires a rewrite. It requires deciding that applications should not hold provider credentials, and then spending an afternoon per service.

This is an engineering perspective on credential architecture, not security advice for a specific environment. Threat figures are cited from published research; treat them as indicative of scale rather than as a forecast for any one organisation.

Sources

Companion checklistProvider key migration: an afternoon per serviceOpen the checklist