Look at what your pipeline can do right now. Not what it does, what it can do.
In most organisations the honest answer is that a CI job can deploy to production, read every database credential, publish to the package registry, and assume an administrative cloud role, because somewhere in the settings there is a static access key that was created during setup and has never expired, never rotated, and never been scoped down.
That key is readable by anyone who can edit a workflow file. Which is anyone who can open a pull request, if branch protections are not exactly right. Which includes any dependency in the build, since a compromised build tool runs with the job’s full environment.
This is the largest unforced security risk in most engineering organisations, and the fix has been available for years. Stop putting long-lived credentials in CI. Have the job prove its identity to the cloud provider at run time and receive a short-lived token scoped to what that specific workflow, on that specific branch, is allowed to do.
Everything else in secrets management, the vault product, the injection mechanism, the encryption scheme, sits downstream of that decision. Get it right and the remaining problems are manageable. Get it wrong and you are buying a better lock for a door that is propped open.
Key takeaways
- OIDC workload federation removes long-lived cloud credentials from CI entirely. The job presents a signed token describing itself, and the cloud exchanges it for short-lived credentials scoped by trust policy.
- The trust policy condition is where this succeeds or fails. A policy that trusts the issuer and the repository but not the branch, environment or workflow is trivially bypassed by a pull request.
- Masking is not a leak prevention mechanism. It is a last-resort filter that fails on transformed, encoded and split values, so the real control is never printing the secret at all.
- Rotation without downtime needs two credentials valid at once. Any scheme with a single active credential has a window where the pipeline is broken, and that window is why nobody rotates.
OIDC workload federation: a standard, not a product
This is the central mechanism, and it is a specification rather than something you buy. Every major CI platform implements the issuing side and every major cloud implements the trusting side.
The flow has four steps.
The CI platform issues a token. When a job runs, the platform mints a short-lived JSON Web Token signed by its own key. The token’s claims describe the job: which repository, which branch or tag, which workflow file, which environment, which event triggered it, and who the actor was. The job cannot forge these claims, because it does not hold the signing key. The platform asserts them.
The job presents the token to the cloud provider. It calls the provider’s token exchange endpoint with the JWT.
The provider verifies and evaluates policy. It fetches the issuer’s public keys from a well-known discovery endpoint, verifies the signature, checks expiry and audience, and then evaluates your trust policy against the token’s claims. That policy is where you say which repository, which branch, and which environment may assume which role.
The provider returns short-lived credentials. The job gets a token that expires in minutes to an hour, scoped to the permissions of the role it was allowed to assume.
No secret was stored anywhere. There is nothing in the CI settings to steal, nothing to rotate, and nothing that outlives the job.
What it gives you
Credentials that expire on their own, which means a leaked one is worth little and only briefly. No stored secret in the CI platform, so compromising the platform’s secret store yields nothing for the cloud. Authorisation that can depend on the branch, so a role that can touch production is unreachable from a feature branch by construction rather than by convention. An audit trail on the cloud side showing which workflow and which commit assumed which role, which is the traceability a static key can never give you. And no rotation work at all for the credentials it replaces, which removes the largest source of rotation debt in most organisations.
What it does not do
It does not cover third parties that have no federation support. A package registry token, a payment provider key, an SMTP credential: many of these are still static strings, and you still need somewhere to keep them.
It does not secure itself. The entire security of the scheme is in the trust policy condition, and a policy matching only the issuer and the repository allows any workflow in that repository, on any branch, including a branch from a pull request, to assume the role. Conditions must pin the subject claim to the specific branch, tag, or deployment environment.
It does not stop a compromised job from using the credentials it legitimately received. A malicious dependency running inside the job has the same short-lived token the job has. Federation narrows the window and the scope; it does not remove the risk.
It does not help with secrets your application needs at run time. Those come from a secret store into the running workload, which is a separate problem with separate tooling.
And it requires the CI platform’s issuer to be reachable and trusted. Self-hosted runners on a self-hosted platform need their own issuer configured and their public keys reachable by the cloud, which is an extra setup step that surprises teams migrating from hosted CI.
The practical adoption path: enumerate every static cloud credential in your CI settings, replace each one with a federated role, then delete the credential. Do the deleting, because a federated role next to an unused static key is not an improvement, and the key is what an attacker will use.
Per-environment scoping and the blast radius of one token
The second structural failure is scope. Most organisations have one set of credentials per provider, shared by every pipeline, because that is the path of least resistance.
The consequence is that a test job running an untrusted contributor’s code has the same access as the production deployment job. Nothing about the pipeline’s purpose narrows what it can reach.
Scoping properly means three separations.
Separate by environment. Development, staging and production credentials must be distinct principals with distinct permissions, and the production ones must be unreachable from any job that is not a production deployment. On platforms with an environment concept, bind production secrets to the production environment and require an approval to enter it. On the cloud side, make the trust policy condition require that environment claim.
Separate by branch or tag. The role that can deploy production should only be assumable from the default branch or from a tag matching a release pattern. This is one line in a trust policy and it eliminates the entire class of attack where somebody opens a pull request containing a workflow change.
Separate by permission. A deployment job needs to push an image and update a service. It does not need to read every secret in the account, list all databases, or modify IAM. Scope the role to the actions the job performs. This takes longer than it sounds because nobody knows exactly what the job does, which is itself informative.
The test for whether you have done this: take a feature branch, add a workflow step that dumps the environment and tries to assume the production role, and open a pull request from a fork. If anything sensitive comes back, your scoping is decorative. Run this deliberately, in a controlled way, before someone runs it accidentally.
The related control is what pull requests from forks can do at all. By default, a fork’s pull request should get no secrets and no federated identity. Workflows that need credentials for external contributions should be split so that the untrusted code runs without access and a separate, trusted workflow handles anything privileged.
Needs first-hand data: Inventory every secret configured in your CI platform, and for each one record where it is used, when it was created, when it was last rotated, and what it can do if stolen. Most teams find secrets nobody can attribute, and those are the ones to revoke first. The count of unattributable secrets is the single best measure of how much of this work you actually need.
How secrets leak into build logs
Build logs are read by more people than production databases, retained longer than anyone intends, and shipped into log aggregation where they are searchable by the whole organisation. They are the most common place secrets escape.
Every CI platform masks registered secret values in log output. Masking is useful and it is not a control you should rely on, because it fails in predictable ways.
Transformation defeats it. A secret that is base64 encoded, URL encoded, JSON escaped, uppercased or embedded in a larger string no longer matches the registered value. The masker compares strings, and the transformed form is a different string.
Splitting defeats it. A value printed across a line boundary, or a multi-line certificate echoed line by line, will not match as a whole.
Derived values are not masked at all. A token exchanged for another token, a signed URL containing a credential, a connection string assembled from parts: none of these are the registered secret, and all of them are just as sensitive.
Debug modes print everything. Shell tracing, verbose HTTP client logging, and framework debug output print headers and environment variables, and they are usually enabled during an incident by someone who is not thinking about the log retention policy.
The controls that actually work, in order:
Do not put secrets in environment variables when you can avoid it. Anything that dumps the environment, including many crash handlers and diagnostic tools, prints them. Writing to a file with restrictive permissions and reading it explicitly is narrower, and passing credentials on a command line is worse than both, because the process table is readable.
Never enable shell tracing in a job that has secrets. If you need it for debugging, turn it off around the sections that handle credentials.
Fetch at the point of use and keep the lifetime short. A secret fetched immediately before the call that needs it, in a step that does nothing else, has a much smaller exposure surface than one exported at the top of the job.
Scan logs, not just code. Secret scanning in repositories is common; scanning build log output for credential patterns is rarer and catches the leaks that scanning code never will.
Assume any leaked secret is burned. The response to a secret appearing in a log is rotation, not deleting the log. The log has already been replicated to the aggregation system, and possibly to a screenshot in a chat thread.
This is the strongest practical argument for federation. A short-lived token printed to a log expires on its own. A static key printed to a log is a permanent liability until someone notices and rotates it, and noticing is the part that does not happen.
Rotating without pipeline downtime
Rotation fails in practice for a specific and fixable reason: most rotation procedures have a window where the old credential is dead and the new one is not yet everywhere, and during that window pipelines break. Once that has happened twice, rotation stops being scheduled.
The fix is overlap. Two credentials valid simultaneously, and a rollover that never has zero valid credentials.
The pattern, for any credential that supports more than one active key:
Create the new credential while the old one still works. Both are now valid.
Update every consumer to the new one. Every CI configuration, every runtime workload, every developer machine, every script somebody wrote and forgot. This is the step that takes real time and it is safe, because the old credential still works throughout.
Verify the old one is unused. This is the step teams skip and it is the one that prevents the outage. Check the provider’s last-used timestamp, or its access logs, and confirm nothing has used the old credential for a full cycle of your longest-running scheduled job. A weekly job that uses the old credential will not show up in a two-hour observation window.
Revoke the old credential. Revoke rather than merely stop using, because an unrevoked old credential is exactly the unattributable secret that shows up in the next audit.
For credentials that do not support two active versions, you need a different strategy: put the secret behind an indirection that can be updated atomically, such as a versioned secret in a store with a rolling alias, and have consumers resolve it at start rather than hold it. That converts a coordination problem into a single atomic update.
Automate the whole sequence or it will not happen. Some secret stores support managed rotation for supported credential types, where the store calls the provider’s rotation API, creates the new version, and tests it. Where that exists, use it. Where it does not, write the rotation as a scheduled pipeline rather than as a runbook, because a runbook is a document that describes work nobody has time for.
And be clear about the real goal. The point of rotation is to bound the lifetime of a compromised credential. Federation achieves the same goal far better by making the lifetime minutes. Every credential you can move to federation is a credential you never have to rotate again, which is why the rotation backlog and the federation project are the same project.
Needs first-hand data: For each long-lived credential in your pipelines, record the provider-reported last-used timestamp and the creation date. Credentials created more than a year ago that are still in use are your rotation backlog; credentials created long ago and never used are your deletion list. Both lists are usually longer than expected and both are actionable immediately.
HashiCorp Vault

Vault is the reference implementation of a general secrets platform: a central store with pluggable authentication backends, fine-grained policy, leases with automatic expiry, full audit logging, and dynamic secrets, where it creates a short-lived database user or cloud credential on demand rather than handing out a stored one. Dynamic secrets are the genuinely distinctive capability, and they are the same idea as federation applied to systems that have no federation support of their own.
Pros
- Dynamic secrets generate short-lived database and cloud credentials on demand, so nothing long-lived exists to steal
- Leases with automatic expiry and revocation give every issued credential a bounded lifetime by design
- Authentication backends for CI platforms, Kubernetes and cloud identity let jobs authenticate without a stored token
- Comprehensive audit logging that records every access, which is what regulated environments actually need
Cons
- Operationally heavy: unsealing, storage backend, high availability, upgrades and disaster recovery all become your problem
- It becomes a hard dependency of every deployment, so an outage stops your pipelines and possibly your applications
- The licence moved to the Business Source License, which is a policy question for some organisations
- Policy and configuration complexity is real, and a misconfigured policy is the failure mode nobody notices
Best for: Organisations with many systems, strict audit requirements, and a platform team that can own a critical availability-sensitive service.
Pricing: Source-available community edition free to self-host, with commercial tiers licensed per client or per node adding namespaces, replication, advanced governance and support, plus a managed cloud offering.
AWS Secrets Manager
AWS Secrets Manager is the native store for teams whose workloads are on AWS. Its argument is not features, it is integration: access is controlled by IAM policies you already write, encryption uses keys you already manage, every access is already in your audit trail, and managed rotation for supported credential types works by invoking a function that performs the rotation for you. Nothing new to operate.
Pros
- IAM-based access control with no separate permission system to maintain or reconcile
- Managed rotation for supported credential types automates the sequence most teams never get to
- Access events land in the existing cloud audit trail alongside everything else
- Combines directly with OIDC federation, so a CI job assumes a role and reads only the secrets that role permits
Cons
- AWS-centric, so it is an awkward fit for multi-cloud estates or on-premises workloads
- Per-secret monthly charges push teams to pack many values into one secret, which defeats fine-grained access control
- Rotation for anything outside the supported types means writing and maintaining your own rotation function
- No native templating or environment inheritance, so per-environment layout is naming convention rather than structure
Best for: Teams running on AWS who want secrets under the same identity and audit model as everything else, with no additional service to operate.
Pricing: Metered per stored secret per month plus a charge per API call, with rotation functions billed as ordinary compute.
Google Secret Manager
Google Secret Manager is the equivalent native service on Google Cloud, with the same core argument: IAM controls access, the platform’s audit logging records it, and the existing key management service encrypts it. Its distinguishing detail is explicit secret versioning with independently enabled, disabled and destroyed versions, which maps cleanly onto the two-credentials-valid-at-once rotation pattern.
Pros
- IAM permissions can be granted per secret, which makes least privilege straightforward to express
- Explicit version lifecycle with enable, disable and destroy supports overlap-based rotation directly
- Access logging integrates with the platform audit trail with no extra configuration
- Works cleanly with workload identity federation, so CI jobs need no stored credential
Cons
- Google Cloud-centric, with the same multi-cloud awkwardness as its peers
- No built-in rotation for credential types, so rotation logic is entirely yours to write
- Metering on access operations means a chatty application or a high-frequency pipeline can generate unexpected call volume
- Sparse structure for organising many secrets across many environments, so you build conventions yourself
Best for: Teams on Google Cloud who want secret access governed by the same IAM and audit systems as the rest of their infrastructure.
Pricing: Metered per active secret version per month plus a charge per access operation, with a free allowance at low volume.
Doppler

Doppler is a secrets platform built specifically around the developer and pipeline workflow rather than around infrastructure. Its model is projects containing environment-specific configurations with inheritance, so a value defined once in a base configuration is overridden only where it differs. Injection happens through a CLI that runs your process with the secrets in its environment, or through integrations that push values into a CI platform or cloud provider.
Pros
- Environment inheritance means shared values are defined once, which removes the copy-paste divergence between environments
- Broad integrations push secrets into CI platforms, cloud providers and container platforms without custom glue
- Change history and audit logging per secret makes it easy to see who changed what and when
- Setup is fast enough that a small team gets real value in an afternoon
Cons
- A third-party service holding every secret you have, which is a trust and availability decision to make explicitly
- Injection into the process environment is convenient and is exactly the exposure pattern that leaks into logs
- Per-seat pricing scales with team size rather than with secret count, which can outpace the value for larger organisations
- Integration-based sync means secrets are copied into other systems, so the source of truth is authoritative but not exclusive
Best for: Small and mid-sized teams who want good environment modelling and fast integration without operating a secrets service.
Pricing: Free tier for individuals and small teams, then per-seat monthly pricing with higher tiers adding SSO, audit retention and access control.
Infisical

Infisical covers similar ground with an open source core that can be self-hosted, which is the main reason to choose it over a purely hosted competitor. It offers environment-scoped secrets, a CLI for injection, integrations with CI platforms and clouds, dynamic secret generation for supported backends, and secret scanning to catch credentials committed to repositories.
Pros
- Open source core with a genuine self-hosting path, so secrets can stay entirely inside your boundary
- Dynamic secrets for supported backends bring short-lived credentials without operating a heavier platform
- Built-in secret scanning catches credentials before they are committed, closing a common leak path
- Environment and folder structure maps naturally onto how teams actually organise configuration
Cons
- Self-hosting means you now operate an availability-critical service, with the same implications as any secrets platform
- Younger project than the established options, so edge-case behaviour and integration depth vary
- Open core boundary shifts, and capabilities you rely on may sit on the commercial side
- Smaller community, so unusual failures have fewer public answers
Best for: Teams who want a modern secrets workflow but need self-hosting for compliance, data residency or cost reasons.
Pricing: Open source core free to self-host, with a hosted service and enterprise tiers priced per seat adding SSO, audit and support.
1Password Secrets Automation
1Password Secrets Automation extends a password manager many organisations already deploy into machine-readable secret delivery. Service accounts and connect servers give pipelines programmatic access to specific vaults, and a CLI resolves secret references at run time. The argument is consolidation: one system for human credentials and machine credentials, one audit trail, one place where offboarding actually removes access.
Pros
- One system for human and machine secrets, so offboarding removes both without a second process
- Secret references resolved at run time keep the actual values out of configuration files and repositories
- Vault-level access scoping is a model your organisation likely already understands and administers
- Strong audit trail across both human and automated access in one place
Cons
- Not designed primarily for infrastructure workflows, so complex environment modelling is less natural than in purpose-built tools
- Connect server deployment adds infrastructure for high-throughput automated access
- Pricing is oriented around user seats, which fits human use better than large-scale machine use
- No dynamic secret generation, so short-lived credentials still need to come from elsewhere
Best for: Organisations already standardised on the product for human credentials who want pipeline secrets in the same place with the same offboarding.
Pricing: Business plan pricing per user seat, with service accounts and connect access included at the appropriate tiers rather than metered separately.
SOPS
SOPS takes the opposite approach to every hosted product here: secrets live encrypted in your git repository, with encryption keys held in a cloud key management service or a hardware-backed key. It encrypts values while leaving keys and structure in plaintext, so an encrypted file still diffs meaningfully in review. There is no service to run and no availability dependency at deploy time beyond the key management service.
Pros
- No secrets service to operate or keep available, which removes an entire class of deployment dependency
- Encrypted files diff readably in code review because structure stays plaintext and only values are encrypted
- Access control delegates to your key management service, so permissions live with your existing cloud identity
- Works identically in air-gapped and on-premises environments where hosted products cannot reach
Cons
- Rotation means re-encrypting and committing files, so every rotation is a code change through review
- Revoking access to someone who already cloned the repository is impossible, since they hold the ciphertext forever
- No audit trail of secret reads, only of key usage, which is coarser than most compliance requirements want
- Git history retains every previously encrypted value, so a compromised key exposes the full history
Best for: GitOps workflows and air-gapped environments where the deployment path must not depend on an external secrets service.
Pricing: Free and open source with no vendor and no bill. The only cost is the cloud key management service that holds the encryption keys, metered per key and per operation.
External Secrets Operator
External Secrets Operator solves one narrow problem well: getting secrets from an external store into Kubernetes as native secret objects. It runs as a controller, reads a custom resource that says which external secret maps to which Kubernetes secret, fetches it using a service account identity, and keeps it refreshed. It is not a store and does not try to be, which is why it appears alongside rather than instead of the others.
Pros
- Decouples the secret store from the workload, so changing stores does not change application manifests
- Kubernetes workloads consume ordinary secret objects with no application awareness of where they came from
- Supports many backends, which makes a multi-cloud or migrating estate manageable from one controller
- Refreshes on an interval, so a rotated secret propagates without a redeployment
Cons
- Materialises secrets as Kubernetes secret objects, which are base64 encoded rather than encrypted at rest unless you configure that separately
- Another controller in the critical path, and its failure surfaces as workloads with stale or missing secrets
- Refresh interval creates a window where the cluster holds an old value after rotation
- Solves delivery only, so you still need a store and all of its decisions
Best for: Kubernetes estates that want workloads to consume native secret objects while the source of truth stays in an external store.
Pricing: Free and open source with no vendor and no bill. Cost is the controller’s compute footprint plus whatever the backing secret store charges.
How to choose
Do these in order. The first two are not tool choices and they matter more than the tool choice.
First, adopt OIDC federation for every cloud credential. This is not optional and it is not a product decision. It removes the majority of long-lived secrets from your pipelines and it removes the majority of your rotation backlog. Do it before evaluating anything below.
Second, scope what remains by environment and branch. Production credentials assumable only from the default branch or a release tag, fork pull requests getting nothing. This is configuration, not procurement.
Then pick a store for what is genuinely left, which is third-party credentials with no federation support and application run-time secrets.
If your workloads are on one cloud, use that cloud’s native secret store. It is already integrated with your identity, encryption and audit systems, and it introduces nothing new to operate.
If you have many systems, strict audit requirements and a platform team, Vault, primarily for dynamic secrets, which give you federation-like short lifetimes for systems that do not support federation.
If you want developer-friendly environment modelling without operating anything, Doppler. If you want the same but self-hosted, Infisical.
If your deployment path must not depend on an external service, or you are air-gapped, SOPS.
If you are already standardised on 1Password for human credentials, its automation tier saves you running a second system and fixes offboarding.
And if you run Kubernetes, External Secrets Operator sits alongside whichever store you chose. It is a delivery mechanism, not an alternative.
| Tool | Type | Runs where | Short-lived credentials | Picks itself when |
|---|---|---|---|---|
| OIDC federation (standard, not a product) | Identity exchange | CI platform and cloud | Yes, by design | Always, for cloud credentials |
| HashiCorp Vault | Platform | Self-hosted or managed | Dynamic secrets | Many systems, strict audit, platform team exists |
| AWS Secrets Manager | Cloud native | AWS | Via federation and rotation | Workloads are on AWS |
| Google Secret Manager | Cloud native | Google Cloud | Via federation | Workloads are on Google Cloud |
| Doppler | Hosted platform | Vendor | No | Environment modelling matters, ops budget does not exist |
| Infisical | Open core platform | Self-hosted or vendor | Dynamic secrets for some backends | Same, but self-hosting is required |
| 1Password Secrets Automation | Hosted platform | Vendor | No | Already standardised for human credentials |
| SOPS | Encrypted files | Your repository | No | GitOps or air-gapped, no service dependency allowed |
| External Secrets Operator | Delivery controller | Your cluster | Inherits from the store | Kubernetes workloads need native secret objects |
Frequently asked questions
Is OIDC federation actually better than a rotated access key?
Yes, and by a wide margin. A rotated key still exists between rotations, is still readable by anyone who can edit CI configuration, and still works from anywhere until revoked. A federated credential is minted per job, expires in minutes, and can be restricted by branch and environment in the trust policy, which a static key cannot express at all. The rotation you were doing manually becomes a property of the system. The one thing to get right is the trust policy condition, which must pin the subject claim, not just the issuer and repository.
Where should I store secrets that have no federation support?
In a secret store, fetched at the point of use by a job that has itself authenticated through federation. That is the composition that matters: federation gets the job an identity, the identity permits reading exactly the secrets that job needs, and the secret never lives in the CI platform’s settings. A third-party API key stored this way is still long-lived, so it still needs rotation, but access to it is now scoped and audited rather than ambient.
How do I stop secrets appearing in build logs?
Do not print them, because masking is a filter that fails on encoded, transformed and split values. Concretely: avoid putting secrets in environment variables where a crash handler or diagnostic tool will dump them, never enable shell tracing in a job handling credentials, fetch secrets immediately before use rather than exporting them at the top, and scan log output for credential patterns rather than only scanning source code. Treat any secret that does appear as compromised and rotate it; deleting the log does not help, because it has already been replicated.
How often should I rotate CI secrets?
The better question is how many you can eliminate, because every credential moved to federation needs no rotation at all. For what remains, rotate on a schedule you can actually automate, and immediately on any suspected exposure or when someone with access leaves. The reason rotation schedules slip is that manual rotation breaks pipelines, so build the overlap pattern, two credentials valid at once, verify the old one is unused across a full cycle of your longest scheduled job, then revoke.
Do self-hosted runners change any of this?
They change the threat model significantly. A self-hosted runner persists between jobs, so anything written to disk, left in a cache, or present in the environment can be read by the next job, including one from a different pull request. Federation helps because the credential expires, but a runner handling untrusted code should get no privileged identity at all. Ephemeral runners that are destroyed after one job are the correct configuration, and the operational implications are covered in self-hosted CI/CD tools.
Does this replace application runtime secrets management?
No. These are two problems that share tooling. Pipeline secrets authorise the build and deploy process; runtime secrets are what the running application needs to talk to its database and its dependencies. The same store usually serves both, but the delivery mechanism differs, and on Kubernetes that delivery is what External Secrets Operator handles. Keeping the two concerns distinct is what stops a pipeline credential quietly becoming an application credential.
Related reading
- Best CI/CD platforms — where federation is configured and how environments gate a deployment.
- Best self-hosted CI/CD tools — runner isolation, which changes the secrets threat model entirely.
- Best software supply chain security and SBOM tools — the signing side of the same identity, and what a compromised pipeline can do.
- Best GitOps tools — where encrypted-in-git secrets fit the deployment model.
- Best infrastructure as code tools — why state files are a credential store you did not plan for.
- Best API authentication tools — the same token exchange mechanics applied to service-to-service calls.