Artifact management is where CI/CD pipelines either scale gracefully or collapse into a tangle of untraceable images, overwritten tags, and 45-minute builds. In my experience running platform teams supporting 80+ microservices, the registry and caching layer gets treated as an afterthought until the first major incident - a production rollback fails because :latest was mutated, or a Helm deploy pulls a chart that no longer exists upstream. Getting artifact management right means treating every build output as an immutable, verifiable, and efficiently cached asset. This article walks through how I design container registries, Helm chart repositories, and build caching for pipelines that need to be fast, reproducible, and auditable.
Why Your Registry Topology Matters More Than Your Registry Vendor
Teams agonize over which registry to pick - ECR, Harbor, Artifactory, GitHub Container Registry - but the topology you choose has a bigger impact on day-to-day reliability than the vendor logo. In my experience, a single monolithic registry with flat namespaces creates permission sprawl and makes cleanup dangerous. I prefer a project or team-based topology with clear isolation boundaries.
A practical pattern I use is registry.company.com/{team}/{app} with separate projects for production-promoted images versus development snapshots. For example, platform/payments-api lives alongside platform/payments-api-candidate. CI pushes every commit to the candidate repository. Only images that pass integration tests and vulnerability scans get promoted to the stable path. This keeps RBAC simple: developers have write access to their team project, but promotion to stable requires a pipeline identity via OIDC, not a human credential.
For multi-region deployments, consider pull-through caches or replicated registries rather than forcing every cluster to pull across regions. On AWS, ECR replication to secondary regions cut our image pull p99 by 40% for workloads running in eu-west-1 while building in us-east-1. Harbor's replication policies give you similar control on-prem. The key is to make the registry close to where you build and close to where you run.
Choosing Between Hosted and Self-Hosted Registries
Hosted registries (ECR, GCR, ACR, GHCR) remove operational overhead but lock you into their auth and retention models. Self-hosted Harbor or Nexus gives you fine-grained retention, vulnerability scanning with Trivy or Clair, and OCI artifact support without per-seat costs. I've found that Harbor is the sweet spot for teams that need on-prem control and want to unify containers and Helm charts in one place. If you're already deep in AWS IAM and need seamless IRSA integration for EKS, ECR is hard to beat - just be aware of its relatively limited lifecycle policy syntax compared to Harbor.
Whatever you choose, enforce authentication via short-lived tokens. Long-lived docker config credentials in CI secrets are a frequent source of leaks. When I wire registry auth into GitHub Actions, I rely on the OIDC federation patterns described in GitHub Actions Advanced Workflows: Matrix Builds, Reusable Actions and OIDC to exchange a workflow token for temporary registry credentials. No static secrets to rotate, and every push is attributable to a specific workflow run.
Immutable Tagging and Image Promotion Strategies That Survive Rollbacks
The single most important rule for container artifact management is immutability. If a tag can be overwritten, it will be overwritten at the worst possible moment. I enforce a three-part tagging scheme for every image build: a content-addressable tag (the git SHA), a human-readable build tag (like v1.4.2-3-gabc123), and a floating environment tag that is only moved by promotion.
For example, a CI build produces:
# Build and tag immutably
docker buildx build \
--tag registry.company.com/platform/payments-api:sha-a1b2c3d \
--tag registry.company.com/platform/payments-api:v1.4.2-3-gabc123 \
--push .
# Promotion is a server-side copy, not a rebuild
crane copy \
registry.company.com/platform/payments-api:sha-a1b2c3d \
registry.company.com/platform/payments-api:stage
# After smoke tests pass in stage, promote to prod
crane copy \
registry.company.com/platform/payments-api:sha-a1b2c3d \
registry.company.com/platform/payments-api:prod
Notice that promotion uses crane copy or skopeo copy - a registry-to-registry manifest copy without re-pulling or re-pushing layers. The SHA tag never moves. Environment tags like :stage and :prod are mutable pointers, but they only ever point to an existing SHA. This gives you instant rollbacks: to revert, you copy the previous SHA back to :prod. No rebuild, no drift.
I also enforce tag immutability at the registry level. In Harbor, enable "immutable tags" matching sha-* and v*.*.*. In ECR, use a lifecycle policy combined with IAM deny on ecr:PutImage when the tag already exists (via a permissions boundary). The Docker Documentation on Dockerfile best practices emphasizes reproducible builds, but reproducibility is meaningless if the registry lets you clobber the artifact after the fact.
What to Store in Labels and Annotations
Don't rely on tags alone for traceability. I add OCI annotations at build time so any image can be traced back to its source without guessing:
LABEL org.opencontainers.image.revision="a1b2c3d"
LABEL org.opencontainers.image.source="https://github.com/org/payments-api"
LABEL org.opencontainers.image.created="2025-08-14T18:32:00Z"
LABEL com.company.pipeline.run-id="123456"
LABEL com.company.pipeline.commit-msg="fix: handle idempotency key collision"
These labels show up in docker inspect and registry catalog APIs, and they make incident response far faster. When an alert fires at 2 AM, you want to know exactly which commit is running, not parse a timestamp from a tag.
Helm Chart Packaging, Versioning and the OCI Registry Shift
Helm charts are artifacts too, and they deserve the same immutability guarantees as container images. I've seen too many teams serve charts from a static S3 bucket or ChartMuseum instance with no versioning discipline, where helm upgrade silently pulls a chart that was overwritten last week.
Chart versioning should follow SemVer strictly, and it should be decoupled from app versioning. In Chart.yaml, bump version for any chart change (templating, values schema) and appVersion for the container it deploys. I automate this in CI: chart version is derived from conventional commits or a chart/Chart.yaml change, not manually edited.
# Chart.yaml - keep version and appVersion distinct
apiVersion: v2
name: payments-api
description: Helm chart for payments-api service
type: application
version: 0.8.3 # chart version - incremented on any chart change
appVersion: "1.4.2" # default app image tag
dependencies:
- name: postgresql
version: "12.8.1"
repository: "oci://registry.company.com/charts/postgresql"
condition: postgresql.enabled
The biggest shift in the last two years is moving Helm charts to OCI registries. Instead of maintaining a separate ChartMuseum or Harbor chart repository, you push charts as OCI artifacts to the same registry that holds your images. This simplifies auth, replication, and backup. Helm 3.8+ supports this natively:
# Package and push as OCI artifact
helm package ./chart --version 0.8.3 --app-version 1.4.2
helm push payments-api-0.8.3.tgz oci://registry.company.com/charts
# Deploy directly from OCI
helm upgrade --install payments-api \
oci://registry.company.com/charts/payments-api --version 0.8.3 \
--set image.tag=sha-a1b2c3d \
--namespace payments --create-namespace
In my experience, OCI-based charts reduce operational toil significantly. You get a single URL scheme, single credential helper, and single retention policy. The Kubernetes Documentation now documents OCI image pull behavior that applies equally to charts when using OCI registries. One caveat: if you use Argo CD or Flux, verify OCI support in your version - Flux's Helm controller has supported OCI since v0.27, but older Argo CD versions required an explicit enablement flag.
Dependency Locking and Provenance
Helm dependencies are a supply chain risk. I always commit Chart.lock and run helm dependency update in CI, failing the build if the lock file drifts. For provenance, enable Helm signing with helm package --sign --key and publish the .prov file alongside the chart. Even if you don't enforce verification on every cluster today, having provenance gives you an audit trail for later.
Building a Unified Artifact Store on OCI Registries
OCI distribution spec has quietly become the universal artifact API. Beyond images and charts, you can store SBOMs, signatures, Terraform modules, and even OPA policies as OCI artifacts. I recommend consolidating on one registry that speaks OCI natively and using it as your artifact control plane.
A unified layout I've used successfully looks like this:
registry.company.com/{team}/{artifact-type}/{name} - for example, registry.company.com/platform/containers/payments-api, registry.company.com/platform/charts/payments-api, and registry.company.com/platform/sboms/payments-api. This keeps RBAC and replication rules consistent. A team gets a namespace, not three separate systems to manage.
Using oras you can push arbitrary artifacts:
# Attach SBOM to an image using OCI referrers
oras attach --artifact-type application/spdx+json \
registry.company.com/platform/containers/payments-api:sha-a1b2c3d \
./sbom.spdx.json
# Push a Terraform module as OCI artifact
oras push registry.company.com/platform/terraform-modules/vpc:1.2.0 \
./module.zip:application/vnd.terraform.module.v1+zip
This unification pays off when you implement supply chain security. Instead of chasing signatures across three systems, you query one registry for all artifacts related to a given SHA. The approach aligns well with the hardening and attestation workflow detailed in CI/CD Security: Pipeline Hardening, SLSA and Software Supply Chain, where the registry becomes the source of truth for what was built, by whom, and with what dependencies.
Making Build Caching Actually Work: Layers, BuildKit and Remote Caches
Build caching is where teams leave the most pipeline time on the table. I've cut 12-minute Docker builds to 90 seconds without changing application code, just by fixing cache invalidation and moving to remote caches.
Start with Dockerfile layer ordering. Every instruction creates a layer, and BuildKit invalidates all subsequent layers when one changes. Put the least frequently changing instructions first:
# Bad: COPY . invalidates the npm install layer on any file change
# COPY . /app
# RUN npm ci
# Good: copy dependency manifests first, install, then copy source
FROM node:20-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci
FROM deps AS build
COPY src ./src
COPY tsconfig.json ./
RUN npm run build
FROM gcr.io/distroless/nodejs20-debian12
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER nonroot
ENTRYPOINT ["node", "dist/server.js"]
The --mount=type=cache instruction is a BuildKit cache mount that persists package manager caches across builds without bloating the final image. Combined with multi-stage builds, it avoids shipping build tools into production.
For CI, local layer caching is not enough - runners are ephemeral. You need a remote cache backend. BuildKit supports registry caches and S3/GHA caches:
docker buildx build \
--cache-from type=registry,ref=registry.company.com/platform/cache/payments-api:buildcache \
--cache-to type=registry,ref=registry.company.com/platform/cache/payments-api:buildcache,mode=max,compression=zstd \
--push \
--tag registry.company.com/platform/containers/payments-api:sha-a1b2c3d .
Use mode=max to cache all stages, not just the final one. I've found registry caches more reliable than tarball caches in GitHub Actions or GitLab, because they are content-addressed and shared across branches. For GitLab runners, the distributed cache configuration covered in GitLab CI Pipeline Optimization: DAG, Cache and Parallel Jobs pairs well with BuildKit registry caching - DAG dependencies ensure dependent jobs reuse the same cache key.
Cache Poisoning and Correctness
A stale cache is worse than no cache. I tag cache refs with both a stable branch key (main-buildcache) and a commit-specific fallback, and I set explicit cache invalidation when lock files change. Avoid --cache-from type=gha with overly broad scopes in monorepos - it can cause a job to reuse layers from an unrelated service. Scope caches per service and per architecture if you build multi-arch images.
Retention, Signing and Scanning: Keeping Your Artifact Estate Sane and Secure
Registries grow forever unless you design retention up front. I've walked into registries with 3TB of untagged images and no way to know what's safe to delete. Implement retention on day one.
Retention rules I apply to every project:
Keep all SHA-tagged images for 90 days, keep SemVer-tagged releases for 1 year, keep main-buildcache(code> layers for 7 days, and prune untagged manifests after 24 hours. In Harbor, this is a retention policy with cron. In ECR, it's a lifecycle policy JSON. Document the policy and make it visible to developers - otherwise they will assume artifacts are permanent and build hidden dependencies on old tags.
Scanning and signing should be blocking gates, not advisory dashboards. I run Trivy or Grype as a post-build step and block promotion if critical CVEs without a waiver are present. For signing, Cosign with keyless OIDC is the least painful option I have found:
# Sign with Cosign keyless (Fulcio + Rekor)
cosign sign --yes registry.company.com/platform/containers/payments-api:sha-a1b2c3d
# Verify in cluster with Kyverno or Sigstore policy-controller
cosign verify \
--certificate-identity-regexp "https://github.com/org/.*" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
registry.company.com/platform/containers/payments-api:sha-a1b2c3d
This creates a Rekor transparency log entry that anyone can audit. When combined with an admission controller, only signed images from trusted builders can run. That's the foundation of SLSA Build L2+ without managing your own PKI.
| Artifact Concern | Anti-Pattern | Recommended Approach |
|---|---|---|
| Image Tagging | Mutable :latest overwritten per commit |
Immutable SHA + SemVer tags, promote via copy |
| Helm Distribution | HTTP ChartMuseum with no auth or versioning | OCI registry push with Chart.lock committed |
| Build Cache | Local cache only on ephemeral runners | Remote registry cache with mode=max and scoped keys |
| Retention | Manual cleanup or never delete | Policy-driven lifecycle: 90d SHA, 7d cache, 1y releases |
| Security Gate | Scan reports ignored, unsigned images allowed | Blocking scan + Cosign keyless signing + admission control |
Finally, treat registry storage as critical infrastructure. Replicate it, back up its metadata, and monitor pull error rates. A registry outage is a deployment outage. I alert on registry_http_requests_duration_seconds and registry_storage_errors_total if self-hosting, and on ECR Throttling metrics if hosted.
Frequently Asked Questions
Should I use one registry for all environments or separate registries per environment?
Use one logical registry with separate projects or repositories per environment stage, not separate physical registries. A single registry gives you unified auth, scanning, and replication. Separate projects like team/app-candidate, team/app-stage, and team/app-prod provide environment isolation while allowing server-side promotion with crane copy. Separate physical registries only make sense for strict air-gapped compliance boundaries.
How do I handle Helm chart dependencies that live in different registries?
Centralize with OCI and use authenticated pulls in CI. Declare dependencies with oci:// URLs in Chart.yaml and run helm dependency update before packaging. Store credentials via helm registry login using the same OIDC-derived token you use for images. Commit Chart.lock to lock digests, and mirror upstream charts you depend on into your own registry to avoid outages when public registries are rate-limited or down.
When does remote build caching hurt more than it helps?
When cache keys are too broad or builds are not reproducible. If multiple services share a single buildcache tag in a monorepo, you get false hits and subtle correctness bugs. Scope caches per service and per platform (linux/amd64 vs linux/arm64). Also, if your Dockerfile copies large contexts that change every commit, cache hit rate will be near zero - fix layer ordering and use .dockerignore before adding remote caching.
Do I need to sign Helm charts if I already sign container images?
Yes, if you want end-to-end supply chain integrity. Image signatures prove the container is trusted, but chart signatures prove the deployment manifest that references that container is trusted. An attacker could publish a chart that points to a legitimate signed image but injects a malicious sidecar or environment variable. Sign both and verify charts at the CD layer with Helm's provenance check or a policy engine in the cluster.