Why use-compiled-releases.yml is Stuck on Stemcell 1.423

Summary

The operations/use-compiled-releases.yml file in cf-deployment references stemcell version 1.423 for all compiled releases, even though newer stemcells are available. This is due to how the CI pipeline is configured to only recompile all releases on major stemcell version bumps.

Root Cause

1. The update-stemcell-and-recompile-releases Job Only Triggers on Major Bumps

In ci/template/update-releases.yml, the job that recompiles all releases is only triggered by stemcell-version-bump-major:

- name: update-stemcell-and-recompile-releases
  public: true
  serial: true
  plan:
  - do:
    - in_parallel:
      # ...
      - get: stemcell
        params:
          tarball: false
        passed:
        - acquire-all-pre-dev-locks   # <-- This is triggered by stemcell-version-bump-major

And acquire-all-pre-dev-locks is triggered by:

- name: acquire-all-pre-dev-locks
  public: true
  plan:
  - in_parallel:
    - get: stemcell-version-bump-major  # <-- Only major bumps!
      trigger: true

2. Individual Release Updates Inherit the Old Stemcell

When individual releases are updated (e.g., when a new version of capi is released), the update job uses a stemcell that has passed: [update-stemcell-and-recompile-releases]:

From ci/template/update-releases.yml lines 370-375:

- get: stemcell
  params:
    tarball: false
  passed:
  - update-stemcell-and-recompile-releases  # <-- Requires stemcell from last major bump

This means every individual release update uses the same stemcell version that was used in the last full recompilation (1.423).

3. The Stemcell Version Comes from the Compiled Tarball Filename

The update-single-compiled-release task calls Go code that extracts release metadata from the tarball filename.

In compiledreleasesops/compiled_releases_opsfile.go:

func getCompiledReleaseForBuild(buildDir, releaseName string) (Release, error) {
    // ...
    release.Version, release.Stemcell.Version, release.Stemcell.OS, err = common.InfoFromTarballName(releaseTarballName, releaseName)
    // ...
}

The tarball filename contains the stemcell version it was compiled against (e.g., capi-1.229.0-ubuntu-jammy-1.423-20260302-122815-07330624.tgz), so the ops file inherits 1.423.

Why No Major Bump Has Occurred

Ubuntu Jammy stemcells are still on version 1.xxx. Since there hasn't been a major version bump (e.g., to 2.0), the update-stemcell-and-recompile-releases job hasn't been triggered since stemcell 1.423.

Impact

• All compiled releases in use-compiled-releases.yml reference stemcell 1.423
• This doesn't break deployments (BOSH allows using compiled releases with newer stemcells of the same OS family)
• However, the compiled releases don't benefit from any fixes/improvements in newer stemcells

Potential Fixes

1. Trigger recompilation on minor bumps too: Modify acquire-all-pre-dev-locks to also trigger on stemcell-version-bump-minor

2. Periodic recompilation: Add a time-based trigger to update-stemcell-and-recompile-releases (e.g., monthly)

3. Threshold-based recompilation: Trigger recompilation when the stemcell version drifts too far from the current compiled version

Relevant Source Code Links

• ci/template/update-releases.yml - Main CI pipeline template
• update-stemcell-and-recompile-releases job (L818-910)
• acquire-all-pre-dev-locks job (L793-816)
• Individual release update stemcell constraint (L370-375)
• runtime-ci: update-single-compiled-release task
• runtime-ci: compiled_releases_opsfile.go
• operations/use-compiled-releases.yml - The affected ops file