Armada: Analysis of helm upgrade/rollback --force/--recreate-pods

What do the flags do

Both rollback and upgrade are moving to a different version of the release, just old vs new, so these flags have same behavior for both rollback and upgrade.

--force

Does 2 things:

1. (as of v2.5.0) Helm always tries to update resources via a kubernetes patch. If the patch fails then normally the error is added to a list of errors to later return and it continues on to the next resource, but with --force it will instead delete and re-create the resource that could not be patched. It uses kubernetes reapers to do the deletes. The failed release update also causes armada to halt.

• original github issue
• code

2. (as of v2.9.0) For upgrade only if it cannot build an updated release and force is specified, it will delete and re-create the entire release.

• original PR

--recreate-pods

Deletes any pods in the release which causes them to get recreated by their controllers (deployments, daemonsets, etc). This should already happen via e.g. a rolling update if the controller's pod template itself changed, so this flag is for cases where that didn't change, but we want to re-initialize the pods anyways. For example, perhaps some dependency of the init containers of the pods has changed, or a config map has changed, and the containers need to re-initialize with it. The pods are deleted directly in quick succession, so could lead to downtime of services while waiting for the controller to restart the pods.

• original github issue
• code

Should we expose within Armada?

--force

Seems reasonable to have the option, defaulted to false. Deleting and re-creating a resource could obviously be disruptive, but in cases where the resource can't be patched, perhaps this is the least disruptive option, at least through helm. Manual intervention directly through kubernetes would probably be required otherwise. If this scenario is encountered in an upgrade/rollback in one environment then it should be possible to predict the need to use this option when promoting to the next environment.

--recreate-pods

Seems reasonable to have the option, defaulted to false. Could be necessary for cases where the pod's controller is not changed, but pod re-initialization is still required, especially config map / secret updates. Restarting just the pods of a certain controller, rather than all pods in a release could be a more targeted way to handle, but that is not available via helm.

How to expose?

Armada chart manifests already have an upgrade key under which could add an options key, to hold options to pass through directly to helm, under which we can add force and recreatePods keys.

upgrade:
  options:
    force: true
    recreatePods: true

If/when armada adds rollback functionality, a parallel structure can be added as well:

rollback:
  options:
    force: true
    recreatePods: true