Simple way to merge yaml files using yq

Unbeknownst to me, yq possesses the ability to merge whole yaml files on top of one another in order. I had been working around this ignorance using scriptlets that would set values as they were discovered while trapsing a file, so I usually avoided doing it altogether. Many of you know I am NOT a fan of complex scripting.

But after doodling around a bit with yq, I found the reduce/ireduce commands and was able to condense down the ordered/layered merging of N yaml files using a very simple yq one liner.

Simple and effective:

This was on yq version 4.44.3, so the command may differ on older versions as I think the ireduce command is more recent.

Consider my files below where defaults.yaml defines a baseline configuration I want to use in a kubernetes command (or similar) and the overrides.yaml file is a bespoke alteration to that configuration that I want to use for a specific use case without modifying the defaults file and having to make a whole new copy of it.

defaults.yaml:

a: 1
b:
  x: 10

overrides.yaml:

b:
  x: 99
c: 3

Using yq I can merge these files into an output with a one line command:

yq ea '. as $item ireduce ({}; . * $item )' defaults.yaml override.yaml

The above command uses ea as short-hand for eval-all, and then loops over all input files (defaults.yaml and override.yaml) and applies the command to them. The command is ireduce which takes an empty config {} and merges * with each item in the loop $item.

The result:

❯ yq ea '. as $item ireduce ({}; . * $item )' defaults.yaml override.yaml
a: 1
b:
  x: 99
c: 3

Cool eh?

** WARNING ** Be aware that this merge will NOT do a deep merge correctly. Even when using the *d token in the ireduce function, if you have an array of differing values, the last version of that array will clobber the value entirely. I spent hours trying to work around this but it seems to be a bug in ireduce currently.

Add in bash-isms to be very powerful

So if I take the above command but want to add a generic list of files that fit a pattern, I can use an inline bash statement to generate the list for me!

yq ea '. as $item ireduce ({}; . * $item )' defaults.yaml $(ls -v overrides_*.yaml)

This looks for any value matching overrides_*.yaml and add them to the argument list in order. So overrides_1.yaml and overrides_2.yaml would be applied in order. This also will do alphabetical too since its just strings.