Git 2.13 introduced conditional configuration includes. For now, the only supported condition is matching the filesystem path of the repository, but that’s exactly what we need in this case.
You can configure your conditional includes by adding them to the bottom of your home directory’s ~/.gitconfig file:
[includeIf "gitdir:~/work/"]
path = ~/.gitconfig-work
[includeIf "gitdir:~/specific-project/"]
path = ~/.gitconfig-specific-project