Adding tracepoints to the Concourse worker's healthcheck on k8s

install go

Make sure go is installed on your worker -- this should be possible to do by running something like

kubectl -n <namespace> exec <worker pod> -- apt-get install -y golang-go

install delve

download delve, with something like

curl -Lo delve.zip 'https://bintray.com/jetbrains/golang/download_file?file_path=com%2Fjetbrains%2Fdelve%2F1.1.35%2Fdelve-1.1.35.zip'
unzip delve.zip
chmod +x dlv/linux/dlv

Copy it to your worker:

kubectl cp dlv/linux/dlv <namespace>/<worker pod>:/usr/local/bin/dlv

attach dlv to worker process

create the file init.dlv with the following contents:

trace startgarden healthchecker.go:51
trace startbaggageclaim healthchecker.go:58
trace endhealthcheck healthchecker.go:65
continue

copy it to the worker and use it to initialize a dlv debugging session:

kubectl cp init.dlv <namespace>/<worker pod>:/tmp/init.dlv
kubectl -n <namespace> exec -it <worker pod> -- \
  bash -c "dlv attach \$(pgrep concourse) --init /tmp/init.dlv"

you should periodically see log lines getting emitted each time the relevant tracepoints are hit, with output like:

> goroutine(42106): [startgarden] github.com/concourse/concourse/worker.(*healthChecker).CheckHealth(*(unreadable input/output error), ("*github.com/concourse/concourse/worker.healthChecker")(0xc000c66a80), net/http.ResponseWriter(*net/http.response) 0xbeef0008)
> goroutine(42106): [startbaggageclaim] github.com/concourse/concourse/worker.(*healthChecker).CheckHealth(*(unreadable input/output error), ("*github.com/concourse/concourse/worker.healthChecker")(0xc000c66a80), net/http.ResponseWriter(*net/http.response) 0xbeef0008)
> goroutine(42106): [endhealthcheck] github.com/concourse/concourse/worker.(*healthChecker).CheckHealth(*(unreadable input/output error), ("*github.com/concourse/concourse/worker.healthChecker")(0xc000c66a80), net/http.ResponseWriter(*net/http.response) 0xbeef0008)

If your healthcheck is becoming seriously slow, there should be a noticeable pause between subsequent log lines -- this will tell you which function is slow!

To end your dlv session, hit Ctrl-C to pause execution and Ctrl-D to detach dlv from the running process. Do not kill the process when detaching.