Before we get into attaching debugger to Go application running on Kubernetes cluster, let's list a few debugging scenarios you might encounter:
debugging running application with stripped symbols(go build -ldflags='-s -w') - probably the hardest scenario. To start debugging session you will need to get debugging symbols package for used build ID.
debugging running application with optimizations applied(go build) - while you will be able to launch debugger session on such process, you may encounter problems with inspecting application state, e.g.:
(dlv) args
r = (unreadable empty OP stack)
w = net/http.ResponseWriter(*net/http.response) 0xbeef0008
debugging running application with debugging symbols and without optimizations(go build -gcflags='all=-N -l') - the easiest case, where you have all required information in format understandable by debugger
I'm going to focus only on the last scenario using Delve.
Let's start with hello world HTTP server application:
main.go:
package main
import (
"net/http"
"fmt"
)
func handleSlash(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello %s!\n", r.Host)
}
func main() {
http.HandleFunc("/", handleSlash)
http.ListenAndServe(":8080", nil)
}
Dockerfile:
FROM scratch
COPY hello-world /
ENTRYPOINT ["/hello-world"]
deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
labels:
app: hw
spec:
replicas: 1
selector:
matchLabels:
app: hw
template:
metadata:
labels:
app: hw
spec:
imagePullSecrets:
- name: registry.local
containers:
- name: hw
image: registry.local/hw:latest
imagePullPolicy: Always
We will need to build application without optimizations:
$> go build -gcflags='all=-N -l'
$> docker build -t registry.local/hw:latest .
$> docker push registry.local/hw:latest
Non-optimized builds are actually used internally by
Delve, if you invoke its debug subcommand.
Now let's create Kubernetes deployment of our application:
$> kubectl apply -f deployment.yaml
We will need separate pod for Delve located on the same node as out application:
$> kubectl get pod hello-world-77457544f4-fglbh -ojsonpath='{.spec.nodeName}{"\n"}'
minikube
pod.yaml:
apiVersion: v1
kind: Pod
metadata:
name: debug
spec:
hostPID: true
containers:
- name: delve
image: golang:1.11-alpine
command: ["/bin/cat"]
tty: true
nodeSelector:
kubernetes.io/hostname: minikube
Some clusters policies might prevent you from creating container in host PID namespace for security.
$> kubectl apply -f pod.yaml
We invoke cat just to sustain pod, so we can enter it with exec:
$> kubectl exec -it debug sh
We will have to locate our application's process and attach Delve to it. Let's find out container ID of hello-world:
$> kubectl get pod hello-world-77457544f4-fglbh \
-ojsonpath='{.status.containerStatuses[?(@.name=="hw")].containerID}{"\n"}' \
| cut -d / -f 3
b686d81723e3c714ddb08fe398dafd62d4456f0211991658b27be1d41d39b0b7
With hostPID: true we have full access to every process on node, even if it's not related to current pod. Docker spawns shim with ID in process name for each container. Busybox version of ps lacks ability to list processes in tree structure, so we will need to install procps and search for hello-world application process.
$> apk update
$> apk add procps
$> ps -eo pid,cmd --forest | grep -A 1 b686d81723e3c714ddb08fe398dafd62d4456f0211991658b27be1d41d39b0b7
7803 \_ docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/b686d81723e3c714ddb08fe398dafd62d4456f0211991658b27be1d41d39b0b7 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
7820 | \_ /hello-world
Now we know that our application is running with process ID 7820. Attaching to it will a matter of installing Delve and writing script to automatically start tracing:
$> apk add git
$> go get -u github.com/derekparker/delve/cmd/dlv
$> cat > trace.dlv
trace handleSlash main.go:10
on handleSlash print w.req.RemoteAddr
continue
$> dlv attach 7820 --init trace.dlv
Type 'help' for list of commands.
Tracepoint handleSlash set at 0x6e826d for main.handleSlash()
/home/jakub/dev/hello-world/main.go:10
> [handleSlash] main.handleSlash() /home/jakub/dev/hello-world/main.go:10 (hits
> goroutine(35):1 total:1) (PC: 0x6e826d)
data.req.RemoteAddr: "127.0.0.1:34868"
Above configuration will let us display source port of every connection handled by hello-world application. It's not really useful, but shows what you can achieve with Delve.
Method described above should be used only in emergency or development environments. It's quite easy to: