# 2018-11-11 Debugging Go in Kubernetes

Before we get into attaching debugger to Go application running on Kubernetes
cluster, let's list a few debugging scenarios you might encounter:

- debugging running application with stripped symbols(*go
  build -ldflags='-s -w'*) - probably the hardest scenario. To start debugging
  session you will need to get debugging symbols package for used build ID.
- debugging running application with optimizations applied(*go build*) - while you will
  be able to launch debugger session on such process, you may encounter problems
  with inspecting application state, e.g.:

  ```
  (dlv) args
  r = (unreadable empty OP stack)
  w = net/http.ResponseWriter(*net/http.response) 0xbeef0008
  ```

- debugging running application with debugging symbols and without optimizations(*go
  build -gcflags='all=-N -l'*) - the easiest case, where you have all required
  information in format understandable by debugger

I'm going to focus only on the last scenario using
[Delve](https://github.com/derekparker/delve).

## Example application

Let's start with *hello world* HTTP server application:

*main.go*:

```
package main

import (
  "net/http"
  "fmt"
)

func handleSlash(w http.ResponseWriter, r *http.Request) {
  fmt.Fprintf(w, "Hello %s!\n", r.Host)
}

func main() {
  http.HandleFunc("/", handleSlash)
  http.ListenAndServe(":8080", nil)
}
```


*Dockerfile*:

```
FROM scratch
COPY hello-world /
ENTRYPOINT ["/hello-world"]
```

*deployment.yaml*:

```
apiVersion: apps/v1
kind: Deployment
metadata:
 name: hello-world
 labels:
   app: hw
spec:
 replicas: 1
 selector:
   matchLabels:
     app: hw
 template:
   metadata:
     labels:
       app: hw
 spec:
   imagePullSecrets:
     - name: registry.local
   containers:
     - name: hw
       image: registry.local/hw:latest
       imagePullPolicy: Always
```


## Deployment

We will need to build application without optimizations:

```
$> go build -gcflags='all=-N -l'
$> docker build -t registry.local/hw:latest .
$> docker push registry.local/hw:latest
```

> Non-optimized builds are actually used internally by `Delve`, if you invoke its
  *debug* subcommand.

Now let's create Kubernetes deployment of our application:

```
$> kubectl apply -f deployment.yaml
```

## Attaching debugger

We will need separate pod for *Delve* located on the same node as out
application:

```
$> kubectl get pod hello-world-77457544f4-fglbh -ojsonpath='{.spec.nodeName}{"\n"}'
minikube
```

*pod.yaml*:

```
apiVersion: v1
kind: Pod
metadata:
 name: debug
spec:
 hostPID: true
 containers:
   - name: delve
     image: golang:1.11-alpine
     command: ["/bin/cat"]
     tty: true
 nodeSelector:
   kubernetes.io/hostname: minikube
```

> Some clusters policies might prevent you from creating container in
  host PID namespace for security.

```
$> kubectl apply -f pod.yaml
```

We invoke *cat* just to sustain pod, so we can enter it with *exec*:

```
$> kubectl exec -it debug sh
```

We will have to locate our application's process and attach *Delve* to it. Let's
find out container ID of *hello-world*:

```
$> kubectl get pod hello-world-77457544f4-fglbh \
   -ojsonpath='{.status.containerStatuses[?(@.name=="hw")].containerID}{"\n"}' \
   | cut -d / -f 3
b686d81723e3c714ddb08fe398dafd62d4456f0211991658b27be1d41d39b0b7
```


### Inside debugging pod

With *hostPID: true* we have full access to every process on node, even if it's
not related to current pod. Docker spawns shim with ID in process name for each
container. Busybox version of *ps* lacks ability to list processes in tree
structure, so we will need to install *procps* and search for *hello-world*
application process.

```
$> apk update
$> apk add procps
$> ps -eo pid,cmd --forest | grep -A 1 b686d81723e3c714ddb08fe398dafd62d4456f0211991658b27be1d41d39b0b7
7803      \_ docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/b686d81723e3c714ddb08fe398dafd62d4456f0211991658b27be1d41d39b0b7 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
7820      |   \_ /hello-world
```

Now we know that our application is running with process ID 7820.
Attaching to it will a matter of installing *Delve* and writing script to
automatically start tracing:

```
$> apk add git
$> go get -u github.com/derekparker/delve/cmd/dlv
$> cat > trace.dlv
trace handleSlash main.go:10
on handleSlash print w.req.RemoteAddr
continue
$> dlv attach 7820 --init trace.dlv
Type 'help' for list of commands.
Tracepoint handleSlash set at 0x6e826d for main.handleSlash()
 /home/jakub/dev/hello-world/main.go:10
 > [handleSlash] main.handleSlash() /home/jakub/dev/hello-world/main.go:10 (hits
 > goroutine(35):1 total:1) (PC: 0x6e826d)
     data.req.RemoteAddr: "127.0.0.1:34868"
```

Above configuration will let us display source port of every connection handled
by *hello-world* application. It's not really useful, but shows what you can
achieve with *Delve*.

## Security considerations

Method described above should be used only in emergency or development
environments. It's quite easy to:

- accidentally break Kubernetes node by messing with it's processes
- break inspected process; *Delve* may noticeably slow-down process, causing
  timeouts
- affect other pods running on node