Debugging Go code with pprof

I was just about done with the Go implementation with hashtree – the only nagging problem was slow upload performance and high CPU and memory usage.
I added the following lines to my main.go code to enable a pprof server in my program:

go func() {
 log.Println(http.ListenAndServe("localhost:6060", nil))

And added the correct import statements:


Next when the programming was running I ran on the command line:

go tool pprof https://localhost:6060/debug/pprof/heap

This brings up an interactive shell – typing in “top” prints out memory consumption of go routines:
Typing in web brings up a svg printout of the memory graph:
This analysis was taken of the current improved version of hashtree, the memory consumption is still quite high but not as high as before:
As you can see the memory consumption is too high – its caused by the “s3Client.PutEncryptedObject”  function used in the uploadFiles function. I reported the error to the developer here.
His was response was prompt and helpful:

"Do not use the encrypted APIs just yet they are being revamped. Memory usage is expected because we have to upload in parts when we don't know the size and we have to buffer up the memory of 64MiB for each Put()
We are revamping the API such that only SSE-C is supported."

Based on the example code he suggested I was able to rewrite the libraries and improve all areas of performance – network, memory, cpu and reduce error rates.
Thanks to the guys and gals at minio-go.
The Go version of hashtree is now nearly ready for production! I will run a few tests on a few GB of data and then I will have confidence enough in it to give it version 1.0 status.

Leave a comment

Your email address will not be published. Required fields are marked *