I was recently investigating a couple of memory leaks issue with long running scripts that capture and process monitoring metrics.
Since those leaks happened over a fairly long period of the time, it was a bit difficult to troubleshoot by simply running the script in PowerShell ISE. Another thing that made the diagnostic process difficult is that PowerShell jobs were used to collect and process data, which makes it a bit hard to set break points in the code to see what gets stuck in memory. PowerShell jobs essentially spawn another process of PowerShell under which your code runs.
I started to see what I could do using Visual Studio. What I ended up doing is the following:
1) Let your PowerShell script run for a little while in order to get into a “bad enough” state
2) Use Process Explorer to create a full dump of the PowerShell process:
You can also generate that dump file directly from Visual Studio by attaching the debugger to the running process. If the process is running on a remote host, you will need to run the Remote Debugger that’s appropriate for your platform. It’s usually located in c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Remote Debugger on the machine that has Visual Studio installed. For example the x64 remote host, you would simply run as Administrator:
\\<vs computer>\c$\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Remote Debugger\x64\msvsmon.exe
Once Remote Debugger is running, you can simply attach to the desired process in Visual Studio:
and then pick the PID of the PowerShell instance you want to debug:
Once Visual Studio is attached, break the execution of the process:
Once the process execution has been suspended, you can then create the dump file using Save Dump As (make sure you select to save the heap):
3) Open the resulting dump file in Visual Studio and select Debug Managed Memory
4) You can then sort by Size (bytes) to figure out which object type ends up using the most memory
5) You can then view the instances of that object type by selecting View Instances
6) Once you are in the instances view, you can select one of the instances, right click on it and select Quick Watch
7) You can now see the content of the object that’s causing the issue, giving you a better reference point in your code to fix the issue
In my case, it turns out that the output of Write-Output at the end of a particular function that was used in my pipeline stayed in memory indefinitely, even though the object was done processing in the pipeline. For instance:
Write-Output -InputObject $body"sent"
After piping the output to Out-Null in my high level script that runs as non-stop scheduled task, the problem was resolved.
Maybe something worth mentioning, it’s a bit hard to dive in the dump file for a PowerShell process as it can be difficult to track back a particular function as the PowerShell script is instantiated as dynamic code in .NET.
Hopefully this post will help someone in the same situation as I was in!