Troubleshooting PowerShell Memory Leaks with Visual Studio

I was recently investigating a couple of memory leaks issue with long running scripts that capture and process monitoring metrics.

Since those leaks happened over a fairly long period of the time, it was a bit difficult to troubleshoot by simply running the script in PowerShell ISE. Another thing that made the diagnostic process difficult is that PowerShell jobs were used to collect and process data, which makes it a bit hard to set break points in the code to see what gets stuck in memory. PowerShell jobs essentially spawn another process of PowerShell under which your code runs.

I started to see what I could do using Visual Studio. What I ended up doing is the following:

1) Let your PowerShell script run for a little while in order to get into a “bad enough” state

2) Use Process Explorer to create a full dump of the PowerShell process:

procexp_create_fulldump

You can also generate that dump file directly from Visual Studio by attaching the debugger to the running process. If the process is running on a remote host, you will need to run the Remote Debugger that’s appropriate for your platform. It’s usually located in c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Remote Debugger on the machine that has Visual Studio installed. For example the x64 remote host, you would simply run as Administrator:

\\<vs computer>\c$\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Remote Debugger\x64\msvsmon.exe

Once Remote Debugger is running, you can simply attach to the desired process in Visual Studio:

vs_dbg_attach_to_process

and then pick the PID of the PowerShell instance you want to debug:

vs_dbg_attach_to_process_select

Once Visual Studio is attached, break the execution of the process:

vs_dbg_attach_to_process_break

Once the process execution has been suspended, you can then create the dump file using Save Dump As (make sure you select to save the heap):

vs_dbg_attach_to_process_savedump

3) Open the resulting dump file in Visual Studio and select Debug Managed Memory

vs_debug_manage_memory

4) You can then sort by Size (bytes) to figure out which object type ends up using the most memory

vs_debug_mm_sort

5) You can then view the instances of that object type by selecting View Instances

vs_debug_mm_view_instances

6) Once you are in the instances view, you can select one of the instances, right click on it and select Quick Watch

vs_debug_mm_quick_watch

7) You can now see the content of the object that’s causing the issue, giving you a better reference point in your code to fix the issue

vs_debug_mm_quick_watch_view

In my case, it turns out that the output of Write-Output at the end of a particular function that was used in my pipeline stayed in memory indefinitely, even though the object was done processing in the pipeline. For instance:

Write-Output -InputObject $body"sent"

After piping the output to Out-Null in my high level script that runs as non-stop scheduled task, the problem was resolved.

Maybe something worth mentioning, it’s a bit hard to dive in the dump file for a PowerShell process as it can be difficult to track back a particular function as the PowerShell script is instantiated as dynamic code in .NET.

Hopefully this post will help someone in the same situation as I was in!

Advertisements

2 thoughts on “Troubleshooting PowerShell Memory Leaks with Visual Studio

  1. Pingback: Troubleshooting PowerShell Memory Leaks with Visual Studio - How to Code .NET

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s