Dotnet dump analysis (performance monitoring of your app)

Many times we had situations where your production app having performance issue but we are not sure where is the performance problem area in code, so by analyzing dump file, you can easily figure out that problem code area can fix that.

Microsfot dotnet now provide multiple tools to analyze your process running on the server. Below I will talk about some of useful dotnet commands to quickly analyze/monitor your dotnet application on server and then you can see detailed demo, how I have collected dump of my running app and analyzed the problem area in code and same I fixed that.

Keep patience, if you are working with dotnet, you will surely enjoy this.

dotnet-counters tool: This tool is used for monitoring your app health high level on the server your app is running. Like using this app you can see how your app having CPU usages, no. of active threads or in pool, GC related etc.. Below are some useful commands, which most of time you would like to use:

1. dotnet tool install --global dotnet-counters  : Command is to install dotnet counter tool.
2. dotnet-counters ps : Command is to see list of dotnet process which can be monitored.
3. dotnet-counters monitor --process-id 24276  --refresh-interval 3 --counters System.Runtime : Command is to monitor respective process id and it will refresh/update the result in every 3 second.
1. If you get some message like "Waiting for initial payload..." and no further result set then try open CMD in administrator mode.
4. dotnet-counters monitor --process-id 24276 --counters System.Runtime[cpu-usage,threadpool-thread-count] : Command is the monitor specific thread/component, like in this example I am want to monitor only cpu usages and no. thread pool active currently for my process. Note: the specific component you use here, please take care of naming convention, it should be in small and replace space with hyphen "-". Fyi - Name you can pick from the list what you got in previous monitoring command (above point 3).
5. dotnet-counters collect --process-id 46364 --format json --counters System.Runtime,Microsoft.AspNetCore.Hosting --dotnet mvc.dll --output C:\Test\dump\counter1.json : Command to monitor specific dll or namespace, like in this case I am trying to monitor core hosting which will provide me the request response details and also in this example I am exporting result into json file.

dotnet-dump tool: This tool is to take dump of your process/app running and you can analyze to see what is going good and what need to look into. Below are some useful commands, which most of time you would like to use:

1. dotnet tool install --global dotnet-dump : Command to install dotnet dump tool.
2. dotnet-dump collect --process-id 24276 --type full --output C:\Test\dump\mydump.dmp --diag : Command to download dump for the process id.
3. dotnet-dump analyze C:\Test\dump\mydump.dmp : Command to analyze dump and when you run this command, it will turn same CMD window into input window where you have to further type/write command to get respective analysis data. So if you are not aware of what command you can run then just type help and enter and it will give you all list of command which you can use to see the analysis result part of that.
1. Help : run this command to see list of commands which you can run to analyze.
2. <respective command> : Run the respective command as per you pull from above point (a.);
4. q or exist : Type q or exit to return from this window to normal CMD window.

dotnet-symbol tool: This tool is to download all the symbols for the process dump file, so that you should be able to debug your dump file and find the root cause of problem and fix that. Below are some useful commands, which most of time you would like to use:

1. dotnet tool install --global dotnet-symbol : Command to install dotnet symbol tool.
2. dotnet-symbol --host-only --debugging C:\Test\dump\mydump.dmp : Command to download all the required symbol for your dump file, so that you can debug your dump.

Now lets moved to detailed demo: I have created dotnet core mvc sample application where I added delay of 50*50 seconds to open the page whenever we request through browser.

So definitely when I will run and try to access this Index action, it will take much time to open and specially when I request same url into multiple tabs in browser then no. of threads will be increase and have performance issue.

So, I will be using "dotnet-counter", "dotnet-dump" and "dotnet-symbol" to monitor this app and also will collect the dump and will try to analyse in Visual Studio and will see, how it will help me to quickly figure out the problem area and I can fix it immediately.

This is my sample application with delay of 50*50 seconds.

I used "dotnet-counter ps" command to list all my running process which I can monitor and in this case process id "46364" which is my sample app process id, which I will monitor and analyse.

Now I used "dotnet-counters monitor --process-id 46364 --refresh-interval 3 --counters System.Runtime" command to monitor my app and see the high level utilization and health of my app. In below result screenshot, if you see I am having 6 active thread which running for my app as per multiple request to browser.

Now I used "dotnet-dump" command to take dump of my process id on my physical location, so that I can analyse the dump.

Further I used "dotnet-symbol" command for my download dump, so that all the required symbols must be available and I can then debug the dump in Visual Studio.

I used "dotnet-dmp analyze" command to analyze the dump on CMD itself instead of Visual Studio and when I run this then it will further ask you to enter command which you want to get analysis data, so in this case I just used "cltrthreads" and it return the list of all the threads being used in this dump.

Now, I want to debug my dump into Visual Studio, which is most preferable option for developer and it give you proper resultset, visual experience.

So open Visual Studio and open your dump file into visual studio. [Use file menu + open file option]. And then click on debug with managed code only option to debug the dump.

Note: When you debug, you might get page like below (error due to missing required symbol, so in that case do checked box checked "Microsoft Symbol Servers" and continue.

On successful debug,  it will open the debug page with all the threads. Where you can analyze the threads and see where is the problem, but

Visual studio provide option to see Parallel stack view of your dump debug thread and you can easily figure out the problem thread or code area. Let see, how.

From Debug menu open Parallel stacks.

Parallel stack view: It will open common threads by grouping and can see how many threads are active.

So in this case, I have max count thread box is 5 and there, when I tried to see, I found with Home controller where is TaskDelay having 5 thread active. Double Click on any of the thread and it will land(open) you to the code view where you can see exact line of code or block of code having this and you can fix that.

This is the code got open when I click on thread from above step, so this if you see the exact line of code which I had forcefully added in my sample application for this demo. So now I can fix this and push new code to server and problem resolved.