General linux performance troubleshooting

Here's a list of commands that you should execute and then share the output to someone who can help you figure out what resource is the bottleneck in your system. Note, it requires the sysstat and procps packages (ubuntu and RHEL and its derivatives):

• uptime
• vmstat 1 10
• iostat -xN 2 10
• mpstat -P ALL 3 10
• pidstat 1 10
• free -mw (or free -m, if your OS doesn't support -w) 

uptime is to see the load averages on the system.

vmstat is mostly used to tell if the system is swapping or not. If you see significant numbers in the 'si' and 'so' columns, your system is most likely swapping (using the hard drive as RAM), which usually slows performance a lot.

iostat is mostly used to determine if your disk subsystem is not able to cope with the load. If you see one or more lines that shows 100 or close almost constantly, it is probably the case.  If it is your swap volume, you probably saw numbers in the 'si' and 'so' columns in the vmstat output.

mpstat shows the % of the CPU time used by different functions of the server. I'll explain the most used ones, in my experience: %usr is applications (usually), %nice is for niced (changed priority) tasks, %sys is kernel tasks like forks and threads creation, %iowait is time spent waiting for disk. See man mpstat for more information.

pidstat shows a per-process, rolling summary, only for active processes.

free shows information about RAM and swap. Total available and used RAM and swap, used, free, shared, buffers and cache. "Recent" versions show an "available" column. This a very important column as it provides the information that is the most important to everyone, in my opinion.  According to the man page, "available" is an estimation of how much memory is available for starting new applications, without swapping. That is usually what we refer to as "free" mem