Another interesting package - ps_mem

I find it hard to see what's using the RAM in a server and ps_mem can help.  It lets you know the memory usage of applications.

I haven't found how to install it on Ubuntu, but on Red Hat and derivatives, it's available in the EPEL repo. 

You can use it by simply executing the ps_mem command, which gives this kind of output:

[root@vps1 ~]# ps_mem
 Private  +   Shared  =  RAM used       Program

   8.0 KiB +  24.0 KiB =  32.0 KiB       agetty (2)
  4.0 KiB +  30.0 KiB =  34.0 KiB       xinetd
  4.0 KiB +  49.5 KiB =  53.5 KiB       lvmetad
  0.0 KiB +  73.5 KiB =  73.5 KiB       saslauthd (2)
  8.0 KiB +  89.0 KiB =  97.0 KiB       pmdadm
 16.0 KiB + 142.5 KiB = 158.5 KiB       systemd-udevd
128.0 KiB +  90.0 KiB = 218.0 KiB       pmdaxfs
152.0 KiB + 120.5 KiB = 272.5 KiB       crond
176.0 KiB + 104.5 KiB = 280.5 KiB       pmdaroot
108.0 KiB + 178.0 KiB = 286.0 KiB       avahi-daemon (2)
340.0 KiB + 128.0 KiB = 468.0 KiB       pmie
328.0 KiB + 330.0 KiB = 658.0 KiB       sendmail.sendmail (2)
764.0 KiB + 104.0 KiB = 868.0 KiB       pmdaproc
672.0 KiB + 230.0 KiB = 902.0 KiB       pmcd
  1.1 MiB + 150.5 KiB =   1.2 MiB       polkitd
  1.4 MiB +  98.5 KiB =   1.5 MiB       dbus-daemon
852.0 KiB + 694.5 KiB =   1.5 MiB       rsyslogd
  1.6 MiB + 108.5 KiB =   1.7 MiB       pmdalinux
  1.7 MiB + 169.5 KiB =   1.9 MiB       systemd
  1.7 MiB + 619.0 KiB =   2.3 MiB       bash (5)
  3.0 MiB + 295.5 KiB =   3.3 MiB       pmlogger
  3.5 MiB + 595.0 KiB =   4.0 MiB       systemd-journald
  2.6 MiB +   1.7 MiB =   4.3 MiB       sshd (6)
  8.3 MiB + 117.5 KiB =   8.4 MiB       systemd-logind
  8.5 MiB +   1.0 MiB =   9.5 MiB       perl (2)
  7.3 MiB +   7.8 MiB =  15.1 MiB       httpd (2)
  6.7 MiB +  11.9 MiB =  18.7 MiB       postgres (10)
226.9 MiB + 406.5 KiB = 227.3 MiB       java
---------------------------------
                        305.0 MiB
=================================

If you add the -s switch, you can get a per-command output, which is may be helpful, for example, when you have multiple java applications running:

[root@vps1 ~]# ps_mem -s
 Private  +   Shared  =  RAM used       Program

   4.0 KiB +  12.0 KiB =  16.0 KiB       /sbin/agetty --noclear tty2 linux
  4.0 KiB +  12.0 KiB =  16.0 KiB       /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
  4.0 KiB +  30.0 KiB =  34.0 KiB       /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid
  4.0 KiB +  43.0 KiB =  47.0 KiB       avahi-daemon: chroot helper
  4.0 KiB +  49.5 KiB =  53.5 KiB       /usr/sbin/lvmetad -f
  0.0 KiB +  73.5 KiB =  73.5 KiB       /usr/sbin/saslauthd -m /run/saslauthd -a pam -n 2 (2)
  8.0 KiB +  89.0 KiB =  97.0 KiB       /var/lib/pcp/pmdas/dm/pmdadm -d 129
 16.0 KiB + 142.5 KiB = 158.5 KiB       /usr/lib/systemd/systemd-udevd
 68.0 KiB + 135.0 KiB = 203.0 KiB       postgres: logger process
128.0 KiB +  90.0 KiB = 218.0 KiB       /var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
  4.0 KiB + 219.0 KiB = 223.0 KiB       sshd: root@pts/0
  4.0 KiB + 219.0 KiB = 223.0 KiB       sshd: root@pts/3
  4.0 KiB + 219.0 KiB = 223.0 KiB       sshd: root@pts/2
  4.0 KiB + 219.0 KiB = 223.0 KiB       sshd: root@pts/1
104.0 KiB + 135.0 KiB = 239.0 KiB       avahi-daemon: running [vps1.local]
152.0 KiB + 120.5 KiB = 272.5 KiB       /usr/sbin/crond -n
 96.0 KiB + 179.0 KiB = 275.0 KiB       postgres: archiver process   last was 000000010000000000000009
176.0 KiB + 104.5 KiB = 280.5 KiB       /var/lib/pcp/pmdas/root/pmdaroot
100.0 KiB + 183.0 KiB = 283.0 KiB       /usr/sbin/sshd -D
140.0 KiB + 163.0 KiB = 303.0 KiB       sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
188.0 KiB + 167.0 KiB = 355.0 KiB       sendmail: accepting connections
340.0 KiB + 128.0 KiB = 468.0 KiB       /usr/bin/pmie -b -h local: -l /var/log/pcp/pmie/vps1/pmie.log -c config.default
348.0 KiB + 149.5 KiB = 497.5 KiB       postgres: stats collector process
312.0 KiB + 277.0 KiB = 589.0 KiB       postgres: autovacuum launcher process
764.0 KiB + 104.0 KiB = 868.0 KiB       /var/lib/pcp/pmdas/proc/pmdaproc -d 3
672.0 KiB + 230.0 KiB = 902.0 KiB       /usr/libexec/pcp/bin/pmcd
796.0 KiB + 318.0 KiB =   1.1 MiB       perl /var/lib/pcp/pmdas/nfsclient/pmdanfsclient.pl
  1.1 MiB + 150.5 KiB =   1.2 MiB       /usr/lib/polkit-1/polkitd --no-debug
200.0 KiB +   1.1 MiB =   1.3 MiB       postgres: wal writer process
  1.4 MiB +  98.5 KiB =   1.5 MiB       /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
852.0 KiB + 694.5 KiB =   1.5 MiB       /usr/sbin/rsyslogd -n
  1.6 MiB + 108.5 KiB =   1.7 MiB       /var/lib/pcp/pmdas/linux/pmdalinux
  1.7 MiB + 169.5 KiB =   1.9 MiB       /usr/lib/systemd/systemd --system --deserialize 19
 88.0 KiB +   1.9 MiB =   2.0 MiB       postgres: writer process
160.0 KiB +   2.0 MiB =   2.1 MiB       /opt/rh/rh-postgresql96/root/usr/bin/postgres -D /var/opt/rh/rh-postgresql96/lib/pgsql/data
  1.7 MiB + 619.0 KiB =   2.3 MiB       -bash (5)
  2.5 MiB + 694.5 KiB =   3.2 MiB       sshd: root@pts/4
584.0 KiB +   2.6 MiB =   3.2 MiB       postgres: checkpointer process
  3.0 MiB + 295.5 KiB =   3.3 MiB       pmlogger -r -T 24h10m -m pmlogger_daily -P -l pmlogger.log -c /var/lib/pcp/config/pmlogger/config.default 20180326.00.10
976.0 KiB +   2.6 MiB =   3.5 MiB       postgres: ugob Syslog ::1(36008) idle
  3.5 MiB + 595.0 KiB =   4.0 MiB       /usr/lib/systemd/systemd-journald
  4.0 MiB +   1.0 MiB =   5.0 MiB       postgres: postgres postgres [local] idle
  8.3 MiB + 117.5 KiB =   8.4 MiB       /usr/lib/systemd/systemd-logind
  7.7 MiB + 728.5 KiB =   8.4 MiB       perl /var/lib/pcp/pmdas/postgresql/pmdapostgresql.pl
  7.3 MiB +   7.8 MiB =  15.1 MiB       /usr/sbin/httpd -DFOREGROUND (2)
226.9 MiB + 406.5 KiB = 227.3 MiB       /usr/bin/java -Xms512m -Xmx1g -XX:MaxPermSize=256m -Djava.security.egd=/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/ --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter
---------------------------------
                        305.0 MiB
=================================

You can also get the information for one PID only.  Here's an example with -s to see the full command line:

[root@vps1 ~]# ps_mem -sp 29280
 Private  +   Shared  =  RAM used       Program

 226.9 MiB + 406.5 KiB = 227.3 MiB       /usr/bin/java -Xms512m -Xmx1g -XX:MaxPermSize=256m -Djava.security.egd=/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/ --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter
---------------------------------
                        227.3 MiB
=================================

Finally, you can use the -w N switch to do a top-like automatic refresh.

That's about it for ps_mem, have fun!

Comments

Post a Comment

Popular posts from this blog

General linux performance troubleshooting

Here's a list of commands that you should execute and then share the output to someone who can help you figure out what resource is the bottleneck in your system. Note, it requires the  sysstat  and  procps  packages (ubuntu and RHEL and its derivatives): uptime vmstat 1 10 iostat -xN 2 10 mpstat -P ALL 3 10 pidstat 1 10 free -mw (or free -m, if your OS doesn't support -w)  uptime is to see the load averages on the system. vmstat is mostly used to tell if the system is swapping or not. If you see significant numbers in the 'si' and 'so' columns, your system is most likely swapping (using the hard drive as RAM), which usually slows performance a lot. iostat is mostly used to determine if your disk subsystem is not able to cope with the load. If you see one or more lines that shows 100 or close almost constantly, it is probably the case.  If it is your swap volume, you probably saw numbers in the 'si' and 'so' columns in the  vmsta...
Read more

Networker automated recovery testing using the REST API - introduction

One feature that is lacking from Networker, compared to some of its competitors, is the built-in automatic recovery testing. However, when there's an API, there is a way.  Networker's REST API is not perfect, but it allows the backup administrator to perform queries about Networker resources (objects). As my workload is going up, I realize that one of the tasks that I tend to skip the most is the periodical recovery tests.  Don't forget that a a backup that is not tested should be considered non-successful. I also found out that my recovery tests were not diverse enough. When I started this project, I knew a little bit about REST APIs, and nothing about JSON processing. With the Networker REST API documentation, and the help of a friend and Networker Support staff, I was able to create HTTP queries with Postman,  cURL and jq . Once I got the queries that I needed, I put them in a bash script that would somehow select one backup, and then restore it. My first attem...
Read more

Tips for being a better system administrator

Here are a few tips that I have discovered or implemented during my career, that can help anyone get better results: Follow the 3-2-1 backup strategy  or better 3 copies of the data 3 different media 1 copy offsite Test your backups regularly. Ideally, automate your recovery tests. Use Whatever-as-code as much as you can. IaC is the first that comes to my mind, but using code to define objects and their properties have several advantages: Auto-documentation of changes Ability to easily rollback (using source code management like Git) Makes it easier for standardization an compliance Ansible, Chef, Puppet, Salt, Terraform, Pulumi are good examples Maintain a changelog of all the major changes in your infrastructure (that isn't already "documented" because you're using IaC). Keep track of your hardware inventory and plan your renewals and warranty extension purchases. Build a spreadsheet with all the components that you manage and make sure that everyone on the sysadmi...
Read more