Investigating a source of an issue can be a hard task, and it is important to have a good set of Linux shell commands on your tool belt. Here is the commands that I have used to mitigate issues on production.

Disk

Check disk space with df

df reports file system disk space usage. That is, it displays the total size, the used and available space for each of your mounted partitions.

Check what is taking the disk space

du estimate file space usage. In the example I use the option -d1 to check only the current directory.

Monitor current I/O

Use iotop to check which processes are doing the most write and reads.

Network

tcpdump

tcpdump is a data-network packet analyzer computer program that runs under a command line interface. It allows the user to display TCP/IP and other packets being transmitted or received over a network to which the computer is attached.[3] Distributed under the BSD license,[4] tcpdump is free software.

Have you ever been in a situation where you don’t know if the source of an issue is on client side, sever side or on the firewall? With tcpdump you can check each package sent and received to pin point where the problem is.

$ tcpdump host 1.1.1.106:20:25.593207 IP 172.30.0.144.39270 > one.one.one.one.domain: 
12790+ A? google.com. 
(28) 06:20:25.594510 IP one.one.one.one.domain > 172.30.0.144.39270: 
12790 1/0/0 A 172.217.15.78 (44)

dig

dig DNS lookup utility. I used for two things: check if the server is being able to resolve an address, and to check if a DNS configuration has been applied.

telnet / nc

Check if application is listening to a given port

Application behavior

strace

Strace captures and records all system calls made by a given process and the signals received by the process.

You can also check a running application by providing its PID.

sudo strace -p 3569

Check processes open files

To list open files of a given process, get its PID using ps aux | grep APP_NAME , then you can list the open files sudo ls -l /proc/PID/fd

Visualize an API request

curl with verbose option (-v) can help you to find issues during a request.

Conclusion

I hope some of this commands were new for you and that this post helps you on a future debugging task.