How to inspect a Linux machine

Linux comes with a large range of binaries you’ll find in known paths like /bin/ or /sbin/.

These binaries are used by the system itself for various operations, but you can use it to get information.

We’ll see some practical examples that will introduce essential concepts you have to know to get started with forensics and other investigations.

Disclaimer

This is not meant to be a full guide but only an introduction with some personal thoughts.

Very basic but still useful commands

Hacking or investigating does not always involve complex and obfuscated code.

Basic commands are helpful to get interesting information:

# basic OS information
uname -a # all info
uname -r # only the kernel version

# get info about the filesystem and mounted devices 
df -h

# scroll in the file that contain all users
less -r /etc/passwd

# scroll in the file that contain all encrypted passwords
sudo less -r /etc/shadow

# who is currently logged in?
w -huis

# something suspicious in recent connections?
last -Fw

# network info and ports
lsof -i TCP # show TCP activity
lsof -i4 # ipv4 activity
lsof -i6 # ipv6 activity
lsof -i -n # open connections
netstat -ano

# inspect processes
ps aux
ps aux | grep root # pipe to search a specific term

Inspect the history

Linux systems usually keep tracks of users’ activities in dotfiles at the root of the home directory:

ls -alh $HOME/.*history

This would give you files that have that pattern in their name, then you might read them:

less -r $HOME/.bash_history

Dive into system logs

It’s usually in /var/log, but it can be elsewhere (e.g. custom config).

In any case, the following files are important:

  • auth.log*
  • syslog*
  • kern.log

Inspect other activities

You can determine (and monitor) which files are opened using known protocols like SSH on port 22:

sudo lsof -r -i :22

It’s also a good idea to read the journal:

journalctl -S -4h

The above command selects events in the last 4 hours.

Automate tests

You might want to use advanced scripts instead or in addition to the manual inspection. I like Lynis and Linpeas, as it’s usually enough for a quick check.

I made my own Bash to download, install and start these fantastic scripts: check linux.

Once the scans are done, you can read the logs in /var/log/checklinux.

Chasing malware

You can go to online community platforms, for example:

You can also leverage popular packages like the following:

  • yara
  • rkhunter

These resource help identify IoCs (Indicators of Compromise), from suspicious packages and software on your system to known malware and rootkits.

Be aware of the magic tricks

Defenders constantly learn new tricks to catch attackers.

However, attackers do the same with defenders with counter-measures and other techniques to cover their tracks.

For example, it’s not uncommon to mess up with metadata (e.g., the “Timestomping” technique), but it leaves traces, so attackers would use it only to delay the detection.

It’s also possible to shift delete files, but it would only skip the recycle bin stage. Experienced investigators can retrieve them pretty quickly.

Instead, cybercriminals would encrypt, perhaps wipe out everything (for example using the shred command). However, even that extreme approach can still leave some traces.

That’s why adversaries tend to use fileless attacks that inject malware in the RAM to keep things volatile.

They exploit existing software and applications on the targeted system, escaping signature-based antivirus and other whitelisting measures.

Level up with memory analysis

Analyzing memory dumps can be hard, especially at the beginning. You might want to use comprehensive Frameworks like volatility.

This simplifies the process significantly, and the wiki contains all prerequisites.

You can extend it with plugins to catch even more artifacts. Indeed, cybercriminals won’t leave steganographic images or password-protected archives behind, like the ones you get in CTFs.

You will only get traces and indirect evidences instead, but that does not mean you cannot catch your adversaries. It’s just way longer and a bit harder.