Spoofing, by definition, means to imitate or trick someone. To understand the spoofing attack, we need to examine the IP packet structure in detail. Many cyber attacks stem from design flaws in the fundamental network designs; packet spoofing is no exception. Please refer to Figure 1.

Figure 1: Ethernet network packet

As we know, a basic Ethernet network packet is essentially a data chunk with various predefined fields such as source and destination MAC addresses, frame checksum code, preambles, etc. In the case of the TCP/IP protocol, the TCP frame encapsulates the IP datagram, and both piggyback on the basic Ethernet packet. The TCP provides the connection-oriented information, while the IP packet contains source and destination addresses and ports. It is the duty of various OSI layers to contribute their bit towards the formation of a complete packet.

As we learnt last month the TCP/IP works on a three-way handshake (SYN, SYN-ACK and ACK),. This handshake establishes a connection between two different network interface cards, which then use the packet sequencing and data acknowledgements to send or receive data. The conversation is formally concluded by using a FIN/FIN-ACK handshake.

The source and IP address field values decide who is talking to whom, while the port fields decide which source application is talking to which destination application. IP address fields in the IP packet are filled up automatically by the upper layers; however, malicious users or programs can modify these fields — and this is exactly where the spoofing comes into the picture (refer to Figure 2).

Figure 2: Packet-spoofing attack

In a very simple packet-spoofing attack, the hacker creates IP packets targeting a destination, but the source IP field is modified so that it does not have the IP address of the hackers’ computer, but in fact, of some other computer, which can be used as a data collector or a sniffer by the hackers.

By its design, the TCP/IP stack running on the destination computer is supposed to respond to the source IP of the received packet. On doing so, the sniffer computer receives the packet. Having said that, if a packet-monitoring software is run on the destination computer, it will show that the packets are coming from this sniffer computer, which in reality is not the case. Thus, the hackers are successful in hiding themselves, and at the same time collecting packet information from the sniffer computer, because their own real source IP is nowhere revealed in this process.

Now let’s understand why hackers prefer spoofing. While hiding on the network is the very first step, a hacker’s intention is always beyond that. By collecting packets on the sniffer computer, hackers can gain a lot of information about the target machine. Vital information, such as open ports, type of operating system, layer 7 applications, cryptography types used, etc, can be collected, while not revealing one’s identity.

Just as an example, a hacker can send a spoofed packet to find out if the target is running a Web server. Once the port information is revealed on the sniffer, another chunk of spoofed packets can be sent to the target to establish a telnet session, and check Web server headers, which would reveal OS information as well as the type of Web server and security support information.

Please note that the hackers need not always have access to the sniffer machine; in fact, using an advanced packet-sniffing tool can help the attackers use only their own machine, send spoofed packets, collect data on the same machine, and still be invisible on the network from an attack viewpoint.

Advanced hackers can use spoofing to find out the state of a firewall too. As we know, the firewall is a stateful packet-filtering process, which works on a set of configurable rules, to decide which packet should be allowed, and which dropped. When a source machine tries to connect to another machine that is protected behind a firewall, the source machine has to first establish a TCP handshake with the firewall.

If the connection is allowed, the firewall itself establishes another connection with the destination machine, and the data transfer occurs with the firewall as a catalyst in this whole process. Also, when the destination machine tries to send an acknowledgement to the source, the firewall is bound to intercept and check whether or not a corresponding request from the source is pending. If it is not, the packet is dropped. All this is true for modern stateful firewalls, but not for old or cheap firewalls, which tend to allow ACK packets, thus exposing spoofing possibilities.

There are three reasons worth mentioning here, which typically make a network vulnerable to spoofing attacks. The first is, the IP packets can be modified very easily using free utilities available. The second reason is that even today, many applications still use source and destination IP address combinations as a secure way of authenticating a packet.

Just as an example, there are Web servers that allow or disallow HTTP requests from a list of configurable IP addresses. Such systems prove to be useless if the packet is spoofed to reflect an IP address which is in the list of allowed IPs. The third reason is that the routers forward traffic based on the destination address, and by default do not care much about the source address.

Packet spoofing types

With this basic knowledge of how spoofing works, let us now dig a bit deeper. At a broad technical level, there are two basic types of spoofing: “blind” spoofing and “non-blind” spoofing. As we discussed earlier, the packet sequence number and acknowledgements help data transfer, so sensing those fields is an important requirement of a spoofing-based attack.

In the blind spoofing type of attack, the attacker sends multiple packets to the target to sample the sequencing numbers. Once the pattern is found out, it becomes easy for the attacker to create another set of spoofed packets, to collect the data being transferred. Whereas, in the non-blind type, the attackers must be on the same subnet as the target, to be able to easily see the sequencing numbers and acknowledgements. Once they get their hands on it, they can break the established connection between the target and the other computer to which it is talking, and re-establish the connection by modifying the sequence numbers to that of the attackers’ own machine.

An extended version of this type of attack is the man-in-the-middle attack, whereby an entire session is stolen to decipher and steal data.

Now let’s take a look at how the basic concept of spoofing could be used by hackers to impact various TCP-based application services.

IP port spoofing

In this type, the source port is modified in order to cheat the NAT devices and firewalls, and also to hide deep in the network. A firewall that is not very well managed can end up leaving stale rules in action, which can potentially lead certain outgoing ports on certain IP addresses to be open. IP port spoofing attacks take advantage of this situation.

This is a serious attack, because the hacker can be outside the network, and still be able to look at internal traffic.

ARP spoofing

Since it is all about modifying or forging packets, modern attackers don’t restrict themselves to IP fields. In this type of attack, which is also known as ARP poisoning, the attackers send spoofed ARP packets on the local area network, to associate the attackers’ own machine’s MAC address with the IP address of another host, which is the target.

Due to this, traffic for the target now reaches the attackers’ machine, due to the binding between IP and MAC addresses. Thus this becomes a local man-in-the-middle attack. The attackers can further hide by forwarding received data to the actual destination, and since there is no data lost in this process, it is almost impossible to find the man-in-the-middle stealing the data.

DNS spoofing

Also known as a DNS poisoning attack, this is more serious. In this case, the domain name system server is spoofed to alter entries of domain names to reflect the attackers’ IP address. This results in sending Web or email traffic to the attackers’ machine. This attack is achieved by creating multiple forged packets wherein the IP, port and service type entries are modified to serve the purpose.

The repercussions of such an attack are very dangerous, whereby a website can be defaced, or email data can be stolen.

Email and Web spoofing

Speaking about Layer 7, the same technique of impersonation can be stretched to crafting fake email addresses, Web requests and hyperlinks. This is usually done to plant a Trojan or a virus and spread it across. Please remember that spoofing is a basic component in planting a denial-of-service attack, mainly because the attacker would always want to hide behind a curtain.

Protecting FOSS systems

While the spoofing attack is a difficult one to tackle, there are a few preventive mechanisms that all network administrators should adopt in their infrastructure. Since the spoofing starts at Layer 2, the real protection has to be implemented in the critical network components such as routers, firewalls, switches, etc.

Deploying the latest stateful firewall, and enabling features in it that fight against source packet spoofing, can be a first level of defense. As a daily chore, network logs from firewalls, routers and switches can be parsed using a script, to see if multiple duplicate ACK packets are being generated.

Also, if an unusual number of SYN packets are being observed, which are not being responded to, that could be a clue to spoofing. The real spoofing detection is to check a bunch of packets for a given set of source and destination IP addresses, and see that there is no other IP address involved in that communication session.

This is a difficult task for a script program to achieve, and for complex networks, deploying an intrusion detection device usually helps to a great extent. There is a thumb rule while inspecting packets, which says that in a network packet log, if an internal LAN IP address is being shown in the log file capturing data for an external interface, that indicates there is a problem.

Linux/FOSS systems come with a built-in, but usually less-known feature called source address verification. It is a kernel feature which, when turned on, starts dropping packets that appear to be arriving from the internal network, but in reality are not. Most of the latest kernels, such as Ubuntu and CentOS, do support it, but if your Linux distro does not, it is time to upgrade. Modifying the hosts.conf file to add nospoof on is another level of defense to try.

In terms of detection, for smaller Linux networks, a nice utility called arpwatch is very useful. This keeps track of MAC and IP addresses, records all changes, and can be scripted to alert administrators about a possible attack. Scripting can also be done to go through network interface logs and look for anomalies in terms of source address forging.

Summary

Packet spoofing is a difficult type of attack to tackle. It can result in serious data loss, and there are ways to detect it and stop it. Configuring firewalls, switches and routers is an important step to prevent networks from spoofing, and network administrators should know about it.

Finance firms are found to be common victims to this kind of attack, and their IT management teams should take the necessary steps to prevent financial losses or damage to their reputation. Implementing IPS devices certainly helps in getting control over the IT network infrastructure security.

The fundamental technique behind a DoS attack is to make the target system busy. In a computer server, when a network packet is being received, all components (right from the network interface card or NIC to the application running under the OS) are participating to ensure successful delivery of that packet. The NIC must monitor the Ethernet frames meant for it, align data and pass it to the network card driver, which then adds its own intelligence and passes it to the OS, which takes it to the application.

Each component involved can exhibit some form of vulnerability, and DoS attacks are devised to exploit one or more of these, to penetrate into the system.

Let’s now understand the basics of the TCP/IP protocol, which uses a handshake between the sender and receiver. Figure 1 shows how a healthy TCP handshake takes place, and how a SYN flood attack compares with it.

Figure 1: A healthy TCP handshake

When the sender wants to communicate, it sends a SYN packet with its own IP address as the source, and the receiver’s IP address as the destination. The receiver responds with a SYN-ACK packet. The sender confirms this by sending an ACK packet. Now, sender and receiver have a guarantee that they can communicate with each other.

The sender then starts sending the actual data in small chunks, and for each data packet received, the receiver sends an ACK back. When the sender sends the final data chunk, it sends a FIN signal, which is acknowledged by the receiver by sending a FIN-ACK back.

If a particular port is not supposed to respond to the request, the receiver responds with an RST packet, which means it is rejecting that request.

As you can see here, the TCP/IP stack software has to deal with complex communication, which does take some CPU and memory resources. To add to this, many handshakes are happening on a server for different source and destination addresses and for various TCP ports. All IP-based protocols such as ICMP ping, telnet, FTP, etc, actually piggyback on this framework to do their job, each working on a different dedicated socket or port.

At the application layers, the OS and the application receiving or transmitting data allocate internal memory buffers and a software process to keep track of what is being sent or received. The OS partially does this job itself, and leaves the rest to the network driver and protocol stack. Each process consumes some CPU time and memory resources.

A DoS attack exploits this situation, by tweaking TCP packets to make the server respond to malicious, fabricated network requests. TCP packets can be forged, or modified to disrupt the basic handshake process, in order to create unexpected network responses. This ultimately results in exhausting all the server resources, which when overwhelmed, stops responding.

There are various ways to do this, each using a different technical approach. Please refer to the following table — it shows you how various DoS techniques map to the OSI model of network layers.

We will now discuss each of these techniques in more detail.

Network layer DoS attacks

The MAC flood

This is a rare Layer 1 attack, in which the attacker sends multiple dummy Ethernet frames, each with a different MAC address. Network switches treat MAC addresses separately, and hence reserve some resources for each request. When all the memory in a switch is used up, it either shuts down or becomes unresponsive.

In a few types of routers, a MAC flood attack may cause these to drop their entire routing table, thus
disrupting the whole network under its routing domain.

The SYN flood

The attacker sends multiple SYN packets; upon receiving SYN-ACK from the target, it does not send ACK, but instead sends more SYN packets. This leaves the TCP/IP stack on the target to conclude that there is a possible network congestion or disconnect, and it waits for a specific amount of time. Thus, multiple partially-open connections are maintained by the stack for some time, in anticipation of an ACK response.

In another SYN flooding type, the SYN is sent with a spoofed source address, which becomes the destination address for the target’s SYN-ACK packet. However, since the system at the spoofed IP never sent that SYN to begin with, it will never send an ACK to this packet, and will simply drop it. The target system is not aware of this, and keeps track of it in anticipation of an ACK.

In both examples, the connection tables and memory resources on the targeted network components are filled up with bogus entries. When the entire table is filled up, the device stops responding.

The ping of death

In this case, a malformed ping packet flood is sent to the target. Since the TCP stack responds only to a certain type of ping packet, it fails to respond to this, exhausting the system resources.

TCP established connection attacks

This is an extension to the SYN flood, the difference being that it uses a complete 3-way TCP handshake. It does not spoof addresses, nor strip the ACK responses, but just establishes a TCP connection and simply does not send any data. Due to this, connections are maintained till time-out; a flood of such connections results in a DoS on the targeted device.

Since the handshake is gracefully done, this attack is difficult to trace, and needs advanced techniques to detect and stop.

Smurf attacks

This is a famous, widely used Layer-3 attack in which the attacker sends ping traffic to IP broadcast addresses. However, the source address is spoofed, and is of the victims’ machines. Thus, routers deliver replies to the victims, which send back with a ping response. On a larger and populated subnet, this can have a devastating effect, and can practically render the routing device non-responsive.

Similar to this is the Fraggle attack, wherein a UDP echo packet is sent instead of TCP

TCP RST attacks

In this new breed of DoS attack, the source IP is spoofed with the victim’s IP address, and this malformed packet is sent to a firewall. This forces the firewall to remember this connection for some time.

This attack is rarely used; its sole purpose is to fool the intrusion detection logic on cheaper firewalls. Unlike the host desktop, a firewall with UTM features works in promiscuous mode, and takes note of each and every packet. However, if the anomaly detection logic is not smart enough, it simply results in piling up connections and eventually rendering itself useless.

Application layer DoS attacks

Besides these network-layer attacks, there are a few that deal with the application layer directly. Such attacks are comparatively easy to detect and fix, but if ignored, can result in heavy downtime.

Application-layer attacks exploit vulnerabilities in the OS and the guest application. Listed below are a few such attacks.

Buffer overflows

This very well-known attack pushes the OS to consume resources to such an extent that it starts leaking memory, becomes sluggish or simply stops responding. It is a myth that a buffer overflow is observed only in Windows OS — in fact, it is very much true for Linux distros too. Applications such as a database, email and Web servers are found to have buffer overflow vulnerabilities.

Web and DNS DoS attacks

Web servers running on TCP port 80 are a common target for DoS attacks. Attackers usually send multiple HTTP requests (not malformed at all) targeted to retrieve enormous amounts of data from the backend database server.

Such request floods make the database server busy, keeping the Web server waiting for data. This creates a pile-up on both servers, which become unresponsive to further requests. This can happen unintentionally too, especially when breaking news is posted and everyone tries to access it at the same time. In case of DNS servers, the attack method is similar to Web servers, but it has serious consequences. If a DNS server becomes non-responsive, it can take down the firm’s entire network.

Distributed DoS attacks

In simple terms, a distributed DoS attack (DDoS) combines multiple attackers using the various techniques discussed above, and can result in catastrophic failure. Please refer to Figure 2 to understand how a typical DDoS attack is planned on a website.

Figure 2: A typical DDoS attack

A lethal combination of spoofed SYN flood and old-style ping-of-death attacks is typically used to disrupt an IT network that is open to the Internet.

In a modern form of DDoS, the attacker injects malicious code into a virus and spreads it to millions of computers, making them zombie machines. On a specific day and time, all infected machines start executing that code, which is usually written to access a website or plant a network-level attack onto a targeted system. Since it is difficult to find out who injected the code into a virus, it is difficult to trace the real attacker.

In another non-hazardous form, a DDoS attack is modified to access the attacker’s website and click advertisements on it, which in turn helps the attacker earn money from ad clicks.

Protecting FOSS systems

Though DoS is one of the oldest and most commonly known attacks, unfortunately, there is no fool-proof solution to stop it because practically, it is difficult to decide which network connection is legitimate, and which one is initiating an attack.

While there are specific tools for a particular type of DoS attack, it boils down to cyber-security design and monitoring to strengthen the network.

Typical symptoms of a DoS attack on a Linux server are a sluggish system or a slow website, sudden and prolonged increase in processor and memory utilisation, excessive disk thrashing without any business activity, slower file transfers, etc.

On a network monitoring system, there could be a large number of TCP packet drops, abnormal TCP resets, broken TCP SYN packets being received, or duplicate ACK packets being sent. Usually, the first-level component impacted is a router, followed by a firewall and other components like switches. Firewalls cannot protect the router, but the firmware of modern routers (e.g., Cisco 7600 or X443) contain patches to protect against DoS attacks. Though modern firewalls have many features to combat DoS attacks, they don’t always help.

Firewalls can certainly protect against network layer-based attacks, but usually fail to protect systems from application-layer attacks like on port 80 (HTTP). Here, an application-level firewall is needed to filter each request and ensure its legitimacy.

For Ubuntu and RHEL, an APF (Advanced Policy Firewall) is a great tool that can help mitigate DoS attacks. Linux FOSS systems are blessed with fantastic network drivers, as well as many built-in features such as a packet-filtering firewall, packet monitoring, network monitoring tools, kernel hardening tools, etc.

For smaller Linux networks, a nice script can be written to SYN Trap open connections and to stop bogus TCP RST connections, as a first line of defence.

For mission-critical corporate Linux networks, deploying an Intrusion Prevention System device (IPS) is the best choice. IPS devices sit on a network in promiscuous mode and use built-in anomaly detection algorithms to intercept and decode each and every packet. Since its intelligence ranges from Layer 2 to Layer 7, it can gauge which packet is legitimate, and which isn’t. It has a great alerting mechanism
to proactively inform and stop DoS and DDoS attacks.

A good combination of IPS devices, UTM firewalls and application layer security can help stop these dreaded attacks.

In today’s world, IT is used almost everywhere, to automate routine tasks and add value to our lives, right from purchasing stocks on the Internet to withdrawing money from an ATM. With its benefits, there are bound to be problems as well. Practically every computer user knows about the threat of viruses. Even the partially computer-literate learn the hard way that their data is not safe but can be corrupted, deleted or stolen — hence, good backups are essential.

While there are many anti-virus software available, there is another category of malicious software called rootkits, which are more dangerous by nature. These are created by some of the world’s smartest programmers, but unfortunately, with bad intentions, hence creating a problem for users.

Malicious software could run as an application, in user-space, or as part of the operating system. Rootkits are typically in the second category, making them powerful, dangerous, tough to detect, and very difficult to get rid of. An unknown person gaining control of your machine and working on it, probably at the same time you are using it, could be dangerous, and invades your privacy. Key-logger kits can steal passwords, credit card numbers, personal details, finance data stored in spreadsheets, confidential business data, etc.

Rootkits are essentially small collections of tools, utilities and scripts. The sole idea behind installing this collection is to be able to gain administrator-level access on the target machine, so that it can either be remotely controlled to harvest secret data, or used to attack other vulnerable machines, to replicate the rootkit on those, and subsequently “own” those systems too.

A rootkit typically contains a combination of network sniffers, log parsers, log-cleaning scripts, scripts to change user privileges, core systems utilities to detect IP addresses, netstat, utilities to kill running processes, scripts to hide code, and a zipped copy of the whole kit, for replication.

How rootkits differ from viruses

Let’s quickly look at the symptoms of both first.

As you can see in the above table, while the dividing line between viruses/Trojans and rootkits is becoming blurred (with the common purpose of causing data loss, or capturing/harvesting confidential data like user names, passwords, email addresses, etc), there are still some distinct differences between them. Although a virus normally runs in “stealth mode”, hiding itself by infecting executables and system files, it still typically runs as an application — which is why anti-virus software can detect and remove it. A Trojan, which is an advanced virus, is meant to hide in a more sophisticated fashion.

A rootkit, on the other hand, subverts part of the operating system to hide itself and gain the maximum control possible. Due to this, it is capable of monitoring as well as performing all activities on a system. It can act as a vehicle for other rootkits and viruses as well. Rootkits turn a computer into a remotely controllable victim, often also making it a spambot to send out unsolicited commercial email.

Infection/installation

Rootkits follow virus-like methods to compromise the system; however, since they need OS-level privileges, installation methods differ a bit. Typically, attackers try the following ways to get access to a system:

• With physical access to the system, attackers try guessing non-complex passwords. If it’s possible to boot the computer from the attacker’s media (CD, USB drive), then the attacker can also try various techniques to recover the root password from the installed OS on the hard disk.
• Attackers can remotely detect OS or network application vulnerabilities that have not been patched by security fixes, and run attacks against these.
• Over the Web, attackers use embedded scripts that sneak into a system via the browser.
• Many spyware software act as easy and sure-fire vehicles to propagate rootkits to systems connected to the Internet.

Usually, a rootkit is packaged as a compressed self-extracting zip file that extracts its contents once it gets on a system. Often, a small kit installer is also packaged, which ensures the health of an installation, gains administrative rights and does what’s needed to run in stealth mode.

A few advanced rootkits are also capable of detecting anti-virus and anti-spyware software, and accordingly change their way of working, or modify their outputs, to avoid being detected. For example, some rootkits copy the kit tools to the root of the file system, but a simple directory listing does not show those files since the rootkit modifies the output of the OS directory listing commands.

Since attackers make every attempt to optimise the kit’s code size, it takes little time to install and activate a rootkit. Rootkits are expected to spread as much as possible, and so almost all contain a distributable copy of themselves. When in action, they use all possible means to scan the local network and find other vulnerable systems.

If LAN security is not properly designed, gaining administrative rights on one system is often enough for a rootkit to easily propagate to other systems on the LAN. Modern kits are designed to detect Internet access, fetch the latest rootkit update, copy it to all infected machines, and then try to propagate over the Internet to other vulnerable or poorly configured systems.

Detection

As mentioned earlier, detecting a rootkit is a seriously difficult job, which requires administrators to go an extra mile when compared to detecting viruses and Trojans. While a few rootkits can be blocked by the latest anti-virus tools, most can still bypass these. Since the rootkit becomes part of the operating system, seemingly rudimentary methods such as booting from an emergency disk or USB drive are very useful in loading a fresh and known “good copy” of the OS, to run a detection tool without interference from the rootkit. Even then, many detection tools only detect, and still cannot remove these, which requires a manual cleanup process.

Detection tools need to adopt a different methodology rather than simple file checks or process checks, because if the rootkit is active, it may fool these legacy detection methods or algorithms. A better approach is to take a snap of different views of a system, and compare those with a baseline snap captured when the detection tool was first installed on a newly-built system.

Newer tools use artificial-intelligence algorithms to detect variations in the system’s status, and hence don’t require any baseline benchmark. There are various detection algorithms, such as signature-based, which are similar to detecting a virus, or integrity-based, wherein files, processes and kernel modules are checked for their binary integrity. There is another effective method, where a memory dump is taken by the tool, and then parsed for anomalies, signatures or trends which are either directly or symptomatically related to a rootkit.

There are many commercial and open source rootkit detection software; let’s take a look at a few popular tools.

McAfee and Symantec provide some protection against letting rootkits get installed, and also provide some level of detection. However, detecting rootkits needs dedicated specialised tools.

In the FOSS world, a famous tool called chkrootkit takes precedence over other tools. It can perform detailed binary checks, file modification validations, and check kernel modules. It works smoothly over a wide range of Linux distributions, making it a must-have tool in the administrator’s toolbox.

Similarly, Tripwire is an important open source tool, which can perform extensive MD5 hash checking, and detect anomalies such as back-door process checks and other local exploits.

Rootkit Hunter is another such famous tool — an elaborate script capable of checking multiple rootkits. It can also check for wrong file permissions and kernel modules, which can be incorporated into a daily scanning job. Upon arrival of a new rootkit, an efficient systems administrator can study it and write a script to detect it.

It is important to note that malware creators often study these detection mechanisms too, and make necessary changes in the newer versions of their rootkits, to make them undetectable by detection tools.

Rootkit examples and the damage they cause

Similar to viruses, there are unfortunately a lot of rootkits for Windows, for commercial-grade Red Hat Enterprise Linux, as well as all open source distributions. In many of these, since the kernel does not change drastically over a period of time, it’s often simple for attackers to write kits that would easily propagate.

Although there are a few hundred dangerous rootkits impacting the FOSS world, we will look at just a few commonly found ones.

Let’s start by mentioning the LRK kit first, because it is one of the oldest, and still active (first detected in 1997, but still found today on vulnerable systems). It has multiple versions, and is known to install very commonly used binaries such as netstat, linsniffer, inetd, ifconfig, etc.

Knark is a kit that is very difficult to detect, as it resides entirely in the kernel. It is very notorious because once active, it hides ports, files and processes from the administrator.

Beastkit is of a fairly new variety, written to target Red Hat distributions, while another dangerous variety, called Illogic, is known to start a process on port 901, which lets attackers telnet into the system and do practically anything that an administrator can with physical access.

It is worth mentioning the rootkits that have caused significant damage to Linux systems: Sneakin, Kitko, Ajakit and Devil. All these kits are capable of using advanced techniques, such as detecting the OS and modifying the kernel on the fly, port NATing to decipher information, key-logging to steal passwords and sneakily emailing them to the attacker’s email address, optimising actions to take least amount of system resources, etc. These kits are also known to spread quickly, and convert infected machines into zombies that participate in spreading the infection further.

As noted earlier, rootkits also open backdoors in the form of ports, IP addresses or kernel processes, which allow attackers to take complete control of the system remotely, even over the Internet.

There are a few “friendly” rootkits that work with each other. If a rootkit trying to invade a system sees that there is already a friendly kit present, it helps by updating older binaries to make it stronger. A few rootkits are known to act as gateways to install viruses and Trojans on a LAN; the rootkit goes in first, makes room, sets user privileges, takes control of the file system, disables any anti-virus that’s running, opens backdoors for viruses, and goes into stealth mode.

A new generation of rootkits is known to install fake SSL certificates to try deciphering HTTPS communications to gain credit card information, which is usually sent over a secure channel. There are some kits that use compromised systems for man-in-the-middle attacks on other systems’ communications, which makes it difficult for cyber security investigators to find the actual attacker.

Designing a continuously improving security infrastructure is very essential to stop the spread of, and damage caused by rootkits. The latest intrusion-detection systems (IDS) can stop rootkits effectively at the doorstep. Senior IT management should be aware of these serious problems, and take aggressive action to secure their networks.

Network administrators usually rely on generic and built-in monitoring tools for network security. Ideally, the network infrastructure is supposed to have carefully designed strategies to scale up monitoring tools and techniques as the network grows, over time. Without this, there can be network performance challenges, downtimes due to failures, and most importantly, penetration attacks. These can lead to monetary losses as well as loss of reputation. Thus, there is a need for best practices to monitor network infrastructure in an agile manner.

Network monitoring challenges

For starters, each of the seven layers of the OSI networking model has its own responsibilities, which call for separate methods of monitoring and security for each layer. Network monitoring is seemingly simple — but in reality, it’s a very complex process. Mixing traditional network monitoring with security monitoring further complicates things from the design perspective, for network architects, network operations teams, and the systems administrators who manage it.

While there are multiple challenges in network monitoring, the most important one is the vast amount of data gathered by the monitoring tool, and the amount of time required to assimilate the information and apply intelligence to it, in order to achieve actionable decisions.

For example, a simple promiscuous packet capture on a network card for barely ten minutes provides us with a few megabytes of data. Finding out which HTTP request failed, and why, is like finding a needle in a haystack. Another important consideration is to identify key areas, often called choke points. The incorrect selection of a choke point can result in erroneous data that doesn’t accurately reflect the current performance scenario. This leads to incorrect capacity planning and security mapping.

One challenge worth mentioning is caused by the unprecedented growth of a network, a result of the organisation’s growth due to business expansion or company mergers. The bigger the network, the tougher it is to visualise the scale of network infrastructure. This can result in performance bottlenecks as well as security vulnerabilities. Finally, failure to incorporate proper monitoring tools is also a challenge to be addressed by senior IT management staff. It has been observed that relying purely on commercial products actually limits a firm’s ability to bring diversification into the network monitoring process.

The use of appropriate FOSS tools and products is highly recommended wherever applicable. As an example, there is hardly any single commercial tool that can relate Layer 7 network monitoring to the underlying Layer 2 packet and help troubleshoot a performance problem.

Network security challenges

From the infrastructure scaling point of view, irrespective of the size of an organisation, network security is often a complex area to deal with. Since network infrastructure contains components like firewalls, routers, managed switches, etc., the configurations and settings for each of these components further add to the complexity.

Also, when faced with the choices of multiple devices offered by many vendors, it is easy for a network architect to get distracted from considering an appropriate solution customised to the network. As the network grows, it can be more prone to vulnerabilities and loopholes, needing tight security policies and careful designs, using cutting-edge technology devices and solutions.

From the security point of view, a new breed of viruses and spyware has emerged recently, which exploits the operating system as well as the networking device’s vulnerabilities, and can take control to cause enough damage. Though there are multiple security solutions available, hackers are often one step ahead of the cybercops.

It is often the case that an organisation is more prone to internal attacks than to attacks originating from outside the firm’s network infrastructure. Preventing such attacks needs the latest techniques, such as the deployment of intrusion detection systems, unified threat management systems (UTMs), etc.

Devices involved

Network monitoring is the term used for health monitoring, whereby monitoring utilities keep a watch on various networking components, to ensure their uptime and overall performance. Network security monitoring is at a level further down the networking layers, whereby utilities capture each and every traffic packet in a promiscuous mode, correlate it with known and unknown attacks and hacking styles, store the log for further “deep-dive” analysis, and report the alerts.

While there are known components such as intrusion-detection devices, UTMs and firewalls, there are multiple software solutions, both off-the-shelf or FOSS-based, which extend the functionalities of these components, and work on the extensive log data gathered by monitoring appliances and devices. We will discuss these solutions in more detail shortly.

Best practices in network security monitoring

Network security monitoring involves collecting network packet data, segregating it among all the 7 OSI layers, and applying intelligent algorithms to get answers to security-related questions. The purpose is to know in real-time what is happening on the network at a detailed level, and strengthen security by hardening the processes, devices, appliances, software policies, etc.

While there is no single list of practices that can cover all possible situations, we can still enumerate some best practices to implement and follow for any network infrastructure:

1. Perform network performance measurement before deploying the security monitoring solution. This is essential because security monitoring can have its own footprint on the network, especially if the monitoring solution is software-based, running on the servers.
2. If possible and affordable, deploy more than one anti-virus solution. Many anti-virus software solutions don’t offer spyware detection, or don’t do it right; hence, a combination is always helpful.
3. Deploy at least one FOSS packet-capturing software on the network. Though the IDS systems do this job partially, there can be situations where the IDS could be too busy to be used as a packet viewer, and a FOSS utility can come in handy for daily chores.
4. Gather monitoring data at a secure place. It is often a mistake to gather security data on a desktop or a server which is easily accessible, making the network vulnerable at that point. Since security applies to the monitoring process too, the data captured must be stored in a secure manner.
5. Monitor all layers; don’t leave anything to chance. Usually, the data link layer is omitted from monitoring; however, since a new wave of attacks can exploit Ethernet frames too, it is important to take this layer into account. The same applies to the network layer, as most internal attacks can easily use it to exploit vulnerabilities.
6. Deploy the IDS behind the firewall, since the firewall filters out everything that is not meant to enter the LAN. This improves the IDS’ efficiency by keeping the clutter away.
7. Capture VLANs separately. Since VLANs are separate TCP broadcast domains, separately gathering and analysing data for each can help detect internal and external security problems quickly.
8. Consider all protocols. Many firms still use NetBIOS internally along with TCP/IP; such a situation demands monitoring all protocols on the wire. There are a few legacy types of attacks based on the NetBEUI protocol, which could be captured.
9. Enable optimal auditing levels on the monitoring devices. Setting up too many audit event captures can easily confuse monitoring solutions while detecting an anomaly, whereas having very few audit logs can render security monitoring useless.

It is a best practice to have an update in the monitoring process whenever a device is added to the network, removed or changed. Also, even if there is no change in the design, monitoring processes should be reviewed in a timely manner, in order to remove errors, and keep pace with changing security scenarios.

Monitoring network security using FOSS solutions

It is worth mentioning a few commercial products used in network security monitoring, before we talk about the FOSS solutions.

Cisco Security Monitoring, Analysis, and Response System (MARS) is a famous solution that falls in the category of Security Threat Mitigation systems (STM). MARS gathers and stores raw network data, and correlates it with intelligence algorithms. IBM Proventia IDS devices fall under the same category, providing richer features and customisable alerts. All these threat monitoring and mitigation appliances enable us to centralise detection, mitigation and reporting on priority threats by leveraging the network and security devices already deployed in a network.

In the FOSS world, the Backtrack Linux distribution is famous among cyber-security groups, as it allows one to write easy, effective and yet industry-standard utilities quickly. Snorby is an open source suite of network security monitoring utilities, which interacts seamlessly with open source IDS solutions such as Snort, Suricata, etc. Snorby contains the necessary binaries to run on various operating systems, including Microsoft platforms.

Siem-Live is another such distribution gaining support and momentum. A good thing about this distribution is that it comes on a bootable CD, relies on generic IDS systems and log collectors, and applies intelligence on it to create customised reports. The event correlation engine of Siem-Live makes it fit for small- and mid-scale networks.

This article will be incomplete if a famous tool called SGuel is not mentioned here. While maintaining the quality of the intelligence engine, this tool comes with a nice GUI and essential bells and whistles, such as real-time event gathering, raw packet capturing, reporting, etc.

Summary

While there are numerous tools and practices available for monitoring the security of a network infrastructure, it is important to apply the industry’s best practices. It is imperative for IT management teams to know what is happening on the network, in real-time, and accordingly introduce the latest technologies. Since cyber security is a process of continuous improvement, there cannot be a single list of best practices, as the quality of practices evolves with network architecture and design, as well as the devices in use.

Network security monitoring is essential for the IT health of an organisation, and a rock-solid monitoring mechanism can help reduce damages to the business. While there are commercial solutions available for this, FOSS also provides real-time monitoring software and cutting-edge intelligence engines. Senior IT management personnel need to form a security monitoring group within the organisation to deploy the best practices and strengthen the corporate network.

For starters, there are seven layers that form the OSI networking model. Beginning at the physical layer, it goes up to the data-link, network, transport, session, presentation, and the application layers. While the physical-layer security is taken care of by surveillance systems, the application-layer security is handled by code instrumentation, which is the job of the developers. In the wake of overall security awareness in the IT world, responsible product managers incorporate code security as a mandated practice in their code-deployment release cycles.

All the intermediate five layers of the OSI model do require security measures too, by some means or the other, irrespective of the software component to which they are catering. Let’s take a look at these layers, which are usually not taken too seriously by security administrators. It is important to note that these layers actually need to be more secure than the layers at the top and bottom, because there are such a variety of attacks on the interim layers that there is no single means, tool or process to control them.

The initial levels of compromising vulnerabilities usually start at the data-link and network layers (alternatively called Layer-2 and Layer-3 attacks). Layer-2 security is susceptible to MAC address spoofing, Layer-2 to flooding attacks, whereas Layer-3 security is subject to IP address-spoofing attacks. Similarly, Layers 4, 5 and 6 are prone to packet-crafting, session-stealing and cryptography-based attacks, respectively.

To understand this better, please refer to Figure 1, which shows various security methodologies and how each of these maps into the OSI layers of networking.

Figure 1: Security methodologies for various layers

While there is no such thing as 100 per cent security, the security design can incorporate carefully thought-out, customised security mechanisms, which can lead to a total security solution in an IT infrastructure. There is a wide range of devices from the security view-point, such as firewalls, intrusion-detection devices, content firewalls, etc., covering each layer, and also overlapping multiple network layers.

Why is typical security not enough for database servers?

Securing database servers needs administrators to go the extra mile in terms of technical design efforts. It is a job to be carried out by sysadmins as well as database admins, to cover all the network layers. Often, the practice is to assume that using OS-level user administration and leveraging the product’s built-in security is enough to protect databases.

Unfortunately, this is not true. While the user- and role-based models with strong passwords for SA accounts, etc., are essential, they exist only at the application layer; relying on them alone defeats the purpose of securing databases.

The technical reason behind why legacy methods fail is because a database is a widely and continuously accessible component, which makes it more vulnerable and susceptible to attacks. Database security requirements touch all networking layers, and hence, need careful design.

As an example, consider a database server that accepts connections from within the corporate network as well as from outside. It is accessed programmatically by the frontend software, as well as directly by admin and development staff to run queries, etc. It is accessed by backup services, anti-virus servers, and other maintenance-based utilities, and thus is widely accessible.

Any of these human or automated means of accessing the database server make it vulnerable to possible attacks originating at various networking touch-points. The latest spyware, and vulnerabilities that revolve around the infamous SQL-injection method, prove that something needs to be done beyond these legacy methods.

This is exactly where the security mechanisms and solutions that span the lower layers in the networking model come into the picture. Let us look at a few moderately advanced methods to secure database servers, followed by some serious solutions.

Securing database servers

NAT and PAT

Using network address translation and port address translation is the first recommended step. Since database products use predefined default ports, it is the first thing hackers look for, and hence should be changed. Even though the database IP address and port is not exposed to the outside world, it is a best practice to change it, to keep spyware and viruses away. Any standard firewall nowadays provides NAT and PAT features.

A demilitarised zone (DMZ)

Please refer to Figure 2, which shows a typical database server farm in a corporate IT infrastructure, hosting mission-critical databases and data-warehousing services.

Figure 2: Database server farm

As we can see, the database server is not exposed to the Internet, but is separated from it by an additional firewall. This design method is well-known, and is called a demilitarised zone (DMZ). In earlier days, it was a practice to keep database servers in the same network as the frontend/Web application servers, making them prone to attacks. An additional layer of security is now achieved by the second firewall, which opens up only the database port for the frontend servers, and not the IP addresses of the entire Internet. If the database carries mission-critical datasets of sensitive information, this method can be clubbed together with the NAT-PAT method, making it even harder for malware to breach the database server.

Content-based firewalls

The latest firewall products are equipped with the capability to look into the data packets flowing in either direction, and provide systems administrators a means to configure actions based on the intercepted content. These firewalls, sometimes called Layer-7 firewalls, are capable of reading SQL queries, query data, validation data, XML data, etc. Deploying such a firewall can certainly reduce virus attacks drastically, because data packets pertaining to ill-behaved operations are dropped and reported.

SSL connections

While SSL technology is commonly used to secure Web-server connections to browsers, it can also be used between frontend servers and the backend database server. SSL uses public-key and private-key cryptography to encrypt all connections between caller and listener. While this solution is a bit costly and takes a toll on data-transfer speeds, it is worth the effort to alleviate man-in-the-middle attacks.

IPSec security

The real challenge while securing a database server is to figure out who should access it. In terms of people, it is as simple as deploying user-ID and password-based access. However, if the network is compromised by a hacker, this won’t help much. A further step, in such a case, would be to deploy IPSec security, whereby each server will connect to the database server using an IPSec token, making it a unique and non-crackable connection.

Securing open source databases

A database administrator always carries out the essential chores to secure a database; however, for Linux-based open source solutions, a little extra effort is required. While there is no single solution to cover all open source database distributions, there are a few good tools for each.

A popular open source database is MySQL, which is used by many commercial websites as their backend. MySQL comes with the necessary scripts to harden the server from the security perspective, such as scripts to remove test databases, lower system and database privileges, enable the built-in Linux firewall to configure default database ports and block others, etc.

MongoDB and CouchDB are two famous database servers running on Apache distributions. Both come equipped with the great feature of IP binding, whereby the database kernel services are bound to one single IP. Adding an open source content firewall or a NAT-PAT solution can help create a very cost-effective and, yet, secure database farm.

The Hypertable open source database, which is modelled after Google’s BigTable DB structure, provides an extensive library of API calls that are used exclusively for adding security features. It is easy to write scripts using those API calls, to securely deploy and configure Hypertable distributions in a big server farm.

Unlike Microsoft SQL Server, Oracle and Sybase products, while designing security for open source products, it is important to incorporate a combination of network security methods, as well as the database product’s application-layer security.

Summary

While much effort goes into securing a database, due diligence is required to secure the server hosting the database too. The security measures taken for the application layer as well as the network layer must go hand-in-hand to keep mission-critical data safe from hackers. A rock-solid change-control process needs to be in place to tighten the IP stack, the patch-management solutions, as well as the physical security.

It is often found that Layer-2 security is not designed adequately, or in some cases is simply absent, which makes all the application-layer security measures futile. Senior IT management and database administrators need to work with systems administrators to amicably formulate an end-to-end security solution. This applies to all database-centric solutions, irrespective of whether they are off-the-shelf or FOSS-based.

In this article, we’ll run through some of my favourite tools for network troubleshooting. If you’re a network admin, you might find these tools useful. However, I have tried, as usual, to favour concepts and description over detailed command information, so even a normal home user might find this article interesting as a casual read. I expect anyone with a serious interest in one of the tools to check out the man pages or other documentation anyway.

All at sea

I’m an avid quizzer, as I’m sure some of you are. I sometimes conduct quizzes too, and one of my favourite questions is: what is the connection between the Sonar equipment used in a submarine and modern networking? Of course, it’s the humble ping command, which was named after the sound that a Sonar makes in a submarine. If you’ve seen Hunt for Red October you’ll know :-) So, continuing the marine theme, ping is the first port of call when you have a network problem, and naturally, everyone knows how to use it to check if some host is up.

Figure 1: The output of a ping command

But is that all ping can do? Even in the normal run, there’s important information. Figure 1 shows a typical ping output.

There’s a very important number that ping shows, called the ‘round trip time’ (RTT). RTT is a measure of how close a host is to you, based on how long it takes a packet to go out and come back again. RTT on a LAN tends to be less than a millisecond, while 2-3 milliseconds (as in Figure 1) is more typical of a wireless network. RTTs on WAN links are more in the 200-600 millisecond range, reflecting the number of routers that they have to go through.

ping can also get you clued into an unreliable connection, by showing a packet loss in the status line (the last line but one above). For instance, it might say “50 packets transmitted, 47 received, 6% packet loss, time 44754ms,” which would indicate a pretty bad connection.

But this information only shows up at the end, after you kill ping. What if you want to keep the ping running for a while and continuously see how reliable the connection is? Well, you can watch the ICMP sequence number to make sure it increases exactly by one and doesn’t skip a few, but that’s too tedious to keep up for a long time. I mean, that’s what computers are for, right—to do the tedious stuff? So can the humble ping command do anything more?

Turns out it can, and in a very imaginative and simple way! The command to use is ping -f -i 1 host. With the -f option, ping prints a “.” for every outgoing packet and a back-space for every reply. Thus the number of dots on the display is the number of ping packets that have not yet been acknowledged by the remote side. A fast and reliable connection will not show you a single dot—every dot will be cancelled by a back-space well before the next dot appears, so the cursor sits on the left of the screen and nothing seems to be happening.

If you see the number of dots increasing gradually, you know there are packet losses happening on the link. It’s actually a pretty cool display, but in order to see it, you have to test it against an unreliable server or an unreliable network. For most home users, the best way to do this is to use a laptop to ping a wireless router, and gradually move the laptop further and further away from the access point.

When you use -f, don’t forget the 1-second interval flag (-i 1). Otherwise, you get what is called a ‘flood ping’, which can look like a Denial of Service attack to the target host, and they might complain (or worse, retaliate). In fact, a fast machine on a fast network can bring down a network using a ‘flood ping’ without an interval specified! However, if the target host is yours or you have permission to do so, it can be fun to try something like ping -f -c 500 -s 1400 host. The laptop + wireless method of simulating a flaky connection is really useful to see this in action. Also, try different values for the packet size (the -s option).

This is not just fun—you’ll start to recognise that this simple dot pattern can clue you into troublesome connections very quickly, although once again, I must repeat that -f without -i 1 should be used very carefully and sparingly, and only on your own hosts.

Who’s dropping the ball?

So you have a flaky connection to your office network…and your VPN keeps dropping off. Or your YouTube feed is constantly stopping to buffer; I mean, we know which is more likely, right?

A ‘flood ping’ tells you there are lots of dropped packets but doesn’t tell you where or who’s responsible. If you’ve ever done a traceroute, you know there are multiple hosts in between yourself and the target, and it may be useful to know where among these hops the packet loss is occurring.

Figure 2: An example mtr output

This is what mtr shows you: it shows where the packet loss is happening, in real time, using ICMP ECHO requests (i.e., ping packets). It’s one of the best tools for figuring out where the problems are, with a simple but really useful display, including a quick online help screen. The default screen looks like Figure 2, once it’s started up, though, of course, it’s continuously updating.

You can quickly see which intermediate router is losing the most packets, as well as which ones are taking the most amount of time to reply. An even more useful display is obtained by hitting j, which shows you packet loss in absolute numbers instead of percentages. More importantly, it also shows you something called ‘jitter’, which means inconsistency in response times. You can also think of jitter as a measure of transient or occasional congestion in that link, causing only delays for now, though if the quality degrades further, there may be packet loss too. Seasoned travellers know that when too many flights show a ‘delayed’ status, sooner or later some will go from ‘delayed’ to ‘cancelled’—this is pretty much the same thing.

Figure 3: Display mode in mtr shows actual timing from the last 50 ping sequences

The best feature of mtr can be seen by cycling the display mode (by pressing d). This is a very interesting display, showing the actual timing results from the last 50 ping sequences (or more, if your screen is wider). A “.” means a reply was received, a “>” means it was received but took a long time, and a “?” means it has not been received yet. If you cycle the display mode again, the display changes to show 6 levels of granularity in the RTT, and a scale at the bottom to say what these levels mean. For example, in Figure 3, a “1” means a reply was received more than 5 but less than 14 ms later, and a “>” means a response packet was received more than 222 ms later.

This is a very cool display—and I can tell you from personal experience that it never fails to impress when you’re trying to prove to someone that the problem is on their router! Most Windows-type admins are left speechless—although that’s probably because they are trying to digest the fact that you don’t need a bloated GUI framework to get useful work done!

Moving up a layer or two

However, it often happens that the system we are trying to trace does not accept ICMP (pings) or UDP (traceroute)—most security conscious admins disable everything that is not absolutely needed, and if it’s a public Web server, it may only allow HTTP/HTTPS (ports 80/443). For times like this, you could just use the traceroute’s -T option, which uses TCP instead of UDP. It works pretty well, although this is not a continuously running program, so it tells you about connectivity and RTT for one round only.

Figure 4: lft shows what's in between and who owns what

However, we may want to find out who owns a particular network. When you need that, lft (Layer Four Trace) is pretty useful. Above and beyond what traceroute can do, lft can show if there are any firewalls in between, as well as what organisation owns those gateways or routers, as you can see in Figure 4.

Thinking local; iftop and iptraf

So let’s say you’ve figured out who or what is slowing down your packets and (hopefully) got someone to fix it. Your traffic is moving pretty smoothly, and everyone is happy.

Actually, some people are too happy—they’re hogging all the bandwidth! You need to find out who they are and have a quick word with them. The only question is: who is hitting the net so badly and what site are they hitting?

Even if you’re not a Simon Travaglia, and you have only your own machine to worry about, perhaps you suddenly noticed a lot of activity on the network monitor (you do use one, right? I suggest conky for low-end machines and gkrellm for all others!) and you’re wondering what program is doing it and why.

While netstat can certainly be used to give you this sort of information, there is another tool that has become a very useful part of my toolkit now, which is called iftop. It’s a pretty old tool, and it hasn’t been updated in a couple of years, but don’t let that stop you from trying it.

Figure 5: iftop -nNPB on a lightly-loaded system

iftop is an interactive program with a number of cool features, all of them accessible by typing some key, and it has a quick one-screen online help in case you forget the keys. Running ifopt -nNPB on a lightly-loaded system might look like the output shown in Figure 5.

The display is quite self-explanatory, except for the last three columns in the main display. These are averages of the data transferred over the previous 2, 10 and 40 seconds, respectively.

The black bars are important. Across the very top is the ‘scale’ for all the bars, and the bars actually represent the 10-second average (the middle column) by default, although pressing “B” will cycle between 2, 10, and 40-second averages. This way you get a visual indication of what hosts and ports are hogging the traffic.

You can do some cool things here—you can choose to look only at outgoing or incoming traffic or perhaps the sum of the two (press t to cycle between these modes). You can aggregate all traffic for each source into one line by pressing s, and for each destination by pressing d. Be sure to read the online help as well as the man page—it’s worth it.

What’s even more cool is that there are two filters to limit the output. Typing l enables a ‘display filter’—the pattern or string you enter will be applied to the host+port field and used to filter lines appearing in the display. This is a literal match: for example, if you type “pop3” as the search filter, then use “N” to disable port number resolution, you’ll have to change the search string to “:110” in order for it to match. The same goes for host names versus IP addresses.

Using l only affects the display; the totals still count all the traffic. On the other hand, you can use f to set a packet filter condition that will stop traffic that does not match, from even coming into the program. For instance, you can type “f” then “port 25” to see only SMTP traffic. This filter can take quite complex conditions, using the same syntax that popular tools like tcpdump, etc, use. Plus, this filter can be specified from the command line too, like:

iftop -nBP -f ‘port 22’

All in all, this is a pretty nice tool to keep an eye on things once in a while or perhaps when someone complains things are a little slow.

iptraf is also a very nice and easy-to-use tool, with a very neat curses GUI. It actually has a lot more features than iptraf—the IP interface monitor shows you TCP and UDP separately, and for TCP it shows you packet flags (making it easy to identify connection attempts that are not succeeding, for instance). Overall statistics for each interface are also available in a separate screen, and on the whole it’s almost a real GUI (using curses), with menus and sub-menus, etc. It also has a very slick filter specification GUI, if you’re not the command line type.

Despite all this, however, I find myself using iftop for day to day use, because iptraf lacks the aggregation, multiple averages, quick and easy filtering, etc, that iftop does. Plus, most of the filters I want are much easier to type into iftop or at the command line.

Some last words

I don’t use these tools every day, but when I needed them, they were really useful. Could I have gotten by without knowing them? Maybe… but that’s not how we think, is it? A workman needs as many tools as he can get his hands on, and these are some of mine.

Among these, iftop remains the one I use more often than the others. It’s actually closer to monitoring than troubleshooting, but they’re all great tools, and exploring them gives you an understanding of what’s happening under the hood as your machine goes about its daily business.