Analysis on the in-house testbed dataset

3. False Alarm Reduction in Signature based IDS: Game Theory Approach

alert tcp EXTERNAL_NET any - >HOME_NET 3372 (msg : “DOS MSDTC attempt”; flow : to_server, established; dsize :>1023; reference : bugtraq, 4006; reference : cve, 2002-0224;

reference : nessus,10939; classtype : attempted - dos; sid : 1408; rev : 10; )

This Snort rule searches for TCP packets coming from any external network from any port to any machine inside the network on port 3372. If any packet with these characteristics is part of an open TCP session and also if the size of this packet is bigger than 1023 bytes, then Snort generates an alarm for DOS MSDTC attempt attack. However, this attack is effective only against the Windows based systems but are ineffective against the Linux based systems. Therefore, operating Snort with default settings, without considering the context of the underlying network environment produces a large number of false alarms.

It can be observed from Table3.7that the accuracy of the proposed false alarm minimiza- tion framework is significantly high for both critical and non-critical vulnerabilities of the IDEVAL dataset. This implies that the proposed framework successfully filters out most of the FP alarms generated by the Snort.

Although, many works [94] [95] in the literature have reportedly pointed out various flaws in the DARPA’s IDEVAL dataset, it still remains one of the few large scale attempt at an objective evaluation of IDS systems. As such, it does provide a basis for making a rough comparison of existing IDS systems under a common set of circumstances and assumptions.

Moreover, in absence of better and openly available benchmark datasets, vast amount of IDS research is based on the experiments performed on the DARPA’s IDEVAL dataset.

The reason for choosing Snort as the default signature based IDS was because of its large community base and its rich set of attack signatures. The attack signatures in Snort are populated from various publicly available vulnerability databases like BugTraq [7], CVE [8], Nmap [9], Nessus [10] etc. This makes Snort an ideal candidate for evaluation of the signature based IDS.

3.4. Performance Analysis

Figure 3.7: Configuration of the in-house testbed network setup

configuration. The host machines in the testbed network were connected in a LAN with a CISCO catalyst 3560 G series switch. Port mirroring facility was enabled at port 8 of the switch to capture the data packets of the testbed network as a tcpdump file. One of the machine in the testbed network running Ubuntu 12.04 and connected to port 11 of the switch was used to generate attacks using a metasploit toolbox [96], which is a freely available open source exploit toolbox. Some of the attacks were also launched from outside the testbed network by external attackers connected through router on port 6 of the switch. The Threat profile of the testbed network was generated using various vulnerability scanners like, Nmap [9], CVE [8] and Nessus [10]. Snort with default set of rules was used as signature based IDS.

Table 3.8: Performance of the proposed framework on the IITG Lab. dataset Critical Vulnerabilities Non-Critical Vulnerabilities Attack Class Alarms After Corr. Acc (%) DR (%) Alarms After Corr. Acc (%) DR (%)

FTP 458 37 100 92.12 353 29 97.65 78.83

SQL 349 25 98.14 92.33 415 32 98.89 75.66

Telnet 328 31 98.85 94.66 456 17 100 83.25

DoS 526 41 100 93.12 786 23 99.24 85.34

Probe 431 37 100 99.12 567 39 99.54 81.34

3. False Alarm Reduction in Signature based IDS: Game Theory Approach

The vulnerabilities in the Threat profile of the testbed network were categorized into 10 different vulnerability sets, with each set containing one or more vulnerabilities. The criticality weights of the vulnerability sets were set between 0.1 to 1. Severe vulnerabilities were assigned to higher criticality weight vulnerability sets. The cost of attacking (Ca) and monitoring (Cm) the network vulnerabilities were both set to 0.002. The false alarm cost was set to 0.003. The detection rate and the false alarm rate of the vulnerability scanners on the subset of the testbed network’s host operating systems were found to be 0.98 and 0.04, respectively. The generated testbed dataset consist of 297 instances of 30 different types of attacks along with normal data traffic collected over a period of 5 hours. Following categories of attacks were considered in the testbed network setup:

• Denial of Service (DoS): Teardrop, Land, Smurf, Ping of death, Win-nuke, Syndrop, Back, Mailbomb, Udpstorm, Arppoison, Crashiis, SYN Flood, tcpreset, selfping, ICMP Flood.

• FTP & SQL: Finger redirect, FTP server overflow, FTP format string, Freeftpd, user- name overflow, SQL server overflow, SQL injection.

• Telnet & Probe: Telnet buffer overflow, Telnet Resolve host conf, Ipsweep, Nmap, Mscan, Reset scan.

Table3.8shows the accuracy and detection rate of the proposed false alarm minimization scheme against different type of attacks on both the critical and non-critical vulnerabilities of the testbed network. It can be observed from the table that the proposed framework achieves highaccuracyacross all categories of attacks for both critical and non-critical vul- nerabilities. This implies that most of the false alarms generated by the Snort were filtered out by the correlation engine of the proposed scheme. The detection rate of the proposed false alarm minimization scheme is relatively high for attacks against critical vulnerabilities, whereas its detection rate for attacks against non-critical vulnerabilities is comparatively low. However, the low detection rate against non-critical vulnerabilities is acceptable as the attacker is very unlikely to attack them due to their low asset values.

We compare the performance of the proposed false alarm minimization scheme with var- ious other frameworks to validate its effectiveness. Table3.9and Table3.10show the per- formance comparison of the proposed false alarm minimization scheme with that of alarm verification based [97], alarm classification based [99], data summarization based [98] and hybrid [63] frameworks on the IDEVAL dataset and the testbed dataset, respectively. The

3.4. Performance Analysis

Table 3.9: Comparison of proposed framework with other false alarm minimization frameworks on the IDEVAL dataset

Alarm verification [97]

Data summarization [98]

Alarm classification [99]

Hybrid [63]

Proposed Scheme

Accuracy(%) 95.57 94.72 95.53 97.91 98.83

Detection Rate(%) 67.51 71.29 66.87 69.23 68.28

Table 3.10: Comparison of proposed framework with other false alarm minimization frameworks on the IITG Lab. dataset

Alarm verification [97]

Data summarization [98]

Alarm classification [99]

Hybrid [63]

Proposed Scheme

Accuracy(%) 97.39 96.17 94.47 95.29 98.55

Detection Rate(%) 89.73 90.78 90.29 90.91 91.87

reason for choosing these frameworks for comparison with the proposed false alarm min- imization framework is because of the similarity of the dataset (DARPA’s IDEVAL dataset) used in these frameworks for their evaluations. Moreover, to the best of our knowledge, there are no other game theory based false alarm minimization frameworks proposed in the literature for signature based IDSs.

It can be observed from Table3.9that the proposed framework has the highest accuracy among all the frameworks on the IDEVAL dataset. However, its detection rate is less than that of frameworks proposed in [98] and [63]. The low detection rate of the proposed framework on the IDEVAL dataset is primarily due to the inability of its signature based IDS (Snort) to detect the attacks in the IDEVAL dataset in the first place and does not necessarily imply poor performance of the proposed framework. The low detection rate of Snort on the IDEVAL dataset can be attributed to the fact that most of the attacks in the IDEVAL dataset are obsolete and Snort no longer contains signatures to detect these attacks.

Similarly, it can be observed from the Table 3.10 that the proposed framework has the least false alarm rate (highest accuracy) amongst all the frameworks on the testbed dataset.

The proposed framework is able to achieve this high accuracy since it uses various network context information parameters like alarm reference numbers, IP addresses, protocol types, port numbers, severity levels of vulnerabilities corresponding to the IDS alarms, OS types etc., along with a game theory-based monitoring strategy to filter out most of the false positive alarms generated by the signature based IDS (Snort). Since all the schemes (except [98]) use a common signature based IDS (Snort), their detection rates are comparable to each other. All the schemes have relatively high detection rate on the testbed dataset

3. False Alarm Reduction in Signature based IDS: Game Theory Approach

since their signature based IDS contains most of the attack signatures to detect the attacks on testbed dataset. However, the proposed framework has a better accuracy compared to other frameworks since it correctly identifies most of the TP alarms, while other frameworks incorrectly classifies some of the normal data traffic as attacks.