Denial of Service (DoS) attacks pose serious threats to the Internet, exerting in tremendous impact on our daily lives that are heavily dependent on the good health of the Internet. This dissertation aims to achieve two objectives:1) to model new possibilities of the low rate DoS attacks; 2) to develop effective mitigation mechanisms to counter the threat from low rate DoS attacks.

A new stealthy DDoS attack model referred to as the "quiet" attack is proposed in this dissertation. The attack traffic consists of TCP traffic only. Widely used botnets in today's various attacks and newly introduced network feedback control are integral part of the quiet attack model. The quiet attack shows that short-lived TCP flows used as attack flows can be intentionally misused. This dissertation proposes another attack model referred to as the perfect storm which uses a combination of UDP and TCP. Better CAPTCHAs are highlighted as current defense against botnets to mitigate the quiet attack and the perfect storm.

A novel time domain technique is proposed that relies on the time difference between subsequent packets of each flow to detect periodicity of the low rate DoS attack flow. An attacker can easily use different IP address spoofing techniques or botnets to launch a low rate DoS attack and fool the detection system. To mitigate such a threat, this dissertation proposes a second detection algorithm that detects the sudden increase in the traffic load of all the expired flows within a short period. In a network rate DoS attacks, it is shown that the traffic load of all the expired flows is less than certain thresholds, which are derived from real Internet traffic analysis. A novel filtering scheme is proposed to drop the low rate DoS attack packets. The simulation results confirm attack mitigation by using proposed technique. Future research directions will be briefly discussed.