2015 Theses Doctoral

Empirical Measurement of Defense in Depth

Boggs, Nathaniel

Measurement is a vital tool for organizations attempting to increase, evaluate, or simply maintain their overall security posture over time. Organizations rely on defense in depth, which is a layering of multiple defenses, in order to strengthen overall security. Measuring organizations' total security requires evaluating individual security controls such as firewalls, antivirus, or intrusion detection systems alone as well as their joint effectiveness when deployed together in defense in depth. Currently, organizations must rely on best practices rooted in ad hoc expert opinion, reports on individual product performance, and marketing hype to make their choices. When attempting to measure the total security provided by a defense in depth architecture, dependencies between security controls compound the already difficult task of measuring a single security control accurately.
We take two complementary approaches to address this challenge of measuring the total security provided by defense in depth deployments. In our first approach, we use direct measurement where for some set of attacks, we compute a total detection rate for a set of security controls deployed in defense in depth. In order to compare security controls operating on different types of data, we link together all data generated from each particular attack and track the specific attacks detected by each security control. We implement our approach for both the drive-by download and web application attack vectors across four separate layers each. We created an extensible automated framework for web application data generation using public sources of English text.
For our second approach, we measure the total adversary cost that is the total effort, resources, and time required to evade security controls deployed in defense in depth. Dependencies between security controls prevent us from simply summing the adversary cost to evade individual security controls in order to compute a total adversary cost. We create a methodology that accounts for these dependencies especially focusing on multiplicative relationships where the adversary cost of evading two security controls together is more than the sum of the adversary costs to evade each individually. Using the insight gained into the multiplicative dependency, we design a method for creating sets of multiplicative security controls. Additionally, we create a prototype to demonstrate our methodology for empirically measuring total adversary cost using attack tree visualizations and a database design capable of representing dependent relationships between security controls.