- 1 -
中国科技论文在线
An Architecture-Centric Approach for Security Technical
Debt Quantification
Yang Jun
1
, Rami Bahsoon
2
, Liu Jiqiang
1**
5
(1. School of Computer Science and Information Technology, Beijing Jiaotong University, Beijing
100044;
2. School of Computer Science, University of Birmingham, Birmingham, UK, B15 2TT)
Brief author introduction:Yang Jun (1990-), female, master graduate student, main research:information security
Correspondance author: Liu Jiqiang(1973-), male, professor,main research: trusted computing,applied
cryptography,security protocol. E-
Abstract: Security managers and architects often have trouble in making decision among various
security technologies when the budget is limited. At this time, a compromise between effective security 10
technologies and limited budget may hurt the system’s long-term health and introduce technology debt.
This paper presents an architecture-centric cost-benefit method for quantifying technology debt caused
by different security deployment solutions, in order to help security managers to manage technology
debts and make decisions. The proposed method is an extension of the Security Attribute Evaluation
Method (SAEM) to reason about the debt in secure architectures. A case study is adopted to exemplify 15
the process of method. The result indicates that the method provides an angle of technology debt
beyond the plain SAEM method to help security manager make choice and manage long-term benefit
and cost.
Key words: Information Security; Technology Debt; Security Architecture
20
0 Introduction
Security managers and architects often have to compromise effective security design
decisions and choices with limited budget. Besides, corporate managers frequently disagree about
architecture design plans which do not directly affect external and invisible qualities, yet they can
still have critical long-term impact on the system, if the risks are ignored. This is because when the 25
budget and resources are limited, decisions tend to be short-term geared [1]. For example, managers
may reject deploying a more expensive but efficient firewall (. circuit gateways firewall),
because they doubt the likelihoods of some attacks to manifest themselves in the future or the
ability of the firewall to withstand refined versions of these attacks. Consequently, the system may
suffer losses in the future because of ill-decisions. Such a situation, which sacrifices long-term 30
health of a system for short-term benefits (. deadline for delivering a release, limited budget,
etc.), may introduce a technical debt (TD) [2].
In commonly used decision making process, stakeholders prefer to choose an adjustment
decision that can satisfy functional requirements under tight resource and time constraints [3].
Specific to security area, since it is impossible to build an absolute security system and budget for 35
security is limited, security managers prefer to deploy most cost-effective security technologies to
reduce threats to an acceptable level [4]. During the decision making process, TD is seldom
considered because it does not cause immediate impact on functions, and it often invisible to the
decision makers [3].
We argue that design and architecture decision for security can benefit from TD analysis and 40
quantification. For sustainable development of a software or system, TD should be managed to
make it visible and under control [5], to help keep a balance between cost and value of the software
project. Making the likely TD visible to the decision makers, it can facilitate the decision making
process. Decision makers can combine TD with other criteria (such as profit and budget) to make
different kind of decisions, such as, whether to choose a decision which can carry positive TD, 45
that can be short term and has the potential to be cleared or to pay off in the long-term.
- 2 -
中国科技论文在线
Although TD can be incurred by compromising software Quality Attributes (QAs), research
effort has to a big extent looked at maintainability and has largely ignored other qualities [6]. Other
QAs, including security has received little attention. Among those limited literature which
discusses security, effort has been concerned with coding TD [7, 8]. There is a general lack of 50
research about how to calculate the TD caused by wrong security technology deployment
decisions.
The aim of this paper is to present an architecture-centric method for calculating and
comparing TD caused by different security deployment solutions in order to help security
architects to manage TD and make decisions which link technical decisions for security with TD 55
criterion. The method extends the Security Attribute Evaluation Method (SAEM) [4]. A case study
is used to evaluate this method. The analysis shows that the method is beneficial to stakeholders
of a system for understanding the different value of short-term and long-term benefits caused by
different security deployment solutions.
The rest of the paper is structured as follows. Section 1 provides the background about 60
technical debt and Security Attribute Evaluation Method (SAEM). Section 2 introduces the
method of quantifying TD to security attribute of architecture, while section 3 adopts a case study
to evaluate the method by describing how the method could be used to select the most suitable
technology set. Section 4 shows the sensitivity analysis on result given in the former section.
Section 5 discusses the advantages, drawbacks and future work of using this method. Section 6 65
covers the review of related works. Section 7 concludes the paper.
1 Background
We give a brief overview of technical debt and Security Attribute Evaluation Method (SAEM)
to pave the way for introducing our new qualitative method for TD in security architecture,
combining TD and SAEM. 70
Technical Debt (TD)
The TD metaphor was firstly introduced by Ward Cunningham in 1992 “for the trade-off
between writing clean code at higher cost and delayed delivery, and writing messy code cheap and
fast at the cost of higher maintenance effort once it is shipped” [9]. Tight resources and budget,
“accelerating the velocity” for new releases and informal development processes are often 75
acknowledged as a source for TD [10]. TD’s discussions have been initially concerned with code
quality. This metaphor has then broadened to cover the entire software development lifecycle [11],
including software architecture (SA), detailed design, coding, documentation, testing and so on. Li
et. al. classifies various types of technical debt covering requirements, architectural, design, code,
test, build, documentation, infrastructure, versioning and defect TD. As the first presented TD type, 80
coding TD is among the primary concerns for researchers [2].
TD has been also classified by intentionality [5]. If the cost of TD is visible and can be
controlled, introducing intentional debt can sometimes be beneficial for “unlocking potentials”,
which can translate to gaining new opportunities and generating business value [12]. Unintentional
TD is generally more problematic type of debt because it can be invisible [13]. If TD remains 85
invisible, TD can build up resulting in potentially significant losses [2]. It is unrealistic to develop a
perfect system without any flaws and debt [14]. Consequently, it is important to identify, track,
evaluate and mitigate the risks of the debt, keeping a balance between cost, short-term and
long-term benefits of the system.
One common-used TD calculation principle is the difference between the ideal solution and 90
- 3 -
中国科技论文在线
the solution that lags behind the ideal option [15]. This principle makes it possible to transform
architecture decision into economic debt. As same as the term “debt” in finance area, technical
debt is also consisted with two part: principle and interest [3]. Principle is defined as the cost
required completing the undone task in order to pay off the debt, while interest is the potential cost
that will be needed to pay in the future, if not completing the task at present [16]. TD is just a 95
metaphor but can reveal insights on the quality of a system and the risks it faces. It can also give
an intuitive view about the economic loss to the system.
Security Attribute Evaluation Method (SAEM)
SAEM is a cost-benefit approach for analyzing security design decisions related on a
quantitative risk and benefit assessment [4]. It can help information system stakeholders have an 100
overall acknowledge of prioritized threats of the system, and then use prioritized threats list and an
evaluation to different aspect of security technologies to select suitable risk mitigation strategies [4].
Participators of SAEM process should contain analysts that acknowledge of SAEM, the security
managers, architects, and other relevant stakeholders [4].
SAEM involves 4 steps [4]: 105
1) A security technology benefit assessment, to provide an effectiveness estimate to
candidate technologies.
2) An evaluation of the effect of security technologies in mitigating risks.
3) A coverage assessment, to choose most suitable technology according engineering design
principle. 110
4) A cost analysis to provide a reference comments for making decision. This step can be
done in parallel with step 3.
Before starting SAEM, a risk assignment is needed to identify a prioritized list of threats and
the consequence caused by these threats, to make security managers have an overall understanding
to the security conditions of the whole system. Butler’s paper uses a multi-attribute risk 115
assessment and results a threat index (TI) for each threats [4]. The multi-attribute risk assessment
process consists of 4 steps [4]:
1) Identify the outcome attributes.
2) Estimated the frequency Freq and outcome attribute values xj for each threat a. At this step,
the analyst also need to estimate the probabilities to the expected, high and low outcome events 120
occur (. Pexp ,Plow ,Phigh).
3) Assess weighting factors of each outcome attributes Wj according to the stakeholders
concerns.
4) Compute threat index TIa for each threat a by the function below:
125
TIa = Freqa * (Plow * ∑j=attributes Wj * Vj(xj low)) + (Pexp * ∑j=attributes Wj * Vj(xj exp))
+ (Phigh * ∑j=attributes Wj * Vj(xj high)) (1)
where Vj(xj) is the value function which normalize attribute value xj.
If the result does not represent the concern of the participators, original input data and 130
assumptions can be revised at any step.
A detailed description of SAEM process is shown in [4]. Especially, a detailed description of
the multi-attribute risk assessment process can be found in [17].
- 4 -
中国科技论文在线
2 An Architecture-centric Method of Technical Debt Quantification
to Security Attribute 135
Motivation
As earlier mentioned, this method is an extension of SAEM to quantify TD of security. The
motivation for proposing this method is based on the weakness of SAEM and the advantage of TD
as a decision-making criterion.
SAEM is based on multi-attribute analysis technique [18], which can summarize values with 140
dissimilar attribute value units (. hours, dollars，Likert scale, etc.) by normalizing them.
However, this multi-attribute risk assignment only gives a relative result of index to show the rank
and severity of threats with respect to each other, and cannot give the actual economic loss caused
by threats. Based on the relative result, SEAM cannot give an intuition view of benefit and cost by
taking a certain security solution, which may not be able to convince corporate managers to accept 145
security managers’ advice.
On the other hand, SAEM is a practical method to compare alternative security design [4], but
it is not consider TD as a criterion to make decision. Since TD has been received increasingly
concerned in the past few years, and there still no researches about security attribute of
architecture, the combination of SAEM and TD metaphor is an interesting research direction to 150
pursue.
Method Overview
According to the above motivation, we provide a method described as below:
1) Risk assignment. The objective of risk assignment is to identify threats and the
consequence of these threats, then to calculate loss of the system caused by each threat. The risk 155
assignment consists of 3 sub-steps.
a) To determine threats and outcome attributes. To a system, outcome attribute can vary,
including direct economic losses (. lost revenue, lost productivity), and indirect economic
losses (. reputation, regulatory penalties). To give an estimate of economic loss caused by
security threats in order to calculate TD in the latter step, all the outcomes should be converted to 160
price. Direct economic losses are easy to convert, while how to convert indirect economic losses is
beyond the scope of this paper.
b) To identify the frequencies and outcome attribute values of each threat. At this step, the
participators also need to provide the probabilities of the expected, low and high bounds for each
outcome attribute for each threat, to estimate how likely an expected, low or high outcome event 165
will happen.
c) To calculate loss. Calculating the optimistic/likely/pessimistic and expectation loss of
the system in given time period by following functions:
Optimistic/Likely/Pessimistic loss of a threat = Frequency* ∑ j=attributes loss j low/expected/high (2) 170
Expectation loss of a threat = plow * optimistic loss + pexpected * likely loss + phigh * pessimistic loss
(3)
Optimistic/Likely/Pessimistic/Expectation loss of the whole system =
Optimistic/Likely/Pessimistic/Expectation loss of a threat (4)
175
The result will provide a ranked list for the severity of threats with the loss they causes.
2) Technical debt quantification. This step aims to choose the optional countermeasures
solution and quantifying technology debt of other choices, in order to help making decision. The
- 5 -
中国科技论文在线
quantification step consists of 5 sub-steps.
a) Choose candidate technologies. Two principles help the analyst to choose the candidate 180
technologies in this step. At first, participators should identify technologies according to the
threats listing in the risk assignment step. Technologies which can mitigate severe threats are more
preferred than technologies which can only mitigate slight threats. Second, participators should
consider adopting technologies according to defense-in-depth principle [4]. Defense-in-depth
classified security technologies into 3 categories according to mechanisms: protection, detection 185
and recovery. If a system has already have a certain kind of technologies (. protection) to a
threat, the security managers may prefer a technology which belongs to another category (.
detection or recovery) rather than the same category. shows some common security
technologies and their classification.
Tab. 1 Technology categories (derived from [4]) 190
Protection Detection Recovery
Packet Filter Firewall (PF FW)
Application Firewall (AP FW)
Smart Cards
Authentication Policy Servers
Virtual Private Network (VPN)
Email Content Inspection
Vulnerability Assessment
Anti-virus Software
Hardened OS
Host-Based IDS
Network Monitors
Net-Based IDS
Auditing
Key Stroke Replicator
Log Analysis Apps
Auditing
Back-up and Recovery Tools
Load Balancing
Key Stroke Replicator
Forensic Software
The choice made in this step is a subjective and very coarse-grained filter. It aims to exclude
the very inappropriate technologies, to reduce the calculation in the later steps.
b) Give the effectiveness estimates of each technology. At this step, the analyst need to
interview security managers to give effectiveness estimates of each technology of each threat. 195
These best-guessing estimates are also according to the participators’ experience and history data.
At this step, participator still can choose to exclude some technologies according to effectiveness
to reduce the calculation.
c) Calculate the reduced loss of the system after using each alternative technology set.
Here the alternative technology sets are the combinations of candidate technologies. For example, 200
5 technologies can combine 32 alternative technology sets (empty set included), including sets
which have one technology, sets which have two technologies, and so on.
After defining technology set, then the loss of a threat after using a certain technology set, the
reduced loss of a threat and the reduced loss of the system can be calculated according functions
as below: 205
Loss of a threat after using a certain technology set = original loss * ∏ j=technology(1-effectivenessj)
(5)
Reduced loss of a threat = original loss - loss of a threat after using a certain technology set (6)
Reduced loss of the system = ∑ reduced loss of a threat (7) 210
d) Find the optimal solution. The definition of the optimal solution is the one who has the
highest net profit (. benefit - cost). The benefit is the reduced loss of the system, and the cost
contains one-time-paid cost (such as the price of buying the facility, cost of deploy, etc.) and the
maintenance cost of the technology (such as the salary for maintenance engineers, licence fee of 215
- 6 -
中国科技论文在线
the technique, etc.) in the given long-term time period (. 5 years).
e) Calculate technical debt of other decisions. In the context of this paper, principle is
defined as the one-time-paid cost of the optimal technology(s), including the price of buying the
facility, cost of deployment, and other one-time-paid fee; while interest is the sum of the reduced
loss after using optimal decision and the reduced payment for the maintenance fee of the optimal 220
decision (include salary for maintenance engineers, licence fee of the technique, etc.) in given time
period. It's worth noting that if the loss and payment for maintenance increase, these items can be
negative.
Data resource
The values of input parameters can be elicited from stakeholders taking into consideration 225
their level of confidence and expertise. This can be done through brainstorming sessions,
questionnaires, surveys, voting etc. Alternatively, similar projects can be examined where the
analyst can look at how these threats had manifested themselves overtime and how given
technologies were able to mitigate these threats. For the latter case, the analyst can provide a
rough estimate of the probability value by inspecting relevant artefacts needed for the analysis. 230
3 Application and Evaluation Using a Case Study
In this section, a case study of a medium system will be given as a means of showing how to
apply the proposed method into a system. The case study is adopted from the one provided by [4].
To simplify the description, we only consider the security technology deployment in this system as
the objectives that need to be maintained. 235
For this case study, the history data of threat frequencies and outcome value, and
effectiveness estimates of technologies are derived from [4]. We provide hypothetical estimates for
the cost of candidate technologies. The estimates are based on per year estimates or observations.
Risk Assignment.
After analyzing the history data of loss caused by threats and discussed with other 240
participates, the security analyst lists the threats the system faces, the attributes of the consequence
and the frequency and attribute values that each threat causes. For simply description, in the case
study, the attributes of loss only contains some lost revenue and lost productivity, both of which is
direct economic loss. Lost revenue can be reported by dollar loss directly, and lost productivity
can be convert from man-hour to dollar by multiple salary of one man-hour (according to [9], 245
$75/man-hour).
In addition, the analyst also provides the probabilities of the expected, low and high bounds
attribute value for each outcome attribute of each threat. Here the initial probabilities are used, .,
expected, low and high bounds are , and respectively.
All the data needed for risk assignment are shown in Tab. 2. These threats are the items that 250
may cause TD.
Tab. 2 Threat Frequencies and Outcome Values
Threats
(Estimated Attacks/year)
Lost Revenue/$ Loss productivity/$
Scanning
(10,220)
Low (p = ) 0
Exp (p = ) 0
High (p = ) 1000 75
Procedural violation Low (p = ) 0 0
- 7 -
中国科技论文在线
(4,380) Exp (p = ) 0 150
High (p = ) 12000 3000
Browsing
(2,920)
Low (p = ) 0 0
Exp (p = ) 0 0
High (p = ) 0 600
Distributed Denial of
Service (DDoS)
(156)
Low (p = ) 0 75
Exp (p = ) 0 225
High (p = ) 0 15000
Password Nabbing
(365)
Low (p = ) 0
Exp (p = ) 0
High (p = ) 0 75
Personal Abuse
(110)
Low (p = ) 0 0
Exp (p = ) 0
High (p = ) 0 300
Then the estimate of loss that caused by each threat will be calculated by (2) - (4), and the
results are shown in Tab. 3, including the estimation of overall loss, and expected, low and high 255
bounds of loss respectively. If the company only needs a rough estimate of the loss, then an
expectation loss is enough; otherwise, expected, low and high bounds of loss are needed.
Tab. 3 Loss of the System (Sum and Separate)
Threat Low (p = ) Expected (p = ) High (p = ) Expectation of loss/$
Scanning 191625 383250 10986500 470120
Procedural violation 0 657000 65700000 1241730
Browsing 0 0 1752000 17520
DDoS 11700 35100 2340000 55809
Password Nabbing 27375
Personal Abuse 0 33000
Total
From , we can see that the worst situation is the one that all security attacks happened in 260
one year result the severest damage, which total loss amounted to $. The best
situation, on contrary, is all attacks only causes the lowest damage, which total loss amount only
$. The most likely loss of the system is there are 10% low intensity attack, 89% of
expected intensity attack and 1% high intensity attack, which brings $ to the system.
A rank for all the threats according to the lost that every threat bring to the system is shown 265
in .
Tab. 4 Rank of Threat
Rank Threat Rank Threat
1 Procedural violation 4 Browsing
2 Scanning 5 Password Nabbing
3 DDoS 6 Personal Abuse
Technical Debt Quantification
It can be possible that during a long time period, type of threats in each time unit and loss
caused by security threats are different. For example, in different quarter of the year, the type and 270
frequency of threats vary. At this time, the loss of the system is the sum of loss in each time unit.
- 8 -
中国科技论文在线
1) Choose candidate technologies
According the rank of threats given in risk assignment and the existing coverage of security
technologies shown as Fig. 1, the security managers most prefer to choose a set of technology that
can defense procedural violation, scanning and DDoS, and for technologies that with similar 275
effectiveness, a technology with protection mechanism for DDoS and a technology with detection
mechanism for procedural violation, scanning is more needed.
Fig. 1 Existing Coverage (derived from [4])
shows the technologies that security manager chosen that can mitigate these threats [4]. 280
Tab. 5 Relevant Technologies
Threat Security technology
Procedural violation Auditing, authentication policy server, forensic software, host-IDS, log
analysis, 1 time password (1xPWD), Biometrics, Smart Card
Scanning Application firewall, vulnerability assessment scanners, packet filter
firewall, host-IDS, network-IDS, hardened OS
DDoS Hardened OS and network monitoring
Browsing Host-IDS, database encryption, authentication policy server, auditing, log
analysis
Password Nabbing 1 time password (1xPWD) , smart cards, hardened OS
Personal Abuse Authentication policy server, forensics, email filters, log analysis,
auditing, Biometrics
2) Give the effectiveness estimates and cost of each technology.
In this step, security manager will give the estimates effectiveness of each technology
according to his/her experience.
The name and the effectiveness of the technologies that the security manager gives are shown 285
in Tab 6. Note that if more than one technology should be combined to mitigate one risk, they
should also be listed as one item in Tab 6.
Note that a technology can have a lot of products. For example, there are a lot of anti-virus
products, and the effectiveness and cost varies. In this condition, different products should be
considered as two different technologies. For simplicity, this case study chooses one product for 290
each technology.
Tab. 6 Effectiveness Estimates
Technology Scanning Procedural
violation
Browsing DDoS Password
Nabbing
Personal
Abuse
- 9 -
中国科技论文在线
Hardened OS 66% 75%
AP FW 75% 5 111
Forensic
Software
100% 40%
Email Filters 100%
Host-IDS 33% 50% 50%
Network-IDS 33%
1xPWD 80% 95%
Biometrics 100% 100%
Smart Card 25% 95%
Log Analysis 40% 30% 40%
DB Encrypt 30%
To simplify the explanation of next step, assuming the security managers try to deploy one or
several technologies chosen from hardened OS, application firewall, host-IDS, biometrics and log 295
analysis.
3) Calculate the reduced loss of the system after using each alternative technology set
After giving the candidate technologies, the alternative technology set can be given out, as
shown in Tab 7.
Tab. 7 Alternative Technology Set 300
Technology Set
No.
Alternative Technology
Set
Technology Set
No.
Alternative Technology Set
0 Keep the status 16 Hardened OS + AP FW +Host-IDS
1 Hardened OS 17 Hardened OS + AP FW +Biometrics
2 AP FW 18 Hardened OS + AP FW +Log Analysis
3 Host-IDS 19 Hardened OS + Host-IDS
+Biometrics
4 Biometrics 20 Hardened OS + Host-IDS +Log
Analysis
5 Log Analysis 21 Hardened OS + Biometrics+ Log
Analysis
6 Hardened OS + AP FW 22 AP FW + Host-IDS +Biometrics
7 Hardened OS + Host-IDS 23 AP FW + Host-IDS +Log Analysis
8 Hardened OS +
Biometrics
24 AP FW + Biometrics+ Log Analysis
9 Hardened OS + Log
Analysis
25 Host-IDS +Biometrics+ Log Analysis
10 AP FW + Host-IDS 26 Hardened OS + AP FW +Host-IDS+
Biometrics
11 AP FW + Biometrics 27 Hardened OS + AP FW +Host-IDS+
Log Analysis
12 AP FW + Log Analysis 28 Hardened OS + AP FW + Biometrics+
Log Analysis
13 Host-IDS +Biometrics 29 Hardened OS + Host-IDS+
Biometrics+ Log Analysis
14 Host-IDS +Log Analysis 30 AP FW +Host-IDS+ Biometrics+ Log
- 10 -
中国科技论文在线
Analysis
15 Biometrics +Log Analysis 31 Hardened OS + AP FW +Host-IDS+
Biometrics+ Log Analysis
The reduced loss of the system can be calculated shown in , according to (5) - (7).
Tab. 8 Reduced Loss of the system (per year)
Technology
set No.
Scanning($) Procedural
Violation($)
Browsing($) DDoS($) Password
Nabbing($)
Personal
Abuse($)
Total($)
1 310279 0 0 41857 6912 0 359048
2 352590 0 0 0 0 0 352590
3 155140 620865 8760 0 0 0 784765
4 0 1241730 0 0 13824 0 1255554
5 0 496692 5256 0 0 631 502579
6 430160 0 0 41857 6912 0 478929
7 363027 620865 8760 41857 6912 0 1041421
8 310279 1241730 0 41857 13824 0 1607690
9 310279 496692 5256 41857 6912 631 861627
10 391375 620865 8760 0 0 0 1021000
11 352590 1241730 0 0 13824 0 1608144
12 352590 496692 5256 0 0 631 855169
13 155140 1241730 8760 0 13824 0 1419454
14 155140 869211 11388 0 0 631 1036370
15 0 1241730 5256 0 13824 631 1261442
16 443347 620865 8760 41857 6912 0 1121741
17 430160 1241730 0 41857 13824 0 1727571
18 430160 496692 5256 41857 6912 631 981508
19 363027 1241730 8760 41857 13824 0 1669198
20 363027 869211 11388 41857 6912 631 1293026
21 310279 1241730 5256 41857 631 13824 1613578
22 391375 1241730 8760 0 13824 0 1655689
23 391375 869211 11388 0 0 631 1272605
24 352590 1241730 5256 0 13824 631 1614032
25 155140 1241730 11388 0 13824 631 1422713
26 443347 1241730 8760 41857 13824 0 1749518
27 443347 869211 11388 41857 6912 631 1373346
28 430160 1241730 5256 41857 13824 631 1733458
29 363027 1241730 11388 41857 13824 631 1672457
30 391375 1241730 11388 0 13824 631 1658949
31 443347 1241730 11388 41857 13824 631 1752777
4) Find the optimal decision 305
As described in section 2, the optimal decision is the one who has the highest net profit (.
benefit - cost). The benefit is the reduced loss of the system, and the cost contains one-time-paid
cost and the maintenance cost of the technique in the given time period. To make description
simplified, assuming that one-time-paid cost contains price of the facilities and deploy, the
maintenance cost contains salary for maintenance engineers and license fee of the technology set. 310
- 11 -
中国科技论文在线
The cost of each alternative technology set is shown as .
Tab. 9 Cost of Alternative Technology Set
Technology
set No.
Price of
purchase($)
cost of
deploy($)
sum of
one-time-paid
cost($)
Salary for
maintenance
engineers($/year)
License fee
($/year)
sum of
maintenance
fee ($/ year)
1 60000 2000 62000 50000 1000 51000
2 50000 2000 52000 50000 1000 51000
3 20000 2000 22000 50000 1000 51000
4 12000 2000 14000 50000 1000 51000
5 15000 2000 17000 50000 1000 51000
6 110000 4000 114000 50000 2000 52000
7 80000 4000 84000 50000 2000 52000
8 72000 4000 76000 50000 2000 52000
9 75000 4000 79000 50000 2000 52000
10 70000 4000 74000 50000 2000 52000
11 62000 4000 66000 50000 2000 52000
12 65000 4000 69000 50000 2000 52000
13 32000 4000 36000 50000 2000 52000
14 35000 4000 39000 50000 2000 52000
15 27000 4000 31000 50000 2000 52000
16 130000 6000 136000 100000 3000 103000
17 122000 6000 128000 100000 3000 103000
18 125000 6000 131000 100000 3000 103000
19 92000 6000 98000 100000 3000 103000
20 95000 6000 101000 100000 3000 103000
21 87000 6000 93000 100000 3000 103000
22 82000 6000 88000 100000 3000 103000
23 79000 6000 85000 100000 3000 103000
24 77000 6000 83000 100000 3000 103000
25 47000 6000 53000 100000 3000 103000
26 142000 8000 150000 100000 4000 104000
27 145000 8000 153000 100000 4000 104000
28 137000 8000 145000 100000 4000 104000
29 107000 8000 115000 100000 4000 104000
30 86000 8000 94000 100000 4000 104000
31 157000 10000 167000 150000 5000 155000
After provided the cost of each alternative technology set, the analysts calculate the net profit
in a reasonable long term (. 5 years). The one which has the highest net benefit will be the 315
optimal decision in the corresponded year period. shows the optional solution in different
time period and their net profit. The optional solution in 1 year is No. 17 set which contains
Hardened OS, application firewall, biometrics facilities. But after 2 years, the optimal solution is
always No. 26, which adds a host-IDS than No. 17. This result shows No .26 set is the optimal
solution to this system in a long term. 320
- 12 -
中国科技论文在线
Best Decision in Different Time Period
Time period Optimal decision Net profit of optimal decision
1 year 17 Hardened OS + AP FW +Biometrics 1496571
2 years 26 Hardened OS + AP FW +Host-IDS+ Biometrics 3141036
3 years 26 Hardened OS + AP FW +Host-IDS+ Biometrics 4786553
4 years 26 Hardened OS + AP FW +Host-IDS+ Biometrics 6432071
5 years 26 Hardened OS + AP FW +Host-IDS+ Biometrics 8077589
shows the calculation result of 5-year net profit for each alternative technology set. 325
Sum of Cost, Reduced Loss of the System and Net Profit of Each Set in 5 Years
Technology set No. Sum of cost Reduced loss of the system Net profit
0 0 0 0
1 317000 1795241 1478241
2 307000 1762950 1455950
3 277000 3923823 3646823
4 269000 6277772 6008772
5 272000 2512896 2240896
6 374000 2394644 2020644
7 344000 5207103 4863103
8 336000 8038452 7702452
9 339000 4308137 3969137
10 334000 5105000 4771000
11 326000 8040722 7714722
12 329000 4275846 3946846
13 296000 7097270 6801270
14 299000 5181849 4882849
15 291000 6307208 6016208
16 651000 5608703 4957703
17 643000 8637855 7994855
18 646000 4907540 4261540
19 613000 8345989 7732989
20 616000 6465129 5849129
21 608000 8067888 7459888
22 603000 8278446 7675446
23 600000 6363026 5763026
24 598000 8070158 7472158
25 568000 7113566 6545566
26 670000 8747589 8077589
27 673000 6866729 6193729
28 665000 8667291 8002291
29 635000 8362285 7727285
30 614000 8294743 7680743
31 942000 8763885 7821885
At this step, sets as follows can be deleted:
• Net benefit <0
- 13 -
中国科技论文在线
• Cost of this set > cost of optimal set, but net profit of this set< net profit of optimal set 330
Because such sets will never be chosen even if the company give adequate budget to the
system, it is pointless to manage TD of such solution. Analysts can use these rules to reduce
workload, but it won’t affect the final result if not using it.
5) Calculate technical debt of the other decisions
As described in Section 2, the principle and interest of each alternative set can be calculated. 335
Principle is the sum of adding purchase deployment cost for changing to optimal solution, while
interest is the sum of reduced loss and maintenance fee if using optimal solution. Then the TD of
each set is the sum of principle and interest. After calculating TD, participators will be able to
make a decision combined with budget. Due to the limited space, we just list TD and budget of
each alternative technology set in 5 years, as shown in and sorted in descending order. 340
According to Tab. 12I participators can make a decision according TD and budget. If the
system pursues a long term benefit, for example 5 years. The best choice is 26. And the budget in
five years should not less than $670000. Otherwise it will cause a technical debt. If the budget is
only $219000, No. 29 will be chosen, but after 5 year, this decision will bring a debt of $
for not choosing the optimal decision. The company can either choose to take such an amount TD 345
and pay off in the future, or add budget to deploy the optimal security solution.
TD of Alterantive Technology Set in 5 Years
Technology
set No.
Purchase
cost
Added cost
for
deployment
Principle Reduced
loss
Reduced
maintenance
fee
Interest Technical
debt
Budget
26 0 0 0 0 0 0 0 670000
28 20000 2000 22000 -5000 665000
17 20000 2000 22000 -5000 643000
29 50000 2000 52000 -5000 635000
19 50000 2000 52000 401600 -5000 396600 448600 613000
30 60000 2000 62000 -5000 614000
22 60000 2000 62000 -5000 603000
8 70000 4000 74000 -260000 336000
11 80000 4000 84000 -260000 326000
21 70000 4000 74000 -10000 608000
24 80000 4000 84000 -10000 598000
13 110000 4000 114000 1650319 -260000 1390319 1504319 296000
25 110000 4000 114000 1634023 -10000 1624023 1738023 568000
20 72000 4000 76000 2282459 -10000 2272459 2348459 616000
23 72000 4000 76000 2384563 -10000 2374563 2450563 600000
15 130000 6000 136000 2440381 -260000 2180381 2316381 291000
4 130000 6000 136000 2469817 -265000 2204817 2340817 269000
16 12000 2000 14000 3138886 -5000 3133886 3147886 651000
7 62000 4000 66000 3540486 -260000 3280486 3346486 344000
14 122000 6000 128000 3565740 -265000 3300740 3428740 299000
10 72000 4000 76000 3642589 -260000 3382589 3458589 334000
18 32000 4000 36000 3840049 -10000 3830049 3866049 646000
9 82000 6000 88000 4439452 -265000 4174452 4262452 339000
12 92000 6000 98000 4471743 -265000 4206743 4304743 329000
3 122000 6000 128000 4823766 -265000 4558766 4686766 277000
- 14 -
中国科技论文在线
5 142000 8000 150000 6234693 -270000 5964693 6114693 272000
6 32000 4000 36000 6352945 -260000 6092945 6128945 374000
1 82000 6000 88000 6952348 -265000 6687348 6775348 317000
2 92000 6000 98000 6984639 -265000 6719639 6817639 307000
0 142000 8000 150000 8747589 -520000 8227589 8377589 0
Additionally, analysts can calculate TD of a certain technology set in different time period to
see the trend of TD. Fig. 2 shows the TD of 5 non-optimal technology sets in 5-year time period. 350
We observe that in this scenario, TD increases from year to year. The decision makers can
introduce TD by choosing a non-optimal solution at the beginning, and choose an appropriate time
to pay off them before accumulating too much to pay off.
Fig. 2. TD quantification comparison for a 5-year period of part technology sets 355
4 Sensitive Analysis
S Sensitivity analysis can be employed to track the effect of slightly changes in the value on
the overall estimates. The participators may be optimistic or pessimistic about their estimates, or
make error in estimating certain values. Sensitivity analysis provides a method allowing the
participators to assess the stability of their estimates. 360
For this case, sensitivity analysis is used to find technologies that tend to be stable and
continue to provide manageable debt despite slight changes in estimates. Since the choice for
optimal decision and other alternative decision is based on the rank of net benefit, the final choice
may not have any change and the calculation results of TD fluctuate in the tolerable range even the
estimates values change. 365
There are 4 estimated values in the whole process, which are attack frequency, outcome
attribute value, probability of an expected/high/low outcome events occurrence, and effectiveness
of technologies. We can apply sensitivity analysis by substituted estimate values to see if different
estimates would result in a different result. For example, the security manager gives the estimate
that the application firewall can reduce about 75% of the scanning attack at the beginning. But 370
he/she is not very confident about the fixed estimate and adds a range for this estimate, which is
more than 50% and less than 80% of scanning attack can be stopped. These values can be replaced
- 15 -
中国科技论文在线
for the initial effectiveness estimate value, to see whether the modified estimate values lead to a
different technology selection and a large change of technology debt results.
5 Discussion and Future Work 375
The SAEM-based method provides a structured cost-benefit analysis technique to quantify
and manage TD of different security technology deployment, in order to help security practitioners
evaluate the selection of security technologies in TD point of view. Since all the input is based on
best-guess of the system, it cannot provide a precise result. But a structured approach which can
provide a result of approximation of the benefit, cost and debt is still meaningful. 380
A comparison between plain SAEM and this method is shown in Tab. 13.
Comparison Between SAEM and this Method
SAEM This method
Multi-technology situation Not consider consider
Probability of over-engineering high low
Benefit representation Percentage Monetary
Provide TD dimension No Yes
Considered the example in [4] where the security managers tries to decide whether to invest
hardened OS, host-based IDS or log analysis software. When using SAEM, the final result is to 385
choose host-based IDS. The reason of making this decision is, host-based IDS and hardened OS
have approximately equal benefit to the system, which reduce % and % risks
respectively; however, host-based IDE provides greater defense-in-depth. After using method
proposed in this paper, the best option is to choose all the three technology, because the net profit
is the highest in long-term, which is $5849129 in five years if using hypothesis cost in TABLE IX. 390
After determining optimal choice, TD of other choice can be calculated. The final result need to
considering budget and TD to make a trade-off.
SAEM does not think about multi-technology investment, and directs participators’ attention
to benefit by assuming the cost of each technology is equal. Cost will determine the final decision
only if two technologies provide similar benefit and defense-in-depth. Considering a situation that 395
a technology t can reduce much more risk of the system than others (. 50%), but the cost also
very high. The result given by SEAM is choosing t. However, SAEM cannot provide a benefit
represented by currency, since it only have a percentage of reducing risk. Deploying t in the
system will cause over-engineering if the economic benefit is lower than the cost.
On the contrary, the method proposed in this paper considers all the combination of 400
technology. In the choice of best solution, this method focus on both benefit and cost by using net
profit, which reduce the possibility of over-engineering security technology in the system.
Meanwhile, the method use currency value to represent benefit and cost, which is easier to
convince corporate managers to accept security managers’ advice than SAEM. Moreover, this
method gives an extra dimension of cost and income to make decisions. Participators are given an 405
intuitive view to the link between decisions and contribution it brings to the system in the future,
by giving the technical debt to the system in long-term period if not choosing optimal decision. It
also gives the ability to see how their estimates affect the choice and future benefit of the system.
To the corporate managers, this method is also useful, which can provide an economic view about
how and why the participators make such a decision. 410
However, this method still has drawbacks, which need future work to solve them. The first
two of the drawbacks are derived from the character of SAEM.
- 16 -
中国科技论文在线
Since the result is based on security managers’ subjective estimate, the experience of security
managers is vital. Inexperienced security managers may have difficulty to give accurate estimate,
which may cause misjudgment. Sensitivity analysis can help to deal with this problem to some 415
extent, but better estimate method still need to be found in order to support inexperienced security
managers to provide estimate value as accurate as possible.
In addition, the effectiveness estimates of technology should be more comprehensive by
considering more situations. In this method, the effectiveness of a technology set is multiplying
effectiveness of each technology in this set. However, the effectiveness of a technology may 420
change if other technology present. For example, the effectiveness of hardened OS and application
firewall to scanning is 66% and 75% respectively, but when hardened OS exists, the effectiveness
of application firewall may not be 75%, thus the effectiveness of a technology set consisting with
hardened OS and application firewall may not be (60%*75%). Additional work needs to be done
to determine the benefit of technology sets which have more than one technology. 425
Finally is complexity of this method. We regard every combination of technologies as one
alternative technology set. But this method will result more and more alternative technology sets
to consider when the candidate technology increase. As describe above, 5 candidate technologies
can be combined to 32 alternative technology sets (empty set included). If there are n candidate
technologies, 2
n
alternative technology sets needed to be calculated. It is better to find another 430
method to deal with such a large amount of technology set. However, in practice a company will
not try to apply a very large amount technology at one time, the analysts can just consider part
technology sets whose cost not exceed budget too much. Furthermore, an automated tool can be
developed to mitigate problem bring by complexity, since the candidate technologies are no more
than dozens. It is tedious to deal with such a large amount input parameters manually, whether the 435
main process or sensitive analysis. The time to deal with the whole process automatically is
acceptable, and certainly less than doing the process manually.
It's worth noting that not deployment certain technology set may cause ethical problems. For
example, credit card information may be leak if the system not deploying certain security facilities.
Although it may not bring direct economic loss, it can causes large reputation loss to this company. 440
It is needed to consider about the threshold when using this method to compromises short and long
term benefit in security. Two ways can be used to deal with such problems. Participators can
follow the requirement of existing standards (. PCI-DSS [19]) to deploy prerequisite technologies
at first, then using this method to deploy optional technologies to strength the system. Or
participators can choose to give a large enough loss estimate to corresponding outcome attribute 445
(. reputation) of certain threats in risk assignment step, to ensure that vital technologies can be
chosen in latter steps.
6 Related Work
Most published TD quantification methods are about code TD. Some researchers believe TD
should be quantified with interest, while others regard TD as “principle”. 450
A Method called Software Quality Assessment based on Lifecycle Expectations (SQALE)
can be applied to estimate code TD [7]. This method tailored the remediation analysis functions to
constitute an estimation model of technical debt, which is the sum of all quality characters’
remediation indices. SQALE also gave an indicator of SQALE debt map on two dimensions,
which are technical debt perspective and business impact perspective, to prioritize remediation 455
actions [7]. One limitation of SQALE is not giving clear justification of remediation indices.
Moreover, SQALE did not provide method to calculate interest. Letouzey and Ilkiewicz [20]
- 17 -
中国科技论文在线
introduced examples to show how SQALE is applied to manage TD and give out remediation
priority actions.
Curtis et al. [8] presented a simple formula to calculate principle of code TD. The formula is a 460
function of the number of must-fix structural problems in code, the time required to fix each
problem, and the cost for fixing a problem. Curtis et al. proposed the average cost of $ per line
for TD by calculating TD-Principle of 745 applications individually, as well as TD-Principle
amount of different technology and language type [8]. Curtis et al. also gave the distribution of
TD-Principle after analyzing 745 applications, showing that 70% TD principle was associated 465
with changeability and transferability, while the remaining related to robustness, security and
performance efficiency [8]. However, Curtis et al. also did not give a method to measure interest of
TD.
Nugroho et al. [15] proposed an approach to quantify TD and interest of maintainability based
on quality rating assessment of software systems. The assessment used a quality model developed 470
at the Software Improvement Group (SIG) for software maintainability. In this article, TD is
defined as the cost to fix quality issues, which is like the definition of “principle” in other papers.
Interest is the additional cost to maintenance due to should-fix quality issues.
Nord et al. [21] gave an architecture-focused and measurement-based approach to manage
architecture rework technical debt. This approach is used to explicit rework tasks at each release at 475
the architecture level to optimize the cost of development [21].
7 Conclusion
The purpose of the proposed technical debt quantification to security attribute method is to
help security managers and other stakeholders to select security technologies which meet a
compromise between short-term and long-term benefit. The SAEM-based method provides a 480
structured cost-benefit analysis technique to quantify and manage TD of different security
technology deployment in order to help security practitioners evaluate the selection of security
technologies and make decision. This method firstly gives a rough risk assignment of the system
based on best-guess estimates of the present security condition (. attack frequencies and
outcomes, and probability of attack frequencies and outcomes), then calculates net profit and TD 485
of alternative solutions based on estimates of effectiveness of technologies. A case study is used to
exemplify the process of method. The result indicated that the method provides an angle of TD
beyond the plain SAEM method to help security manager make choice and manage long-term
benefit and cost.
References 490
[1] Brown N, Cai Y, Guo Y, et al. Managing technical debt in software-reliant systems[C].Proceedings of the
FSE/SDP workshop on Future of software engineering research. ACM, 2010: 47-52
[2] Li Z, Avgeriou P, Liang P. A systematic mapping study on technical debt and its management[J]. Journal of
Systems and Software, 2015, 101: 193-220.
[3] Seaman C, Guo Y, Izurieta C, et al. Using technical debt data in decision making: Potential decision 495
approaches[C]//Proceedings of the Third International Workshop on Managing Technical Debt. IEEE Press, 2012:
45-48.
[4] Butler S A. Security attribute evaluation method: a cost-benefit approach[C].Proceedings of the 24th
international conference on Software engineering. ACM, 2002: 232-240.
[5] Lim E, Taksande N, Seaman C. A balancing act: what software practitioners have to say about technical debt[J]. 500
Software, IEEE, 2012, 29(6): 22-27.
[6] ISO/IEC 25010:2011, Systems and software engineering -- Systems and software Quality Requirements and
Evaluation (SQuaRE) -- System and software quality models [S]. ISO, Geneva, 2011.
[7] Letouzey SQALE Method - Definition Document, Version [OL]. January 2012.
[Online; accessed 1 May 2016] 505
[8] Curtis B, Sappidi J, Szynkarski A. Estimating the size, cost, and types of Technical Debt[C].Proceedings of the
- 18 -
中国科技论文在线
Third International Workshop on Managing Technical Debt. IEEE Press, 2012: 49-53
[9] Cunningham W., The WyCash portfolio management system[C]. Proceedings of the 7th Object-Oriented
Programming Systems, Languages, and Applications (OOPSLA'92), ACM, Vancouver, British Columbia, Canada,
1992, pp. 29-30. 510
[10] Klinger T, Tarr P, Wagstrom P, et al. An enterprise perspective on technical debt[C].Proceedings of the 2nd
Workshop on managing technical debt. ACM, 2011: 35-38.
[11] Lia, Z., Liangb, P., & Avgerioua, P. Architecture viewpoints for documenting architectural technical debt. in
press.
[12] Allman E. Managing technical debt[J]. Communications of the ACM, 2012, 55(5): 50-55. 515
[13] Klinger T, Tarr P, Wagstrom P, et al. An enterprise perspective on technical debt[C].Proceedings of the 2nd
Workshop on managing technical debt. ACM, 2011: 35-38.
[14] J. Higgs, The Four Grades of Technical Debt[OL], 2011;
[Online; accessed May 1,2016]
[15] Nugroho A, Visser J, Kuipers T. An empirical model of technical debt and interest[C].Proceedings of the 2nd 520
Workshop on Managing Technical Debt. ACM, 2011: 1-8.
[16] Seaman C, Guo Y. Measuring and monitoring technical debt[J]. Advances in Computers, 2011, 82: 25-46
[17] Butler S A, Fischbeck P. Multi-attribute risk assessment[C].Symposium on Requirements Engineering for
Information Security. 2002.
[18] Yoon K P, Hwang C L. Multiple attribute decision making: an introduction[M]. Sage publications, 1995. 525
[19] PCI Security Standards Council. Payment Card Industry Data Security Standard (PCI DSS) [S]. PCI
Security Standards Council, 2015.
[20] Letouzey J L, Ilkiewicz M. Managing technical debt with the sqale method[J]. IEEE software, 2012 (6):
44-51.
[21] Nord R L, Ozkaya I, Kruchten P, et al. In search of a metric for managing architectural technical debt[C]. 530
Software Architecture (WICSA) and European Conference on Software Architecture (ECSA), 2012 Joint Working
IEEE/IFIP Conference on. IEEE, 2012: 91-100.
一种以体系结构为中心的安全技术债535
务量化方法
杨珺1，Rami Bahsoon2，刘吉强1
（1. 北京交通大学计算机与信息技术学院，北京 100044；
2. 计算机学院，伯明翰大学，英国伯明翰，B15 2TT）
摘要：当预算有限时，如何选择合适的安全技术来保护系统安全，是安全管理员与架构师经540
常面临的问题。此时，因预算的限制而选用欠佳的安全技术从长远看可能会危害系统健
康，并会产生技术债务。本文将现有的一个成本-收益方法——安全属性评价方法（Security
Attribute Evaluation Method, SAEM）进行扩展，结合技术债务的概念提出了一套成本-收益
方法，用来评估不同的安全解决方案所造成的技术债务，以协助安全管理员和公司决策者
根据需求做出恰当的信息安全决策，并对技术债务进行管理。针对某企业的案例研究表545
明，相较于传统的 SAEM方法而言，本方法可以使得决策者从技术债务的角度进行决策以及
对长期收益和成本进行管理。
关键词：信息安全；技术债务；安全体系结构
中图分类号：TP309
550
