- 1 -
中国科技论文在线
QoE Evaluation for Adaptive Streaming Based on
Psychological Recency Effect
LIU Qianhong, LIU Yitong, YANG Dacheng**
(Wireless Theory and Technology lab, Beijing University of Posts and Telecommunications, 5
Beijing, 100876)
Brief author introduction:Qianhong Liu(1990-), Male, Main research: QoE research on multimedia service
Abstract: A new method based on the recency effect, a psychological phenomenon that the recent
information is more prominent in short-term memory, is put forward for the QoE evaluation of adaptive
streaming, which takes bitrate adaption and users’ experiencing habits into account. Different from the
traditional solutions, the new method uses the divide-and-conquer strategy to simplify the QoE 10
evaluation of adaptive streaming into calculating scores of CBR (Constant Bit Rate) segments and
integrating these scores to give the final QoE evaluation based on the recency effect. A subjective test
has been conducted to confirm the great influence of the recency effect on the integral QoE evaluation,
and then a mathematical model is developed to measure its influence. Based on the model, a solution is
proposed to integrate the scores of CBR segments into the QoE of adaptive streaming service. As a 15
result, the Pearson correlation coefficient between the scores of the new method and the subjective
MOS reaches , and the new method is verified to be sensitive to bitrate adaption.
Key words: Comunication and Information system, the Recency effect, DASH, QoE, subjective test
0 Introduction
With the development of communication technology and the popularization of mobile 20
platforms, there is an explosive demand for high-quality video service. In a report by Cisco [1],the
real-time video takes 50% of the Internet traffic at peak periods, and it is predicted that by 2015,
various forms of video will exceed 90 percent of global consumer traffic, and almost 66 percent of
the world’s mobile traffic will be video. As a result, video communication over mobile networks
brings great challenges due to limitations in bandwidth and difficulties in maintaining high 25
reliability, quality, and latency demands. At present, one of the key solutions is adaptive streaming
which is an increasingly promising method to deliver video to end users allowing enhancements in
QoE (Quality of Experience) and network bandwidth efficiency.
There exists various adaptive streaming solutions and the industry is undergoing
standardizing solutions referred to as DASH [2] (Dynamic Adaptive Streaming over HTTP). DASH 30
presets new challenges and opportunities for content developers, service providers, network
operators and device manufactures. One of these important challenges is to develop evaluation
methodologies and performance metrics to accurately assess user QoE for adaptive streaming
services, and effectively utilizing these metrics for service provisioning and optimizing network
adaptation. 35
Currently, the research about this aspect is rather limited. The mainstream is to extract
metrics (KPIs, KQIs, .) from network and image parameters, and integrate them into a score.
For example, in [3], packet size, transmission time and bit rate are considered to measure the
representation level. Another effective method is the application of neural network [4]. However,
these existing methods seem to ignore the influence of bitrate distribution caused by rate adaption, 40
the most direct and the most obvious characteristic of adaptive streaming.
Perhaps, a new approach can be achieved from users’ viewing behavior. When viewing an
adaptive streaming service, viewers’ feelings fluctuate with its time-varying quality in real-time
- 2 -
中国科技论文在线
playback, and these mutative feelings then turn out the integral QoE scores finally. This presents 45
the following two significant phenomena, first of which is that viewers make an appraisal for the
quality (definition, fluency, etc.) of current part, and the second is that the integral evaluation is
the result of the partial appraisals after accumulation and aggregation. These phenomena are
caused by the structural characteristic of adaptive streaming, which is composed with loading part,
CBR (Constant Bit Rate) segments, buffering, skipping in the time domain. Generally speaking, 50
CBR segment of one certain bitrate possesses comparatively stable definition and fluency, which
results in comparatively fixed viewing feelings. In terms with buffering and skipping, their length
and frequency are the very factors that affect experiencing, but the latest adaptive streaming
technology, like DASH, can restrain them well. Then the problem is how to simulate the process
of partial appraisals’ accumulation and aggregation (we call the process ‘integrating’). Here the 55
related psychological effects take importantly guiding role. There has been much successful
research about QoE about CBR streaming [5], so our studies focus on the descriptive model for the
recency effect to be used and the divide-and-conquer integrating process.
In this paper, research focuses on the influence of the recency effect, one of the most
important psychological effects that are closely related to subjective experience. Part II illustrates 60
our study about the recency effect in detail. A crowd test is conducted to verify the influence of
the recency effect, and a step further is establishing a scientific description for the recency effect.
In part III, the recency effect is applied to achieve QoE scores with the help of divide-and-
conquer strategy. And here compares the result of new model with that of the traditional method.
It is convincing that new model performs much better. 65
1 Descriptive model for the recency effect
The serial position effect
The serial position effect [6], a term coined by Hermann Ebbinghaus [7]. When asked to recall a
list of items in any order (free recall), people tend to begin recall with the end of the list, recalling
those items best (the recency effect), and among earlier list items, the first few items are recalled 70
more frequently than the middle items (the primacy effect
Fig. 1 The serial position effect
Primacy Intermediate Recency
0
1
Timeline
Re
ca
lli
ng
Q
ua
lit
y
- 3 -
中国科技论文在线
Primacy Effect
The primacy effect is a cognitive bias that the first items presented in a series to be 75
remembered better or more easily, or for them to be more influential than those presented later in
the series.
Recency Effect
The recency effect is the phenomenon that the most recently presented items or experiences
will most likely be recalled best. The longer the inter-presentation interval is, the clearer the 80
recency effect is. The reason is that the information presented earlier blurs gradually, so the recent
information seems prominent in memory.
Both effects are significant in perceptual evaluation of multimedia service, for the importance
of each part seems to be different. However, the recency effect dominates during experiencing
long video service or content-based video service (content is much more important than video 85
quality). Therefore, the research focused on the influence of the recency effect.
Experimental Verification for the Recency Effect
An ingenious control test has been conducted to collect data related to the recency effect.
The latest adaptive streaming solutions (DASH .) partition a multimedia file into segments
and deliver to a client using HTTP. Each segment contains different encoded versions of different 90
bitrates, and the media presentation description (MPD) describes segment information (timing,
URL, media characteristics such as video resolution and bitrates). When a user request DASH
service, the client gets MPD at first, and then requests segments of proper bitrate successively
according to the current network condition and hardware capacity. In the client, the received
streaming is composed seamlessly by adaptive bitrates. 95
According to the structural characteristics of DASH, the testing samples were set with
different bitrate distributions. The last several segments had rising, falling and convex bitrate
distributions as required.
The parameters of some testing samples and results are as follows:
Tab. 1 Bitrate distribution of testing samples 100
Bitrate Distribution NO.
Seg 1-5 Seg 6 Seg 7 Seg 8
Subjective
MOS
TS 1 R2 R1
TS 2 R2 R6
TS 3 R2 R1 R1
TS 4 R2 R1 R6
TS 5 R2 R6 R1 3
TS 6 R2 R1 R1 R6
TS 7 R2 R1 R6 R1
TS 8 R2 R6 R1 R1
R1:256kbps R2:512kbps R6:1538kbps
Segment duration: 3s Total duration: 24s
The same beginning segments to wipe out the disruption of the primacy effect.
In order to remove the influence of the primacy effect, the segment 1~5 are constantly
512kbps. According to the subjective test, with the average bitrate, video samples of rising bitrate
distribution (TS 4 and TS 6) get the highest scores, and samples with falling bitrate distribution
- 4 -
中国科技论文在线
(TS 5 and TS 8) get the lowest scores. It reveals that the high-quality segment has a greater effect 105
in the recent position than in early position. It is convincing that the recency effect plays an
important role in QoE evaluation of adaptive streaming.
New Model for the Recency Effect
According to the previous section, it’s known that the recency effect actually exists in
experiencing adaptive streaming. The recency effect simulates the ‘forgetting law’ of Hebbian 110
synapses. In order to measure the ‘importance’ (weight coefficient) of each segment under the
influence of recency effect, a monotonically increasing function is rational and necessary.
We propose a recency function with good ascending and convergence:
2 2
1( ) , [0, ]
1 ( )recency
f t t T
T t
Note: stands for the strength of the recency effect. As shown in , the choosen 115
recency function has excellent derivative function, which corresponds with the characteristics of
the recency effect.
Fig. 2 The Recency Function
In the case of no rebuffing, seg(i) lasts from ti-1 to ti. the duration is ti – ti-1. 120
Then, the “importance” of seg(i) is calculated as follows:
1
( )i
i
t
i recencyt
f f t dt
2 2( 1)
1 , [1, ]
1 ( )
i T
i T
dt t N
T t
1 arctan[ ( 1)] arctan[ ( )]T N i T N i
2
1 arctan
1 ( ) ( 1)( )
T
T N i N i
125
0 ti-1 ti T
0
1
Time
Re
ce
nc
y
In
te
ns
ity
seg(i)
- 5 -
中国科技论文在线
Here an important metric is defined as the strength of the recency effect after discretization.
T
Then,
2arctan 1 ( 1)( )i
Tf
N i N i
(4)
After normalization, we can the weight for each segment. 130
2
1
arctan[ ]
1 ( 1)( ) , [1, ]
arctan( )
i
i N
i
f N i N iw i N
Nf
(5)
Definition for recency factor
Both of and are used to describe the strength of recency effect. The difference lies in that
is applied when the adaptive streaming is handled as discrete segment serial sequence, and is
applied in temporal continuous scenes. 135
In fact, and are the same in terms of usage, and they can be transformed into each
according to formula (3). Then the problem is finding a solution to determine their values. Both
metrics are closely related to several factors, such as the type of video content, the pattern of
services, the duration of services and etc. In this paper, an intelligent and effective method is
offered to estimate . 140
Imitating the definition of 3dB bandwidth, we can define that the last m segments
(seg(N-m+1)~seg(N)) weight over a percentage of .
(5)
1
arctan( )
arctan( )
N
formula
i
i N m
mw
N
(6)
A fact can be useful that when the service is long enough, scilicet, N is big enough, the value
of arctan(Nλ) approaches to л/2. 145
lim arctan( )
2N
N
Then,
1 tan( )
2m
Here we see that the value of depends on the number of chosen segments m and required
percentage . Additionally, psychological researches show that, 7±2 units are the capacity of 150
short-term memory [8]. In this way, the problem has been simplified. According to video’s content
and length, we decide the weight percentage of the last 7 segments, and then estimate required
recency effect strength.
As shown in , the different weight percentages lead to the different recency factors,
- 6 -
中国科技论文在线
which correspond to various scenarios of adaptive streaming. Generally speaking, those services 155
whose contents are more important than quality (like news, sports matches) tend to highlight the
quality of the latest playback, so they should get bigger recency factor. On the contrary, movie
streaming and documentary streaming pay more attention to their overall quality, and they need
more gentle recency curve.
160
Fig. 3 Weight percentage and recency factor
2 Intergrating into service QoE scores
The new method is based on the divide-and-conquer strategy. Since the involvement of
bitrate adaption, skipping and buffering, it is a tough issue to establish a direct model to achieve
service QoE scores. However, many previous researches have focused on the QoE evaluation of 165
CBR streaming, and have got fruitful achievements. If we realize this fact, and that the adaptive
streaming is composed with CBR segments of several different bitrates, there might be a shortcut
to solve the problem. Therefore, the new method includes three parts at least:
Evaluation of CBR segments
In each CBR segment, the rate and other parameters stay stable in a short period of time (3s, 170
in DASH), it is comparatively easy to draw objective evaluation scores with several effective
methods, such as VQM [9], VMOS [5], SSIM [10] and so on. Here it is unnecessary to put forward a
new method.
Evaluation of buffering and skipping
In the traditional adaptive streaming, there exists buffering and skipping affecting the 175
playback experience. A model can be established to measure the influence of buffering and
skipping with their frequency and duration. However, the current excellent adaptive algorithms
(DASH, .) can restrain buffering and skipping, so here we pay little attention to this issue.
Integrating into the QoE of adaptive streaming service
According to the previous discussion, users’ partial feelings of each part turn out the final 180
evaluation after accumulation and aggregation, which is a complicated process involved with
0 5 10 15 20 25 30
0
Serial Number
W
ei
gh
t
alpha=50%,lamda=
alpha=60%,lamda=
alpha=70%,lamda=
alpha=80%,lamda= 7 segments
- 7 -
中国科技论文在线
multiple factors. However, the serial position effects are known to be one of the most significant
roles. The influence of the primacy effect weakens over time, but on the contrary, the influence of
the recency effect strengthens.
Once the evaluation scores of each CBR segments are obtained, they can be integrated into 185
service QoE scores based on users’ experiencing habits and psychological effect.
In the previous section, a weight model of the recency effect has been built. This model describes
the importance weight of each segment in the overall experience.
We use the vector W
to store weight of segments, and the vector P
to save partial scores
of segments. When N segments have been played, both vectors can be updated as follows: 190
1 2 3[ , , , , ]NW w w w w
1 2 3[ , , , , ]NP p p p p
Then, the integral QoE score of the played N segments is calculated:
1
( )
NT
i i
i
S N W P w p
� � (11)
As the adaptive streaming plays, N increases gradually, and we get the real-time service QoE 195
evaluation dynamically. When the service ends, the S(N) turns out to be the final QoE score.
Now, we expect to examine the performance of new method with the data of the subjective
test referred to in Part II. The configuration of testing samples is listed in Table II. The subjective
test is strictly conducted as the requirements in [11] and [12]. There are totally 29 testers joining in
the test, and 232 votes have been collected for the 8 adaptive streaming samples. We have also 200
collected the scores of each segment.
Here, we expect to put emphasis on the adaptive part of the testing samples, and due to the
short duration, we define in formula (8), m = 3, = 50%, so = is applied to calculate the
weight vectorW
.
Tab. 2 Bitrate Distribution of Testing Samples (2) 205
Bitrate Distribution NO.
Seg 1-5 Seg 6 Seg 7 Seg 8
Subjective
MOS
TS 1 R2 R1 256
TS 2 R2 R6 1538
TS 3 R2 R1 R1 256
TS 4 R2 R1 R6 897
TS 5 R2 R6 R1 897
TS 6 R2 R1 R1 R6 683
TS 7 R2 R1 R6 R1 683
TS 8 R2 R6 R1 R1 683
R1:256kbps R2:512kbps R6:1538kbps
Rising: TS 4, TS 6 Falling: TS 5, TS 8 Convex: TS 7
As a comparison, the average MOS scores are also got as the traditional way.
- 8 -
中国科技论文在线
Fig. 4 Performance comparison
As shown in , compared with the traditional averaging process that is usually used in 210
CBR streaming, the QoE scores after integrated under the recency effect are closer to the
subjective MOS scores.
The Pearson correlation coefficient of new method is , a distinct promotion against
of average MOS.
In terms of the sensitivity to bitrate adaption, we made following comparison: 215
Fig. 5 Comparison in sensitivity to bitrate adaption
From , the traditional averaging process has no sensitivity to the bitrate adaption, but the
integrating scores correspond with the subjective MOS very well, which reveals that the new
method has taken bitrate adaption into account. 220
3 Conclusion
In the QoE evaluation research of CBR streaming, it’s fruitful to draw objective QoE scores
from various parameters, but in the QoE evaluation of adaptive streaming, introducing
TS 4 TS 5 TS 6 TS 7 TS 8
1
2
3
4
5
Bitrate Distribution
M
O
S
Average MOS
Subjective MOS
Integrating MOS
TS 8 TS 3 TS 1 TS 7 TS 5 TS 6 TS 4 TS 2
1
2
3
4
5
Bitrate Distribution
M
O
S
Subjective MOS
Average MOS
Integrating MOS
- 9 -
中国科技论文在线
psychological effect into QoE evaluation can be a creative and effective method. This paper
discusses the practicability of applying serial position effect to QoE evaluation of adaptive 225
streaming, and verifies that the recency effect takes an important part by subjective test. Therefore,
a mathematical model is built to describe the recency effect. With the help of the model, the
divide-and-conquer method simplifies the tough QoE evaluation of adaptive streaming into two
easy parts: the evaluation of CBR segments and integrating into the service QoE. The results of
experimental testing show that the new method is sensitive to bitrate adaption, and the integrating 230
scores match the subjective MOS well.
References
[1] Cisco C V N I. Global Mobile Data Traffic Forecast Update, 2010-2015[J]. Cisco Visual Networking Index
(VNI) Forecast, 2011. 235
[2] Stockhammer T. Dynamic adaptive streaming over HTTP--standards and design principles[C]. Proceedings of
the second annual ACM conference on Multimedia systems. ACM, 2011. 133-144.
[3] Oyman O, Singh S. Quality of experience for HTTP adaptive streaming services[J]. Communications
Magazine, IEEE, 2012, 50(4). 20～27.
[4] Singh K D, Hadjadj-Aoul Y, Rubino G. Quality of Experience estimation for adaptive HTTP/TCP video 240
streaming using H. 264/AVC[C]. Consumer Communications and Networking Conference (CCNC), 2012 IEEE.
IEEE, 2012: 127～131.
[5] Shen Y, Liu Y, Qiao N, et al. QoE-based evaluation model on video streaming service quality[C]. Globecom
Workshops (GC Wkshps), 2012 IEEE. IEEE, 2012. 1314～1318.
[6] Murdoch JR B B. The serial position effect of free recall1[J]. Journal of experimental psychology, 1962, 64(5): 245
482-488.
[7] Ebbinghaus H. Memory: A contribution to experimental psychology[M]. Teachers college, Columbia
university, 1913.
[8] Miller G. The magical number seven, plus or minus two: Some limits on our capacity for processing
information[J]. The psychological review, 1956, 63: 81-97. 250
[9] Xiao F. DCT-based video quality evaluation[J]. Final Project for EE392J, 2000: 769.
[10] Wang Z, Lu L, Bovik A C. Video quality assessment based on structural distortion measurement[J]. Signal
processing: Image communication, 2004, 19(2): 121-132.
[11] BT I T U R R. 500-11. Methodology for the subjective assessment of the quality of television pictures[J].
International Telecommunication Union, Geneva, Switzerland, 2002: 53-56. 255
[12] ITU-T RECOMMENDATION P. Subjective video quality assessment methods for multimedia applications[J].
1999.
基于心理近因效应的自适应流媒体260
QoE 评估方法
刘前红，刘奕彤，杨大成
（北京邮电大学无线理论与技术实验室，北京市 100876）
摘要：本旣提剖乜种基于心理学近因效应盠自适应流媒体疄户体验质量（QoE）评估昕洱。
近因效应是指垄短期记忆争，越是最近盠信息越清晰。乪传统盠评估昕洱乩同盠是，该旣265
提剖盠新昕洱考虑到了码率变化和疄户体验习惯，利疄剢治盠思想，将自适应流媒体盠
QoE 评估先简化亖固定码率剢段盠评估，然后基于近因效应将其罘合起来。该硰竒进行了
乜项伖群亗观测试，测试旌据证明了近因效应垄自适应流媒体观看过程争盠存垄悃。之
后，本旣对近因效应进行旌学建模，并疄该模型把自适应流盠固定码率剢段评估得剢罘合亖
整体盠QoE得剢。经过验证，新盠评估昕洱将测试样本盠评估得剢乪亗观测试得剢 Pearson270
相关系旌提高到 ，并对码率变化有很好盠敏感悃。
关键词：通信与信息系统；近因效应；DASH；QoE；主观测试
中图分类号：
