- 1 -
中国科技论文在线
A perception-motivated Interpolation algorithm for dynamic
image sequences#
WANG Qian, DU Junping**
(School of computer science, Beijing University of Posts and Telecommunications, Beijing 5
100876)
Foundations: National Basic Research Program of China (973 Program) 2012CB821200 (2012CB821206), the
National Natural Science Foundation of China (No. 91024001, No. 61070142), and the Beijing Natural Science
Foundation (No. 4111002)
Brief author introduction:Wang Qian(1989-),Male,Master,Artificial Intelligence
Correspondance author: DU Junping(1963-),Female,Professor,Artificial Intelligence.
Abstract: Image interpolation is a classical problem in image processing. In this paper, we propose a
perception-motivated interpolation algorithm for dynamic image sequences, which attempts to acquire
high quality interpolation results in accordance with human visual perception. This algorithm mainly
consists of two stages. First, the salient region detection method based on histogram contrast is used to 10
capture attention regions of images. Then a partition interpolation model is presented to improve the
interpolation quality of attention regions. Conducted experiments have shown that our algorithm can
spend less time to produce satisfactory image sequences.
Key words: image interpolation; salient region; iterative interpolation; image gradient;
0 Introduction 15
With the development of the computer network technology, image processing has become more and
more popular in people’s daily life. In face of this situation, how to improve the result of image
processing attracts tens of thousands of researchers. Image interpolation is a basic step for understanding
and analyzing image contents, and it plays a key role in the image processing. Better interpolation result
leads to better image processing. So this technique is widely used in aerospace, medical imaging, remote 20
sensing and other image processing. The conventional image interpolation algorithm can be roughly
divided into two categories: transformation and iteration [1]. The former interpolation algorithm is to
directly compute the values of the unknown pixels using the value of neighboring pixels, such as the
nearest neighbor, bicubic interpolation [2] and bilinear interpolation [3]. Transformation interpolation is
widely used in commercial software. The latter interpolation is to make full use of image properties to 25
iteratively calculate the values of interpolated pixels. These methods are often used in research projects.
The typical iteration interpolation algorithm is ICBI (Iterative Curvature Based Interpolation) [4], which
can achieve higher image quality but longer processing time. However, we find that the above
approaches have only focused on the nature characteristics of image sequences and paid little attention to
interpolation results from the view of human perception. In dynamic image sequences, people cannot 30
notice every detail of each frame. The moving area in each image is more attractive to human perception.
Base on this, dynamic image can be divided into different areas. Implement corresponding interpolation
in different areas can be more efficient. Taking human perception into consideration is therefore
beneficial to develop an algorithm to obtain high quality interpolation results of image sequences with
high computing speed. 35
In summary, the main contributions of this paper list as follows: (1) we propose a
perception-motivated interpolation algorithm based on visual saliency detecting technique for dynamic
image sequences;(2) we present the salient region detection method based on histogram contrast to
capture attention regions in accordance with human visual perception;(3) We demonstrate a partition
interpolation model combining ICBI with Bilinear Interpolation to enhance the quality and computing 40
efficiency of the interpolated frames.
The rest of the paper is structured as follows. Section 2 discusses the related work. Section 3
illustrates the proposed method. Section 4 presents experimental work to demonstrate the effectiveness
of our method. The last section concludes the paper.
1 Related work 45
Visual saliency technique has been used in many computer tasks recently, such as image
segmentation, image resizing and image representation. The latest saliency detection method called
- 2 -
中国科技论文在线
histogram-based contrast (HC) was proposed in 2011[5] and it can efficiently yield full resolution
saliency maps. The idea of HC is to compute the salient value of each pixel through the contrast among
pixels, shown as (1). 50
( ( )) ( ( ), ( ))
i U
X G k D G k G i
∈
=∑ (1)
In this formula, ( ( ), ( ))D G k G i represents the pixel difference between pixel k and pixel i in an
image G, which is calculated by Euclidean distance of RGB channels [6]. U represents the set of pixels in
G. Then a threshold can be set and the salient region can be determined on the basis of the threshold.
This technique can judge the important content of the image without any prior knowledge. So we adopt 55
it to obtain the appropriate attention region.
ICBI interpolation is an iterative interpolation algorithm and it was proposed by Giachetti et al. [4] in
method mainly contains two steps. The first step is to calculate the value of the pixels with
odd coordinates through the pixels in diagonal direction. And the second step is to complete the values
of the pixels with one odd coordinate. If the image is enlarged to higher multiples, we only iteratively do 60
the above stages. The advantage of this algorithm is energy calculations of pixels, as shown in (2).
1 2 3(2 1,2 1) (2 1,2 1) (2 1,2 1) (2 1,2 1)E i j uE i j vE i j wE i j+ + = + + + + + + + + (2)
Where i and j denote the coordinates in low-resolution image, and E1, E2, and E3 denote curvature
continuity term, curvature enhancement term and isolevel curves smoothing term respectively. A lot of 65
experiments have demonstrated the good performance of their algorithm and in this paper, we utilize it
to construct partition interpolation model.
2 Perception-motivated interpolation algorithm
We propose the perception-motivated interpolation algorithm (PMI) and the algorithm architecture is
shown in Figure 1. The main detailed implementation is divided into the salient region detection and 70
partition interpolation model construction. In the first stage, we use the salient region detection
approach to capture attention regions for dynamic image sequences. And in the second stage, we
establish the partition interpolation model to enhance the quality of obtained attention regions. Through
above stages, the high quality image sequences can be obtained.
1
( ) ( , )
L
g g j
j
Sa I C I I
=
= ∑ 1:argmax ( | )it
i
t t t
vs
vs p vs vs=
nM n
RM
n
BM
n
LM
n
TM
75
Fig. 1 The architecture of the proposed algorithm
Salient region detection
Let ( , )nf x y be the n-th frame of a dynamic image sequence and
nM be its attention regions.
According to the spatial and temporal characteristics, nM is decomposed as ( , )n s tM M M= , in which
sM and tM denotes the spatial attention region and temporal attention region respectively. 80
First, we use HC method to compute sM . Specifically, the saliency value ( )gSa I of pixel gI is
obtained through color distance function, shown as (3).
1
( ) ( , )
L
g g j
j
Sa I C I I
=
= ∑ (3)
Where ( , )j nI f x y∈ , j gI I≠ . L is the total number of pixels in ( , )nf x y , and ( , )g jC I I is the color
distance between gI and jI . Then, we adopt visual tracking method
[7] to obtain tM . In particular, the 85
obtained sM are divided into overlapped patches, putting into the target
set
1
{ , .. ., | ( , )}
i N i n
VS vs vs vs vs f x y= ∈ . N represents the total number of divided patches. These target patches
are tracked under Bayesian inference framework, as shown in (4).
- 3 -
中国科技论文在线
( )1:i
t
i
t t t
vs
vs arg max p vs vs= (4)
Where tvs denotes the patch state of n-frame, 1:( | )
i
t t
p vs vs represents posterior probability of itvs . 90
Finally, nM is the union of sM and tM , and four border values can be computed using (5).
1: 1:
1: 1:
( , ) ( , )
( , ) ( , )
min , max ,
min , max ,
n n n n
i i t i i t
n n n n
i i t i i t
n n n n
L i L R i Rx y vs x y vs
n n n n
T i T B i Bx y vs x y vs
M x M x
M y M y
ε ε
ε ε
∈ ∈
∈ ∈
= − = +
= − = + (5)
Where parameters { nLM , nRM , nTM , nBM } correspond to nM denote left border, right border, top
border and bottom border respectively and parameters { Lε , Rε , Tε , Bε } denote the predefined minimal
deviation values. 95
Partition interpolation model construction
In this section, the partition interpolation model is constructed to simulate the effects of visual
perception. We adopt the different methods for the different regions and then all the pixels from all
regions are combined to form high-resolution image sequences.
Curvature iterative interpolation in attention regions 100
We improve ICBI interpolation algorithm and propose a new curvature iterative interpolation
algorithm by taking the gradient information of pixel into consideration. By calculating the second
derivative in one of the diagonal direction, we can obtain the reference pixels for interpolation and use
curvature iterative interpolation in attention regions.
We define ( , )D x y as the enlarged image. Because low changing rate of second-order derivative 105
represents low variation trend of pixel value. 1 (2 1, 2 1)E i j+ + , the first energy term, equals to the sum
of the change of second-order derivative in local region which can ensure the interpolation results’
local curvature continuity. In order to ensure the efficiency of the algorithm, we used the derivation
forms to represent the second-order derivatives [8] in selected direction, as (6).
"
1
"
2
(2 1,2 1) (2 2,2 2) 2 (2 1,2 1) (2 2,2 2)
(2 1,2 1) (2 2,2 2) 2 (2 1,2 1) (2 2,2 2)
D i j D i j D i j D i j
D i j D i j D i j D i j
+ + = − − − + + + + +
+ + = − + − + + + + − (6) 110
We calculate the sum of local image region’s change rate of second-order derivative in different
directions with (7). Coefficient x represents the proportion of particular direction’s second-order
derivative’s change rate of all directions, which can ensure the similarity among adjacent pixels. Then
we calculate the gradient of local region in different directions to avoid the image too smooth and
guarantee the edge sharpness. So the second energy term is defined as (8). 115
{ } { }" " " "1 1 1 2 2( (2 ,2 ) (2 ,2 ) (2 ,2 ) (2 ,2 ) ) . . 1,1 , 1,1E x D i j D i j D i j D i j s tα β α β α β= − + + + − + + ∈ − ∈ − (7)
2 (2 ,2 ) (2 2,2 2) (2 ,2 2) (2 2, )E D i j D i j D i j D i j= − − + + − + − + (8)
From previous experiments, we found pixel’s change rate of brightness in local region is smaller than
overall region. So the third energy term is the sum of brightness change rate of local region as formula (9)
and ( ),O i j is the brightness value of a pixel. 120
2 2
3
2 2
(2 ,2 )(2 1,2 1) 1
(2 1,2 1)
O i jE i j
O i jα β
α β
=− =−
+ ++ + = −+ +∑ ∑ (9)
According to the requirement for edge sharpening and removing jagged edge, we choose the suitable
coefficient u, v, w to iterate formula (2) and obtain the optimal (2 1, 2 1)D i j+ + . In the second step of
ICBI, we calculated the second-order derivative value of horizontal and vertical direction, and set the
mean value of the pixel in the direction with the smaller result as the missing pixel value. 125
Second-order derivative bilinear interpolation in other regions
For other regions, we propose a second-order derivative bilinear interpolation. This method uses
second-order derivative to indicate the local pixel continuity and calculate the interpolated pixel by using
- 4 -
中国科技论文在线
the second-order derivative value in different directions as weights. Use ( , )R i j representing the other
region, and the region is enlarged by two times, the original pixels are in the even-numbered position, 130
that is (2 , 2 ) ( , )D i j R i j= .
For the pixels to be interpolated and the original pixels that are in the same row or column, we use
the spatial distance as weight value, and set the most nearest four original pixels’ weighted average value
to the missing pixel. For pixels (2 1, 2 1)D i j+ + , there is no original pixel on the horizontal or vertical
direction. We use the second-order derivative in diagonal direction as the weights, as (10). 135
" "
2 1
1 2" " " "
1 2 1 2
(2 1,2 1) (2 1,2 1) (2 1,2 1)D DD i j R i j R i j
D D D D
+ + = + + + + ++ + (10)
Where
1
(2 1, 2 1)R i j+ + and
2
(2 1, 2 1)R i j+ + are the weighted average value using the spatial distance
as weight in diagonal directions. In formula (10), the weight in selected direction is the ratio of the
second-order derivative in the other direction and the sum of second-order derivative in both directions.
For those regions in which the change rate of the two pixels in the two directions is similar to each other, 140
the missing pixel can be directly assigned with the mean value.
3 Experiment results and analysis
In our experiments, we used Matlab 2010(b) simulation tools. And we used a PC with Pentium(R) 4
CPU and memory as runtime environment. In order to verify the validity of the
proposed interpolation algorithm, we do some comparative experiments using two sequences of standard 145
images named plane and space. We generated the low resolution image sequences by down-sampling the
standard high resolution image sequences. The maximum number of iterations was set to 200 and
parameters {u, v, w} were {, , }.We divided each image into three color channels, and then
calculated its interpolation result separately.
Figure 2 and Figure 3 show the interpolation results using the different interpolation algorithms. 150
Figure 2(a) and Figure 3(a) are the standard high resolution frames, Figure 2 (b) and Figure 3(b) are the
bilinear interpolation results, Figure 2(c) and Figure 3 (c) are ICBI interpolation results, and Figure 2 (d)
and Figure 3(d) are the results of the proposed algorithm. It can be seen that the results of bilinear
interpolation have obvious jagged structure in detail. While the results of ICBI and ours have high image
smoothness and no obvious jagged structure. But our algorithm has more detail information. So the 155
proposed algorithm shows comparatively better performance in terms of visual quality.
�� �� �� �
(a) The standard frame (b) bilinear interpolation (c) ICBI interpolation
(d) Proposed algorithm
Figure 1. The interpolation results of space 160
�� �� �� �
(a) The standard frame (b) bilinear interpolation (c)
ICBI interpolation (d) Proposed algorithm
- 5 -
中国科技论文在线
Figure 2. The interpolation results of plane
The performance of the proposed algorithm can be measured using PSNR and MSSIM. Table 1 and 165
Table 2 show the comparison results of PSNR and MSSIM using three interpolation algorithms. It can
be found that the bilinear interpolation has the worst performance and lost information most seriously.
ICBI and the proposed algorithm can generally achieve the highest PSNR and MSSIM value. Table 3
shows consuming time using the above three algorithms, including total time consuming and average
time consuming. We can see that the obtained time using our method is lower than those by ICBI. 170
Because ICBI calculates a large number of useless pixels, while our method only focuses on calculating
the salient regions. In summary, our algorithm can spend less time to produce high quality image
sequences.
TAB. 1 COMPARISON OF PSNR AND MSSIM WHEN APPLIED TO PLANE
Bilinear ICBI Proposed Algorithm
PSNR MSSIM PSNR MSSIM PSNR MSSIM
Frame 1
Frame 2
Frame 3
Frame 4
Frame 5
Frame 6
Frame 7
Frame 8
Frame 9
Frame 10
TAB. 2 COMPARISON OF PSNR AND MSSIM WHEN APPLIED TO SPACE 175
Bilinear ICBI Proposed Algorithm
PSNR MSSIM PSNR MSSIM PSNR MSSIM
Frame 1
Frame 2
Frame 3
Frame 4
Frame 5
Frame 6
Frame 7
Frame 8
Frame 9
Frame 10
TAB. 1 COMPARISON OF TIME CONSUMING
Image
sequences
Total time
using Bilinear
Total time
using ICBI
Total time
using PMI
Average time
using
Bilinear
Average time
using ICBI
Average time
using PMI
Plane
Space
4 Conclusion
In this paper, we have presented a novel perception-motivated interpolation algorithm for dynamic
image sequences. The main feature of this algorithm is to obtain the satisfactory interpolation results
- 6 -
中国科技论文在线
according to spatial and temporal correlations in image sequences. The implementation process involves 180
two steps: salient region detection and partition interpolation model construction. The former captures
attention regions using the salient region detection approach, and the latter adopts curvature iterative
interpolation and second-order derivative bilinear interpolation respectively to acquire high quality
images. Experiments have shown that our method can yield encouraging performance not only in terms
of image visualization but also in terms of quantitative measures. 185
Acknowledgements
This work was supported by the National Basic Research Program of China (973 Program)
2012CB821200 (2012CB821206), the National Natural Science Foundation of China (no. 91024001, no.
61070142), and the Beijing Natural Science Foundation (no. 4111002).
190
References
[1] Chunjing Li, Duanduan Ma, and Hui Li. An Algorithm of Image Reconstruction Based on Rational
Interpolation with Rectangle-Hole Structure[J]. In 2012 Fourth International Conference, 2012, 4(1): 80-83.
[2] Widyantara , Wirawan W, and Hendrantoro G. Wyner-Ziv video coding with improved motion field 195
using bicubic interpolation[J]. In Instrumentation, Communications, Information Technology, and Biomedical
Engineering(ICICI-BME), 2011, 2(3): 218-222.
[3] Heechang Kim, Sangjun Par, Jin Wang, Yonghoon Kim, Jechang Jeong. Advanced Bilinear Image
Interpolation Based on Edge Features[J]. Advances in Multimedia, 2009, 7(4): 33-36.
[4] Giachetti A, and Asuni N. Real-Time Artifact-Free Image Upscaling[J]. IEEE Transactions on Image 200
Processing, 2011, 20(10):2760-2768.
[5] Lee YJ, Song JB, Choi JH. Performance Improvement of Iterative Closest Point-Based Outdoor SLAM by
Rotation Invariant Descriptors of Salient Regions[J]. JOURNAL OF INTELLIGENT & ROBOTIC SYSTEMS,
2013, 71(16): 3-4.
[6] Jeyapoovan, T, Murugan, M. Surface roughness classification using image processing[J]. MEASUREMENT, 205
2013, 46(7): 2065-2072.
[7] Jia X, Lu H, Yang. Visual tracking via adaptive structural local sparse appearance model[J]. CVPR, 2012, 6(1):
1822-1829.
[8] Zongying Li. The image encryption algorithm based on the generalized second-order directional derivative[J].
Intelligent Computing and Intelligent Systems(ICIS), 2010, 2(1): 550–553. 210
基于显著性区域的运动图像序列插值
算法
汪谦，杜军平 215
（北京邮电大学计算机学院, 北京 100876）
摘要：插值是图像处理中的经典问题。本文针对图像序列中有目标运动的插值问题，提出了
一种基于人眼视觉感知的插值算法。在该算法中我们将图像分为显著性区域和非显著性区
域，对显著性区域采用改进的二次曲率迭代插值算法，并结合二次曲率连续性和图像的局部
区域亮度变化率，保证图像的平滑和去锯齿状。对非显著性区域，引入二次导数连续性作为220
权值，采用改进的双线性插值求取插值结果，保证了算法的整体时效性。本文提出的算法在
主观效果和客观指标上与单纯的 ICBI插值相比在计算效率上有明显提高。
关键词：图像插值；视觉感知；迭代插值；图像梯度；
中图分类号：
225
