-1-
Trilateral Filtering for Image Interpolation
Junhua Liu
Institute for Signals and Information ProcessingLanzhou University，Lanzhou（730000）
Abstract
A new image interpolation scheme called trilateral filtering is presented in this paper. The method first
gives a primary estimate of unknown pixels by their four nearest neighbours with proper weights, which
are determined by the corresponding edge information and spatial distance. Then photometric
similarities between the primary estimate of unknown pixels and their neighbours are introduced to
further refine the weights, based on which, the final estimate of unknown pixels obtained. Experimental
results show that the proposed scheme produces fewer artifacts and sharper edges compared to
conventional methods, and still preserves simplicity and efficiency compared to more advanced ones.
Keywords: Image Interpolation； Image Magnification；Trilateral Filtering
1. Introduction
Image interpolation is the process to generate a high-resolution image from its given low-resolution
version. It can be applied in many fields, such as medical imaging and forensics, image demosaicking,
image geometry transform, video size conversion, etc. Sharpness of edges and freedom from artifacts
of interpolated images are two critical factors to measure the performance of an interpolation scheme,
and computation efficiency is also an important factor [2].
Many interpolation schemes have been proposed, and they can be generally divided into linear
methods and nonlinear ones. Linear methods estimate the values at unknown pixels by a linear
combination of their neighbours which are already known. These methods include nearest neighbour
interpolation, bilinear interpolation, bicubic interpolation, kernel based interpolation and so on, and
they are widely used for their computational efficiency [1], [4], [11]. However, since they are based on
space-invariant functions, they can’t capture the fast evolving statistics around edges, thus produce
blurring and annoying artifacts in the interpolated images [5].
To improve subjective quality of the interpolated images, nonlinear interpolation schemes are
developed, and they are usually based on local features of the given image. [2], [7] employ local edge
direction information to estimate unknown pixels. [5] assumes local covariance of low resolution
image can adequately approximate the covariance of high resolution (interpolated) one, and thus could
be used to estimate the weights of the unknown pixel's neighbours. [10] applies interpolation error
theorem to refine the linearly interpolated value of the unknown pixel, where the error is locally
estimated by its neighbours. [14] combines local vector ordering and spatial information for color
image interpolation. Some schemes (. [12]) iteratively modify the primarily interpolated image to
give better visual effects. Generally speaking, nonlinear methods usually consume more time than
linear ones, and will cause undesirable effects when input image structures do not conform to their
hypothetic models.
In this paper, we propose an intuitive non-iterative method for image interpolation. The idea is
motivated by the facts that if we want the interpolated edges with fewer artifacts, convex corners
-2-
should be suppressed, and if we want the interpolated edges to be sharp, a fine tuning of the primary
estimate of unknown pixels may be needed. Based on this idea, we formulate our scheme. The scheme
first estimates the unknown pixel value by a combination of its four neighbours with proper weights,
which are determined by the corresponding neighbour's edge information and spatial distance. Then
the weights are refined by photometric similarities between the primarily estimated unknown pixel
value and its corresponding neighbours. As edge information, spatial distance and photometric
similarity are all used to determine the weights, herein the name trilateral filtering comes. Simulation
results show that the scheme is very effective in edge sharpness and artifacts suppression.
The paper is organized as follows: Section 2 presents the formulation of the trilateral filtering
interpolation scheme, and section 3 gives experimental results. We conclude this paper in section 4.
2. Trilateral Filtering Interpolation
Edge Information
Suppose one edge between two objects in an image is a continuous line (gray) as shown in Figure 1,
then after digitallization of the image, we would get the white and black pixels on each side. If we
recover the edge with conventional interpolation methods (. nearest neighbour interpolation), one
possible edge would be the black curve, which gives annoying artifacts. An intuitive idea is that if
edge convex corner pixels ( ( , 1)i j + , ( 1, )i j+ , etc.) are positioned to be closer to the interpolated edge,
there should be fewer artifacts. So edge convex corner pixels should be given less importance while
performing interpolation, in another word, they should be suppressed.
Figure 1 Artifacts suppression along edges.
1 2 1
0 0 0
1 2 1
− − −⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
1 0 1
2 0 2
1 0 1
−⎡ ⎤⎢ ⎥−⎢ ⎥⎢ ⎥−⎣ ⎦
(1)
-3-
2 1 0
1 0 1
0 1 2
− −⎡ ⎤⎢ ⎥−⎢ ⎥⎢ ⎥⎣ ⎦
0 1 2
1 0 1
2 1 0
⎡ ⎤⎢ ⎥−⎢ ⎥⎢ ⎥− −⎣ ⎦
(2)
To extract information of edges and edge convex corners, we apply Sobel masks [8] to detect
horizontal and vertical directional edges which shown in (1). We also introduce similar masks [8] to
detect diagonal directional edges as shown in (2). Let ,
h
i jE and ,
v
i jE represent horizontal and vertical
edge indicators at pixel ( , )i j , then:
, 1, 1 1, 1, 1 1, 1 1, 1, 12 2
h
i j i j i j i j i j i j i jE f f f f f f+ − + + + − − − − += + + − − −
, 1, 1 , 1 1, 1 1, 1 , 1 1, 12 2
v
i j i j i j i j i j i j i jE f f f f f f− + + + + − − − + −= + + − − −
Where ,i jf is image value at pixel ( , )i j . Let
1
,
d
i jE and
2
,
d
i jE be 135
o and 45o directional edge
indicators at pixel ( , )i j , then the values are:
1
, , 1 1, 1 1, 1, 1, 1 , 12 2
d
i j i j i j i j i j i j i jE f f f f f f+ + + + − − − −= + + − − −
2
, 1, 1, 1 , 1 , 1 1, 1 1,2 2
d
i j i j i j i j i j i j i jE f f f f f f− − + + − + − += + + − − −
The four edge indicators are combined to form the final information ,i jE of edge and edge convex
corners at pixel ( , )i j by (3). The larger ,i jE is, the more likely pixel ( , )i j is an edge or edge
corner pixel.
1 2
, , , , ,| | | | | | | |
h v d d
i j i j i j i j i jE E E E E= + + + (3)
As edges and edge corners should be suppressed, so we should design a decreasing function of ,i jE
to determine weights, and here we employ the exponent function. Let pixel ( , )i j be one nearest
neighbour of ( , )x y (gray square point) as shown in Figure 1, where ( , )x y is the mapping point
from the interpolated image coordinates to the original low-resolution one, the first partial weight of
( , )i j can be formulated as:
, ,1 ,exp{ }i j i jw Eα= − (4)
where α is a tunable parameter to control suppression degree.
-4-
Spatial Distance
Using spatial distance information to interpolate unknown pixels is widely accepted in conventional
image interpolation schemes [4]. Here we simply borrow bilinear interpolation strategy, which is very
simple, yet possesses the advantage to keep image values smooth.
As illustrated in Figure 1, ( , )i j is one nearest neighbour of ( , )x y , then let dx and dy denote
the horizontal and vertical distance between them, the second partial weight of ( , )i j could be written
as:
, ,2 (1 )(1 )i jw dx dy= − − (5)
where | |dx x i= − and | |dy y i= − .
Primary Estimation
Previously we have developed two partial weights, and now it's time to combine them together to
give a primary estimate of unknown pixels. Assume ( , ), , 0, 1i k j l k l+ + = are four nearest
neighbours of ( , )x y as in Figure 1, then the primary estimate of ( , )x y is:
1 1
, ,1 , ,2 ,0 0
, 1 1
, ,1 , ,20 0
i k j l i k j l i k j lk l
x y
i k j l i k j lk l
w w f
f
w w
+ + + + + += =
+ + + += =
= ∑ ∑∑ ∑ (6)
This primary estimate of the interpolated image has fewer artifacts. Although edges are sharper than
bilinear interpolation method in many areas, blurring effects still exist, and in the next subsection, we
will seek techniques to refine this estimation.
Photometric Similarity Refinement
As illustrated in Figure 2, horizontal axis denotes pixel positions for 1-D signal, and vertical axis
denotes pixel values. The values at white circles are already known, and values between them are to be
estimated. With the previously developed interpolation scheme, the dashed curve would be obtained,
since edges information at position i and 1i + are the same, and (6) is degenerated to linear
interpolation. Hence blurring is produced. If we want to get a sharper edge, the interpolated curve
should be like the solid one.
Figure 2 Illustration of blurring and deblurring.
One approach to achieve this goal is to introduce photometric similarity to adjust the primarily
-5-
estimated dashed curve, which means that if the estimated value is closer to the already known pixel
value, the pixel should be given more contribution. Here we still use the exponent function to form our
third partial weight of ( , )i j :
2
, ,3 , , ,exp{ ( ) }i j i j i j x yw E f fβ= − − (7)
where β is a parameter to tune the sharpness, and ,i jE is introduced to ensure refinement only done
at edge pixels. This has the advantage to give sharper edges, and still preserves continuity in smooth
area.
Final Estimation
In this sub-section, we combine the previously developed three partial weights together to give a final
estimation of unknown pixels. Let
, , ,1 , ,2 , ,3i j i j i j i jw w w w=
then the pixel value at ( , )x y is:
1 1
, ,0 0
, 1 1
,0 0
i k j l i k j lk l
x y
i k j lk l
w f
f
w
+ + + += =
+ += =
= ∑ ∑∑ ∑ (8)
where the relative position of ( , )i j and ( , )x y is shown in Figure 1, and , ,1i jw , , ,2i jw , , ,3i jw are
given in (4), (5), (7) respectively.
Implementation
Suppose the size of the original input image I is M N× , and the zooming factors in the vertical and
horizontal direction are p and q respectively, then the size of the interpolated image J is H W× ,
where H p M= × and W q N= × . To estimate marginal pixels in J , padding of the original image
may be needed, since we need all the mapping points from the interpolated image J to the original
image I have four neighbours whose values are known. The procedure of the proposed interpolation
scheme now can be described as follows:
z Detect edge information ,i jE of image I with (3).
z For each pixel ( , )m n in the interpolated image J , decide its mapping position ( , )x y in the
original image I , where /x m p= and /y n q= . Let i be the integer part of x , and j be
the integer part of y , then ( , ), , 0, 1i k j l k l+ + = are four nearest neighbours of ( , )x y . A
primary estimate of ( , )x y could be obtained by (6), and a refined version of ( , )x y could be
computed by (8).
-6-
3. Experiment Results and Analysis
As mentioned in the introduction that freedom from artifacts and sharpness of edges are two critical
factors in the perceived quality of the interpolated image, however, there are little objective criteria to
properly reflect them, therefore we shall only rely on subjective evaluation to assess the performance
of the proposed scheme. Another important factor is the computational efficiency, and here we also
compare time consuming of various interpolation schemes with ours.
In the experiments, we adopt the widely used Cameraman and Lena as our benchmark images. Both
of the original images size are 256 256× , and the zooming factor is 4 4× . The parameters α , β in
our interpolation scheme are selected to be and 3 respectively. We implemented nearest neighbour
interpolation, bilinear interpolation [4], bicubic interpolation [1], triangular interpolation [7], EASE
interpolation [10] and our proposed interpolation scheme with C programming language. We also
borrowed simulation programs of NEDI [5] and iNEDI [11] which are written with Matlab by their
authors as references.
Figure 3 shows simulation results of Cameraman, and Figure 4 of Lena by different interpolation
schemes. According to the results, the proposed interpolation scheme produces sharper edges and
fewer artifacts (shoulder in Figure 3, hat edge in Figure 4, etc.), yet still keeps continuity in smooth
area (face in Figure 3, .) compared with other interpolation schemes. One exception of our scheme
is that in the very fine texture areas (single pixel line, for example), artifacts along edges (see hat band
of Figure 4) still exist due to the insufficiency of edge and edge convex corner information
representation extracted by Sobel and similar masks in section .
(a) Nearest neighbour (b) Bilinear (c) Bicubic (d) EASE
(e) Triangulation (f) NEDI (g) iNEDI (h) Proposed
Figure 3 Portions of image Cameraman magnified with 4 4× by different interpolation schemes.
-7-
(a) Nearest Neighbour (b) Bilinear (c) Bicubic (d) EASE
(e) Triangulation (f) NEDI (g) iNEDI (h) Proposed
Figure 4 Portions of image Lena magnified with 4 4× by different interpolation schemes.
Table 1 shows time consuming of different interpolation schemes, which are executed on our HP
laptop computer with CPU AMD Sempron 2800+ and memory 512M inside. As can be seen, the
nearest neighbour interpolation is most efficient, and our scheme consumes about an order of
magnitude higher than conventional schemes. However, it is much efficient than more advanced ones
(NEDI, iNEDI), even though we consider the differences of executive efficiency between C and
Matlab language.
Table 1 Time consuming of different interpolation schemes (The first six schemes are implemented with C
programming language and the last two are with Matlab).
Interpolation Scheme Cameraman [s] Lena [s]
Nearest neighbour
Bilinear
Bicubic
Triangulation
EASE
Proposed
-8-
NEDI
iNEDI
4. Conclusion and future works
In this paper, we present an intuitive trilateral filtering scheme for digital image interpolation. The
scheme first estimates primary values of unknown pixels by their four nearest neighbours with proper
weights, which are determined by the corresponding neighbour's edge information and spatial distance.
The primary estimates of unknown pixels are further refined by introducing photometric similarities of
their neighbours to give sharp edges. Experimental results show that the proposed scheme gives
sharper edges and fewer artifacts than compared interpolation schemes in most parts of the
interpolated images except very fine texture areas. The shortages will be studied further in future
works.
Acknowledgments
Thanks to X. Li and N. Asuni for providing their implementation of NEDI and iNEDI interpolation
schemes to the public, and the author also would like to thank Yafeng Ma and Yao Wang for their
kindly checking on this paper.
References
[1] R. G. Keys (1981) Cubic Convolution Interpolation for Digital Image Processing, IEEE Trans. Acoustics,
Speech and Signal Processing, 29(6), pp. 1153-1160.
[2] J. P. Allebach and P. W. Wong (1996) Edge-Directed Interpolation, Proc. Int. Conf. Image Processing, vol.
3, pp. 707-710.
[3] C. Tomasi and R. Manduchi (1998) Bilateral Filtering for Gray and Color Images, Proc. 6th Int. Conf.
Computer Vision, pp. 839-846.
[4] T. M. Lehmann, C. Gonner and K. Spitzer (1999) Survey: Interpolation Methods in Medical Image
Processing, IEEE Trans. Medical Imaging, 18(11), pp. 1049-1075.
[5] X. Li and M. T. Orchard (2001) New Edge-Directed Interpolation, IEEE Trans. Image Processing, 10(10),
pp. 1521-1527.
[6] W. T. Freeman, T. R. Jones and E. C. Pasztor (2002) Example-Based Super-Resolution, IEEE Computer
Graphics and Applications, 22(2), pp. 56-65.
[7] D. Su and P. Willis (2003) Demosaicing of Color Images Using Pixel Level Data-Dependent Triangulation,
Proc. Theory and Practice of Computer Graphics, pp. 16-23.
[8] A. Bovik (2005) Handbook of Image and Video Processing, 2nd ed, Elsevier Inc.
[9] S. Yang and K. Hong (2005) Bilateral Interpolation Filters for Image Size Conversion, IEEE Int. Conf.
Image Processing (ICIP), pp. 986-989.
[10] Y. Cha and K. Seongjai (2007) The Error-Amended Sharp Edge (EASE) Scheme for Image Zooming, IEEE
Trans. Image Processing, 16(6), pp. 1496-1505.
[11] N. Asuni and A. Giachetti (2008) Accuracy Improvements and Artifacts Removal in Edge Based Image
Interpolation, Proc. 3rd Int. Conf. Computer Vision Theory and Applications (VISAPP'08), pp. 58-65.
[12] A. Giachetti and N. Asuni (2008) Fast Artifacts-Free Image Interpolation, Proc. Conf. British Machine
Vision (BMVC).
[13] A. Bourquard, P. Thevenaz, K. Balac and M. Unser (2008) Consistent and Regularized Magnification of
Images, IEEE Int. Conf. Image Processing (ICIP), pp. 325-328.
[14] L. H. Jin, D. H. Li and E. M. Song (2009) Combining Vector Ordering and Spatial Information for Color
Image Interpolation, Image and Vision Computing, 27(4), pp. 410-416.
-9-
Author Brief Introduction：
Junhua Liu received the B. S degree in electronics and information science from Lanzhou University,
Lanzhou, China, in 2007, where he is currently pursuing the M. S degree in signal processing. His
research interests include speech recognition and image processing.
