遥感图像融合

J Indian Soc Remote Sens (September2015) 43(3):467–473DOI 10.1007/s12524-014-0430-4

RESEARCH

ARTICLE

A Novel Remote Sensing Image Fusion Approach Research Based on HSV Space and Bi-orthogonal Wavelet Packet Transform

Wenxing Bao &Xiaoliang Zhu

Received:15December 2013/Accepted:11November 2014/Publishedonline:8February 2015#Indian Society of Remote Sensing 2015

Abstract Acquiring as much spatial and spectral informa-tion as possible is a challenge of remote sensing image fusion between panchromatic (PAN)image and multispec-tral (MS)image. Wavelet packet analysis (WPA)can offer precise result for image process and visual characteristics of human eye are considered sufficiently in HSV (Hue,Saturation, Value) color space. A fusion approach of re-mote sensing images combining Region Classification and Modulus Lifting based on WPA applied in HSV color space is presented in the paper. Firstly, there can be a HSV model gained after converting the original MS im-age into HSV space. The Value ingredient that extracted from the HSV model is decomposed into various frequen-cy domains via Bi-orthogonal wavelet packet transform at the third scale. The PAN image is also decomposed in the same way as the former meanwhile. Secondly, according to respective features of different frequency bands, the Region Classification rule is used in the low frequency band, and the high frequency coefficients are obtained by adopting the Modulus Lifting rule. A fused Value ingre-dient can be provided by inverse wavelet packet trans-form. Lastly, a merged image produces after finishing the space transform from HSV to RGB. By comparing with other fusion methods carefully, analysis result reveals that the proposed algorithm can achieve the most satisfactory spectrum effect and spatial information.

Keywords Image fusion . HSV . Bi-orthogonal wavelet packet transform . Region classification . Modulus lifting

W. Bao (*) X. Zhu

School of Computer Science and Engineering, Beifang University of Nationalities, No.204Wenchang, North-Street, Xixia District, Yinchuan, Ningxia 750021, China e-mail:[email protected]

Introduction

In recent years, multi-source remote sensing data fusion has gradually become a promising and forward research subject in related application field with the profound development of remote sensing technology. Multi-source remote sensing im-age fusion (Pajaresand de la Cruz 2004)(Baoand Wang 2011) refers to such a technique that uses certain scheme to merge organic and complementary data information from multi-sensor images of the same scene perfectly into a new image. It enables the gained composition to be more beneficial to practical application. Ordinarily the image fusion of remote sensing is implemented between the panchromatic (PAN)image and the multispectral (MS)one. According to the document (Naet al. 2004), the former has higher spatial resolution and single waveband, whereas the latter owns lower spatial resolution and multiple wavebands which are usually regarded as pseudo color. How to fuse the two kinds of image better has already been a hot topic.

In early time, a large number of traditional image fusion methods such as PCA (PrincipalComponent Analysis) algo-rithm and HIS (Hue,Intensity, Saturation) transform played a positive role. They operate in space domain and only overlap original data pixel by pixel, which are simple and intuitive but not accurate. Gradually, more and more scholars introduce multi-resolution analysis to image fusion area. Multi-resolution analysis (Huet al. 2008) that called multi-scale analysis as well is a sophisticated theory including pyramid transform, wavelet analysis (WA),wavelet packet analysis (WPA),super wavelet analysis, and so on. The WA-based (Chenet al. 2008) image fusion approach, which is more compact and renders better performance than the conventional methods mentioned above, is able to decompose the image signal into multi-frequency fields which are low-low band at low frequency field, as well as low-high, high-low, and high-high bands at high frequency fields. WPA (Chenet al.

4682008) is further evolution of WA and a more accurate transform tool because it can decompose the low-frequency information and the high-frequency one in all wavelet packet sub-bands at the same time. But for WA, only low frequency band can be decomposed fur-ther so that it losses some useful detail information of signal.

The image fusion method presented in Literature (Sunet al. 2012) is just based on WA with region segment rule utilized in low-frequency domain and Modulus Maxima based on aver-age gradient applied in high frequency domain. But this method had spectrum distorted and image edges blurred. The WAP-based image fusion algorithm which we put forward has some improved points. Firstly, the MS im-age is transformed into HSV model. Secondly, the Value component extracted from the HSV model of the MS image and the PAN image are transformed independently into one low-frequency domain and 63high-frequency bands by Bi-orthogonal wavelet packet transform at the third scale. The Region Classification rule is used in the low frequency band, and the Modulus Lifting rule is applied for high frequency coefficients. A fused Value ingredient is provided by inverse wavelet packet trans-form. Lastly, a new merged image produces after finishing the space transform from HSV to RGB. We have performed a series of tests to show that the objec-tive characteristics and the subjective visual effect of the new method are superior to other methods mentioned in the paper.

The Presented Theory HSV Color Space Principle

The HSV color space pattern similar to HIS model is based on human vision sense perception about color and approximates to human eyes vision system much more than the RGB color space. It has also three channels that represent different factors of color space which are Hue, Saturation, and Value. Obviously, HSV means the abbre-viation of Hue, Saturation, and Value. Hue indicates spectrum wavelength that decides the essence of color, Saturation indicates the purity of color, and Value indi-cates the brightness of light (Zhanget al. 2010). Among the three channels of the HSV color space model, the ingredient Value is the prime part that is used really in image processing. Sometimes image disposing in the RGB color space cannot exert the advantage of color apperceive, so many scholars usually use the mutual transform between HSV and RGB to accomplish peculiar image process and get excellence.

J Indian Soc Remote Sens (September2015) 43(3):467–473

Wavelet Packet Analysis

WPA belongs to multi-resolution (i.e.multi-scale) anal-ysis theory which makes signal processing become easy. WPA originates from WA which is a relatively recent branch of mathematics. In fact, Wavelet is a waveform which owns a number of mathematically available fea-tures preferable to the primitive sine or cosine function (Chitadeand Katiya 2012). Since image is a 2-D signal and WA can be regarded as a special case of WPA, we mostly concentrate on 2-D WA and 2-D WPA theory (SunYankui 2005).

WA decomposes an image into four sub-images that have the same size through the low-pass decomposed filter (L)and the high-pass decomposed filter (H).Through the first level of decomposing a image, we get an approximate sub-image, which corresponds to Low-Low (LL)frequency band, and three detail sub-images, which correspond respectively to Low-High (LH)fre-quency band, High-Low (HL)frequency band, and High-High (HH)frequency band. Only the LL band can be iteratively decomposed into four smaller sub-images whereas other detail bands cannot be decomposed again. Conversely, the source image can be reconstructed in different level through the low-pass reconstructed filter (L*) and the high-pass reconstructed filter (H*).

WPA theory is the extension of WA. The distinctive differ-ence between the two theorems is the three high frequency zones (i.e.LH, HL, and HH) of wavelet packet transform can be also decomposed into four sub-images having the same size respectively when its low frequency domain is divided further at the given scale, whereas the three high frequency zones of wavelet transform cannot do that as the previous paragraph said. It enables wavelet packet transform to provide finer information and it is beneficial to sign process (Caoet al. 2003). Similarly, the original signal may be obtained by inverse wavelet packet transform which uses a series of sub-images decomposed individually in the low and the high frequency bands.

Figure 1displays the structure of wavelet packet tree which is a completely quad-tree. It makes it out that the discrimina-tion between WA and WPA is WPA can decomposed contin-uously high frequency signals. In Fig. 2, we can tell a series of frequency bands in wavelet packet fields. LL i j means sub-image with low frequency in horizon and low frequency in vertical, LH i j means low frequency in horizon and high fre-quency in vertical, HL i j means sub-image with high frequency in horizon and low frequency in vertical, HH i j means sub-image with high frequency in horizon and high frequency in vertical, where j is the decomposed scale and i is the order number of sub-image in certain decomposed scale.

Bi-orthogonal wavelet packet is also called semi-orthogonal wavelet packet which is of good symmetry

J Indian Soc Remote Sens (September2015) 43(3):467–473Fig. 1Wavelet packet decomposition

quad-tree

469

and exact. Its difference from orthogonal wavelet packet is the wavelet functions of decomposition and recon-struction are diverse, so the same FIR filter is not used for decomposition and reconstruction at the same time. Owing to the important property rendering some redun-dant in the process of signal shift between the two interfacing decomposed scales, Bi-orthogonal wavelet packet may be used to reconstruct the original signal more finely than orthogonal wavelet packet.

The Presented Method The Scheme Flow Description

This image fusion approach combines the Region Clas-sification rule and the Modulus Lifting rule based on wavelet packet analysis followed by transforming the primitive MS image into the HSV color space. After drawing the constituent Value from HSV model of the MS image, the Value component and the PAN image are decomposed at the third scale through bi-orthogonal wavelet packet transformation with the ‘9–7’type filter (Duncanand Mihn 2006) respectively. According to the features of data sources, the Region Classification rule is used for the low frequency band at the third decomposed scale and the Modulus Lifting rule is exploited in the high frequency domains at the same

Fig. 2Wavelet packet decomposition structure

scale. Figure 3shows the image fusion scheme flow of the presented method in the paper.

Step one The MS image needs to be transformed from the

RGB space to the HSV one. Only the component Value is extracted from the converted image in HSV space, whereas the H and the S component are reserved meanwhile.

Step two The Bi-orthogonal wavelet packet decomposition

is applied independently to the derived Value in-gredient and the original panchromatic image. We choose three as the number of decomposition lev-el. Therefore, there are 64sub-images produced which include one approximation sub-image and 63detail sub-images for every image source.

Step three We utilize the Region Classification rule to the

approximation part while the Modulus Lifting rule is used for the coefficients of detail sub-images.

Step four We can gain the fused Value component by

implementing inverse wavelet packet decompo-sition (i.e.wavelet packet reconstruction).

Step five The fused Value, combining the H and the S, is

organized as a new composition. The final fusion result produces with the new composition trans-formed from the HSV space to the RGB one.

Fusion Rule in Low Frequency Domain

It is foremost how to obtain the fused approximate coefficients from low frequency band of source image, because low frequency coefficients are the decided fac-tor of image body which is close to original image very much. In order to get fully reliable information from low frequency domain of sub-image and improve the image fusion performance, we use the fusion rule based on Region Classification for the low frequency band. The topic of Region Classification is that low frequency band of source image is categorized logically into two blocks which are homogeneous domain and detail do-main so that it is feasible to judge which domain the

470

Fig. 3Flow diagram of the presented fusion method

J Indian Soc Remote Sens (September2015) 43(3):467–473

current coefficient is. In fact, homogeneous domain is the real low frequency area. However, detail domain is such a block in which all coefficients represent the quite rich spectrum information of low frequency. We, there-fore, apply the coefficient absolute maxima to detail domain and the weighted average to homogeneous do-main. The specific step of dividing low-frequency band into homogeneous domain and detail domain is as follow.

At first, Supposed that there exist two low frequency sub-images f A and f B drew from source matrixes A and B decomposed by Bi-orthogonal WPA at the third scale, according to eq. (1), the average gradient grad k n (x,y) for the nearest-neighbor n x n split window of which the pixel (x,y) is the centre in the sub-image f A or f B is computed and defined as a pixel value of the new matrix which is called ‘coefficient measure image ’M k , where k represents the data source A or B and n represents 3.

grad k n ðx ; y Þ¼

1ðn −1Þ

x ; y

X i ¼x −1; j ¼y −1

r ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi .

ðf k ði þ1; j Þ−f k ði ; j ÞÞ2þðf k ði ; j þ1Þ−f k ði ; j ÞÞ2

2; n ¼3ð1Þ

Furthermore, after choosing the appropriate threshold val-ue t, each coefficient in image M k is compared with t via eq.

(2) and it makes the black and white image g k (x,y) produce, where the parameters k and n have the same meaning as above. The right degree of threshold value plays a positive role in improving quality of fused image. Through repeated tests, the parameter t takes 0.1.

1; grad k n ðx ; y Þ≥t ð2Þg k ðx ; y Þ¼k

0; grad n ðx ; y Þ

1; g A ðx ; y Þ&&g B ðx ; y Þ¼1

ð3ÞR ðx ; y Þ¼

0; g A ðx ; y Þ&&g B ðx ; y Þ¼0In the ‘binary value resolution image ’R(x,y), the value of every pixel (x,y) has a state:one or zero. The sign f A (x,y) refers to a pixel (x,y) of the low frequency sub-image f A and

the sign f B (x,y) refers to a pixel (x,y) of the low frequency sub-image f B . If R(x,y) is equal to one, then it means that f A (x,y) and f B (x,y) fall inside the detail domains of the sub-images f A and f B respectively. The acquirement of the corresponding fused approximate coefficient P L (x,y) adheres to the absolute maximum rule depending on eq. (4). We can found P L (x,y) is either f A (x,y) or f B (x,y) whose absolute value is maximal when comparing them. The foot mark L of P L (x,y) indicates low frequency area.

P L ðx ; y Þ¼ðf A ðx ; y Þþf b ðx ; y ÞÞÃ0:5

ð4Þ

Similarly, if R(x,y) is equal to zero, then it means that f A (x,y) and f B (x,y) fall inside the homogeneous domains of the sub-images f A and f B respectively. The corresponding fused approximate coefficient P L (x,y) is the average value for f A (x,y) and f B (x,y), which follows eq. (5). The foot mark L of P L (x,y) is the same as above.

P L ðx ; y Þ¼

f A ðx ; y Þ; j f A ðx ; y Þj≥j f B ðx ; y Þj f B ðx ; y Þ; j f A ðx ; y Þj

ð5Þ

J Indian Soc Remote Sens (September2015) 43(3):467–473471

Fusion Rule in High Frequency Domain

In the complete quad tree of wavelet packet decomposition at the third scale, the nodes that range from (3,1) to (3,63) are the 63high-frequency sub-images which can make full use of detail information from source. To highlight the edge and contour contents of the fused image and improve the clarity of the fused image, we use the rule which is named ‘Modulus Shifting ’based on eq. (6) for high frequency sub-bands.

P Hi ðx ; y Þ¼

d ÃD Ai ðx ; y Þ; j D Ai ðx ; y Þj≥j D Bi ðx ; y Þj d ÃD Bi ðx ; y Þ; j D Ai ðx ; y Þj

ð6Þ

The sign D ki (x,y) points to the detail coefficient of pixel (x,y) in the ith high-frequency sub-image from original image k, where k still means the data source A or B as well as the variable i indicates the serial number of sub-images in high frequency fields of the third decomposition level whose value ranges from 1to 63. For example, the node (3,1) is corre-sponding to D k1(x,y) of eq. (6) and the node (3,63) matches with D k63(x,y). We can gain the fused high-frequency coef-ficient P Hi (x,y) by multiplying ratio d and modulus D ki (x,y) whose value is either D Ai (x,y) or D Bi (x,y) by following the absolute maximum rule to compare them. The foot mark H of P Hi (x,y) indicates high frequency area. It is important to select a suitable value for the ratio d, which is 1.2in this case. That is because the parameter d not only decides the spatial definition of the fused image but also affects the spectral effect of the fused result to some extent. If d is larger than 1.2, then it may result in the serious spectrum discrepancy. If d is smaller than 1.2, then it cannot enhance the spatial quality of the fused image.

fetched from Sand Lake area in certain city are used as the experimental materials. The MS image is taken from the fourth, the fifth, and the third wave band of origin, which are corresponding to the three colors:red, green, and blue. The PAN image just comes from single wave band. It is of impor-tance and feasibility to identify the boundaries of sand or water and determine features of vegetation, sand, water body, wet land, and so on through these wave bands. In order to illustrate the validness and efficiency of the proposed ap-proach, there are three existing methods rendered and compared.

A series of tests are conducted in MATLAB platform by designing related simulation program. The situation of these experimental results shows in Fig. 4a, b are the source images that Fig. 4a is the panchromatic image and Fig. 4b is the multispectral image. These two remote sensing images possess different spectral features and spatial resolution. Before fusion, they have to be regis-tered and matched strictly so that the corresponding pixels are co-aligned. The four fused images utilized respectively the PCA fusion method, the wavelet packet fusion method, the fusion method about reference (Sunet al. 2012) and the presented fusion method display in Fig. 4c-f successively.

Subjective Analysis and Judgment of Fusion Effect Quality assessments of fused images are often measured through human visual perception and analysis. From the fused image in Fig. 4c , we can clearly see that PCA fusion method can reserve spectrum information much better, but cannot gain higher legibility of detail and texture of the fused image. In Fig. 4d shows the fused image using the WPA fusion method that based on the weighted average in the low frequency field and the absolute maximum in the high frequency fields. Its final result including spectrum features and space clarity is not as grace as the former. Obviously, the legibility of the fusion data in Fig. 4e based on reference (Sunet al. 2012) is higher than that of the PCA fusion method, but its spectrum level is much lower than PCA. By visual comparison, the

proposed

Experimental Result Analysis

In the section, the TM low resolution multispectral image and the TM high resolution panchromatic image of the same scene

(a) Panchromatic (b) Multi-image

spectrum image

(c) Fused image (d) Fused image (e) Fused image by

by using the PCA by using the W PA method method

using the reference

[6] methodFig. 4The sources and the fused images of the different methods (a )

Panchromatic image (b ) Multi-spectrum image (c ) Fused image by using the PCA method (d)Fused image by using the WPA method (e ) Fused

image by using the reference Sun et al. (2012) method (f ) Fused image by using the presented method

472Table 1

Objective evaluation results of different images

Average gradient 15.72598.27277.38827.823916.192615.643515.935615.648015.529815.572716.152915.749215.930917.154516.563016.7661

Space frequency 31.383516.038914.415314.365332.435132.094231.479930.898930.827030.636231.727531.312231.144033.593433.023432.7803

J Indian Soc Remote Sens (September2015) 43(3):467–473

image Wave name band Fig. 4a -Fig. 4b R

G B

Fig. 4c R

G B Fig. 4dR

G B

Fig. 4e R

G B Fig. 4f R

G B

Information entropy 7.79267.68967.82267.81197.54767.40407.77707.78797.81877.86257.79537.82177.88747.76907.82147.8381

Standard deviation 75.477471.973571.775464.760479.885681.401074.305372.030174.705968.231277.009775.140765.488876.065576.354769.5088

Mean value

116.4437146.6923116.4780106.3500141.3824111.6965101.1782131.5692116.4523111.3517149.2934117.0993127.4893146.7934117.2480106.6052

These images offered in Table 1can be explicated as follow. Figure 4a means the Panchromatic image; Fig. 4b means the Multi-spectrum image; Fig. 4c means the fused image by using the PCA method; Fig. 4d means the Fused image by using the WPA method; Fig. 4e means the Fused image by using the reference Sun et al. (2012) method; Fig. 4f means the Fused image by using the presented method

scheme is superior to the three methods above, which not only improves the spatial resolution of the fused image in Fig. 4f , but also reserves the spectrum information greatly, and has the best fusion effect among these fusion methods.

Objective Analysis and Judgment of Fusion Effect

Quantity analysis and measure have to be carried out by following several numeric evaluation criteria below, which mean Objective Analysis and Judgment of fusion effect, ow-ing to the random of subjective judgment. In the experiment, we can choose the five objective criteria such as the average gradient (Yanget al. 2010), space frequency, information entropy, standard deviation, and mean value to assess the performance of the three channels R, G, and B of the fused images obtained with different algorithms. The comparison and estimation results of performance parameters list in Table 1below. Larger average gradient and space frequency indicate better legibility, detail and texture content of fused image. Larger entropy means the fused image has gotten much richer information from original images. Larger standard deviation reflects the wider grey range of the image. The more approximate to the original MS image mean value of fused image is, the better fusion effect is, which makes known that reserves much more spectrum information.

From the indicated data in Table 1it is obvious that the average gradient and space frequency of the fused image

based on the presented fusion method are the greatest, and its entropy ranks the third and standard deviation lists the second in the row, and its mean value is the closest to the original multi-spectrum ’s. It is an explicit statement that the represented method makes the spectrum discrepancy least when it improves the definition of the fused result nicely. In addition, the run time operated by the reference (Sunet al. (2012) method is multiplied dozens of times as much as the presented method but those of other methods are similar. Therefore, the synthetic effect of the presented fusion method is the most advantaged one among the several methods.

Conclusion

The theory and application of WA and WPA lead to improve-ment of multi-source remote sensing image fusion technology to large degree. The new fusion method presented in the paper is combined Region Classification and Modulus Lifting based on wavelet packet analysis followed by the HSV color space transform, and it exerts a positive efficiency. However, diverse data sources own their characteristics. It is the key fact that aiming at special problem, how to develop a fusion method of more preponderance is our study target of next step. This paper will accelerate our deeper research on remote sensing image fusion scheme. Furthermore we can precast the study and application of WA and WPA theory will continue to play an important role in remote sensing image fusion area.

J Indian Soc Remote Sens (September2015) 43(3):467–473

Acknowledgments This work is supported by National Natural Sci-ence Foundation of China (61162013).

References

Bao, W. X., &Wang P. (2011).Remote sensing image fusion based on

wavelet packet analysis. IEEE 3rd International Conference on Communication Software and Networks (ICCSN),359–362.

Cao, W., Li, B. C., &Zhang, Y . (2003).A remote sensing image fusion

method based on PCA transform and wavelet packet transform.IEEE Int. Conf. Neural Networks 8Signal Processing Nanjing China, December, 14–17, 976–981.

Chen, H., Liu, Y . Y ., &Wang, Y . J. (2008).Anovel image fusion method

based on wavelet packet transform. IEEE International Symposium on Knowledge Acquisition and Modeling Workshop, 462–465Chitade, A. Z., &Katiya, S. K. (2012).Multiresolution and multispectral

data fusion using discrete wavelet transform with IRSI images:cartosat-1, IRS LISS III and LISS IV . Journal of the Indian

473

Duncan, D. Y . P., &Minh, N. D. (2006).Directionalmulti-scalemodeling

of images using the contourlet transform. IEEE Transactions on Image Processing, 15(6),1610–1620.

Hu, G., Liu, Z., Xu, X. P., Gao, R. (2008).Research and recent develop-ment of image fusion at pixel level. Application Research of Computers, 25(3),650–654.

Na, Y ., Lin Shi, L., &Yang, W. H. (2004).Wavelet packet transform and

remote sensing image fusion. ACTA PHOTONICA SINICA, 33(6),736–738.

Pajares, G., &de la Cruz, J. M. (2004).A wavelet-based image fusion

tutorial. Pattern Recognition, 37, 1855–1872. Society of Remote Sensing, 40(1),121–128.

Sun, Yankui. (2005).Analysis and Application of wavelet. (pp:245-255)

Beijing:China Machine Press, (1).

Sun, P., Deng, L., &Nie, J. (2012).A multi-scale remote sensing image

fusion method based on wavelet decomposition. Application of Remote Sensing Technology, 27(6),844–849.

Yang, S. Y ., Wang, M., Jiao, L. C., et al. (2010).Image fusion based on a

new contourlet packet. Information Fusion, 11(2),78–84.

Zhang, Z., Wang, Y . P., &Xue, G. X. (2010).Digital image process and

machine vision. Posts and telecom press, 1, 233–269.