Title Window Image Reflection Removal by Blind Image Separation

Author(s) Mohammad Reza Alsharif; Mahdi Khosravy; Yamashita,Katsumi

Citation 琉球大学工学部紀要(71)

Issue Date 2010

URL http://hdl.handle.net/20.500.12000/18509

Rights

Window Image Reflection Removal by Blind Image Separation

Mohammad Reza Alsharif1, Mahdi Khosravy1 and Katsumi Yamashita21Department of Information Engineering, University of the Ryukyus, Okinawa, Japan

2Graduate School of Engineering, Osaka Prefecture University, Osaka, JapanE-mail : 1asharif@ie.u-ryukyu.ac.jp, mahdikhosravy@yahoo.com

2yamashita@eis.osakafu-u.ac.jp

Abstract

A novel second order statistics approach to blind im-age separation has been deployed to remove reflectionimage from window glass images. The reflection im-age and the image from behind the window glass areconsidered as blind sources. Since sources are un-correlated, their low frequency as well as high fre-quency halfbands are uncorrelated too. It is shown thatthe un-mixing matrix is obtained by simultaneous di-agonalization of covariance matrices of low and highhalfbands of the mixtures. The proposed techniquehas been compared with AMUSE [1] and SOBI [2] astwo other well known second-order statistics BSS tech-niques. Three different numerical metrics approves theefficiency dominance of the proposed technique.

Keywords: Reflection removal, Blind image separation,second-order statistics, covarriance matrix, subband linear fil-ters.

1 Introduction

Although images normally do not mix together in awell engineered system, blind image separation is aproblem encountered in various applications such asdocument image restoration, astrophysical componentseparation, analysis of functional Magnetic ResonanceImaging and removal of spurious reflections. In takingphotograph from the objects behind the window glass,the taken photo is a mixture of the image of the objectsbehind the glass and the reflection image of the objectsin front of the glass. Such event happens in windowshopping, aquarium photography and any other photog-raphy with a glass window between camera and objects.Also in photography of inside the water from above ofwater surface, the photo is mixture of inside the waterand reflection of the sky and clouds appeared on thewater surface.

Blind source separation (BSS) [3, 4] is a well knowntechnique to recover multiple original sources fromtheir mixtures. WhileK unknown source signals upontransmission through a medium have been mixed to-

gether and mixture signals are collected byM sen-sors, BSS try to blindly separate the sources frommixed observed data without any prior information ofthe source signals and mixing process. In mathemat-ical model, theM × 1 vector x of observed signalsx(t) = (x1(t) x2(t) · · · xM (t))T is multiplicationof K × 1 vectors of unknown source signalss(t) =(s1(t) s2(t) · · · sK(t))T by unknown mixing matrixA ∈ RM×K :

x(t) = As(t) + n(t) (1)

wheren(t) is a possible additive noise. Given only theobservation signalsx(t) = (x1(t) x2(t) · · · xM (t))T ,the solution of BSS problem seeks for the bestW ∈

RK×M as un-mixing matrix to extract signals as muchas possible close to unknown source signals as follows:

y(t) = Wx(t) (2)

wherey(t) is K ×1 vector of extracted signalsy(t) =(y1(t) y2(t) · · · yK(t))T . W is obtained in the waythat the estimated sources to be spatially uncorrelatedor as independent as possible, in the sense of minimiz-ing various functions that measure independence [5-9]. This measure is based upon second-order statistics(SOS) or higher-order statistics (HOS). Many second-order BSS approaches have been presented in the litera-ture [10- 18]. One of the well known second-order BSSmethods is an algorithm for multiple unknown signalsextraction (AMUSE) [1]. AMUSE converts the prob-lem of blind identification of mixing matrix to simulta-neous diagonalization (SD) problem of two covariancematrices of mixtures and their delayed signals. Theother well known second-order BSS method is second-order blind identification (SOBI) algorithms [2]. SOBIemploys joint diagonlization of covariance matrix ofmixtures and covariance matrices of different delayedmixtures to obtain un-mixing matrix.

In this paper, the reflection over window glass isblindly separated from the image of the objects be-hind the glass by using a BSS approach by deploy-ing second-order statistics of the sources in two dif-ferent views at the same time. The proposed BSS isbased on assumption that sources are mutually uncor-related. When sources are uncorrelated, their subbands

1

obtained by linear filters are uncorrelated too. The si-multaneous diagonalization (SD) of covariance matri-ces of the low halfbands and high halfbands of imagemixtures has been deployed to remove glass reflectionof window images. The proposed BSS method has beencompared with AMUSE and SOBI.

The rest of the paper is as follows. In section 2, theBSS by SD of high and low halfbands has been de-scribed. Experimental results and discussions are pre-sented in section 3. Finally, Section 4 concludes thepaper.

2 BSS by simultaneous diagonal-ization of low and high half-bands

The proposed BSS is based on second-order statisticsof low and high halfbands of the uncorrelated sources.In order to extend the theory of the proposed BSS,first we express some properties of responses of a lin-ear filters to uncorrelated signals. Consider a vec-tor of signalss(t) = [s1(t), s2(t), · · · , sK(t)]T , andthe vector of responses of a linear filter to them ass(t) = [s1(t), s2(t), · · · , sK(t)]T . For responses of alinear filter to a vector of signals, the following proper-ties can be obtained;

• Property 1 If signalss(t) are mutually uncorre-lated, thens(t) the response signals of a linear fil-ter to them are also mutually uncorrelated.

E [sisj] = 0, i 6= j =⇒ E [sisj ] = 0 i 6= j (3)

So, the covariance matrix of the response signalsis a diagonal matrix;

C ss = diag (E [s1s1] , E [s2s2] , · · · , E [sK sK ])

• Property 2 Let y(t) and x(t) are respectivelyresponses of a linear filter toy(t) and x(t). Ify(t)=Ux(t), U ∈ RK×M . Then, we have

y(t) = Ux(t) (4)

and C yy = UC xxUT (5)

whereC xx and C yy are respectively covariancematrices ofx(t) andy(t).

Let’s remember the above properties and go backto the BSS problem. Vector ofM mixture signalsx(t) = [x1(t), x2(t), · · · , xM (t)]T has been observedby M sensors. BSS looks for the best estimate of un-mixing matrix WK×M in order to recover the vectorof K unknown source signalsy(t) = [y1(t), y2(t), · · ·, yK(t)]T by Eq.(2).

y(t) = Wx(t) ≈ s(t) (6)

Considery(t) andy(t) are respectively low and highhalfbands of the recovered signals byW. y(t) andy(t)have been obtained by a linear lowpass filter and a lin-ear highpass filter, respectively. Throughout this sec-tion the following assumptions are made unless statedotherwise:

• Assumption 1 : The mixing matrixA ∈ RM×K

is a full column rank matrix withM ≥ K, and itis the same for all frequency contents of signals.

• Assumption 2 : Sources are mutually uncorrelatedwith different autocorrelation functions.

E [sisj] = 0, i 6= j (7)

E [sisi] 6= E [sjsj ] i 6= j (8)

• Assumption 3 : Autocorrelation functions of lowhalfbands of the sources are different from auto-correlation functions of their high halfbands.

E [sisi] 6= E [sisi] (9)

If W is the un-mixing matrix, based on assumption 2 re-covered source signals are mutually uncorrelated. Also,since low and high halfbands are obtained by linear fil-ters, from property 1 , the covariance matrices ofy(t)andy(t) are diagonal matrices;

Cyy = E[yyT

](10)

= diag (E [y1y1] , E [y2y2] , · · · , E [yK yK ]) ,

Cyy = E[yyT

](11)

= diag (E [y1y1] , E [y2y2] , · · · , E [yK yK ]) ,

whereCyy andCyy are covariance matrices ofy(t) andy(t). Furthermore, since from Assumption 3 the lowand high halfbands of the sources have different auto-correlation functions, bothCyy andCyy are non-zerodistinct matrices. Taking into accounty(t) = Wx(t)and considering the property 2, we have

Cyy = WCxxWT (12)

Cyy = WCxxWT (13)

whereCxx andCxx are covariance matrices ofx andx. SinceCyy andCyy are distinct matrices, it followsfrom Eqs.(12) and (13) thatCxx andCxx are distinctmatrices too. Moreover, Eqs.(12) and (13) convey thatsimultaneous diagonalization of two distinct matricesCxx andCxx estimates the un-mixing matrixW up toits re-scaled and permuted version.

For the sake of simplicity, the problem of simultane-ous diagonalization is converted to generalized eigen-value decomposition (GEVD) [1] as follows. Since, in

2

the case of separation,Cyy andCyy are distinct diag-onal matrices, their multiplicationRyy = CyyCyy is adiagonal matrix;

Ryy = CyyCyy (14)

= diag (E [y1y1]E [y1y1], E [y2y2]E [y2y2], · · ·)

(· · · , E [yK yK ]E [yK yK ]) .

On the other hand, from Eqs. (12) and (13)

Ryy =[WCxxWT

] [WCxxWT

](15)

= WCxx[WT W

]CxxWT (16)

If we convert the problem to GEVD, it will allow us tomake the assumption of orthogonality as follows:

• Assumption 4 : The un-mixing matrixW can becomposed of orthogonal separating vectors.

By applying Assumption 4 (WT W = I ) to Eq.(15),we have

CxxCxx = W−1RyyW. (17)

SinceRyy is a diagonal matrix, Eq.(17) conveys that:The un-mixing matrix W is the matrix which diagonal-izes CxxCxx.

Based on the result of Eq.(17) and by using diago-nalization theorem [19], the un-mixing matrixW is ob-tained as follows.The un-mixing matrix W is composed of eigenvectors ofCxxCxx.

Since, CxxCxx is a symmetric square matrix, itseigenvectors are orthogonal, and Assumption 4 is ap-proved.

The algorithm of blind source separation by simul-taneous diagonalization of two covariance matrices ofresponses of two different linear filters to the mixturesignals can now be described by the following steps .

The Algorithm Outline:

• Step 1 : Obtainx(t) andx(t) the responses of twodifferent linear filtersF and F to observed mix-ture signalsx(t).

• Step 2 : ComputeCxx andCxx the covariance ma-trices ofx(t) andx(t) as well as their multiplica-tion Rxx = CxxCxx.

• Step 3 : Find the generalized eigenvector matrixV for the generalized eigenvalue decomposition;

RxxV = V D (18)

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

−80

−70

−60

−50

−40

−30

−20

−10

0

Normalized Frequency (×π rad/sample)

Mag

nitu

de (d

B)

Magnitude Response (dB)

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

−80

−70

−60

−50

−40

−30

−20

−10

0

Normalized Frequency (×π rad/sample)

Mag

nitu

de (d

B)

Magnitude Response (dB)

Figure 1: Frequency responses of the deployed lowpassfilter (top) and highpass filter (bottom).

• Step 4 : The un-mixing matrix up to its re-scaledand permuted version is given byW = V on thecondition that all eigenvalues are real and distinct.

The quality of separation depends on how much theassumptions 2 and 3 are strong. The more different au-tocorrelation functions of the low halfbands from au-tocorrelation functions of the high halfbands results inbetter separation result.

3 Results and Discussion

To empirically evaluate potential usefulness of the pro-posed method, some evaluation tests. We have usedIIR elliptic lowpass and highpass filters to obtain lowand high halfbands of the mixtures, respectively. Inthis simulation, in all of the cases the lowpass filter is aminimum order filter with a normalized passband fre-quency of 0.2, a stopband frequency of 0.25, a pass-band ripple of 1 dB, and a stopband attenuation of 60dB, and the highpass filter is also a minimum orderfilter with a normalized passband frequency of 0.3, astopband frequency of 0.25, a passband ripple of 1 dB,and a stopband attenuation of 60 dB. The above men-tioned frequencies of the filters specifications are nor-malized with respect to sampling frequency of the sig-nals. Since, from Niquist sampling theorem the band-width of the signals is maximally considered as halfthe sampling frequency, to obtain halfbands of the mix-tures, the optimum stop frequency of both lowpass andhighpass filters isfS

4 , wherefS is the sampling fre-quency. Fig. 1 shows frequency responses of the de-ployed lowpass and high pass filters.

We have compared the performance of the proposedmethod with two well known second-order BSS al-

3

gorithms, AMUSE (Algorithm for Multiple UnknownSignals Extraction) [1] and SOBI (Second Order BlindIdentification) [2]. Both AMUSE and SOBI jointlyexploit the second order statistics (correlation matri-ces for different time delays) and temporal structure ofsources. In this simulation, the optimal delay is1 timesample in AMUSE BSS, and number of matrices whichare jointly diagonalized at each run of SOBI algorithmis 4.

We have evaluated the method with a real experi-ment. Two images have been taken of the same view ofstreet from inside the window glass of a room. Both im-ages belong to the same time of the day in different lightconditions of the room. Figure 2 (last page) shows thetaken photos and reflection removed photos as well asextracted reflection by the above mentioned BSS meth-ods. There exist many possible performance criterionin the literature [20]. Most of evaluation methods are byusing sources or global matrix (multiplication of mix-ing matrix and demixing matrixG = AW ). But here,since, mixing matrix and sources are not available, itis not possible to use most of evaluation methods men-tioned in literature. We have just real mixed observedimages and separated ones. In order to evaluate theseparation performance, we have used three numeri-cal indexes for measuring similarity of recovered im-ages obtained by each method. We have measured thecorrelation, mean absolute dsitance (MAD) and meansquared distance (MSD) between two recovered imagesobtained by each method. Each of the less correlation,the more MAD and the more MSD individually showsthe less similarity of recovered images as well as thebetter performance of separation. Table.1 shows thesenumerical evaluations result in reflection removal fromreal reflected window images of figure.2. As it is seenall the three indexes approve higher performance of theproposed method.

3.1 Robustness of reflection removal

In addition to real experiment of Fig.2, the robustnessof the method to noise and different images and reflec-tions has been artificially investigated too. To aim this,we have used a set of 15 different gray-scale images assource images and reflections. They were randomly se-

Table 1: Performance comparison of the proposedmethod with AMUSE and SOBI in reflection removalof real window glass images of figure 2.

BSS Methods

AMUSE SOBI ProposedCorrelation : 0.8704 0.8707 0.8583

MAD : 4.5892e + 03 4.5569e + 03 6.2269e + 03

MSD : 30.5779 30.5089 36.7917

0.01 0.03 0.05 0.07 0.09 .11 0.130.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

0.65

ind(G

)

Noise variance

Figure 3:The averagedind(G) Vs. variance of noisein separation mixed images by AMUSE (plus - black),SOBI (circle - pink), And SD of low and high halfbands(star - blue).

lected and mixed each other with full rank random ma-trix. The mixing matrix was the same for all methods.At each noise level, the performance of the methodshave been compared by averaged result of 200 MontCarlo runs. Since the mixing matrix was available forevaluation, we used theind(G) [20] the index based onglobal matrix. The global matrix is obtained by multi-plication of mixing and de-mixing matricesG = WA.This performance index is as follows

ind(G) = 1K(K−1)

[∑K

i

(∑K

j

|Gi,j|2

maxl

|Gi,l|2− 1

)]

[+

∑K

j

(∑K

i

|Gi,j|2

maxl

|Gl,j |2− 1

)](19)

whereK is dimension ofG. Indeed, this non-negativeindex is zero in the case of perfect separation, and itsthe smaller value indicates the more proximity to thedesired solutions.Fig.3 shows the averagedind(G) Vs. variance of noisein separation of gray-scaled images. As it is seen, theproposed method is less affected by noise level than theothers and it has the highest performance in comparisonto the others.

4 conclusion

In this paper, the reflection over window glass imageshas been removed by simultaneous diagonalization oflow and high halfbands of the mixtures. Decomposingthe mixtures to the set of low subband and high sub-band mixtures enables us to have two different views ofthe mixtures. The simultaneous diagonalization in twodifferent views results in higher performance of sepa-ration. The proposed second order technique has beencompared with AMUSE [1] and SOBI [2] in a real ex-periment. Furthermore, it robustness to noise and dif-

4

ferent sources has been compared to the others. Threedifferent numerical metrics approves dominance of theproposed BSS to the others in removing the reflectionfrom window glass images.

References

[1] L. Tong, V. C. Soon, Y. F. Huang, and R. Liu,“AMUSE: a new blind identification algorithm.” inProc. IEEE ISCAS, vol.3, pp. 1784-1787 , New Or-leans, LA, 1990.

[2] A. Belouchrani, M. G. Amin, and K. Abed-Meraim, “Direction finding in correlated noisefields based on joint block-diagonalization ofspatio-temporal correlation matrices.”IEEE SignalProcessing Letters, 4(9), September 1997.

[3] P. Common, “Independent component analysis, Anew concept?”,Signal Process., Vol. 36, pp. 287–314, 1994.

[4] A. Cichocki, S. Amari, “Adaptive Blind Signal andImage Processing.”John Wiley & Sons, New York,USA 2000.

[5] A. Hyvrinen, “ New approximations of differentialentropy for independent component analysis andprojection pursuit”,Adv. Neural Inform. Process.Systems vol. 10, pp. 273-279, 1998.

[6] A. Hyvrinen, “Survey on independent componentanalysis”,Neural Comput. vol. 2, pp. 94-128, 1999.

[7] A. J. Bell, & T. J. Sejnowski, “An information-maximization approach to blind separation andblind deconvolution”,Neural computation,vol. 7,pp. 1129–1159, 1995.

[8] H. Mathis, T. Hoff, M. Joho, “Blind separationof mixed-kurtosis signals using an adaptive thresh-old nonlinearity”, inProc. Independent ComponentAnalysis and Blind Signal Separation, Helsinki,Finland, pp. 19-22, 2000.

[9] M. Khosravy, M. R. Asharif and K. Yamashita,“A PDF-matched Modification to Stones Measureof Predictability for Blind Source Separation” ,6th International Symposium on Neural Networks,Wuhan, China, Springer-Verlags Lecture Notesin Computer Science (ISNN 2009, Part I, LNCS5551), pp. 219-228, 2009.

[10] R. Gharieb and A. Cichocki, “Second-order statis-tics based blind source separation using a bank ofsubband filters”,Digital Signal Process. Vol. 13,pp. 252-274, 2003.

[11] A. Belouchrani, K. A. Meraim, J. F. Cardoso andE. Moulines, “A blind source separation techniqueusing second-order statistics”.IEEE Trans. Sig-nal Processing, vol. 45(2), pp. 434-444, February1997.

[12] K. A. Meraim, E. Moulines, P. Loubaton, “Pre-diction error method for second-order blind identi-fication.” IEEE Trans. Signal Processing, Vol. 45,pp. 694-705, March 1997.

[13] A. Cichocki and R. Thawonmas “On-line algo-rithm for blind signal extraction of arbiturarily dis-tributed, but temporally correlated sources usingsecond order statistics.”Neural Processing Letters,Vol. 12, pp. 91–98, August 2000.

[14] S. Choi, A. Cichocki and A. Belouchrani “Secondorder nonstationary source separation.”Journal ofVLSI Signal Processing, Vol. 32, no. 1-2, pp. 93–104, Aug. 2002.

[15] W. Liu, D. Mandic, A. Cichocki, “Blind Second-Order Source Extraction of Instantaneous NoisyMixtures”, IEEE Transactions on Circuits and Sys-tems II: Express Briefs 53(9), pp. 931–935, 2006.

[16] S. Choi, A. Cichocki, A. Beluochrani, “Secondorder non-stationary source separation”,J. VLSISignal Process. vol. 32 (1/2) pp. 93-104, 2002.

[17] M. Khosravy, M.R. Asharif and K. Yamashita,“A Probabilistic Short-length Linear Predictabil-ity Approach to Blind Source Separation”,The23rd International Technical Conference on Cir-cuits/Systems, Computers and Communications(ITC-CSCC), pp. 3181–384, Yamaguchi, Japan,2008.

[18] M. Khosravy, M. R. Asharif and K. Yamashita, “APDF-matched Short-term Linear Predictability ap-proach to Blind Source Separation”,InternationalJournal of Innovative Computing, Information andControl (IJICIC), Vol. 5, No. 11(A), pp. 3677-3690, November 2009.

[19] G. L. Peterson , J. S. Sochacki, “Linear Algebraand Differential Equation”.Addison-Wesley Pub-lishing Company, Boston, 2002.

[20] E. Moreau, “A generalization of joint-diagonalization criteria for source separation”,IEEE Transactions on Signal Processing, Vol. 49,pp. 530–541, Mar 2001.

5

Imageswithreflections

AMUSE

SOBI

Proposed

Figure 2: Real window images with reflection (1st row), separated window images and reflections by AMUSE (2nd

row), by SOBI (3rd row) and by Simultaneous Diagonalization of low and high halfbnds ( bottom row).

6