Optical Engineering 4512, 127201 December 2006 Depth from mo(2)

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

LinandChang:Depthfrommotionanddefocusblur

blursimultaneouslyfromanimagepair.Amodelwaspro-posedandusedtoobtaintherelationshipbetweentheaf netransformationandthelevelofblur.Thederivedequationwasthensolvedusinganiterativeapproach.Althoughtheexperimentalresultswerepresentedwithseveralrealandcomplicatedimages,onlythechangesindefocusduetothemotionalongtheopticalaxiswereconsidered.Motionblurcausedbytherelativemotionbetweenthecameraandthesceneduringnonzerocameraexposuretimewasnotexplic-itlytakenintoaccount.Furthermore,theirformulationbasedonanaf nemodelrequirestwoimagesforbothde-focusblurandaf nemotionrecovery.Inthecurrentre-search,wearemoreinterestedindepthrecoveryfromasingleblurredimage.Theimageblurismodeledasopticaldefocusand/ortheimagedegradationcausedbytheobjectmotionduringnonzeroimageacquisitiontime.Thus,somepracticalissuesrelatedtotheproblem,suchastheimageformationandmotionblur,arenotcompletelyaddressedRef.15.

InFox’searlywork,16theconceptofrangefromablur-acutestereopairwasintroduced.Aspecialhardwarewasdesignedtocaptureboththeblurredandunblurredimagesfordepthmeasurements.Itwasbasicallyaconventionalstereomethodincorporatedwithtranslationmotionblurin-formation.Moreover,theauthormainlydescribedthethe-oreticalaspectoftheproposedideawithoutexperimentalresults.Recentworkrelatedto3-DreconstructionfromblurimagecueswasgivenbyFavaro,Burger,andSoatto.17Theyproposedavariationalapproachtorecoverthedepthmapandradianceofasceneusingadefocusedandmotion-blurredimagesequence.Theblurinformationwasusedtominimizethediscrepancybetweenthemeasureddefocusedimagesandtheoutputsynthesizedbythediffusionprocess.Althoughanoff-the-shelfcameracanbeused,multipleim-agesarestillrequiredforanisotropicdiffusion.Further-more,theirapproachisonlyvalidundertheassumptionofLambertiansurfacewithuniformillumination.

Incontrasttothepreviouswork,weuseonlyasinglemotionanddefocusblurredimagefordepthrecovery.Depthcalculationisaccomplishedbyidentifyingtheblurextentcausedbylateralcameramotionanddistance-varyingdefocus.Inthisresearch,animageblurmodelforuniformlinearmotionandopticaldefocusisformulatedbasedongeometricoptics.Theblurextentoftheimageisthenestimatedandusedfordepthmeasurement.

Theworkorganizedasfollows.Section2introducesthetheoryofimageformationwithdefocusandmotionblur.InSec.3,wepresentthemethodsforblurparameterestima-tionandcamerafocuscalibration.Section4providestheimplementationdetailsandexperimentalresults.Severaltypesofrealimageswithdifferentexperimentalsetupsareusedtovalidatetheproposedtechnique.Finally,Sec.5concludestheworkandpointstopossibledirectionsoffu-tureresearch.

2TheoreticalFormulation

Asimplecameramodelconsistingofathinlensandanimageplaneisusedtoderivesomefundamentalcharacter-isticsoffocusingbasedongeometricoptics.TheexistingalgorithmsfordeterminingobjectdepthfromimagefocusordefocususuallyformulatetheopticalblurduetotheFig.1Cameramodelfordefocusblur.

thiswork,boththeimageblurcausedbythecameramotionandopticaldefocusaretakenintoaccount.We rstmodelthemotionbluronly,i.e.,theobjectisinfocusifthereisnorelativemotionbetweenthecameraandthescene,andthenconsiderthedefocusbluraswell.

2.1GeometricCameraModel

Inthissectionwebrie ydescribethecameramodelforthegeometricimageformationprocess.InterestedreaderscouldrefertoRefs.21and22formoredetailedinforma-tiononthistopic.AsillustratedinFig.1,acamerasystemconsistingofasingleconvexlenswithfocallengthfisconsidered.TherelationshipbetweenthepositionofapointPinthesceneandthecorrespondingfocusedpositionP intheimageisgivenbythewellknownlensformula111+=,pqf

1

wherepisthedistanceoftheobjectfromthelensononeside,andqisthedistanceoftheimageplanefromthelensontheotherside.

IfweconsideranobjectpointQwithadistancezfromthelens withoutlossofgenerality,weassumezisgreaterthanp ,thenEq. 1 canberewrittenas111+=,zz f

2

wherez isthedistanceofthevirtualfocuspositionfromthelens.Furthermore,thecorrespondingimagepointofQismodeledasablurcirclecenteredatCaccordingtogeo-metricoptics.FromEq. 2 andtherelationdq z =,Dz

thediameterdoftheblurcircleisgivenbyd=Dq

3

111

,fzq

4

whereDisthediameterofthelensandqisthedistancefromthelenstotheimageplane.23Sincethedistanceqis

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

d=

Dpf11

.

p fpz

5

Thatis,thesizeoftheblurcircleforanyscenepointlo-catedatadistancezfromthelenscanbecalculatedbyEq. 5 .

SimilarderivationsofEq. 5 canbefound,forexample,inRefs.20and24.Itisclearthattheblurcirclediameterddependsonlyonthedepthzif xedcamerasettingsofD,f,andparegiven.Inthiswork,wemakeseveralfurtherobservationsontheimagingmodel.First,Eq. 5 canberewrittenasd=c wherec=

Df

.p f

7

s =s cos +sin tan .

10

cp,z

6

Fig.2Cameramodelforpuremotionblur.

FromEq. 6 ,thesizeoftheblurcircleislinearlyrelatedtotheinversedistanceoftheobject.Moreover,theblurcirclediameterd→0asthedistancez→p,andconverselyd→casz→ .Inthelattercase,theconstantcgivenbyEq. 7 representsthemaximumdiameteroftheblurcirclewhentheobjectapproachesin nity.

IfwerewriteEq. 6 asafunctionofd,thenthedepthzoftheobjectisgivenbyz=cp

.c d

8

SubstituteEq. 9 withEqs. 10 and 1 ,andthedistancepisgivenby

s

p=1+ cos +sin tan f,

x

11

whichisindependentofthedistanceqbetweenthelensandtheimageplane.

Sincetheangle isusuallysmallifonlythecentralpartoftheimageisconsidered,especiallyforp f,Eq. 11 canbeapproximatedbyp=1+

s

cos f.x

Inthepreviousequation,thefocusingrangepcorrespond-ingtoa xedlenspositionatqfromtheimageplanecanbeobtainedfromcamerafocuscalibration.Thisisalsoanim-portantobservation,sincetheconstantccanbeeithercom-puteddirectlyfromEq. 7 withaperturediameterofthecameraormeasuredfromthesizeoftheblurcircleastheobjectapproachingin nity.Consequently,thedistancezcanbeacquiredfromEq. 8 byobservingtheblurcirclesize.

2.2ModelforMotionBlur

Differentfromopticaldefocus,motionblurisaresultoftherelativemotionbetweenthecameraandthesceneduringtheimagingprocess.AsshowninFig.2,supposeanobjectmovesadistancesfromPtoQ,andtheanglebetweenthemotiondirectionoftheobjectandtheimageplaneofthecamerais .Ifwefurtherassumethatthe3-DpointQisinthesamedepthof eldasthepointP,sothatthereisnodefocusblurassociatedwiththeimagepointQ whentheobjectPisinfocus.ThenthedistancebetweenQ andthelensisgivenbyq,andwehaves x=,pq

9

12

SupposethedistancebetweenQ andtheimagecenteris ,thentan =

11

=

qfpf

ifp f.

Foracharge-coupleddevice CCD sensorsizeof11mm

diagonalandacamerafocallengthof9mm,tan islessthan7 10 2mmifQ appearsinsidethecentral20%oftheimage. Thus,thedepthcanbecomputedbytherelativedisplacementandmovingdirectionoftheobjectmotion.Furthermore,iftheangle isnotsigni cant,i.e.,therela-tivemotionisapproximatelyparalleltotheimageplane,thenEq. 12 canbewrittenasp=1+

sf.x

13

bysimilartriangles.Let betheanglebetweenQQ andtheopticalaxisofthelens,thenthedisplacementscanbeItshouldbenotedthatifasimplepinholecameramodelisconsidered,thedepthpisgivenbysf/x.Thus,thereexistsadifferenceofthefocallengthfwhencomparedtothecameramodelwitha niteaperture.

Now,supposetheobjectundergoesuniformlinearmo-tionduringtheimagingprocesswithcameraexposuretimeofTseconds,i.e.,theobjectmovesfromPtoQatthe

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

s x=,zq

18

wherexisthedifferencebetweenP andQ ,thecentersoftheblurcirclesassociatedwithPandQ,respectively.SimilartothederivationsofEqs. 11 and 12 ,thedis-tancezoftheobjectisgivenbyz=

spf cos +sin tan vTpf cos +sin tan

=,

x p f x p f

19

or

Fig.3Cameramodelforbothmotionanddefocusblur.

z=

vT

cos f,p=1+x

spfcos vTpfcos

=

x p f x p f

if 0deg, 20

14

andp=1+

vT

f,x

15

wherepisthefocusingrangecorrespondingtothedistanceqbetweenthelensandtheimageplane.Thatis,thedis-tancezcanbeobtainedprovidedthatalloftheparametersf,v,p,T,x,and areknown.Forthecasethattheobject’smotiondirectionisparalleltotheimageplane,Eq. 20 canbefurthersimpli edtoz=

spfvTpf

=,

x p f x p f

21

respectively,wherexcorrespondstotheblurextentduetotherelativemotion.Sincethefocallengthandexposuretimearegivenbythecamerasettings,thedistanceoftheobjectcanbeobtainedfromEqs. 14 or 15 ifthemovingspeedvoftheobjectisknownandtheextentofmotionblurxcanbeidenti ed.Moreover,foranytwoobjectswiththesamemotiondirectionandmovingspeedwithrespecttothecamera,theirrelativedepthcanbecomputedbyp2x1vTcos +x2

=,p1x2vTcos +x1

16

providedthattheobjectsarelocatedwithinthesamedepthof eld i.e.,out-of-focusblurcanbeignored .

Inmostpracticalsituations,thedisplacementoftheob-jectss=vTismuchlargerthantheblurextentsx1andx2intheimage.Thus,Eq. 16 canbefurtherreducedtop2x1

=.p1x2

17

bysetting aszero.DifferentfromEqs. 11 – 15 ,whichdealwiththecasewithoutdefocusblur,theparameterpinEqs. 19 – 21 representsthefocusingrangewithrespecttoa xedlenspositionwiththedistanceqfromtheimageplane.Itcanbeobtainedfromfocuscalibrationfor xedparametersettingsofthecamera.25However,theparameterxrepresentingthedisplacementbetweenthecentersofde-focusblurcirclesappearsmoredif culttoidentify.

FromEq. 20 ,itisclearthattherelativedepthoftwoobjectsatdifferentdistancesisgivenbyz2x1

=,z1x2

22

Thatis,exceptfortheextentsofmotionblur,neitherthecameraparametersnorthemovingspeedarerequiredforthecalculationofrelativedepth.Althoughtheimagingmodelassumestheobjectsareinmotion asshowninFig.2 ,therelativemotioncanbeachievedbymovingthecam-eralaterally.

2.3ModelforBothMotionandDefocusBlur

AsshowninFig.3,ifweconsideranobjectPwithadistancezfromthecamera assumingzislargerthanp ,thenthecorrespondingimagewillbeablurcirclecenteredatP andthecirclesizeisgivenbyEq. 4 .SupposetheobjectmovesfromPtoQwithadisplacementsandanwithouttheassumptionsx1,x2 srequiredforEq. 17 .FromboththeEqs. 17 and 22 ,therelativedepthoftwoobjectsisinverselyproportionaltothemotionblurextents,providedthattherelativemotiondirectionisthesame.Ifweconsideraspecialcasewheretheobjectisveryfaraway,orthefocusingrangeissetasin nity i.e.,p→ ,thenEq. 19 canbesimpli edasz=

vTf

cos +sin tan ,x

23

whichcorrespondstoapinholecameramodel.Inthiscase,theparameterxisnotonlythedistancebetweenthecentersoftheblurcircles,butalsotheblurextentwithoutthepres-enceofdefocusblur.

Ifboththedefocusandmotionblurareconsidered,theblurextentcausedbymovinganobjectfromPtoQisgivenbythedisplacementofthecentersofblurcirclesplus

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

LinandChang:Depthfrommotionanddefocusblur

diametersoftheblurcirclesassociatedwithP andQ ,respectively.Thentheblurextentcausedbybothmotionandopticaldefocusisx+

d1+d2

,2

24

2.4PointSpreadFunction

Theobservedimageg x,y iscommonlymodeledastheoutputofa2-Dlinearspace-invariantsystem,whichischaracterizedbyitspointspreadfunction PSF ,h x,y .Moreprecisely,thedegradedimageg x,y canbeformu-latedasg x,y =

wherexisthedisplacementoftheblurcircles,andd1andd2aregivenbyEq. 4 .IfthedepthchangebetweenPandQisnotsigni cant,Eq. 24 canbeapproximatedbyx+d,wheredistheaverageofd1andd2.Inpractice,theGaussianblurmodelisextensivelyused4,6,18,23,26andtheblurparameterisgivenby =kd,wherekisacameraconstant.

Supposetheobservedblurextentisx intheimage,i.e.,x =x+kd,thenthedepthzcanbederivedifEq. 20 canbewrittenasafunctionofx .SubstitutingEq. 4 withEqs. 1 and 20 gives

fxDpf11

=D,d=

p fpzp fscos

h x ,y f , d d , 30

whereh x,y isalinearshift-invariantPSF,andf x,y is

theidealimage.Ifweconsideralosslesscamerasystem,then

h x,y dxdy=1, 31

25

andthePSFofimagedegradationcausedbyout-of-focusblurcanbewrittenasapillboxfunction4d222

,ifx+y

4,h x,y = d2

0,otherwise

wheredispositiveifz pandnegativeifz p.Forthe

formercase,Eq. 25 canberewrittenasx=

scos kDf

x ,

scos kDp f

32

26

sincex =x+kd.Finally,thedepthzoftheobjectisgivenbyz=

scos kD pf vTcos kD pf

=,

x p f kDfx p f kDf

27

accordingtogeometricoptics,wheredisthediameterof

theblurcirclegivenbyEq. 5 .

Foranimagewithanidealstepedge,thecorrespondingblurimageisgivenbythe1-Dconvolutiong x =f x *l x ,

wherel x isthelinespreadfunction4d if x d 4x,

2.l x = d2

0,otherwise

33

fromEq. 20 .Similarly,wehavez=

scos +kD pf vTcos +kD pf

=,

x p f +kDfx p f +kDf

34

ifzissmallerthanp.

Therearesomeobservationsfromthepreviousequa-tions.First,ifapinholecameramodelisconsidered i.e.,D→0 ,thenx=x byEq. 26 .Consequently,Eq. 27 isreducedtoEq. 20 .Second,ifthedisplacementoftheobjectismuchlargerthanthesizeofthecameraaperture,i.e.,s D,thenxcanbeapproximatedbykDf

,x x

p f

28

However,duetoaberrations,diffraction,andnonidealitiesofthelenses,aGaussianPSFiscommonlyusedinsteadofEq. 32 ,andthelinespreadfunctionisgivenbyx2

l x =

2 exp 2 2,

1

35

where isthespreadparameter.

Itisshownthat isproportionaltotheblurcircledi-ameterd,i.e.,

whichisindependentoftheangle .Furthermore,itcanbereducedtokDf

,x x p

29

=kdfork 0, 36

andcanbedeterminedbyanappropriatecalibrationprocedure.27Thus,Eq. 8 canberewrittenasz=c p

,c

37

iff p.FromEq. 29 ,itisclearthatdefocusblurisnegligibleiftheaperturesizeandfocallengtharerelativelysmallcomparedtothedisplacementoftheobjectandthe