MODEL-AIDED CODING USING 3-D SCENE MODELS IN MOTION-COMPENSA(3)

ICIP'2000, Vancouver, September 2000. We show that traditional waveform-coding and 3-D model-based coding are not competing alternatives but should be combined to support and complement each other. Both approaches are combined such that the generality of w

Fiveanimationparametersareestimated:translationinthreedi-rections,rotationintheimageplaneandopeningoftheclap.Allpixelsshowingaparticularobjectcontributetothecorrespondingsystemoflinearequationsfortheestimationofanimationparam-eters.Theclassi cationofthepixelstoindividualobjectsisdeter-minedexploitingknowledgefromthesynthetic3-Dscene.

4.EXPERIMENTALRESULTS

Experimentsareconductedwiththetwoself-recordednaturalCIFsequencesClapperBoardandIllumination.Rate-distortioncurvesaremeasuredbyvaryingtheDCTquantizerparameterovervalues

ableproducingandthesame.Bit-streamsPSNRvaluesareasgeneratedattheencoder.thatareThedecod-dataforthe rstintra-codedframeandtheinitial3-Dmodelareex-cludedfromtheresultsthussimulatingsteady-statebehavior,i.e.,wecomparetheinter-framecodingperformanceofbothcodecsexcludingthetransitionphaseatthebeginningofthesequence.Tospecifythecodingperformanceoftheproposedmodel-aidedcodec(MAC),wecompareittotheH.263testmodel,TMN-10(AnnexesD,F,I,J,andTenabled).

Forthespecialcaseofhead-and-shouldersequences,bit-ratesavingsof35%atthelowbit-rateendcorrespondingtoacodinggainof2-3dBPSNRarereported[1,8].Ifthelightinginthescenechangesthiscodinggainisreduced,sincethemodelframesnolongerrepresenttheoriginalvideoframescorrectly.Theaddi-tionalestimationofthelightingsituation,however,allowstoadapttheilluminationconditioninthesyntheticscenetotherealworld.The

in

Fig.3.Rate-distortionplotforthesequenceIlluminationillustrat-ingtheachievedimprovementwhenusinganilluminationestima-tor(ILE).

Fig.3forthesequenceIllumination.Duringtheacquisitionofthissequence,onenaturallightsourcewasmovedtoaltertheillumina-tionconditions.Twoexperimentsareperformed.Forthe rstone,onlytheFAPsareestimatedtocreateamodelframe.Forthesec-ondexperiment,weadditionallyestimatetheilluminationparam-etersandgeneratemotion-andillumination-compensatedmodelframes.AsshowninFig.3,thegaininPSNRforthemodel-aidedcodercomparedtotheTMN-10isabout1dBifnoilluminationcompensationisperformed.Anadditionalgainofabout1.5dBisachievedwhenexploitingilluminationinformation.

Inasecondexperiment,thein uenceofunknownobjectsinthesceneisinvestigated.Fig.4showsthe rstframesofthehead-and-shouldersequenceClapperBoard.Duringthe rst50frames,thefaceisoccludedbyanobjectthatcannotberepresentedbythe3-Dmodelsavailableatthedecoder.Asaresult,thecorrespond-

Fig.4.Frames0,11,22,33,44,and55ofthesequenceClapperBoard.

ingmodelframesdonotcontainthisadditionalobjectasshown

inFig.7a).Sincepredictionfromthepreviousdecodedframeandresidualcodingprovidesuswithrobustnessagainstmodelfail-ures,themodel-aidedcoderrepresentstheentireframecorrectly(Fig.7b)).Thecodingef ciencyofthemodel-aidedcoder,how-ever,dropsdownduringthe rstframesasshowninthetempo-ralevolutionofthePSNRinFig.5.IfthefaceisvisibleagainthemodelframecanbeexploitedandthePSNRrecoversshowinghighforthe

Fig.5.

TemporalevolutionofPSNRforthesequenceClapperBoard.BothcodersuseaDCTquantizerparameterof31.entiresequenceisdepictedinFig.6.Bit-ratesavingsof33%atthelowFig.6.Rate-distortionplotforsequenceClapperBoard.framesisillustratedinFig.7.Imageb)showsframe54encodedwiththemodel-aidedcoder,whilec)correspondstotheTMN-10coderatthesamebit-rate.Figure7d)showsaframefromtheTMN-10coderthathasthesamePSNRasthemodel-aidedframe.EventhoughthePSNRisthesame,thesubjectivequalityofthe