, . . . http://cvsp.cs.ntua.gr . , .

  ,       .  . . http://cvsp.cs.ntua.gr   . , .

, . . . http://cvsp.cs.ntua.gr . , . . CVSP --

. () 3 7 . + 2-5 . + . : . , . , / ( )

- & / & : http://cvsp.cs.ntua.gr

(McGurk & MacDonald)) () : / -

/ , :

- (King et al., Deng) : N (Articulatory Gestures, Browman & Gold)stein) ... (.. Bell, 1867))

G. Papand)reou, A. Katsamanis, V. Pitsikalis, and) P. Maragos, Ad)aptive Multimod)al Fusion by Uncertainty Compensation with Application to Aud)io-Visual Speech Recognition, IEEE Trans. ASLP, 2009 : &

() :

1 2 :

. Face detector System Overview Adaboost-based, @5 fps Image Acquisition Firewire color camera, 640x480 @25 fps (Re)initialization

Face tracking & feature extraction Real-time AAM fitting algorithms GPU-accelerated processing OpenGL implementation HMM-based backend Transcription :

; (.. ) , , , ...

: , , (Knill & Richard)s) (.. Ernst et al.) // Maragos et al., Cross-Mod)al Integration, Springer 2008 : :

: Wiener Kalman ; : :

SNR= 20d)B SNR= 5d)B : o (Gaussian Mixture Mod)el - GMM) S

. : : C X : C X

Y !

GMM S p c | x1:s p (c)1 s ,c N xs ; s ,c , s ,c C X :

p ys | xs N ys ; xs e , s , e, s S M s ,c p c | y1:s p (c ) s ,c ,m N ys ; s ,c ,m e, s , s ,c ,m e , s s 1 m 1 C X Y GMM

1- (y1 y2), 2 S ws : b (c | y ) p ( c ) p( y | c)

1:s s 1 : S p c | y1:s p (c)1 N ys ; s ,c , s ,c e ,s PoG : w N x; , N x; , w 1

S b c | y1:s p (c)1 N ys ; s ,c , w s ,c : 1 s ,c ws ,c e ,s 1 1 s ,c

EM- C

Q( ,pXCX ) [log p( X ,{C}| ) | X , pXCX ] X C X Y Q( , pXCX ) [log p(Y ,{ X , C}| ) | Y , pXCX ]

Markov () & Viterbi () - () ( frame) C1 C2

C3 C4 X1 X2 X3 X4 C1

C2 C3 C4 X1

X2 X3 X4 Y1 Y2 Y3 Y4

Mel Frequency Cepstral Coefficients (MFCCs): Pre-emphasis STFT | . | Mel-scale log( . ) DCT (e.g. SPLICE, ALGONQUIN) MFCC (VTS) X noisy f ( X clean , N ) MFCC MFCC

+ X clean X E Deng, Droppo, Acero, IEEE Tr. SAP, 2005 - 1 2 3

C1 C2 C3 X1 X2 X3 Multistream-

Product- : Asynchronous-HMM, Coupled)-HMM, Dynamic Bayesian Networks, CUAVE

. : CUAVE: 36 (30 , 6 ) 5 10 : 1500 (30x5x10) : 300 (6x5x10) babble - NOISEX HMMs (- , 8 , 1 /, ) HTK (

) AV A

/ AV-W-UC vs. A-UC 28.7 %

AV-UC vs. AV AV-W-UC vs. AV-W 20 % Product-HMM Prod)uct-HMM vs.

Multistream-HMM 1.2 % : &

: MUSCLE (NoE) & HIWIRE (STREP) - A. Katsamanis, G. Papand)reou, and) P. Maragos, Face Active Appearance Mod)eling and) Speech Acoustic Information to Recover Articulation, IEEE Trans. ASLP, 2009 -

: : () : , , MOCHA CSTR, Univ. Edinburgh

(, 1 /1 ), 460 TIMIT (2- 9 ) 30 - phoneme

37 y, x : prior : Yehia, Rubin & Vatikiotis-Bateson, Speech Comm., 1998

CCA . (CCA) CCA : : . : 40

Viterbi Markov -> Hiroya & Honda, IEEE TSAP 2004 : /

: . : HMM / MS-HMM: () : / . : Visemes ( ) ( ) MOCHA

- ( ) (//) : :

: . 51 Katsamanis et al. EUSIPCO 2008 / CVSP (. )

: X-rays, (. . ) Audiovisual Speech Inversion Articulatory Parameter Extraction Articulatory Speech Synthesis Articulatory Model

Training - : : ()

: , , : ASPI (FET) & ()

!

: http://cvsp.cs.ntua.gr

Recently Viewed Presentations

  • Warm Up: Rhyming Game Please stand in circles

    Warm Up: Rhyming Game Please stand in circles

    What rhymes with cat?" and show students a picture of a cat. You might scaffold students by changing the expected response. Before students can come up with a rhyming word on their own, they have to be able to identify...
  • Nursing Leadership & Management Theories and Styles of

    Nursing Leadership & Management Theories and Styles of

    He/she has neither a high regard for creating systems for getting the job done, nor for creating a work environment that is satisfying and motivating. ... Leader strives to increase follower's self-esteem and make job more interesting. 2. Directive leadership...
  • Adaptive Insertion Policies for High-Performance Caching Moinuddin K.

    Adaptive Insertion Policies for High-Performance Caching Moinuddin K.

    Other Policies IPC Improvement Outline Summary Questions DIP vs. LRU Across Cache Sizes DIP with 1MB 8-way L2 Cache Interaction with Prefetching mcf snippet art snippet health mpki swim mpki DIP Bypass DIP (design and implementation) Random Replacement (Success Function)...
  • POWERPOINT JEOPARDY - Lancaster High School

    POWERPOINT JEOPARDY - Lancaster High School

    * * * * * * * * * * * * * * * * * * * * * * Multiply Add and subtract Rounding Forms Place Value Digits 50 40 30 20 10 10 20 30 40...
  • Housing Opportunities for Persons with AIDS

    Housing Opportunities for Persons with AIDS

    More than one million Americans are living with HIV, with an estimated 56,300 Americans becoming infected with HIV each year. The households affected by the disease are typically among the lowest income households.
  • 2007 O que  o SciFinder? Ferramenta do CAS

    2007 O que o SciFinder? Ferramenta do CAS

    * * * * * * * * * * * This by means is not an exhaustive list of all of the smarts which drive this database, but it is some of what will make a difference in what...
  • Corporate Analysis

    Corporate Analysis

    Projet : Ecole Européenne des Langues et des Cultures Missions de l'Ecole 2 missions transversales : Promouvoir les langues et les cultures Créer un portail d'informations « langues et cultures » 5 missions spécifiques : Fédérer et ouvrir l'offre en...
  • Harvard MRSEC DMR-1420570 20172018 Crushing Soda Cans: Predicting

    Harvard MRSEC DMR-1420570 20172018 Crushing Soda Cans: Predicting

    Past experimental tests with cylindrical shells have suggested that defects strongly reduce the buckling resistance of thin-walled structures, thus, predicting the critical buckling loads is very challenging. A team at the Harvard MRSEC led by Rubinstein, Hutchinson, and Brenner investigated...