qiodZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z ddl Z ddlmZddlmZddlmZmZmZddlmZdd lmZmZdd lmZmZdd lmZdd l m!Z!dd l"m#Z# ddl$m%Z%ddl&m'Z'ddl(m)Z*ddl+m,Z,dZ1GddeZ2GddeZ3GddZ4y#e-$r7Z.ej^de.ej^de0de.dZ.[.wwxYw)a HeyGen implementation for Pipecat. This module provides integration with the HeyGen platform for creating conversational AI applications with avatars. It manages conversation sessions and provides real-time audio/video streaming capabilities through the HeyGen API. N)Enum) AwaitableCallableOptionalUnion)logger) BaseModel) AudioRawFrame ImageRawFrame StartFrame)FrameProcessorSetup) HeyGenApiNewSessionRequest) LiveAvatarApiLiveAvatarNewSessionRequest)StandardSessionResponse)TransportParams)BaseTaskManager)rtc)VideoBufferType)connect)ConnectionClosedOKz Exception: zEIn order to use HeyGen, you need to `pip install pipecat-ai[heygen]`.zMissing module: i]ceZdZdZdZdZy) ServiceTypezEnum for HeyGen service types.INTERACTIVE_AVATAR LIVE_AVATARN)__name__ __module__ __qualname____doc__rrP/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/services/heygen/client.pyrr6s(-Kr"rcdeZdZUdZegedfed<eegedfed<eegedfed<y)HeyGenCallbacksaCallback handlers for HeyGen events. Parameters: on_connected: Called when the bot connects to the LiveKit room. on_participant_connected: Called when a participant connects. on_participant_disconnected: Called when a participant disconnects. N on_connectedon_participant_connectedon_participant_disconnected)rrrr rr__annotations__strr!r"r#r%r%=sI2y.//&uio'=>>!)3%4*@!AAr"r%ceZdZdZdddddedej dedee e e fd ee d e d ed dfd ZdZded dfdZd4dZdeded dfdZd4dZdZdZded dfdZd4dZded dfdZded dfdZd4dZd4d Z d4d!Z!e"d efd"Z#e"d efd#Z$d$e%ded dfd%Z&d&Z'd'Z(ded dfd(Z)d)ed dfd*Z*d)ed dfd+Z+d,e,jZfd-Z.d,e,j^fd.Z0d/Z1d0Z2d1Z3d2e4jjfd3Z6y)5 HeyGenClientaA client for interacting with HeyGen's Interactive Avatar Realtime API. This client manages both WebSocket and LiveKit connections for real-time avatar streaming, handling bi-directional audio/video communication and avatar control. It implements the API defined in https://docs.heygen.com/docs/interactive-avatar-realtime-api The client manages the following connections: 1. WebSocket connection for avatar control and audio streaming 2. LiveKit connection for receiving avatar video and audio Attributes: HEY_GEN_SAMPLE_RATE (int): The required sample rate for HeyGen's audio processing (24000 Hz) NF)session_request service_typeconnect_as_userapi_keysessionparamsr-r. callbacksr/returnc6||ntj|_|w|u|tjk(r(t |t st jdd}n:|tjk(r't |tst jdd}|7|jtjk(rtdd}n t d}|jtjk(rt|| |_ nt|| |_ d|_ d|_ d|_||_d |_d |_d |_||_||_d|_d|_d|_d|_d|_d|_d |_d |_d |_d |_||_y) aInitialize the HeyGen client. Args: api_key: HeyGen API key for authentication session: HTTP client session for API requests params: Transport configuration parameters session_request: Configuration for the HeyGen session (optional) service_type: Type of service to use callbacks: Callback handlers for HeyGen events connect_as_user: Whether to connect using the user token or not (default: False) NzeService type is LIVE_AVATAR but session_request is not SessionTokenRequest. Ignoring session_request.zjService type is INTERACTIVE_AVATAR but session_request is not NewSessionRequest. Ignoring session_request.Shawn_Therapist_publicv2) avatar_idversionz$1c690fe7-23e0-49f9-bfba-14344450285b)r8)r1rF) rr _service_typer isinstancerrwarningrr_apir_heyGen_session _websocket _task_manager_params_in_sample_rate_out_sample_rate _connected_session_request _callbacks _event_queue _event_task _video_task _audio_task_video_frame_callback_audio_frame_callback_send_interval_next_send_time_audio_seconds_sent_transport_ready_connect_as_user)selfr0r1r2r-r.r3r/s r#__init__zHeyGenClient.__init__Zs0)4L+:X:X   &<+C{666z!<@{#'!?!?? !2IA#'  "!![%C%CC"36 # #>D#   !?!? ?!'7;DI%gw?DIBF8<   ! /#59%)"%)"   #& % !0r"cK|jj|jd{|_t j d|jj t j d|jjt j d|jjt j d|jjt jd|jjd|jjt jdy7w)NzHeyGen sessionId: zHeyGen realtime_endpoint: zHeyGen livekit URL: zHeyGen livekit token: z5Full Link: https://meet.livekit.io/custom?liveKitUrl=z&token=zHeyGen session started) r> new_sessionrFr?rdebug session_idws_url livekit_url access_tokeninforSs r# _initializezHeyGenClient._initializes%)YY%:%:4;P;P%QQ )$*>*>*I*I)JKL 1$2F2F2M2M1NOP +D,@,@,L,L+MNO -d.B.B.O.O-PQR CDDXDXDdDdCeelmqnBnBnOnOmP Q   ,- Rs)ED>DEsetupcK|jtjdy|j|_ |j d{t j|_|jj|j|j|d|_ y7[#t$r;}tjd||jd{7Yd}~yd}~wwxYww)zSetup the client and initialize the conversation. Establishes a new session with HeyGen's API if one doesn't exist. Args: setup: The frame processor setup configuration. Nz"heygen_session already initializedz::event_callback_taskzFailed to setup HeyGenClient: )r?rrW task_managerrAr^asyncioQueuerH create_task_callback_task_handlerrI Exceptionerrorcleanup)rSr_es r#r_zHeyGenClient.setups    + LL= > "// !""$ $ $ ' D #11==++D,=,=>&-. D  % ! LL9!= >,,.  !sM4C/B( B& AB(%C/&B(( C,1+C'CC'"C/'C,,C/cK |jE|jj|jjd{d|_d|_|j rB|j r5|j j|j d{d|_yyy7a7#t$r"}tjd|Yd}~yd}~wwxYww)zgCleanup client resources. Closes the active HeyGen session and resets internal state. NFzException during cleanup: ) r?r> close_sessionrXrErIrA cancel_taskrfrrgrSris r#rhzHeyGenClient.cleanups ;##/ii--d.B.B.M.MNNN'+$"'D$6$6((44T5E5EFFF#' %7 O G ; LL5aS9 : : ;sSC>B(B$AB(B& B(!C$B(&B(( C1C CCCframeaudio_chunk_sizecBK|jrtjdytjd|jjxs |j|_|jj xs |j |_||jz dz |_tjd|j|jd{|jd{|j|jjy7B7,w)a|Start the client and establish all necessary connections. Initializes WebSocket and LiveKit connections using the provided configuration. Sets up audio processing with the specified sample rates. Args: frame: Initial configuration frame containing audio parameters audio_chunk_size: Audio chunk size for output processing zheygen client already startedNzHeyGenClient startingzHeyGenClient send_interval: )r@rrWrBaudio_in_sample_raterCaudio_out_sample_raterDrN _ws_connect_livekit_connect_call_event_callbackrGr&)rSrnros r#startzHeyGenClient.starts ?? LL8 9  ,.#||@@^ED^D^ $ B B aeFaFa/$2G2GG1L 3D4G4G3HIJ   ##%%% !!$//">">? !%s$CDDD0D1+DDcKtjd|jd{|jd{|j d{y7577 w)zStop the client and terminate all connections. Disconnects from WebSocket and LiveKit endpoints, and performs cleanup. zHeyGenVideoService stoppingN)rrW_ws_disconnect_livekit_disconnectrhr]s r#stopzHeyGenClient.stopsU  24!!###&&(((lln $(s3)A'A!A'A#A'A%A'#A'%A'cK |jrtjdytjdt|jj d{|_d|_|jj|jd|_ y7A#t$r+}tj|d|d|_Yd}~yd}~wwxYww) z%Connect to HeyGen websocket endpoint.z"HeyGenClient ws already connected!NzHeyGenClient ws connecting)uriTHeyGenClient_Websocketnamez initialization error: ) r@rrWwebsocket_connectr?rYrErArd_ws_receive_task_handler _receive_taskrfrgrms r#rtzHeyGenClient._ws_connect s # AC LL5 7$5((//%DO#DO!%!3!3!?!?--/6N"@"D   # LLD6!8< ="DOO #sKC!B"C8B"B AB"C B"" C+!C CCCcZK|jr_ |jjd{}tj|}|j |d{|jr^yy7A7#t $rYyt$r"}tjd|Yd}~yd}~wwxYww)z#Handle incoming WebSocket messages.Nz$Error processing WebSocket message: ) rEr@recvjsonloads_handle_ws_server_eventrrfrrg)rSmessageparsed_messageris r#rz%HeyGenClient._ws_receive_task_handlersoo  $ 4 4 66!%G!422>BBB oo6B%   CA3GH sb B+A3A/-A3A1A3 B+-B+/A31A33 B(<B+>B(B#B+#B((B+eventcK|jd}|dk(rtjd|ytjd|yw)z&Handle an event from HeyGen websocket.typez agent.statez'HeyGenClient ws received agent status: z(HeyGenClient ws received unknown event: N)getrrWtrace)rSr event_types r#rz$HeyGenClient._handle_ws_server_event,sDYYv&  & LLB5'J K LLCJ<P QsA A cK d|_|jr"|jjd{d|_y7 #t$r$}t j |d|Yd}~2d}~wwxYw#d|_wxYww)z*Disconnect from HeyGen websocket endpoint.FNz disconnect error: )rEr@closerfrrgrms r#ryzHeyGenClient._ws_disconnect4sq ##DOoo++---#DO . : LLD6!4QC8 9 9 :#DOsIA?0AAAA?A A0 A+&A3+A00A33 A<<A?rc8K|jstj|dy |jr7|jj t j |d{yy7#t$r}tjd||d}~wwxYww)z#Send a message to HeyGen websocket.z$ websocket is not connected anymore.Nz+Error sending message to HeyGen websocket: ) rErrWr@sendrdumpsrfrg)rSrris r#_ws_sendzHeyGenClient._ws_send?s LLD6!EF G  oo**4::g+>????  LLFqcJ KG s:&B=A/&A-'A/+B-A// B8BBBevent_idcKtjd|j|jd|dd{y7w)zInterrupt the avatar's current action. Stops the current animation/speech and returns the avatar to idle state. Useful for handling user interruptions during avatar speech. zHeyGenClient interruptzagent.interruptrrN)rrW_reset_audio_timingrrSrs r# interruptzHeyGenClient.interruptKsB  -.   "mm)$    s=AAAcKtjd|jdtt j dd{y7w)zStart the avatar's listening animation. Triggers visual cues indicating the avatar is listening to user input. z"HeyGenClient start_agent_listeningzagent.start_listeningrN)rrWrr*uuiduuid4r]s r#start_agent_listeningz"HeyGenClient.start_agent_listeningZsA  9:mm/ -    sAA A Ac~K|jdttjdd{y7w)zoStop the avatar's listening animation. Returns the avatar to idle state from listening state. zagent.stop_listeningrN)rr*rrr]s r#stop_agent_listeningz!HeyGenClient.stop_agent_listeninggs4 mm. -    s 3=;=cd|_y)zHIndicates that the output transport is ready and able to receive frames.TN)rQr]s r#transport_readyzHeyGenClient.transport_readyss $r"c|jS)z`Get the output sample rate. Returns: The output sample rate in Hz. )rDr]s r#out_sample_ratezHeyGenClient.out_sample_ratews$$$r"c|jS)z^Get the input sample rate. Returns: The input sample rate in Hz. )rCr]s r#in_sample_ratezHeyGenClient.in_sample_rates###r"audiocKtj|jd}|jd||dd{|j d{y77w)zSend audio data to the agent speak. Args: audio: Audio data in base64 encoded format event_id: Unique identifier for the event zutf-8z agent.speak)rrrN)base64 b64encodedecoder_write_audio_sleep)rSrr audio_base64s r# agent_speakzHeyGenClient.agent_speaksf''.55g> mm%%$    %%'''  (s"=A!AA!AA!A!c d|_d|_y)z%Reset audio timing control variables.r:rN)rPrOr]s r#rz HeyGenClient._reset_audio_timings#&  r"cK|jdkrF|xj|jz c_tj|jz|_ytj}t d|j|z }|dkDr=t j|d{|xj|jz c_ytj|jz|_y7Kw)z7Simulate audio playback timing with appropriate delays.g@Nr)rPrNtime monotonicrOmaxrbsleep)rS current_timesleep_durations r#rzHeyGenClient._write_audio_sleeps  # #c )  $ $(;(; ; $#'>>#3d6I6I#ID ~~' Q 4 4| CD A --/ / /  D$7$7 7 #'>>#3d6I6I#ID  0sB C0"C.#A C0chK|j|jd|dd{y7w)zSend signaling that the agent has finished speaking. Args: event_id: Unique identifier for the event zagent.speak_endrN)rrrs r#agent_speak_endzHeyGenClient.agent_speak_ends5   "mm)$    s (202participant_idcKtjd|d|j||_|jtj dy|j r||j jvr|j j|}|jjD]}|jtjjk(s+|j8tjd|jtj |j|j}|j"j%|j'|d|_yyyyw) zCapture audio frames from the HeyGen avatar. Args: participant_id: Identifier of the participant to capture audio from callback: Async function to handle received audio frames zcapture_participant_audio: z, sample_rate: NLTrying to capture more than one audio stream. It is currently not supported.z+Starting audio capture for existing track: track sample_rateHeyGenClient_Receive_Audior)rrWrCrMrKr= _livekit_roomremote_participantstrack_publicationsvalueskindr TrackKind KIND_AUDIOrsid AudioStreamrArd_process_audio_frames)rSrcallback participant track_pub audio_streams r#capture_participant_audioz&HeyGenClient.capture_participant_audios<  ).)9I]I]H^ _ &."    ' NN^     .D4F4F4Z4Z"Z,,@@PK(;;BBD  >>S]]%=%==)//B]LL#Ny}}o!^_#&??'oo4;O;O$L(,'9'9'E'E22<@Gc(F(D$ #[ sCE% E%!BE%cKtjd|||_|jtjdy|j r||j j vr|j j |}|jjD]}|jtjjk(s+|j8tjd|jtj|j}|j j#|j%|d|_yyyyw)zCapture video frames from the HeyGen avatar. Args: participant_id: Identifier of the participant to capture video from callback: Async function to handle received video frames zcapture_participant_video: Nrz+Starting video capture for existing track: HeyGenClient_Receive_Videor)rrWrLrJr=rrrrrrr KIND_VIDEOrr VideoStreamrArd_process_video_frames)rSrrrr video_streams r#capture_participant_videoz&HeyGenClient.capture_participant_videos  2>2BCD%-"    ' NN^     .D4F4F4Z4Z"Z,,@@PK(;;BBD  >>S]]%=%==)//B]LL#Ny}}o!^_#&??9??#CL'+'9'9'E'E22<@Gc(F(D$ #[ sCE  E A8E streamcHK tjd|23d{} |j}t|j}t ||j d}|jr%|jr|j|d{u7p7#t$r"}tjd|Yd}~d}~wwxYw6n.#t$r"}tjd|Yd}~nd}~wwxYwtjdy#tjdwxYww)z)Process audio frames from LiveKit stream.z"Starting audio frame processing...N)rr num_channelszError processing audio frame: zError processing audio frames: zAudio frame processing ended.) rrWrnbytesdatar rrQrMrfrg)rSr frame_event audio_frame audio_dataris r#rz"HeyGenClient._process_audio_framess  ; LL= >%+ G GkG"-"3"3K!&{'7'7!8J"/($/$;$;%&#K ,,1K1K"88EEE GF GLL#A!!EFFG&,  @ LL:1#> ? ? @ LL8 :FLL8 :sD"CCBCCA%B B BCCB B?B:5C:B??CD C/ C*%D*C//D2D"DD"cK tjd|23d{} |j}|jtj k7r|j tj }tt|j|j|jfd}|jdz|_ |jr%|jr|j|d{77#t $r"}tj"d|Yd}~d}~wwxYw6n.#t $r"}tj"d|Yd}~nd}~wwxYwtjdy#tjdwxYww) z)Process video frames from LiveKit stream.z"Starting video frame processing...NRGB)imagesizeformatiz)Error processing individual video frame: zError processing video frames: zVideo frame processing ended.)rrWrnrrRGB24convertr rrwidthheight timestamp_usptsrQrLrfrg)rSrr video_frame image_frameris r#rz"HeyGenClient._process_video_framessS ; LL= >%+ R RkR"-"3"3K#''?+@+@@&1&9&9/:O:O&P #0#K$4$45)//1C1CD$#K '2&>&>$&FKO,,1K1K"88EEE# R"F RLL#LQC!PQQR%&,( @ LL:1#> ? ? @ LL8 :FLL8 :sE<DDC*DDB?C.$C,%C.)D*D,C.. D7DDDDE" E 'E?E"E  E" E<"E99E<c K tjdjjt j _j jddtjffd }j jddtjdtjdtjffd }j jd dtjdtjdtjfd }j jd dtjffd }jsjjnjj}j jjj|d {tjdj jtjdj j j"tjdt%j j&j j&j)D]}tjd|j"d|j*j-j.j0|j*|j2j)D]>}tjd|j"d|j4d|j@y 7#t6$r)}tj8d|d _Yd }~y d }~wwxYww)zConnect to LiveKit room.z-HeyGenClient livekit connecting to room URL: participant_connectedrc ztjd|jd|j|jj D]>}tjd|jd|j d|j@jjj|jy)NzParticipant connected - SID: , Identity: zAvailable track - SID: , Kind: , Name: ) rrWridentityrrrrrvrGr')rrrSs r#r'z?HeyGenClient._livekit_connect..on_participant_connected9s 3KOO3DLQ\QeQePfg"-!?!?!F!F!HILL1)--HXX`ajaoao`pq))OO<R>Rr"track_subscribedr publicationc|jtjjk(rjuj it jd|jtj|}jjj|d_y|jtjjk(rjjut jd|jtj |j"}jjj%|d_yyyy)Nz+Creating video stream processor for track: rrz+Creating audio stream processor for track: rr)rrrrrLrJrrWrrrArdrrrMrKrrCr)rrrrrrSs r#on_track_subscribedz:HeyGenClient._livekit_connect..on_track_subscribedFs(JJ#--":"::22>((0LL#N{N_!`a#&??5#9L'+'9'9'E'E22<@Gc(F(D$JJ#--":"::22>((0LL#N{N_!`a#&??DL`L`#aL'+'9'9'E'E22<@Gc(F(D$ 1?;r"track_unsubscribedcbtjd|jd|jy)NzTrack unsubscribed - SID: r)rrWrr)rrrs r#on_track_unsubscribedz.on_track_unsubscribedas+  9+//9J(SXS]S]R^_`r"participant_disconnectedctjd|jd|jj j j |jy)Nz Participant disconnected - SID: r)rrWrrrvrGr()rrSs r#r(zBHeyGenClient._livekit_connect..on_participant_disconnectedisQ 6{6G|T_ThThSij))OO??AUAUr"Nz(Successfully connected to LiveKit room: zLocal participant SID: zNumber of remote participants: zExisting participant - SID: rzExisting track - SID: rrzLiveKit initialization error: )rrWr?rZrRoomronRemoteParticipantTrackRemoteTrackPublicationrRlivekit_agent_tokenr[rrlocal_participantrlenrrrrvrGr'rrrfrg) rSr'rrr(r[rrris ` r#ruzHeyGenClient._livekit_connect1s[ & LL?@T@T@`@`?ab "%D     " "#: ; c6K6K <    " "#5 6 yy  77 !22 7 4   " "#7 8 ayy a 77 a!22 a9 a   " "#= > 9N9N ? ,,$$88))66  $$,,T-A-A-M-M|\ \ \ LLCDDVDVD[D[C\] ^ LL243E3E3W3W3[3[2\] ^ LL1#d6H6H6\6\2]1^_  $11EELLN   2;??2CR>R"-!?!?!F!F!HILL0x GWW_`i`n`n_op  ]( & LL9!= >!%D   &sBM;F>MME>MM;M M8M3.M;3M88M;c`K tjd|jr4|jj |jd{d|_|j r4|jj |j d{d|_|j rTtjd|j jd{d|_tjdyy77n7&#t$r"}tjd|Yd}~yd}~wwxYww)zDisconnect from LiveKit room.zStarting LiveKit disconnect...NzDisconnecting from LiveKit roomz+Successfully disconnected from LiveKit roomzLiveKit disconnect error: ) rrWrJrArlrKr disconnectrfrgrms r#rzz HeyGenClient._livekit_disconnects ; LL9 :((44T5E5EFFF#' ((44T5E5EFFF#' !! >?((33555%)" JK "GG 6 ; LL5aS9 : : ;sfD.A D C:?D C<A DC> D8D.:D<D>D D+ D&!D.&D++D.cB|jj|g|y)z,Queue an event callback for async execution.N)rH put_nowait)rSrargss r#rvz!HeyGenClient._call_event_callbacks $$h%6%67r"queuecK |jd{^}}||d{|j87$7w)z1Handle queued callbacks from the specified queue.N)r task_done)rSr rr s r#rez#HeyGenClient._callback_task_handlers?&+iik 1 XD/ ! ! OO  1 !sA<A>AA)r4N)7rrrr r*aiohttp ClientSessionrrrrrrr%boolrTr^r r_rhr intrwr{rtrdictrryrrrrrpropertyrrrrrrrrrrrrrrrurzrvrbrcrer!r"r#r,r,Ks (\`.2 %[0[0&& [0  [0 "%(CEV(V"WX [0{+[0#[0[0 [0z .!!4!!0;"@@s@t@.#$ R4RDR # d t           %%%%$$$(u((($! J$  c  d  cPT>cPT8;#//;2;#//;:]&~;08'--r"r,)5r rbrrrrenumrtypingrrrrrlogururpydanticr pipecat.frames.framesr r r "pipecat.processors.frame_processorr .pipecat.services.heygen.api_interactive_avatarrr&pipecat.services.heygen.api_liveavatarrr pipecat.services.heygen.base_apir!pipecat.transports.base_transportr"pipecat.utils.asyncio.task_managerrlivekitr"livekit.rtc._proto.video_frame_pb2rwebsockets.asyncio.clientrrwebsockets.exceptionsrModuleNotFoundErrorrirgrfHEY_GEN_SAMPLE_RATErr%r,r!r"r#r%s 77 CWE=>,BF8  $  Bi Bf f ;,FLL;qc"#FLLXY &qc* ++,s,B''C#,2CC#