qi58.dZddlZddlmZddlmZmZddlZddlm Z ddl m Z ddl m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZmZdd lm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&m'Z'm(Z(m)Z)dd l*m+Z+ddl,m-Z-dZ.eGdde+Z/Gdde Z0y)a HeyGen implementation for Pipecat. This module provides integration with the HeyGen platform for creating conversational AI applications with avatars. It manages conversation sessions and provides real-time audio/video streaming capabilities through the HeyGen API. N) dataclass)OptionalUnion)logger)create_stream_resampler) AudioRawFrameBotStartedSpeakingFrame CancelFrameEndFrameFrame ImageRawFrameOutputAudioRawFrameOutputImageRawFrameOutputTransportReadyFrameSpeechOutputAudioRawFrame StartFrameTTSAudioRawFrameTTSStartedFrameUserStartedSpeakingFrameUserStoppedSpeakingFrame)FrameDirectionFrameProcessorSetup) AIService)NewSessionRequest)LiveAvatarNewSessionRequest)HEY_GEN_SAMPLE_RATEHeyGenCallbacks HeyGenClient ServiceType)ServiceSettings)TransportParamsgffffff?ceZdZdZy)HeyGenVideoSettingsz&Settings for the HeyGen video service.N)__name__ __module__ __qualname____doc__O/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/services/heygen/video.pyr#r#8s0r)r#c PeZdZUdZeZeed<dddddedejde e e e fde ed e ed df fd Zd effd ZfdZdZdefdZdefdZdefdZdefdZdeffd Zdeffd Zdeffd Zde de!ffd Z"d e#fdZ$dZ%dZ&d Z'd!Z(de)fd"Z*d#Z+xZ,S)$HeyGenVideoServiceaA service that integrates HeyGen's interactive avatar capabilities into the pipeline. This service manages the lifecycle of a HeyGen avatar session by handling bidirectional audio/video streaming, avatar animations, and user interactions. It processes various frame types to coordinate the avatar's behavior and maintains synchronization between audio and video streams. The service supports: - Real-time avatar animation based on audio input - Voice activity detection for natural interactions - Interrupt handling for more natural conversations - Audio resampling for optimal quality - Automatic session management Args: api_key (str): HeyGen API key for authentication session (aiohttp.ClientSession): HTTP client session for API requests session_request (NewSessionRequest, optional): Configuration for the HeyGen session. Defaults to using the "Shawn_Therapist_public" avatar with "v2" version. _settingsN)session_request service_typesettingsapi_keysessionr.r/r0returnc td}||j|t| dd|i|||_||_d|_d|_t|_ d|_ ||_ ||_ d|_ d|_d|_y)aInitialize the HeyGen video service. Args: api_key: HeyGen API key for authentication session: HTTP client session for API requests session_request: Configuration for the HeyGen session service_type: Service type for the avatar session settings: Runtime-updatable settings. HeyGen has no model concept, so this is primarily used for the ``extra`` dict. **kwargs: Additional arguments passed to parent AIService N)modelr0Frr()r apply_updatesuper__init___api_key_session_client _send_taskr _resampler_is_interrupting_session_request _service_type_other_participant_has_joined _event_id_audio_chunk_size) selfr1r2r.r/r0kwargsdefault_settings __class__s r*r8zHeyGenVideoService.__init__Xs*+6    ) )( 3 ="2=f=  /3 2613 % /)-2*!"r)setupc Kt||d{t|j|jt ddddt |j|jt|j|j|jd|_ |jj|d{y77w)aSet up the HeyGen video service with necessary configuration. Initializes the HeyGen client, establishes connections, and prepares the service for audio/video processing. This includes setting up audio/video streams, configuring callbacks, and initializing the resampler. Args: setup: Configuration parameters for the frame processor. NiT)audio_in_sample_rateaudio_in_enabledvideo_in_enabledaudio_out_enabledaudio_out_sample_rate) on_connectedon_participant_connectedon_participant_disconnected)r1r2paramsr.r/ callbacksconnect_as_user)r7rHrr9r:r!rr?r@r _on_connected_on_participant_connected_on_participant_disconnectedr;)rDrHrGs r*rHzHeyGenVideoService.setup~sgmE"""#MMMM"%*!%!%"&&9 !11++%!//)-)G)G,0,M,M !#  &ll  ''') #( (s"B>B:BB>4B<5B><B>cKt|d{|jjd{d|_y7.7w)zClean up the service and release resources. Terminates the HeyGen client session and cleans up associated resources. N)r7cleanupr;)rDrGs r*rYzHeyGenVideoService.cleanups? goll""$$$  $sAA!AA AAc6Ktjdyw)z%Handle bot connected to LiveKit room.z$HeyGen bot connected to LiveKit roomNrinforDs r*rUz HeyGenVideoService._on_connecteds :;sparticipant_idc$Ktjd||jsdd|_|jj ||j d{|jj ||jd{yy747w)z$Handle participant connected events.zParticipant connected TN)rr\rAr;capture_participant_video_on_participant_video_framecapture_participant_audio_on_participant_audio_datarDr^s r*rVz,HeyGenVideoService._on_participant_connecteds ,^,<=>1115D .,,88 @ @  ,,88 ? ?   2  s$ABB -BBBBc<Ktjd|yw)z'Handle participant disconnected events.zParticipant disconnected Nr[rds r*rWz/HeyGenVideoService._on_participant_disconnecteds //?@As video_framecKt|j|j|j}|j |d{y7w)z/Handle incoming video frames from participants.)imagesizeformatN)rrhrirj push_frame)rDrfframes r*raz.HeyGenVideoService._on_participant_video_frames?###!!%%  ooe$$$AA A A  audio_framecKt|j|j|j}|j |d{y7w)z-Handle incoming audio data from participants.)audio sample_rate num_channelsN)rrprqrrrk)rDrnrls r*rcz-HeyGenVideoService._on_participant_audio_datas?)###//$11  ooe$$$rmrlcKt||d{ttdzdz |_|j j||jd{|j d{y7e77 w)aStart the HeyGen video service and initialize the avatar session. Creates necessary tasks for audio/video processing and establishes the connection with the HeyGen service. Args: frame: The start frame containing initialization parameters. N)r7startintrrCr;_create_send_taskrDrlrGs r*rvzHeyGenVideoService.startsugmE"""!$&9A&=%C!Dll  (>(>???$$&&& # @&s4BA<ABA>B6B7B>BBcKt||d{|jd{|jd{y7577 w)zStop the HeyGen video service gracefully. Performs cleanup by ending the conversation and cancelling ongoing tasks in a controlled manner. Args: frame: The end frame. N)r7stop_end_conversation_cancel_send_taskrys r*r{zHeyGenVideoService.stopsLgl5!!!$$&&&$$&&& "&&1AA AAAAAAAcKt||d{|jd{|jd{y7577 w)zCancel the HeyGen video service. Performs an immediate termination of the service, cleaning up resources without waiting for ongoing operations to complete. Args: frame: The cancel frame. N)r7cancelr|r}rys r*rzHeyGenVideoService.cancelsLgnU###$$&&&$$&&& $&&r~ directionc~Kt|||d{t|tr3|j d{|j ||d{yt|t r=|jjd{|j ||d{yt|tr5|jj|j ||d{yt|tr|j|d{yt|tr|jd{yt|tr3|j!d{|j ||d{y|j ||d{y77f7O77777v7O777w)aProcess incoming frames and coordinate avatar behavior. Handles different types of frames to manage avatar interactions: - UserStartedSpeakingFrame: Activates avatar's listening animation - UserStoppedSpeakingFrame: Deactivates avatar's listening state - TTSAudioRawFrame: Processes audio for avatar speech - Other frames: Forwards them through the pipeline Args: frame: The frame to be processed. direction: The direction of frame processing (input/output). N)r7 process_frame isinstancer_handle_user_started_speakingrkrr;stop_agent_listeningrtransport_readyr_handle_audio_framerstart_ttfb_metricsr stop_ttfb_metrics)rDrlrrGs r*rz HeyGenVideoService.process_frameshg#E9555 e5 6446 6 6//%3 3 3 7 8,,335 5 5//%3 3 3 8 9 LL ( ( *//%3 3 3 / 0**51 1 1  /))+ + + 6 7((* * *//%3 3 3//%3 3 3- 6 7 3 5 3 4 1 + + 3 3sF=F"'F=F%F=F(2F= F+ F=&F.'AF=+F1,)F=F3(F=>F5?(F='F7(F=F9F=F;F=%F=(F=+F=.F=1F=3F=5F=7F=9F=;F=cy)zzCheck if the service can generate metrics. Returns: True if metrics generation is supported. Tr(r]s r*can_generate_metricsz'HeyGenVideoService.can_generate_metricss r)c6Kd|_|jj|jd{|j d{d|_|j d{|jj d{y7^7H7+7 w)aSHandle the event when a user starts speaking. Manages the interruption flow by: 1. Setting the interruption flag 2. Signaling the client to interrupt current speech 3. Cancelling ongoing audio sending tasks 4. Creating a new send task 5. Activating the avatar's listening animation TNF)r>r; interruptrBr}rxstart_agent_listeningr]s r*rz0HeyGenVideoService._handle_user_started_speaking#s!%ll$$T^^444$$&&& %$$&&&ll00222 5&&2sE0BBB B B)B*!B B BBBBcbKd|_|jjd{y7w)zEnd the current conversation and reset state. Stops the HeyGen client and cleans up conversation-specific resources. FN)rAr;r{r]s r*r|z$HeyGenVideoService._end_conversation4s& .3*ll!!!s %/-/cK|js>tj|_|j |j |_yyw)z2Create the audio sending task if it doesn't exist.N)r<asyncioQueue_queue create_task_send_task_handlerr]s r*rxz$HeyGenVideoService._create_send_task<s:!--/DK"..t/F/F/HIDOsA Ac~K|jr+|j|jd{d|_yy7 w)z+Cancel the audio sending task if it exists.N)r< cancel_taskr]s r*r}z$HeyGenVideoService._cancel_send_taskBs4 ??""4??3 3 3"DO  3s +=;=cVK|jj|d{y7w)zQueue an audio frame for processing. Places the audio frame in the processing queue for synchronized delivery to the HeyGen service. Args: frame: The audio frame to process. N)rput)rDrls r*rz&HeyGenVideoService._handle_audio_frameHskkooe$$$s )')cK|jj}t}d|_ t j |j jtd{}|jryt|tr|jt|j|_|jj|j |j"|d{}|j%|t'||j(k\rn|d|j(}||j(d}|jj+t-||jd{t'||j(k\rn|j j/k7477>#tj0$rT|jE|jj3|jd{7d|_|j5YowxYww)zHandle sending audio frames to the HeyGen client. Continuously processes audio frames from the queue and sends them to the HeyGen client. Handles timeouts and silence detection for proper audio streaming management. N)timeout)r;out_sample_rate bytearrayrBrwait_forrgetAVATAR_VAD_STOP_SECSr>rrstridr=resamplerprqextendlenrC agent_speakbytes task_done TimeoutErroragent_speak_endclear)rDrq audio_bufferrlrpchunks r*rz%HeyGenVideoService._send_task_handlerSsll22  {  )%..t{{/@J^__((e%56~~-),UXX"&//":": U%6%6 #E!''.l+t/E/EE ,-Et/E/E F'3D4J4J4L'M "ll66uU|T^^TTT l+t/E/EE  %%''_U'' )>>-,,66t~~FFF%)DN &&(  )s)H6F"F#F3H4A*FFA=FFF:FHFFFAH$G'%HHHH)-r$r%r&r'r#Settings__annotations__raiohttp ClientSessionrrrrrr8rrHrYrUrVrWr rarrcrrvr r{r rr rrboolrrr|rxr}rrr __classcell__)rGs@r*r,r,?s;*#H""\`.226$#$#&& $# "%(CEV(V"WX $# {+ $#./$# $#L(!4(@< c BB%]%%M% ' ' ' ' '+ '#4#4>#4Jd3""J # %/B %$)r)r,)1r'r dataclassesrtypingrrrlogururpipecat.audio.utilsrpipecat.frames.framesrr r r r r rrrrrrrrr"pipecat.processors.frame_processorrrpipecat.services.ai_servicer.pipecat.services.heygen.api_interactive_avatarr&pipecat.services.heygen.api_liveavatarrpipecat.services.heygen.clientrrrrpipecat.services.settingsr !pipecat.transports.base_transportr!rr#r,r(r)r*rs!"7"S1LN 6=  /   x)x)r)