qiQ90dZddlmZmZmZddlZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZddlmZmZmZddlmZddlmZdd lm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(m)Z)Gd de$Z*Gdde&Z+Gdde)Z,Gdde(Z-y)a;HeyGen implementation for Pipecat. This module provides integration with the HeyGen platform for creating conversational AI applications with avatars. It manages conversation sessions and provides real-time audio/video streaming capabilities through the HeyGen API. The module consists of three main components: - HeyGenInputTransport: Handles incoming audio and events from HeyGen conversations - HeyGenOutputTransport: Manages outgoing audio and events to HeyGen conversations - HeyGenTransport: Main transport implementation that coordinates input/output transports )AnyOptionalUnionN)logger) AudioRawFrameBotConnectedFrameBotStartedSpeakingFrameBotStoppedSpeakingFrame CancelFrameClientConnectedFrameEndFrameFrameInputAudioRawFrameInterruptionFrameOutputAudioRawFrame StartFrameUserStartedSpeakingFrameUserStoppedSpeakingFrame)FrameDirectionFrameProcessorFrameProcessorSetup)NewSessionRequest)LiveAvatarNewSessionRequest)HeyGenCallbacks HeyGenClient ServiceType)BaseInputTransport)BaseOutputTransport) BaseTransportTransportParamsceZdZdZdedeffd Zdeffd ZfdZ de ffd Z de ffd Z deffd Zd efd ZdefdZxZS)HeyGenInputTransportzInput transport for receiving audio and events from HeyGen conversations. Handles incoming audio streams from participants and manages audio capture from the Daily room connected to the HeyGen conversation. clientparamsc Pt||fi|||_||_d|_y)zInitialize the HeyGen input transport. Args: client: The HeyGen transport client instance. params: Transport configuration parameters. **kwargs: Additional arguments passed to parent class. FN)super__init___client_params _initializedselfr#r$kwargs __class__s U/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/transports/heygen/transport.pyr'zHeyGenInputTransport.__init__8s- *6*  !setupcKt||d{|jj|d{y7(7w)znSetup the input transport. Args: setup: The frame processor setup configuration. Nr&r1r(r,r1r.s r/r1zHeyGenInputTransport.setupK< gmE"""ll  ''' #'A?"AAAAcKt|d{|jjd{y7'7w)z"Cleanup input transport resources.Nr&cleanupr(r,r.s r/r9zHeyGenInputTransport.cleanupT6goll""$$$ $A=!A?AAframecKt||d{|jryd|_|j|d{y727w)z{Start the input transport. Args: frame: The start frame containing initialization parameters. NT)r&startr*set_transport_readyr,r=r.s r/r?zHeyGenInputTransport.startYsN gmE"""     &&u--- # .s!A A ,A A A  A cKt||d{|jjd{y7'7w)zpStop the input transport. Args: frame: The end frame signaling transport shutdown. Nr&stopr(rAs r/rDzHeyGenInputTransport.stoph: gl5!!!ll!!! "!A>!AAAAcKt||d{|jjd{y7'7w)zyCancel the input transport. Args: frame: The cancel frame signaling immediate cancellation. Nr&cancelr(rDrAs r/rIzHeyGenInputTransport.cancelq: gnU###ll!!! $!rFparticipant_idcK|jjrGtjd||jj ||j d{yy7w)zStart capturing audio from a participant. Args: participant_id: The participant to capture audio from. z6HeyGenTransport start capturing audio for participant N)r)audio_in_enabledrinfor(capture_participant_audio_on_participant_audio_datar,rKs r/start_capturing_audioz*HeyGenInputTransport.start_capturing_audiozs\ << ( ( KKPQ_P`a b,,88 ? ?   ) sAA#A!A# audio_framecKt|j|j|j}|j |d{y7w)z'Handle received participant audio data.)audio sample_rate num_channelsN)rrUrVrWpush_audio_frame)r,rSr=s r/rPz/HeyGenInputTransport._on_participant_audio_datasA"###//$11  ##E***sAA A A )__name__ __module__ __qualname____doc__rr r'rr1r9rr?r rDr rIstrrRrrP __classcell__r.s@r/r"r"1sm "" "&(!4(% . ."""+" # +M+r0r"ceZdZdZdedeffd Zdeffd ZfdZ de ffd Z de ffd Z deffd Zej"fded effd Zded effd ZdedefdZxZS)HeyGenOutputTransportzOutput transport for sending audio and events to HeyGen conversations. Handles outgoing audio streams to participants and manages the custom audio track expected by the HeyGen platform. r#r$c ^t||fi|||_||_d|_d|_y)zInitialize the HeyGen output transport. Args: client: The HeyGen transport client instance. params: Transport configuration parameters. **kwargs: Additional arguments passed to parent class. FN)r&r'r(r)r* _event_idr+s r/r'zHeyGenOutputTransport.__init__s6 *6*  "r0r1cKt||d{|jj|d{y7(7w)zoSetup the output transport. Args: setup: The frame processor setup configuration. Nr3r4s r/r1zHeyGenOutputTransport.setupr5r6cKt|d{|jjd{y7'7w)z#Cleanup output transport resources.Nr8r:s r/r9zHeyGenOutputTransport.cleanupr;r<r=c2Kt||d{|jryd|_|jj||jd{|j |d{|jj y7z7:7#w)z|Start the output transport. Args: frame: The start frame containing initialization parameters. NT)r&r?r*r(audio_chunk_sizer@transport_readyrAs r/r?zHeyGenOutputTransport.starts gmE"""     ll  (=(=>>>&&u--- $$& # ?-s4BBABBB1B2 BBBcKt||d{|jjd{y7'7w)zqStop the output transport. Args: frame: The end frame signaling transport shutdown. NrCrAs r/rDzHeyGenOutputTransport.stoprErFcKt||d{|jjd{y7'7w)zzCancel the output transport. Args: frame: The cancel frame signaling immediate cancellation. NrHrAs r/rIzHeyGenOutputTransport.cancelrJrF directioncK|tjk(rt|tr<|jt j dt|j|_nDt|tr4|jj|jd{d|_t|5||d{y7$7w)zPush a frame to the next processor in the pipeline. Args: frame: The frame to push. direction: The direction to push the frame. Nz"self._event_id is already defined!)r DOWNSTREAM isinstancer rcrwarningr]idr r(agent_speak_endr& push_framer,r=rkr.s r/rrz HeyGenOutputTransport.push_frames 11 1%!89>>-NN#GH!$UXXE#:;ll224>>BBB!%g  222C2s$BCCC:C;CCc<Kt|||d{t|trG|jj |j d{|j||d{t|tr=|jjd{|j||d{yt|tr=|jjd{|j||d{yy7777s7[7*7w)aeProcess frames and handle interruptions. Handles various types of frames including interruption events and user speaking states. Updates the HeyGen client state based on the received frames. Args: frame: The frame to process direction: The direction of frame flow in the pipeline Note: Special handling is implemented for: - InterruptionFrame: Triggers interruption of current speech - UserStartedSpeakingFrame: Initiates agent listening mode - UserStoppedSpeakingFrame: Stops agent listening mode N) r& process_framernrr( interruptrcrrrstart_agent_listeningrstop_agent_listeningrss r/ruz#HeyGenOutputTransport.process_frames g#E9555 e. /,,((8 8 8//%3 3 3 e5 6,,446 6 6//%3 3 3 7 8,,335 5 5//%3 3 39 6 8 3 6 3 5 3s{DD>JJJ Ks=AAA)rYrZr[r\rr r'rr1r9rr?r rDr rIrrmrrrrurboolr}r^r_s@r/raras  *(!4(% '' """+"JXIbIb3e33&44>46-@Tr0rac.eZdZUdZdZeed<dZeed<y) HeyGenParamszConfiguration parameters for the HeyGen transport. Parameters: audio_in_enabled: Whether to enable audio input from participants. audio_out_enabled: Whether to enable audio output to participants. TrMaudio_out_enabledN)rYrZr[r\rMr~__annotations__rr0r/rrs"d!"t"r0rceZdZdZeddddfdej dededeedeedee e e fd ee ffd Z d Zd efd Zd efdZdefdZdefdZdefdZdefdZxZS)HeyGenTransportaTransport implementation for HeyGen video calls. When used, the Pipecat bot joins the same virtual room as the HeyGen Avatar and the user. This is achieved by using `HeyGenTransport`, which initiates the conversation via `HeyGenApi` and obtains a room URL that all participants connect to. Event handlers available: - on_client_connected(transport, participant): Participant connected to the session - on_client_disconnected(transport, participant): Participant disconnected from the session Example:: @transport.event_handler("on_client_connected") async def on_client_connected(transport, participant): ... Nsessionapi_keyr$ input_name output_namesession_request service_typec Ft|||||_t|||||t |j |j |j|_d|_ d|_ d|_ |jd|jd|jdy)aInitialize the HeyGen transport. Sets up a new HeyGen transport instance with the specified configuration for handling video calls between the Pipecat bot and HeyGen Avatar. Args: session: aiohttp session for making async HTTP requests api_key: HeyGen API key for authentication params: HeyGen-specific configuration parameters (default: HeyGenParams()) input_name: Optional custom name for the input transport output_name: Optional custom name for the output transport session_request: Configuration for the HeyGen session service_type: Service type for the avatar session Note: The transport will automatically join the same virtual room as the HeyGen Avatar and user through the HeyGenClient, which handles session initialization via HeyGenApi. )rr) on_connectedon_participant_connectedon_participant_disconnected)rrr$rr callbacksNron_client_connectedon_client_disconnected) r&r'r)rr _on_connected_on_participant_connected_on_participant_disconnectedr(_input_output_HeyGen_participant_id_register_event_handler) r,rrr$rrrrr.s r/r'zHeyGenTransport.__init__3s8 JKH #+%%!//)-)G)G,0,M,M   7; 8< &*# $$^4 $$%:; $$%=>r0cK|jdd{|jr,|jjtd{yy7=7w)z%Handle bot connected to LiveKit room.rN)_call_event_handlerrrrrr,s r/rzHeyGenTransport._on_connectedgsK&&~666 ;;++(():)<= = =  7 =s!AA6AAAArKcKtjd|d|dk7r|j|d{yy7w)NHeyGen participant z disconnectedheygen)rdebug_on_client_disconnectedrQs r/rz,HeyGenTransport._on_participant_disconnectedmsA *>*:-HI X %..~> > > & >s 3><>cKtjd|d|dk7rJ|j|d{|jr$|jj |d{yyy767 w)Nrz connectedr)rr_on_client_connectedrrRrQs r/rz)HeyGenTransport._on_participant_connectedrsi *>*:*EF X %++N; ; ;{{kk77GGG & ;Gs!3A0A,.A0$A.%A0.A0ryc~|js&t|j|j|_|jS)zGet the input transport for receiving media and events. Returns: The HeyGen input transport instance. r#r$)rr"r(r)rs r/inputzHeyGenTransport.inputys+ {{.dll4<<XDK{{r0c~|js&t|j|j|_|jS)zGet the output transport for sending media and events. Returns: The HeyGen output transport instance. r)rrar(r)rs r/outputzHeyGenTransport.outputs+ ||0 T\\ZDL||r0 participantcK|jd|d{|jr,|jjtd{yy7=7w)zHandle client connected events.rN)rrrrr r,rs r/rz$HeyGenTransport._on_client_connectedsO&&' ?? HcH~AcA NNr0r).r\typingrrrrlogururpipecat.frames.framesrrr r r r r rrrrrrr"pipecat.processors.frame_processorrrr.pipecat.services.heygen.api_interactive_avatarr&pipecat.services.heygen.api_liveavatarrpipecat.services.heygen.clientrrrpipecat.transports.base_inputrpipecat.transports.base_outputr!pipecat.transports.base_transportrr r"rarrrr0r/rs (' cbLNUU<>L\+-\+~A/AH #? #uNmuNr0