qiDdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z ddl Z ddlmZddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZm Z ddl!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(m)Z)dd l*m+Z+ ddl,Z,dd l-m.Z.ddl/m0Z0m1Z1ddl2m3Z3m4Z4m5Z5dZ:dZ;dZ<GddeZ=Gdde0Z>Gdde.Z?GddZ@Gdde$ZAGd d!e&ZBGd"d#e(ZCy#e6$r7Z7ejpde7ejpde9de7dZ7[7wwxYw)$a Small WebRTC transport implementation for Pipecat. This module provides a WebRTC transport implementation using aiortc for real-time audio and video communication. It supports bidirectional media streaming, application messaging, and client connection management. N)deque)Any AwaitableCallableListOptional)logger) BaseModel) CancelFrameClientConnectedFrameEndFrameFrameInputAudioRawFrameInputTransportMessageFrameOutputAudioRawFrameOutputImageRawFrameOutputTransportMessageFrame!OutputTransportMessageUrgentFrame SpriteFrame StartFrameUserImageRawFrameUserImageRequestFrame)FrameDirection)BaseInputTransport)BaseOutputTransport) BaseTransportTransportParams)SmallWebRTCConnection)VideoStreamTrack)AudioStreamTrackMediaStreamError) AudioFrameAudioResampler VideoFramez Exception: zNIn order to use the SmallWebRTC, you need to `pip install pipecat-ai[webrtc]`.zMissing module: camera screenVideo microphonecheZdZUdZeeegedfed<ee gedfed<ee gedfed<y)SmallWebRTCCallbacksaCallback handlers for SmallWebRTC events. Parameters: on_app_message: Called when an application message is received. on_client_connected: Called when a client establishes connection. on_client_disconnected: Called when a client disconnects. Non_app_messageon_client_connectedon_client_disconnected) __name__ __module__ __qualname____doc__rrstrr__annotations__rZ/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/transports/smallwebrtc/transport.pyr)r)=sQc3Z4899!#8"99T?"JKK$&;%)rDwaitrRrPsamplesframes r5recvzRawAudioTrack.recvzs# ??Q ;;$//D4E4E"EFTDaxmmD)))    --557ME6fkkm!!$'$../E--RXX6''a(8H --OO #,,Q0A0AB 4111 %*sA(E?*E<+DE?) r-r.r/r0r<r_rTrk __classcell__rFs@r5r7r7Ks $52r4r7c.eZdZdZfdZdZdZxZS) RawVideoTrackzzCustom video stream track for WebRTC output. Handles video frame queuing and conversion for WebRTC transmission. cpt|||_||_t j |_y)zInitialize the raw video track. Args: width: Video frame width in pixels. height: Video frame height in pixels. N)r;r<_width_heightrKQueue _video_buffer)rDwidthheightrFs r5r<zRawVideoTrack.__init__s,   $]]_r4c:|jj|y)zAdd a video frame to the transmission buffer. Args: frame: The video frame to queue for transmission. N)rt put_nowaitrDrjs r5add_video_framezRawVideoTrack.add_video_frames %%e,r4c|K|jjd{}tj|jtj j |j|jdf}tj|d}|jd{\|_ |_ |S77w)zReturn the next video frame for WebRTC transmission. Returns: A VideoFrame ready for WebRTC transmission. NrVrgb24format)rtgetr`raimageuint8reshaperrrqr$rcnext_timestamprdrg)rD raw_frame frame_datarjs r5rkzRawVideoTrack.recvs ,,0022 ]]9??"((CKK \\4;; * '' 7C,0+>+>+@%@" 5? 3&As"B<B8BB<$B:%B<:B<)r-r.r/r0r<rzrkrlrms@r5roros --r4rocjeZdZdZej ej ej ejdZde de fdZ de jdede jfd Zd efd Zd Zd edefdZd edefdZdefdZdZdZd eezfdZdZdZdZ de!defdZ"dZ#e$defdZ%e$defdZ&y)SmallWebRTCClientzWebRTC client implementation for handling connections and media streams. Manages WebRTC peer connections, audio/video streaming, and application messaging through the SmallWebRTCConnection interface. )yuv420pyuvj420pnv12graywebrtc_connection callbacksc&|_d_|_d_d_d_d_d_d_d_ d_ d_ d_ d_ jjddtffd }jjddtffd }jjd dtffd }jjd dtd t ffd }y)zInitialize the WebRTC client. Args: webrtc_connection: The underlying WebRTC connection handler. callbacks: Event callbacks for connection and message handling. FNr connected connectionclKtjdjd{y7w)NzPeer connection established.)r debug_handle_client_connectedrrDs r5 on_connectedz0SmallWebRTCClient.__init__..on_connecteds& LL7 8//1 1 1 )424 disconnectedclKtjdjd{y7w)NzPeer connection lost.)r r_handle_peer_disconnectedrs r5on_disconnectedz3SmallWebRTCClient.__init__..on_disconnecteds& LL0 1002 2 2rclosedclKtjdjd{y7w)NzClient connection closed.)r r_handle_client_closedrs r5 on_closedz-SmallWebRTCClient.__init__..on_closeds& LL4 5,,. . .rz app-messagemessagecZKj||jd{y7w)N)_handle_app_messagepc_id)rrrDs r5r*z2SmallWebRTCClient.__init__..on_app_messages#**7J4D4DE E Es +)+)_webrtc_connection_closing _callbacks_audio_output_track_video_output_track_audio_input_track_video_input_track_screen_video_track_params_audio_in_channels_in_sample_rate_out_sample_rate_leave_counter_audio_in_resampler event_handlerrr)rDrrrrrr*s` r5r<zSmallWebRTCClient.__init__s1#4 ##' #' >B>B?C  "&# $$( . .{ ; 2+@ 2 < 2  . .~ > 3.C 3 ? 3  . .x 8 /(= / 9 /  . .} = F-B FS F > Fr4 frame_array format_namereturnc|jdr|Stjj|}|t d|t j ||S)aXConvert a video frame to RGB format based on the input format. Args: frame_array: The input frame as a NumPy array. format_name: The format of the input frame. Returns: The converted RGB frame as a NumPy array. Raises: ValueError: If the format is unsupported. rgbzUnsupported format: ) startswithrFORMAT_CONVERSIONSrrJcv2cvtColor)rDrrconversion_codes r5_convert_framez SmallWebRTCClient._convert_framesX  ! !% ( +>>BB;O  "3K=AB B||K99r4 video_sourcecK |tk(r |jn |j}|tjdd{B tj |j dd{}|t|tstjdd{|j j"}|j%|}|j'||}~|j)}~t+|jj,||j.|j0fd }||_|j4|_~~|E7 7#tj$rG|jjr'|r%|jrtjdd}Y4t$rtjdd}YVwxYw70w) amRead video frames from the WebRTC connection. Reads a video frame from the given MediaStreamTrack, converts it to RGB, and creates an InputImageRawFrame. Args: video_source: Video source to capture ("camera" or "screenVideo"). Yields: UserImageRawFrame objects containing video data from the peer. N{Gz?@timeoutz;Timeout: No video frame received within the specified time.zBReceived an unexpected media stream error while reading the video.r~RGB)user_idrsizer)CAM_VIDEO_SOURCErrrKr[wait_forrk TimeoutErrorr is_connected is_enabledr warningr! isinstancer$rname to_ndarrayrtobytesrrrurvtransport_sourcerd) rDr video_trackrjrr frame_rgb image_bytes image_frames r5read_video_framez"SmallWebRTCClient.read_video_frames #33''--  "mmD))) %..{/?/?/A3OO}Juj$AmmD))),,++K**+*>K++KEI#++-K+//55!kk5<<0 K ,8K (#iiKO e*P'' ++88:##..0NN#`a# cd  *s`=GE G(E.E /E3*GG B,G EAG $G'G G G  GcK |jtjdd{+ tj|jj dd{}|t|tstjdd{|j|jk7r|j j#|n|g}|D]y}|j%j't(j*}|j-}~t/||j|j0}|j2|_~|{~J7&7#tj $r[|j jr;|jr/|jjrtjdd}YZt$rtjdd}Y|wxYw7Vw)zRead audio frames from the WebRTC connection. Reads 20ms of audio from the given MediaStreamTrack and creates an InputAudioRawFrame. Yields: InputAudioRawFrame objects containing audio data from the peer. Nrrrz;Timeout: No audio frame received within the specified time.zBReceived an unexpected media stream error while reading the audio.)audiorE num_channels)rrKr[rrkrrrrr rr!rr"rErrresamplerastyper`rbrrrrd)rDrjframes_to_processprocessed_frame pcm_array pcm_bytes audio_frames r5read_audio_framez"SmallWebRTCClient.read_audio_frameZs&&.mmD))) %..t/F/F/K/K/MWZ[[}Juj$AmmD))) $$(<(<<((11%8W  $5 "+668??I %--/ 0# $ 4 4!%!8!8 #()) !! " _)\'' ++88:////::<NN#`a# cd  *s^&G)EG)2E!E"E&*G)G&B>G)EA*G#=G)G#G)"G##G)rjcK|jr:|jr.|jj|jd{yy7w)zWrite an audio frame to the WebRTC connection. Args: frame: The audio frame to transmit. Returns: True if the audio frame was written successfully, False otherwise. NTF) _can_sendrrTrrys r5write_audio_framez#SmallWebRTCClient.write_audio_framesD >>  8 8**::5;;G G G HsAAAAc|K|jr(|jr|jj|yyw)zWrite a video frame to the WebRTC connection. Args: frame: The video frame to transmit. Returns: True if the video frame was written successfully, False otherwise. TF)rrrzrys r5write_video_framez#SmallWebRTCClient.write_video_frames4 >>  8 8  $ $ 4 4U ;s:<rcK|j|_|jxs |j|_|jxs |j|_||_|xjdz c_tdd|j|_ yw)zSet up the client with transport parameters. Args: _params: Transport configuration parameters. frame: The initialization frame containing setup data. rZs16rXN) audio_in_channelsraudio_in_sample_rateraudio_out_sample_raterrrr#r)rDrrjs r5setupzSmallWebRTCClient.setupsw#*";";&;;Yu?Y?Y ' = = \A\A\  q #1%AUAU#V sB B cK|jjrytjd|jj d{y7w)z Establish the WebRTC connection.NzConnecting to Small WebRTC)rrr infoconnectrDs r5rzSmallWebRTCClient.connects@  " " / / 1  02%%--///sAAAAc>K|xjdzc_|jdkDry|jrd|jsWtjdd|_|j jd{|jd{yyy77 w)z Disconnect from the WebRTC peer.rZrNzDisconnecting to Small WebRTCT) rr is_closingr rrr disconnectrrs r5rzSmallWebRTCClient.disconnects q     "    T__ KK7 9 DM))446 6 6002 2 2 &5  7 2s$A7B9B:BBBBcxK|jr&|jj|jyyw)zSend an application message through the WebRTC connection. Args: frame: The message frame to send. N)rrsend_app_messagerrys r5 send_messagezSmallWebRTCClient.send_messages/ >>   # # 4 4U]] C s8:cK|jsy|jj|_|jj |_|jj |_|jjr@t|j|_ |jj|j|jjr_t|jj|jj |_|jj%|j"|j&j)|jd{y7w)z'Handle client connection establishment.N)rE)rurv)rraudio_input_trackrvideo_input_trackrscreen_video_input_trackraudio_out_enabledr7rrreplace_audio_trackvideo_out_enabledrovideo_out_widthvideo_out_heightrreplace_video_trackrr+rs r5rz*SmallWebRTCClient._handle_client_connecteds|| "&"9"9"K"K"M"&"9"9"K"K"M#'#:#:#S#S#U << ) )'4AVAV'WD $  # # 7 78P8P Q << ) )'4ll224<<;X;X(D $  # # 7 78P8P Qoo11$2I2IJJJsEE( E&!E(cRKd|_d|_d|_d|_d|_yw)z"Handle peer disconnection cleanup.N)rrrrrrs r5rz+SmallWebRTCClient._handle_peer_disconnecteds."&"&#' #' #' s%'cKd|_d|_d|_d|_d|_|j s.|j j|jd{yy7w)z!Handle client connection closure.N) rrrrrrrr,rrs r5rz'SmallWebRTCClient._handle_client_closeds]"&"&#' #' #' }}//889P9PQ Q Q QsAA#A!A#rsendercXK|jj||d{y7w)%Handle incoming application messages.N)rr*rDrrs r5rz%SmallWebRTCClient._handle_app_messages oo,,Wf===s *(*c8|jxr |j S)z2Check if the connection is ready for sending data.)rrrs r5rzSmallWebRTCClient._can_send s  8%88r4c6|jjS)ztCheck if the WebRTC connection is established. Returns: True if connected to the peer. )rrrs r5rzSmallWebRTCClient.is_connecteds&&3355r4c|jS)zCheck if the connection is in the process of closing. Returns: True if the connection is closing. )rrs r5rzSmallWebRTCClient.is_closings}}r4N)'r-r.r/r0rCOLOR_YUV2RGB_I420COLOR_YUV2RGB_NV12COLOR_GRAY2RGBrrr)r<r`ndarrayr1rrrrboolrrrrrrrrrrrrrrrrpropertyrrr3r4r5rrs7))**&&"" +F*?+FL`+FZ:"**:3:2:::.>3>@7r -@ T  -@ T  W? W0 3 D03TT DK*( R>>c>96d66Dr4rceZdZdZdedeffd Zdedeffd Z de ffd Z d Z de ffd Zdeffd Zd Zd efdZdefdZdefdZefdefdZxZS)SmallWebRTCInputTransportzInput transport implementation for SmallWebRTC. Handles incoming audio and video streams from WebRTC peers, including user image requests and application message handling. clientparamsc t||fi|||_||_d|_d|_d|_g|_d|_y)zInitialize the WebRTC input transport. Args: client: The WebRTC client instance. params: Transport configuration parameters. **kwargs: Additional arguments passed to parent class. NF) r;r<_clientr_receive_audio_task_receive_video_task_receive_screen_video_task_image_requests _initializedrDrrkwargsrFs r5r<z"SmallWebRTCInputTransport.__init__)sO *6*  #' #' *.'<>"r4rj directioncKt|||d{t|tr|j |d{yy7/7w)zProcess incoming frames including user image requests. Args: frame: The frame to process. direction: The direction of frame flow in the pipeline. N)r; process_framerrrequest_participant_image)rDrjrrFs r5rz'SmallWebRTCInputTransport.process_frameAsMg#E9555 e2 3007 7 7 4 6 8s!A A(A A A  A cnKt||d{|jryd|_|jj |j |d{|jj d{|j|d{|js:|j jr$|j|j|_|jsA|j jr*|j|jt|_ yyy7777w)zStart the input transport and establish WebRTC connection. Args: frame: The start frame containing initialization parameters. NT)r;startrrrrrset_transport_readyraudio_in_enabled create_task_receive_audiorvideo_in_enabled_receive_videorrDrjrFs r5r"zSmallWebRTCInputTransport.startMs gmE"""     ll  u555ll""$$$&&u---''DLL,I,I'+'7'78K8K8M'ND $''DLL,I,I'+'7'78K8KL\8]'^D $-J' # 6$-sGD5D,AD5D/!D5:D1;D5D3BD5/D51D53D5cK|jr*|j|jd{d|_|jr+|j|jd{d|_yy7C7w)zStop all background tasks.N)r cancel_taskrrs r5 _stop_tasksz%SmallWebRTCInputTransport._stop_tasksbsl  # #""4#;#;< < <'+D $  # #""4#;#;< < <'+D $ $ = =s!+A5A15A5#A3$A53A5cKt||d{|jd{|jj d{y7?7)7 w)zStop the input transport and disconnect from WebRTC. Args: frame: The end frame signaling transport shutdown. N)r;stopr,rrr)s r5r.zSmallWebRTCInputTransport.stopksP gl5!!!   ll%%''' " '1AAAA!AAAAAcKt||d{|jd{|jj d{y7?7)7 w)zCancel the input transport and disconnect immediately. Args: frame: The cancel frame signaling immediate cancellation. N)r;cancelr,rrr)s r5r1z SmallWebRTCInputTransport.cancelusP gnU###   ll%%''' $ 'r/c ,K |jj}|23d{}|s |j|d{&7!76y#t$r<}t j |d|j jd|dYd}~yd}~wwxYww)z7Background task for receiving audio frames from WebRTC.N exception receiving data:  ())rrpush_audio_frame Exceptionr errorrFr-)rDaudio_iteratorres r5r&z(SmallWebRTCInputTransport._receive_audios [!\\::.>##$78/D +.4 6sDDsourcecK|tk(rG|js;|jjr%|j |j |_y|t k(rL|js@|jjr*|j |jt |_y|tk(rN|jsA|jjr*|j |jt|_ yyyyw)zCapture media from a specific participant. Args: source: Media source to capture from. ("camera", "microphone", or "screenVideo") N) MIC_AUDIO_SOURCErrr$r%r&rrr'r(rJr)rDrKs r5capture_participant_mediaz3SmallWebRTCInputTransport.capture_participant_medias & &,, --(,'7'78K8K8M'ND $ & &,, --(,'7'78K8KL\8]'^D $ ) )33 --/3.>.>##$78/D +.4 *sC>D)r-r.r/r0rrr<rrrrr"r,r r.r r1r&r1r(rrHrr rrNrlrms@r5rr"s "!" "0 8 8> 8__*,(((+( [$[$[LPcP!5J!J'  r4rceZdZdZdedeffd Zdeffd Zde ffd Z de ffd Z de ezfd Zded efd Zded efd ZxZS) SmallWebRTCOutputTransportzOutput transport implementation for SmallWebRTC. Handles outgoing audio and video streams to WebRTC peers, including transport message sending. rrc Pt||fi|||_||_d|_y)zInitialize the WebRTC output transport. Args: client: The WebRTC client instance. params: Transport configuration parameters. **kwargs: Additional arguments passed to parent class. FN)r;r<rrrrs r5r<z#SmallWebRTCOutputTransport.__init__s/ *6*  "r4rjcFKt||d{|jryd|_|jj |j |d{|jj d{|j|d{y77B7"7 w)zStart the output transport and establish WebRTC connection. Args: frame: The start frame containing initialization parameters. NT)r;r"rrrrrr#r)s r5r"z SmallWebRTCOutputTransport.starts gmE"""     ll  u555ll""$$$&&u--- # 6$-sFB!BAB!B!B!:B;B!BB!B!B!B!cKt||d{|jjd{y7'7w)zStop the output transport and disconnect from WebRTC. Args: frame: The end frame signaling transport shutdown. N)r;r.rrr)s r5r.zSmallWebRTCOutputTransport.stop,s: gl5!!!ll%%''' "'A>!AAAAcKt||d{|jjd{y7'7w)zCancel the output transport and disconnect immediately. Args: frame: The cancel frame signaling immediate cancellation. N)r;r1rrr)s r5r1z!SmallWebRTCOutputTransport.cancel5s: gnU###ll%%''' $'rTcVK|jj|d{y7w)zSend a transport message through the WebRTC connection. Args: frame: The transport message frame to send. N)rrrys r5rz'SmallWebRTCOutputTransport.send_message>s ll''...s )')rcTK|jj|d{S7w)zWrite an audio frame to the WebRTC connection. Args: frame: The output audio frame to transmit. Returns: True if the audio frame was written successfully, False otherwise. N)rrrys r5rz,SmallWebRTCOutputTransport.write_audio_frameH#\\33E:::: (&(cTK|jj|d{S7w)zWrite a video frame to the WebRTC connection. Args: frame: The output video frame to transmit. Returns: True if the video frame was written successfully, False otherwise. N)rrrys r5rz,SmallWebRTCOutputTransport.write_video_frameSrXrY)r-r.r/r0rrr<rr"r r.r r1rrrrrrrrrlrms@r5rPrPs~ "!" "(.."(((+(/03TT/ ;-@ ;T ; ;-@ ;T ;r4rPc eZdZdZ ddededeedeeffd Zde fdZ de fd Z d e ezfd Zd efd Zd edefdZdZdZefdefdZefdefdZxZS)SmallWebRTCTransportaWebRTC transport implementation for real-time communication. Provides bidirectional audio and video streaming over WebRTC connections with support for application messaging and connection event handling. Event handlers available: - on_client_connected(transport, client): Client connected to WebRTC session - on_client_disconnected(transport, client): Client disconnected from WebRTC session - on_client_message(transport, message, client): Received a data channel message Example:: @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): ... rr input_name output_namecPt|||||_t|j|j |j |_t||j|_ d|_ d|_ |jd|jd|jdy)a9Initialize the WebRTC transport. Args: webrtc_connection: The underlying WebRTC connection handler. params: Transport configuration parameters. input_name: Optional name for the input processor. output_name: Optional name for the output processor. )r]r^)r*r+r,Nr*r+r,) r;r<rr)_on_app_message_on_client_connected_on_client_disconnectedrrr_input_output_register_event_handler)rDrrr]r^rFs r5r<zSmallWebRTCTransport.__init__rs JKH .// $ 9 9#'#?#?  )):DOOL ;? =A  $$%56 $$%:; $$%=>r4rc|js1t|j|j|j|_|jS)zGet the input transport processor. Returns: The input transport for handling incoming media streams. r)rcrrr _input_namers r5inputzSmallWebRTCTransport.inputs8 {{3 dll1A1ADK{{r4c|js1t|j|j|j|_|jS)zGet the output transport processor. Returns: The output transport for handling outgoing media streams. rg)rdrPrrrhrs r5outputzSmallWebRTCTransport.outputs8 ||5 dll1A1ADL||r4rjcK|jr3|jj|tjd{yy7w)znSend an image frame through the transport. Args: frame: The image frame to send. Nrd queue_framer DOWNSTREAMrys r5 send_imagezSmallWebRTCTransport.send_image7 <<,,**5.2K2KL L L  L:AAAcK|jr3|jj|tjd{yy7w)znSend an audio frame through the transport. Args: frame: The audio frame to send. Nrmrys r5 send_audiozSmallWebRTCTransport.send_audiorqrrrrcK|jr#|jj|d{|jd||d{y7 7w)rNr*)rcrH_call_event_handlerrs r5r`z$SmallWebRTCTransport._on_app_messagesG ;;++..w7 7 7&&'7&III 8Is!+AAAA AAcK|jd|d{|jr,|jjtd{yy7=7w)z Handle client connection events.r+N)rvrc push_framer rDrs r5raz)SmallWebRTCTransport._on_client_connectedsP&&'<>OPPP ;;++(()=)?@ @ @  Q @s!AA6AAAAcDK|jd|d{y7w)z#Handle client disconnection events.r,N)rvrys r5rbz,SmallWebRTCTransport._on_client_disconnecteds&&'?ARSSSs   rcrK|jr%|jj|d{yy7w)zCapture video from a specific participant. Args: video_source: Video source to capture from ("camera" or "screenVideo"). rKNrcrN)rDrs r5capture_participant_videoz.SmallWebRTCTransport.capture_participant_video2 ;;++77|7L L L  L ,757 audio_sourcecrK|jr%|jj|d{yy7w)zCapture audio from a specific participant. Args: audio_source: Audio source to capture from. (currently, "microphone" is the only supported option) r|Nr})rDrs r5capture_participant_audioz.SmallWebRTCTransport.capture_participant_audiorr)NN)r-r.r/r0rrrr1r<rrirPrkrrrprrtrr`rarbrr~rMrrlrms@r5r\r\_s,%)%) !?0!? !?SM !? c] !?F 0  2 M&9K&GMM&9MJSJ#J A T - M M- M Mr4r\)Dr0rKrerA collectionsrtypingrrrrrnumpyr`logurur pydanticr pipecat.frames.framesr r r rrrrrrrrrrr"pipecat.processors.frame_processorrpipecat.transports.base_inputrpipecat.transports.base_outputr!pipecat.transports.base_transportrr)pipecat.transports.smallwebrtc.connectionrraiortcraiortc.mediastreamsr r!avr"r#r$ModuleNotFoundErrorr:r8r7rrJrMr)r7rorrrPr\r3r4r5rs' ;; ><>LK,'F99 # O9 OK$K\,$,^WWt [ 2[|\;!4\;~FM=FMY,FLL;qc"#FLLab &qc* ++,s4C##D(2DD