qil8UdZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z ddlmZddlmZddlmZddlmZmZdd lmZdd lmZmZdd lmZerdd lm Z eZ!eZ"eZeZeGd dZ#e e!e#fZ$e e%d<GddZ&y)a|Universal LLM context management for LLM services in Pipecat. Context contents are represented in a universal format (based on OpenAI) that supports a union of known Pipecat LLM service functionality. Whenever an LLM service needs to access context, it does a just-in-time translation from this universal context into whatever format it needs, using a service-specific adapter. N) dataclass) TYPE_CHECKINGAnyListOptional TypeAliasUnion)logger) NOT_GIVEN)NotGiven)ChatCompletionMessageParam#ChatCompletionToolChoiceOptionParam)Image) AdapterType ToolsSchema) AudioRawFrame)OpenAILLMContextc&eZdZUdZeed<eed<y)LLMSpecificMessagezA container for a context message that is specific to a particular LLM service. Enables the use of service-specific message types while maintaining compatibility with the universal LLM context format. llmmessageN)__name__ __module__ __qualname____doc__str__annotations__r\/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/processors/aggregators/llm_context.pyrr2s H LrrLLMContextMessagecJeZdZdZed'dZdeefdeee de e zde e zfdZ ed dd d ed ed eede fdZed dd d ededeeefded eede f dZed dd d edeed ede fdZedee fdZdee fdZd(deedee fdZede e zfdZede e zfdZde fdZdee fdZdee fdZefde e zfd Z de e zfd!Z!dd d"dedeeefded eed ef d#Z"dd$deed efd%Z#ede e zde e zfd&Z$y)) LLMContextzManages conversation context for LLM interactions. Handles message history, tool definitions, tool choices, and multimedia content for LLM conversations. Provides methods for message manipulation, and content formatting. returncdddl}|j5|jd|jdtdddd|j }t |trtgtj|i}t|j||jS#1swYhxYw) aCreate a universal LLM context from an OpenAI-specific context. NOTE: this should only be used internally, for facilitating migration from OpenAILLMContext to LLMContext. New user code should use LLMContext directly. .. deprecated:: 0.0.99 `from_openai_context()` is deprecated and will be removed in a future version. Directly use the universal `LLMContext` and `LLMContextAggregatorPair` instead. See `OpenAILLMContext` docstring for migration guide. Args: openai_context: The OpenAI LLM context to convert. Returns: New LLMContext instance with converted messages and settings. rNalwayszfrom_openai_context() (likely invoked by create_context_aggregator()) is deprecated and will be removed in a future version. Directly use the universal LLMContext and LLMContextAggregatorPair instead. See OpenAILLMContext docstring for migration guide. stacklevel)standard_tools custom_tools)messagestools tool_choice)warningscatch_warnings simplefilterwarnDeprecationWarningr- isinstancelistrrSHIMr# get_messagesr.)openai_contextr/converted_toolss r from_openai_contextzLLMContext.from_openai_contextIs&   $ $ &   ! !( + MMF#   ).. ot ,)!1A1A?0SO#002!&22  %  s *B&&B/Nr,r-r.c\|r|ng|_tj||_||_y)zInitialize the LLM context. Args: messages: Initial list of conversation messages. tools: Available tools for the LLM to use. tool_choice: Tool selection strategy for the LLM. N) _messagesr#_normalize_and_validate_tools_tools _tool_choice)selfr,r-r.s r __init__zLLMContext.__init__vs+?G(B.8.V.VW\.] =Hruser)roletextrCurlrDchg}|r|jd|d|jdd|id||dS)zCreate a context message containing an image URL. Args: role: The role of this message (defaults to "user"). url: The URL of the image. text: Optional text to include with the image. rDtyperD image_urlrE)rHrIrCcontent)append)rCrErDrKs r create_image_url_messagez#LLMContext.create_image_url_messages=  NNFD9 : 5#,GH11rformatsizeimagecKjdfd}tj|d{}drndd|}tj |||S7(w)aCreate a context message containing an image. Args: role: The role of this message (defaults to "user"). format: Image format (e.g., 'RGB', 'RGBA', or, if already encoded, the MIME type like 'image/jpeg'). size: Image dimensions as (width, height) tuple. image: Raw image bytes. text: Optional text to include with the image. zimage/cr}nLtj}tjj |d|j }t j|jd}|S)NJPEG)rNutf-8) ioBytesIOr frombytessavegetvaluebase64 b64encodedecode)bytesbuffer encoded_imagerNrPimage_already_encodedrOs r encode_imagez5LLMContext.create_image_message..encode_imagese$e499&9P)",,U3::7CM rNzdata:z image/jpegz;base64,)rCrErD) startswithasyncio to_threadr#rM) rCrNrOrPrDrar_rEr`s ``` @r create_image_messagezLLMContext.create_image_messagesk(!' 1 1( ; !&// ==  5f<HQ^P_`22#D2QQ >s2A#A!)A#z Audio follows audio_framescKd|dg}fd}tj|d{}|jd|ddd||d S7 w) aCreate a context message containing audio. Args: role: The role of this message (defaults to "user"). audio_frames: List of audio frame objects to include. text: Optional text to include with the audio. rDrGc dj}dj}djdD}tj5}t j |d5}|jd|j||j||j|dddtj|jjd}ddd|S#1swYExYw#1swYSxYw)Nrrc34K|]}|jywN)audio).0frames r zHLLMContext.create_audio_message..encode_audio..sBEEKKBswbr'rT) sample_rate num_channelsjoinrUrVwaveopen setsampwidth setnchannels setframerate writeframesrZr[rYr\)rprqdatar^wf encoded_audiorfs r encode_audioz5LLMContext.create_audio_message..encode_audios&q/55K'?77L88B\BBD TYYvt,)OOA&OOL1OOK0NN4( ) !' 0 01B C J J7 S  T! )) T! s% C8#AC,(:C8,C5 1C88DN input_audiowav)ryrN)rHr}rJ)rcrdrL)rCrfrDrKr|r{s ` r create_audio_messagezLLMContext.create_audio_messages^#D12 ! &// == %(5G  11>s$A A!A c"|jS)a1Get the current messages list. NOTE: This is equivalent to calling `get_messages()` with no filter. If you want to filter out LLM-specific messages that don't pertain to your LLM, use `get_messages()` directly. Returns: List of conversation messages. )r7r@s r r,zLLMContext.messagess  ""rcddl}|j5|jd|jdtdddd|j S#1swY|j SxYw)aGet messages suitable for persistent storage. NOTE: the only reason this method exists is because we're "silently" switching from OpenAILLMContext to LLMContext under the hood in some services and don't want to trip up users who may have been relying on this method, which is part of the public API of OpenAILLMContext but doesn't need to be for LLMContext. .. deprecated:: 0.0.92 Use `get_messages()` instead. Returns: List of conversation messages. rNr&zPget_messages_for_persistent_storage() is deprecated, use get_messages() instead.r'r()r/r0r1r2r3r7)r@r/s r #get_messages_for_persistent_storagez.LLMContext.get_messages_for_persistent_storagesm   $ $ &   ! !( + MMb"     ""   ""s *AA/llm_specific_filterc| |jS|jDcgc]#}t|tr|j|k(r|%}}t |t |jkrt j d|d|Scc}w)aGet the current messages list. Args: llm_specific_filter: Optional filter to return LLM-specific messages for the given LLM, in addition to the standard messages. If messages end up being filtered, an error will be logged; this is intended to catch accidental use of incompatible LLM-specific messages. Returns: List of conversation messages. z> !~~ c#56#''EX:X    !C$7 7 LLNObNccef !  s(Bc|jS)zFGet the tools list. Returns: Tools list. )r>rs r r-zLLMContext.tools.s{{rc|jS)zjGet the current tool choice setting. Returns: The tool choice configuration. r?rs r r.zLLMContext.tool_choice7s   rrc:|jj|y)zAdd a single message to the context. Args: message: The message to add to the conversation history. N)r<rL)r@rs r add_messagezLLMContext.add_message@s g&rc:|jj|y)zAdd multiple messages to the context. Args: messages: List of messages to add to the conversation history. N)r<extendr@r,s r add_messageszLLMContext.add_messagesHs h'rc"||jddy)zReplace all messages in the context. Args: messages: New list of messages to replace the current history. N)r<rs r set_messageszLLMContext.set_messagesPs %qrc8tj||_y)z|Set the available tools for the LLM. Args: tools: A ToolsSchema or NOT_GIVEN to disable tools. N)r#r=r>)r@r-s r set_toolszLLMContext.set_toolsXs !>>uE rc||_y)zxSet the tool choice configuration. Args: tool_choice: Tool selection strategy for the LLM. Nr)r@r.s r set_tool_choicezLLMContext.set_tool_choice`s (r)rDrCcvKtj|||||d{}|j|y7w)aAdd a message containing an image frame. Args: format: Image format (e.g., 'RGB', 'RGBA', or, if already encoded, the MIME type like 'image/jpeg'). size: Image dimensions as (width, height) tuple. image: Raw image bytes. text: Optional text to include with the image. role: The role of this message (defaults to "user"). )rCrNrOrPrDN)r#rer)r@rNrOrPrDrCrs r add_image_frame_messagez"LLMContext.add_image_frame_messagehsB&#77f4u48   ! s 979)rDcpKtj||d{}|j|y7w)zAdd a message containing audio frames. Args: audio_frames: List of audio frame objects to include. text: Optional text to include with the audio. )rfrDN)r#rr)r@rfrDrs r add_audio_frames_messagez#LLMContext.add_audio_frames_messages5#77\X\7]] !^s 646ct|tr |js|jstS|S|turtSt dt |)zNormalize and validate the given tools. Raises: TypeError: If tools are not a ToolsSchema or NotGiven. zJIn LLMContext, tools must be a ToolsSchema object or NOT_GIVEN. Got type: )r4rr*r+r TypeErrorrH)r-s r r=z(LLMContext._normalize_and_validate_toolssW e[ )''0B0B  L i  \]abg]h\ij r)r8rr$r#rj)%rrrr staticmethodr:r rrr!rr LLMContextToolChoicerArrMtupleintr]rer5rrpropertyr,rr7r-r.rrrrrrrr=rrr r#r#As* * \7;(17@ I4 123IX%I*H4 I"" 222sm 2  22*# $R$R$RCHo $R  $R sm $R $R$RLo%2%2-1--@%2HK%2 %2%2N #$01 # ##T:K5L#6! !N_I`!4{X-!1H<!!'#4'(T*;%<(%T*;%<%9BF{X5F(+?(+J(#""CHo "  " sm ""2AP "#M2 ":= "[8-C V^H^rr#)'rrcrZrUrs dataclassesrtypingrrrrrr logurur openai._typesr OPEN_AI_NOT_GIVENr OpenAINotGivenopenai.types.chatr rPILr%pipecat.adapters.schemas.tools_schemarrpipecat.frames.framesr1pipecat.processors.aggregators.openai_llm_contextrLLMStandardMessagerrr!rr#rrr rs !GG84J/R0:     %%79K%KL9L[[r