qi>dZddlmZddlmZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZeGd d e ZGd d eZy)zNVIDIA NIM API service implementation. This module provides a service for interacting with NVIDIA's NIM (NVIDIA Inference Microservice) API while maintaining compatibility with the OpenAI-style interface. ) dataclass)Optional) LLMTokenUsage) LLMContext)OpenAILLMContext)OpenAILLMSettings)OpenAILLMService)_warn_deprecated_paramceZdZdZy)NvidiaLLMSettingszSettings for NvidiaLLMService.N)__name__ __module__ __qualname____doc__M/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/services/nvidia/llm.pyr r s(rr c |eZdZUdZeZeed<dddddededeed eeffd Z d e e zffd Z d e fdZxZS)NvidiaLLMServicea]A service for interacting with NVIDIA's NIM (NVIDIA Inference Microservice) API. This service extends OpenAILLMService to work with NVIDIA's NIM API while maintaining compatibility with the OpenAI-style interface. It specifically handles the difference in token usage reporting between NIM (incremental) and OpenAI (final summary). _settingsz#https://integrate.api.nvidia.com/v1N)base_urlmodelsettingsapi_keyrrrc td}|tdtd||_||j|t |d|||d|d|_d|_d|_d|_ d|_ y) aInitialize the NvidiaLLMService. Args: api_key: The API key for accessing NVIDIA's NIM API. base_url: The base URL for NIM API. Defaults to "https://integrate.api.nvidia.com/v1". model: The model identifier to use. Defaults to "nvidia/llama-3.1-nemotron-70b-instruct". .. deprecated:: 0.0.105 Use ``settings=OpenAILLMSettings(model=...)`` instead. settings: Runtime-updatable settings. When provided alongside deprecated parameters, ``settings`` values take precedence. **kwargs: Additional keyword arguments passed to OpenAILLMService. z&nvidia/llama-3.1-nemotron-70b-instruct)rNr)rrrrFr) r r r apply_updatesuper__init___prompt_tokens_completion_tokens _total_tokens_has_reported_prompt_tokens_is_processing)selfrrrrkwargsdefault_settings __class__s rrzNvidiaLLMService.__init__*s2-3[\   "7,=w G%*  "    ) )( 3 a8FVaZ`a"#+0(#rcontextcKd|_d|_d|_d|_d|_ t ||d{d|_|jdkDs|jdkDrb|j|jz|_t|j|j|j}t |!|d{yy77#d|_|jdkDs|jdkDrc|j|jz|_t|j|j|j}t |!|d{7wwxYww)aProcess a context through the LLM and accumulate token usage metrics. This method overrides the parent class implementation to handle NVIDIA's incremental token reporting style, accumulating the counts and reporting them once at the end of processing. Args: context: The context to process, containing messages and other information needed for the LLM interaction. rFTN) prompt_tokenscompletion_tokens total_tokens) rr r!r"r#r_process_contextrstart_llm_usage_metrics)r$r(tokensr's rr-z!NvidiaLLMService._process_contextXsO "#+0(" >'*73 3 3"'D ""Q&$*A*AA*E%)%8%84;R;R%R"&"&"5"5&*&=&=!%!3!3 g5f===+F 4>#(D ""Q&$*A*AA*E%)%8%84;R;R%R"&"&"5"5&*&=&=!%!3!3 g5f===+FsK$EC CC BEC EC E BE EEEr/cK|jsy|js'|jdkDr|j|_d|_|j|j kDr|j|_yyw)aAccumulate token usage metrics during processing. This method intercepts the incremental token updates from NVIDIA's API and accumulates them instead of passing each update to the metrics system. The final accumulated totals are reported at the end of processing. Args: tokens: The token usage metrics for the current chunk of processing, containing prompt_tokens and completion_tokens counts. NrT)r#r"r*rr+r )r$r/s rr.z(NvidiaLLMService.start_llm_usage_metricsxsk"" //F4H4H14L"("6"6D /3D ,  # #d&=&= =&,&>&>D # >sA-A/)r rrrr Settings__annotations__strrrrrr-rr. __classcell__)r's@rrrsw!H  >#04 ,$,$ ,$ } ,$ ,- ,$\>.>.K>@?M?rrN)r dataclassesrtypingrpipecat.metrics.metricsr*pipecat.processors.aggregators.llm_contextr1pipecat.processors.aggregators.openai_llm_contextr pipecat.services.openai.base_llmrpipecat.services.openai.llmr pipecat.services.settingsr r rrrrr=sO "1AN>8<  )   o?'o?r