qi, DUdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddl mZ e jj!dehd Zeeed<ehdZeeed<eez Zeeed<e eefZdedej<dededef dZdede fdZ!dedeedeede de eee ff dZ"eGddZ#dee#defd Z$y#e$rC e j$dd n*#eef$r Ze j,d ed YdZ[ndZ[wwxYwYwxYw)!aText processing utilities for sentence boundary detection and tag parsing. This module provides utilities for natural language text processing including sentence boundary detection, email and number pattern handling, and XML-style tag parsing for structured text content. Dependencies: This module uses NLTK (Natural Language Toolkit) for robust sentence tokenization. NLTK is licensed under the Apache License 2.0. See: https://www.nltk.org/ Source: https://www.nltk.org/api/nltk.tokenize.punkt.html N) dataclass) FrozenSetListOptionalSequenceTuple)logger) sent_tokenizeztokenizers/punkt_tab punkt_tabT)quietz4Failed to download NLTK 'punkt_tab' tokenizer data: aE. This data is required for sentence tokenization features. The download failed due to filesystem permissions. To resolve: pre-install the data in a location with appropriate read permissions, or set the NLTK_DATA environment variable to point to a writable directory. See https://www.nltk.org/data.html for more information.>՜՞։؏؛؟۔।॥໌།༎၊။።፧፨។៕。！．；？｡…!.;?SENTENCE_ENDING_PUNCTUATION>r&r'r(r)r*"_LATIN_SENTENCE_ENDING_PUNCTUATION'UNAMBIGUOUS_SENTENCE_ENDING_PUNCTUATIONtextmatcholdnewreturnc|j}|j}|||j||}|d||z||dz}|S)aReplace occurrences of a substring within a matched section of text. Args: text: The input text in which replacements will be made. match: A regex match object representing the section of text to modify. old: The substring to be replaced. new: The substring to replace `old` with. Returns: The modified text with the specified replacements made within the matched section. N)startendreplace)r.r/r0r1r4r5 replacements F/opt/pipecat/venv/lib/python3.12/site-packages/pipecat/utils/string.py replace_matchr9jsR KKME ))+CuS/))#s3K <+ %ST 2D Kc|j}|syt|}|sy|d}t|dk(r@||k(r;|r|dtvr t|St |D]\}}|t vs|dzcSyt|dkDr t|Sy)aFind the position of the end of a sentence in the provided text. This function uses NLTK's sentence tokenizer to detect sentence boundaries in the input text, combined with punctuation verification to ensure that single tokens without proper sentence endings aren't considered complete sentences. Args: text: The input text in which to find the end of the sentence. Returns: The position of the end of the sentence if found, otherwise 0. r)rstripr lenr+ enumerater-)r. sentencesfirst_sentenceichs r8match_endofsentencerE}s ;;=D d#I q\N 9~~5 DH ;;t9  t_ EAr<<1u   9~>"" r:tags current_tagcurrent_tag_indexc |r|\}}|||dvr dt|fS||fS|D]g\}}||dj|}||dj|}|dk(r |dk(rd|fcS||kDr||ft|fcS||k(sZdt|fcSd|fS)aParse text to identify start and end tag pairs. If a start tag was previously found (i.e., current_tag is valid), wait for the corresponding end tag. Otherwise, wait for a start tag. This function returns the index in the text where parsing should continue in the next call and the current or new tags. Args: text: The text to be parsed. tags: List of tuples containing start and end tags. current_tag: The currently active tags, if any. current_tag_index: The current index in the text. Returns: A tuple containing None or the current tag and the index of the text. Nr)r?count) r.rFrGrH_end_tag start_tagstart_tag_count end_tag_counts r8parse_start_end_tagsrPs0 7 d,-. .#d)$ $.//#% 701288C./066w? a MQ$6+, , } ,(#d)4 4  -#d)$ $% # $$r:c,eZdZUdZeed<eed<dZy)TextPartForConcatenationaClass representing a part of text for concatenation with concatenate_aggregated_text. Parameters: text: The text content. includes_inter_part_spaces: Whether any necessary inter-frame (leading/trailing) spaces are already included in the text. r.includes_inter_part_spacescT|jd|jd|jdS)Nz(text: [z], includes_inter_part_spaces: ))namer.rS)selfs r8__str__z TextPartForConcatenation.__str__s,))HTYYK/NtOnOnNoopqqr:N)__name__ __module__ __qualname____doc__str__annotations__boolrXr:r8rRrRs I $$rr:rR text_partscrdd|sSdtffd }|D]}|jss |||jr r ||2|jssdz ||Ndjs"|jdjsdz ||j S)aConcatenate a list of text parts into a single string. This function joins the provided list of text parts into a single string, taking into account whether or not the parts already contain spacing. This function is useful for aggregating text segments received from LLMs or transcription services. Args: text_parts: A list of text parts to concatenate. Returns: A single concatenated string. Fpartc<|jz |jy)N)r.rS)rdlast_includes_inter_part_spacesresults r8 append_partz0concatenate_aggregated_text..append_parts $))*.*I*I'r: r=r)rRr.rSisspacestrip)rarhrdrfrgs @@r8concatenate_aggregated_textrlsF&+#  J2J yy      * */N  009X cMF  ":%%' ! 0D0D0F#   5D\\^F Mr:)%r\re dataclassesrtypingrrrrrnltklogurur nltk.tokenizer datafind LookupErrordownloadOSErrorPermissionErroreerror frozensetr+r]r^r,r- StartEndTagsMatchr9intrErPrRrlr`r:r8rs  !== '  IINN)*/8,./Ys^.d6??Z5["IcN[  "DD(3S#X BHH3SS&3 c3 c3 l)% )% < )%,')% )%  8L !3 &' )%X  r r  r ?D1I,J?s?c     k. _ %  B1#FG G     s;CDC21D2D9DDDDD