qi\eVddlZddlmZddlZddlmZmZmZmZm Z ddl m Z ddlm Z ddl mZddlmZdd lmZmZdd lmZdd lmZdd lmZmZmZdd lmZddlmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0Gdde0eZ1Gdde.Z2Gdde'Z3Gdde&Z4Gdde jjZ6Gdd e"Z7Gd!d"e!Z8Gd#d$e*Z9Gd%d&e)Z:Gd'd(e,Z;ed)*Gd+d,e;Z<Gd-d.e+Z=gd/Z>y)0N)Callable) Tokenizerdecoders normalizerspre_tokenizers processors)Unigram)nn)create_bidirectional_mask)BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TokenizersBackend)TransformersKwargsauto_docstringcan_return_tuple)merge_with_config_defaults)capture_outputs)LlamaAttentionLlamaRotaryEmbeddingapply_rotary_pos_embeager_attention_forward)ParakeetCTCConfigParakeetEncoderConfig)ParakeetEncoderBlock ParakeetEncoderConvolutionModuleParakeetForCTCParakeetPreTrainedModel)ParakeetProcessor) T5Tokenizerc ZeZdZ d dZ d deeezdededzdedef dZy) LasrTokenizerNc R||_|q|D cgc]} dt| vs| } } t| dkr!|t|D cgc]} d| d c} z }nC|dkDr>|t| k7r0t d|d|dt|D cgc]} d| d } } | }|||_nbt|dft|dft|dfd g|_t|dz d d D]#} |j j d| ddf%tt|j d d |_ |$tj||j_ tjtjtj dddg|j_t%j ddd|j_t)j*d|||||d| t-j.ddggdd|j0fg|j_ycc} wcc} wcc} w)Nz rzBoth extra_ids (z!) and additional_special_tokens (zm) are provided to LasrTokenizer. In this case the additional_special_tokens must include the extra_ids tokens)▁gr F)unk_id byte_fallbackr*alwaysT) replacementprepend_schemesplit) eos_token unk_token pad_token extra_idsadditional_special_tokens$A)r7r8z$Br8)singlepairspecial_tokens) _extra_idsstrlenrange ValueError _vocab_scoresappendrr _tokenizerr Precompiled normalizerrSequenceWhitespaceSplit Metaspace pre_tokenizerrdecoderr__init__rTemplateProcessing eos_token_idpost_processor) selfr2r3r4_spm_precompiled_charsmapr5r6vocab vocab_filekwargsx extra_tokensis W/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/lasr/modular_lasr.pyrLzLasrTokenizer.__init__,s]$ % 0'@[!LTWXYTZDZA[< 1$)yIY-ZA 1#Q.?-ZZ)Q9L0A#A &yk1RSlRmn 8=Y7GH!j1-HLH(4 %  !&D Y%Y%Y% "D  9q="b1 D""))Zs!+-**+* &k\-ZIsHH H H$ token_idsskip_special_tokensclean_up_tokenization_spaces group_tokensreturnc t|tr|g}|r%tj|Dcgc]}|d }}|Dcgc]}||jk7s|}}t j |f|||d|Scc}wcc}w)Nr)rYrZr[) isinstanceint itertoolsgroupby pad_token_idr_decode)rPrYrZr[r\rT token_grouptokens rXrdzLasrTokenizer._decodeys i %" I ;D;L;LY;WXKQXIX)2PuUd>O>O5OUP P ((   3)E     YQs A4A9A9)r8zzNdNNN)FNT) __name__ __module__ __qualname__rLr`listboolr>rdr<rXr%r%+so"&"&K `%*48!  c? " '+Tk      rmr%c eZdZy) LasrProcessorNrhrirjr<rmrXrorosrmroc\eZdZdZddddddddd d d d d dd d d dddddgddgddffd ZxZS)LasrEncoderConfiga This is the configuration class to store the configuration of a [`LasrEncoder`]. It is used to instantiate a `LasrEncoder` model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 512): Dimension of the layers and the hidden states. num_hidden_layers (`int`, *optional*, defaults to 17): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer encoder. intermediate_size (`int`, *optional*, defaults to 2048): Dimension of the "intermediate" (often named feed-forward) layer in the Transformer encoder. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the encoder and pooler. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the attention layers. convolution_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in convolutions of the conformer's convolution module. conv_kernel_size (`int`, *optional*, defaults to 32): The kernel size of the convolution layers in the Conformer block. subsampling_conv_channels (`int`, *optional*, defaults to 256): The number of channels in the subsampling convolution layers. subsampling_conv_kernel_size (`int`, *optional*, defaults to 5): The kernel size of the subsampling convolution layers. subsampling_conv_stride (`int`, *optional*, defaults to 2): The stride of the subsampling convolution layers. num_mel_bins (`int`, *optional*, defaults to 128): Number of mel features. dropout (`float`, *optional*, defaults to 0.1): The dropout ratio for all fully connected layers in the embeddings, encoder, and pooler. dropout_positions (`float`, *optional*, defaults to 0.0): The dropout ratio for the positions in the input sequence. layerdrop (`float`, *optional*, defaults to 0.1): The dropout ratio for the layers in the encoder. activation_dropout (`float`, *optional*, defaults to 0.1): The dropout ratio for activations inside the fully connected layer. attention_dropout (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention layers. max_position_embeddings (`int`, *optional*, defaults to 10000): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. layer_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the layer normalization layers. feed_forward_residual_weights (`tuple[float, float]`, *optional*, defaults to `[1.5, 0.5]`): The residual weights for the feed forward layers. conv_residual_weights (`tuple[float, float]`, *optional*, defaults to `[2.0, 1.0]`): The residual weights for the convolution layers. batch_norm_momentum (`float`, *optional*, defaults to 0.01): The momentum for the batch normalization layers. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. Example: ```python >>> from transformers import LasrEncoderModel, LasrEncoderConfig >>> # Initializing a `LasrEncoder` configuration >>> configuration = LasrEncoderConfig() >>> # Initializing a model from the configuration >>> model = LasrEncoderModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` This configuration class is based on the LasrEncoder architecture from Google Health AI. You can find more details and pre-trained models at [TODO/TODO](https://huggingface.co/TODO/TODO). isiluF rg?r)i'g{Gz?gư>g?g?g@g?g{Gz?Nc ||_||_||_||_||_t |did|d|d|d|d|d|d|d|d | d | d | d | d | d|d|d|d|d|d|||`|`y)N hidden_sizenum_hidden_layersnum_attention_headsintermediate_size hidden_actattention_biasconvolution_biasconv_kernel_sizesubsampling_conv_channels num_mel_binssubsampling_conv_kernel_sizesubsampling_conv_stridedropoutdropout_positions layerdropactivation_dropoutattention_dropoutmax_position_embeddingsinitializer_ranger<) rope_parameterslayer_norm_epsfeed_forward_residual_weightsconv_residual_weightsbatch_norm_momentumsuperrLsubsampling_factor scale_input)rPr|r}r~rrrrrrrrrrrrrrrrrrrrrrT __class__s rXrLzLasrEncoderConfig.__init__s8 /,-J*%:"#6   # / !4 0  "  *  . . '@ & *F %<  0   2! "0# $%<% &0) .  #  rm)rhrirj__doc__rL __classcell__rs@rXrrrrsfK^"%%& ! %'*Cj"Cj 3::rmrrcJeZdZdZ ddeezffd ZedZxZ S) LasrCTCConfiga This is the configuration class to store the configuration of a [`LasrForCTC`]. It is used to instantiate a Lasr CTC model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PreTrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 512): Vocabulary size of the model. ctc_loss_reduction (`str`, *optional*, defaults to `"mean"`): Specifies the reduction to apply to the output of `torch.nn.CTCLoss`. Only relevant when training an instance of [`LasrForCTC`]. ctc_zero_infinity (`bool`, *optional*, defaults to `True`): Whether to zero infinite losses and the associated gradients of `torch.nn.CTCLoss`. Infinite losses mainly occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance of [`LasrForCTC`]. encoder_config (`Union[dict, LasrEncoderConfig]`, *optional*): The config object or dictionary of the encoder. pad_token_id (`int`, *optional*, defaults to 0): Padding token id. Also used as blank token id. Example: ```python >>> from transformers import LasrForCTC, LasrCTCConfig >>> # Initializing a Lasr configuration >>> configuration = LasrCTCConfig() >>> # Initializing a model from the configuration >>> model = LasrForCTC(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` This configuration class is based on the Lasr CTC architecture from Google Health AI. You can find more details and pre-trained models at [TODO/TODO](https://huggingface.co/TODO/TODO). encoder_configc 0t|d|||||d|y)N) vocab_sizectc_loss_reductionctc_zero_infinityrrcr<)rrL)rPrrrrrcrTrs rXrLzLasrCTCConfig.__init__Cs0  !1/)%    rmc4|jjdzS)Nr)rr)rPs rXinputs_to_logits_ratioz$LasrCTCConfig.inputs_to_logits_ratioUs""::A==rm)rsmeanTNr) rhrirjrdictrrrLpropertyrrrs@rXrr!sCF!37  00  $>>rmrc\eZdZdeffd Zdej dej fdZxZS)LasrEncoderSubsamplingconfigc(t|tj|j|j |_tj|j |j |j|j|_ tj|j |j|j|j|_ tj|j|j |_ tj|_y)N) kernel_sizestride)rrLr Linearrr|dense_0Conv1drrconv_0rconv_1dense_1ReLUact_fn)rPrrs rXrLzLasrEncoderSubsampling.__init__[s yy!4!4f6H6HI ii      ;;11   ii     , ,;;11   yy!A!A6CUCUV ggi rminput_featuresr]c,|j|j|}|jdd}|j|j|}|j|j |}|jdd}|j |S)Nr'r)rr transposerrr)rPr hidden_statess rXforwardzLasrEncoderSubsampling.forwardmsz DLL$@A %//15  DKK $>?  DKK $>? %//15 ||M**rm) rhrirjrrrLtorchTensorrrrs@rXrrZs+ 0 $+ell+u||+rmrc eZdZy)LasrEncoderRotaryEmbeddingNrpr<rmrXrrvsrmrc eZdZdedeffd Z d dejdeejejfdzdejdzde e d eejejff d Z xZ S) LasrEncoderAttentionr layer_idxc4t|||d|_y)NF)rrL is_causalrPrrrs rXrLzLasrEncoderAttention.__init__zs +rmNrposition_embeddingsattention_maskrTr]c |jdd}g|d|j}|j|j|j dd}|j |j|j dd}|j |j|j dd} |\} } t||| | \}}tj|jjt} | |||| |f|jsdn |j|jd|\} }| j g|dj#} |j%| } | |fS)Nr+r'rr))rscaling)shapehead_dimq_projviewrk_projv_projrr get_interfacer_attn_implementationrtrainingrrreshape contiguouso_proj)rPrrrrT input_shape hidden_shape query_states key_states value_statescossinattention_interface attn_output attn_weightss rXrzLasrEncoderAttention.forward~sk$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &S#7 jRUWZ#[ j(?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((rmNN) rhrirjrrr`rLrrtuplerrrrrs@rXrrys0SIM.2 ")||")#5<<#=>E") t+ ") +, ") u||U\\) * ")rmrc&eZdZddeffd ZxZS)LasrEncoderConvolutionModulerct|||d|_tj|j |j |_y)Nsame)momentum)rrLpaddingr BatchNorm1dr|rnorm)rPr module_configrs rXrLz%LasrEncoderConvolutionModule.__init__s7 / NN6#5#5@Z@Z[ rmN)rhrirjrrrLrrs@rXrrs\0\\rmrc eZdZdedeffd Z d dejdejdzdejdzdee d ejf d Z xZ S) LasrEncoderBlockrrcTt||||j|_|j|_t j |j |jd|_t j |j |jd|_ t j |j |jd|_ t j |j |jd|_ t j |j |jd|_ y)NF)bias) rrLrrr LayerNormr|rnorm_feed_forward1 norm_self_att norm_convnorm_feed_forward2norm_outrs rXrLzLasrEncoderBlock.__init__s +-3-Q-Q*%+%A%A""$,,v/A/A6CXCX_d"e\\&*<*S>SZ_`f&8&8&:O:OV[\"$,,v/A/A6CXCX_d"e V%7%79N9NUZ[ rmNrrrrTr]c 0|}|j|j|}|jd|z|jd|zz}|j|}|jd|||d|\}}||z}|j |j ||} |jd|z|jd| zz}|}|j|j|}|jd|z|jd|zz}|j|}|S)Nrr')rrr)rr<) feed_forward1rrr self_attnconvrr feed_forward2rr) rPrrrrTresidualnormalized_hidden_statesr_ conv_outputs rXrzLasrEncoderBlock.forwardsJ!**4+B+B=+QR  . .q 1H >> from transformers import AutoProcessor, LasrEncoder >>> from datasets import load_dataset, Audio >>> model_id = TODO >>> processor = AutoProcessor.from_pretrained(model_id) >>> encoder = ParakeetEncoder.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> encoder_outputs = encoder(**inputs) >>> print(encoder_outputs.last_hidden_state.shape) ``` r')devicer)pr) target_length)r inputs_embedsrFT)rr)last_hidden_state)rrrarangerr unsqueezer functionalrrr_get_output_attention_maskr rrrandrrr ) rPrrrTrrr encoder_layerto_dropdropout_probabilitys rXrzLasrEncoder.forwards>7 ?? 5<< (;(;A(>}G[G[\ffghi S --mt||VZVcVc-d mm##C4+A+ADMM#Zmm##C4+A+ADMM#Z  %!<<^[h[n[nop[q<rN2;;')  "[[ MG}}&+jjn#&7"G -!!#1),c ! !    m4 ??rmr)rhrirjrr__annotations__base_model_prefixrLrrrrrrrrr rrrs@rXr r s !0"/3?@ ?@ t+?@+, ?@  ?@ ?@rmr ceZdZfdZxZS) LasrForCTCc 8tjdi|S)a Example: ```python >>> from transformers import AutoProcessor, LasrForCTC >>> from datasets import load_dataset, Audio >>> model_id = TODO >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = LasrForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> predicted_ids = model.generate(**inputs) >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> print(transcription) ``` r<)rgenerate) super_kwargsrs rXr'zLasrForCTC.generateOs, uw/,//rm)rhrirjr'rrs@rXr%r%Ns 00rmr%)r%r rrorrrr%)?racollections.abcrr tokenizersrrrrrtokenizers.modelsr r masking_utilsr modeling_outputsr modeling_utilsrrprocessing_utilsrtokenization_utils_tokenizersrutilsrrr utils.genericrutils.output_capturingrllama.modeling_llamarrrrparakeet.configuration_parakeetrrparakeet.modeling_parakeetrrr r!parakeet.processing_parakeetr"t5.tokenization_t5r#r%rorrrModulerrrrrrr r%__all__r<rmrXr;s!$ SS%6/F&>II75vvV =,d K!2d N % H-HV6>%6>r+RYY+8')T\#C\.+.b1& X@%X@  X@v004 rm