Iui0dZddlmZmZmZddlZddlmZm Z m Z Gdde jZ Gdde jZ Gd d e jZGd d e jZGd de jZGdde jZGdde j$ZGdde j(ZGdde j,ZGdde j0ZGdde j,ZGdde j0Zy)z0Declares specification of the Transformer model.)OptionalTupleUnionN)attention_spec common_spec model_specc/LeZdZddejj dej jddddddddddddddddfdedede d e d ejd ed ej d e de de de de de de ede ede ede de e jde de de ede e de f.dZy)TransformerEncoderSpecTFN' num_layers num_headspre_norm no_final_norm activationnum_source_embeddingsembeddings_mergelayernorm_embeddingrelative_positionrelative_attention_biasffn_glurms_normmulti_query_attention num_heads_kvhead_dim rotary_dimrotary_interleaverotary_scaling_typerotary_scaling_factor rotary_basesliding_windowqk_normpre_post_layer_normc | r||dk7r tdd}| |_tjdj ||_||_tjdj ||_tjdj ||_t|Dcgc]}tjc}|_ d|_ | s| st|_|r|stj | |_|rtj | |_|)tjdj ||_t|Dcgc]}t)| | | | ||||||||| c}|_ycc}wcc}w) a'Initializes a Transformer encoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. head_dim: Number of dimensions per attention head. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. sliding_window: Max sequence length to retain in KV Cache. qk_norm: Apply layer normalization to the query and key projections. pre_post_layer_norm: Add post layer norm for each pre norm layer. Nr 5Enabling multi_query_attention implies num_heads_kv=1int16int8Trint32) rrrrrrrrrrr r"r#) ValueErrorrnpdtypetyperrrrrangerEmbeddingsSpec embeddingsscale_embeddingsPositionEncoderSpecposition_encodings LayerNormSpec layer_normrr!TransformerEncoderLayerSpeclayer)selfr rrrrrrrrrrrrrrrrrrr r!r"r#_s T/opt/pipecat/venv/lib/python3.12/site-packages/ctranslate2/specs/transformer_spec.py__init__zTransformerEncoderSpec.__init__ swx !'LA,= KL%:"'*// :  ((6*// ; " 0 5 56F G278M2N -.K & & ( !% )@&9&;D # M)77JDO '2'@'@('SD $  %"$((7"3"8"8"HD $:&!  ("3(?!)!%"3$7&;'$7     s -FF )__name__ __module__ __qualname__r ActivationRELUEmbeddingsMergeCONCATintboolrrRotaryScalingTypefloatr;r:r r s #-8-C-C-H-H%&8C8S8S8Z8Z$)"'(-&+&*"&$("&JN'("(,"'$)1f f f  f  f  ** f  #f &55f "f  f "&f f f  $f smf 3-!f "SM#f $ %f &&n&F&FG'f ( %)f *+f ,! -f .$/f 0"1f rHr cJeZdZdejj ddddddddddddddddddddddddddddddddf"ded ed ed ejd ed ededededededededededededeededee jde de deded ed!ed"ed#ed$eed%eed&eed'eejd(eed)eed*ed+eefHd,Z ed-Zy).TransformerDecoderSpecTFr Nr rr rrrrwith_encoder_attentionrproject_in_outrralignment_layeralignment_headsrralibialibi_use_positive_positions scale_alibirrrrr original_max_position_embeddingsmax_position_embeddingsparallel_residualshared_layer_normr#rrrr! quant_typequant_group_size quant_bitsr" external_pre_post_encoder_layersc% lt|_|r|s td|r td|r||dk7r tdd}tjdj ||_||_tjdj ||_tjdj | |_ tjdj | |_ tj|_ d|_tj |_||_||_||_|)tjd j ||_| s| s|s|t-|_|r|stj0| |_|rtj0| |_tj6|_t;|D%cgc]G}%t=d$id |d | d | d| d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|#d|$Ic}%|_d|_ |xs||k7|jd <|r2tj6|_!tj6|_"| r.| |jd!<|"|jd"<|!|jd#<yycc}%w)%a. Initializes a Transformer decoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. with_encoder_attention: Enable the encoder attention sublayers. no_final_norm: Disable the final layer norm in the pre-norm architecture. project_in_out: Add linear transformations after the embedding layer and before the final layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: Add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. sliding_window: Max sequence length to retain in KV Cache. quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) external_pre_post_encoder_layers: if the encoder attention pre and processing is done outside the attention. z/The GPT-J block expects a pre-norm architecturez-The GPT-J block does not have cross attentionNr r%r&r'Tr)r(rLrrrrrrrrr rSrTrUrVr#rrr!r"rZFrquantization_typequantization_bitsquantization_group_sizerG)#dict_configr*r+r,r-rrrrNrOrr/r0r1rOPTIONAL scale_outputsrPrQrRr!r2r3r4r5r LinearSpec projectionr.TransformerDecoderLayerSpecr7start_from_zero_embedding project_in project_out)&r8r rrrrrLrrMrrrNrOrrrPrQrRrrrrr rSrTrUrVr#rrrr!rWrXrYr"rZr9s& r:r;zTransformerDecoderSpec.__init__uspv  !RSS% !PQQ 'LA,= KL'*// :  ((6*// ;!xx055oF!xx055oF%446 $'00 ,H)&  %"$((7"3"8"8"HD !+"&9&;D # M)77JDO '2'@'@('SD $%0020:&/ .- ( '= "3 )@    "   &  #4 %8 '< ( 2R )@ #4 #4 %8  *! ""# $ .% & ' (2R)   2*/&0E1 I % ,- )446DO*557D  0:DLL, -0:DLL, -6FDLL2 3 E sA J1c|jSN)r`r8s r:configzTransformerDecoderSpec.configs ||rH)r<r=r>rr?r@rCrDrrrErF Quantizationr;propertyrlrGrHr:rJrJts -8-C-C-H-H$)'+#$"'(-! -2!$("&JN'("01'("'"'$)&+&*"&(,9=*.$(;@KdGdGdG dG  ** dG " dG!%dGdGdG dG"&dGdGdGdGdG !dG"'+#dG$%dG&SM'dG( )dG*&n&F&FG+dG, %-dG./dG0+.1dG2"%3dG4 5dG6 7dG8"9dG: $;dG<sm=dG>3-?dG@! AdGB[556CdGD#3-EdGFSMGdGHIdGJ+34.KdGLrHrJc jeZdZ d deededeejdededef dZ y) r6Nrrrrr r#ctjd|||||||| | | | |  |_t|||_|rt j ||_t j ||_t j ||_ t j ||_ t|jdt|jdyy)NT) self_attentionrrrrrr!rrrrr r"glurr(r5) rMultiHeadAttentionSpecrqFeedForwardSpecffnrr4input_layer_normpost_attention_layer_normpre_feedforward_layer_normpost_feedforward_layer_normdelattr)r8rrrrrrr!rrrrr r"r#s r:r;z$TransformerEncoderLayerSpec.__init__!s"-CC/$;%)!/ 3"7# #wB $/$=$=x$PD !-8-F-F!.D */:.G.G!/D +0;/H/H!0D , D'' 6 DHHl + rH)FFFFNNNNTNr r FF) r<r=r>rrCrDrrErFr;rGrHr:r6r6 s~ %$("&JN'("$)/,SM/, /,&n&F&FG/, %/,/,"/,rHr6c<eZdZ ddZy)reNcdtjd||||||| | | | |||||_|r"tj||||||du|_t |||_| rz|rt j|_n2t j|_ t j|_ t|jdt|j d|rt j||_ t j||_ |r8|r6t j||_ t j||_ t j||_t j||_t|jdt|j dyy)NT)rqrrrrrrrr rSrTrrr!r"F)rrrr!r"has_normrrr5r()rrtrq attentionrurvrr4rVrwrxr{*external_post_encoder_attention_layer_norm)external_pre_encoder_attention_layer_normryrz)r8rLrrrrrrrrr rSrTrUrVr#rrr!r"rZs r:r;z$TransformerDecoderLayerSpec.__init__Ts.-CC/$;!/ 3"7#-M$;%) $ "+BB!)!-9UB DN#wB  )4)B)B)D&(3(A(A(C%1<1J1J1L. D'' 6 DHHl + $/$=$=x$PD !-8-F-F!.D *&*J--x@? --x@> /:.G.G!/D +0;/H/H!0D , D'' 6 DHHl +1 rH)TFFFFNTNr r rrFFFNNNFFr<r=r>r;rGrHr:rereSsF $ % )* !!).+W,rHreceZdZddZy)ructj||_tj|_tj|_|rtj|_yy)Nr()rr4r5rclinear_0linear_1linear_0_noact)r8rsrs r:r;zFeedForwardSpec.__init__sM%33XF#..0 #..0 "-"8"8":D  rHN)FFrrGrHr:rurus;rHruceZdZdZy)r2c.tj|_yrj)rra encodingsrks r:r;zPositionEncoderSpec.__init__s#,,rHNrrGrHr:r2r2s-rHr2c0eZdZdZddeeffd ZxZS)TransformerConfigz%Configuration for Transformer models.layer_norm_epsilonc (t|dd|i|y)zInitializes the configuration for Transformer models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNrGsuperr;r8rkwargs __class__s r:r;zTransformerConfig.__init__ I,>I&IrHrjr<r=r>__doc__rrFr; __classcell__rs@r:rrs/J8E?JJrHrc TeZdZdZdedeffd Zedddejjdddejjdddddf d e eeeeffd ed ed ed edejdedededejdededededefdZedZedZdZdZdZxZS)TransformerSpeczDescribes a Transformer model. The specification is invariant to hidden dimensions but requires to explicitly set the number of layers and attention heads. encoderdecoderc t|ts tdt|ts tdt|||_||_|jjd|j jy)zInitializes a Transformer model specification. Args: encoder: The encoder specification. decoder: The decoder specification. 1encoder argument must be a TransformerEncoderSpec1decoder argument must be a TransformerDecoderSpecrN) isinstancer TypeErrorrJrr;rrr` add_attributer)r8rrrs r:r;zTransformerSpec.__init__sm'#9:OP P'#9:OP P    "" #T\\%G%G rHFTrKr r rwith_relative_positionrrrrNrOrrrrrrrct|ttfr|\}}n||}}t|||||| | | || | || }t |||||| || ||| || }|||S)aCreates a Transformer model specification. Args: num_layers: Number of encoder and decoder layers, or a 2-tuple if the number is different. num_heads: Number of attention heads. with_relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layer as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention. ) rrrrrrrrrrr) rrrrrrrNrOrrr)rlisttupler rJ)clsr rrrrrrNrOrrrrrrrnum_encoder_layersnum_decoder_layersrrs r: from_configzTransformerSpec.from_configsV j4- 05? 2  25? 2 (  '!"7- 34$;"7  )  '! 34$;++"7  7G$$rHcy)NrrGrks r:namezTransformerSpec.name7s rHcy)NrGrks r:revisionzTransformerSpec.revision;rHctSrj)rrks r:get_default_configz"TransformerSpec.get_default_config?s  ""rHc|jjDcgc]}|jjdc}Scc}wNrrr0weightshape)r8specs r:get_source_vocabulary_sizez*TransformerSpec.get_source_vocabulary_sizeBs/151H1HI !!!$IIIs <c\|jjjjdSrrr0rrrks r:get_target_vocabulary_sizez*TransformerSpec.get_target_vocabulary_sizeE#||&&--33A66rH)r<r=r>rr rJr; classmethodrr?r@rArBrrCrrDrrnrrrrrrrs@r:rrsn  - 8N * (-#-8-C-C-H-H! %&8C8S8S8Z8Z$)(-&+!O%#uS#X./O%O%!% O%  O%  O% **O%O%O% #O%&55O%"O%"&O%O%O% $!O%O%b!!#J7rHrc0eZdZdZddeeffd ZxZS)TransformerDecoderModelConfigz-Configuration for Transformer decoder models.rc (t|dd|i|y)zInitializes the configuration for Transformer decoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNrGrrs r:r;z&TransformerDecoderModelConfig.__init__LrrHrjrrs@r:rrI7J8E?JJrHrc@eZdZdZdeffd Zedejjddddddddddddddd d dddddddddddfd e d e d e d ejde de de de de de de de de de e de de e jdedede de de de d e d!e d"e e d#e e d$e e d%e ej d&e e d'e e d(e f>d)Zed*Zed+Zd,Zd-ZxZS).TransformerDecoderModelSpecz3Describes a Transformer decoder model (e.g. GPT-2).rct|ts tdt|||_|j j jD]!\}}|jj||#y)z|Initializes a Transformer decoder model specification. Args: decoder: The decoder specification. rN) rrJrrr;rrlitemsr`r)r8rkeyvaluers r:r;z$TransformerDecoderModelSpec.__init__Ysh '#9:OP P  ,,--335 3JC LL & &sE 2 3rHTFNr r rr rrrrrrMrrrrPrQrRrrrrr rSrTrUrVr#rrrr!rWrXrYr"c t||fid|d|d|ddd|d|d|d | d | d | d | d | d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|} || S) a! Creates a Transformer decoder model specification. Args: num_layers: Number of decoder layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. no_final_norm: Do not apply layer normalization after the last decoder block. project_in_out: Add a linear layer after the embedding layer and another one before the final output projection. with_relative_position: Enable relative position representations modules. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. head_dim: Number of head sliding_window: max sequence length to retain KV cache quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) rrrrLFrrMrrrrPrQrRrrrrr rSrTrUrVr#rrrr!rWrXrYr")rJ)!rr rrrrrrMrrrrPrQrRrrrrr rSrTrUrVr#rrrr!rWrXrYr"rs! r:rz'TransformerDecoderModelSpec.from_configgsRX)  ! ! " ! !4 ! $) ! (! *! 5! ! ! ! *F! $! "! 0!! "!4#! $#8%! &$'! (.N)! *%<+! ,0-! .0/! 0!41! 2#83! 4&5! 67! 8*9! :";! <.=! >"?! @A! F7|rHcy)NrJrGrks r:rz TransformerDecoderModelSpec.name'rHcy)NrGrks r:rz$TransformerDecoderModelSpec.revisionrrHctSrj)rrks r:rz.TransformerDecoderModelSpec.get_default_config ,..rHc\|jjjjdSrrrks r:get_vocabulary_sizez/TransformerDecoderModelSpec.get_vocabulary_sizerrH)r<r=r>rrJr;rrr?r@rCrDrrrErFrmrrnrrrrrrs@r:rrVsK= 3 6 3 -8-C-C-H-H$)#$',-2!$("&JN'("01'("'"'$)&+&*"&(,9=*.$(Annn n  ** n " nnn!%nnnn'+nnSMn  !n"&n&F&FG#n$ %%n&'n(+.)n*"%+n, -n. /n0"1n2 $3n4sm5n63-7n8! 9n:[556;n<#3-=n>SM?n@Ann`((/7rHrc0eZdZdZddeeffd ZxZS)TransformerEncoderModelConfigz-Configuration for Transformer encoder models.rc (t|dd|i|y)zInitializes the configuration for Transformer encoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNrGrrs r:r;z&TransformerEncoderModelConfig.__init__rrHrjrrs@r:rrrrHrceZdZdZdej j fdededej ffd Z e dZ e dZ d Z d ZxZS) TransformerEncoderModelSpecz2Describes a Transformer encoder model (e.g. BERT).Fr pooling_layerpooling_activationcPt|ts tdt|||_|j jd|j j|rCtj|_ tjdj||_yy)zInitializes a Transformer encoder model specification. Args: encoder: The encoder specification. pooling_layer: Add the pooling layer. pooling_activation: The activation to apply after the pooling layer. rrr'N)rr rrr;rr`rrrrc pooler_denser+r,r-pooler_activation)r8rrrrs r:r;z$TransformerEncoderModelSpec.__init__s'#9:OP P   "" #T\\%G%G   + 6 6 8D %'XXf%5%:%:;M%ND " rHcy)Nr rGrks r:rz TransformerEncoderModelSpec.namerrHcy)Nr rGrks r:rz$TransformerEncoderModelSpec.revisionrrHctSrj)rrks r:rz.TransformerEncoderModelSpec.get_default_configrrHcb|jjdjjdSrrrks r:rz/TransformerEncoderModelSpec.get_vocabulary_sizes(||&&q)0066q99rH)r<r=r>rrr?Tanhr rDr;rnrrrrrrs@r:rrsw< $5@5K5K5P5P O'OO(22 O4((/:rHr)rtypingrrrnumpyr+ctranslate2.specsrrr LayerSpecr rJr6rerur2SequenceToSequenceModelConfigrSequenceToSequenceModelSpecrLanguageModelConfigrLanguageModelSpecrrrrGrHr:rs6))EEg Z11g TiZ11iX0,*"6"60,fX,*"6"6X,v;j**;-*..- J @@ J}7j<<}7@ JJ$B$B JN7*">">N7b JJ$B$B J):*">">):rH