qi1 d dl Zd dlZddlmZ ddlmZ ddlmZ ddl m
Z
m Z ddl m
Z
e j e Zded ed
ed ed ed
ej& f dZ e
d G d de ZdgZy) N ) hertz_to_mel)SequenceFeatureExtractor) BatchFeature)
TensorTypelogging)requires num_mel_binsnum_spectrogram_bins sample_ratelower_edge_hertzupper_edge_hertzreturnc t j }d}|dz }t j d||| |d } t | d ddt j f }
t j t |d t |d | dz | } | dd t j ddf | dd
t j ddf | dd t j ddf }}
} |
| z
|
| z
z }||
z
||
z
z }t j
dt j || }t j ||d gd d gg j | S ) z.NumPy-port of the JAX mel weight matrix logic. g @ )dtypeNkaldi) mel_scale r ) npfloat64linspacer newaxismaximumminimumpadastype)r
r r r
r r internal_dtype
bands_to_zero
nyquist_hertzlinear_frequenciesspectrogram_bins_meledgeslower_edge_mel
center_melupper_edge_mel lower_slopes upper_slopesmel_weights_matrixs b/opt/pipecat/venv/lib/python3.12/site-packages/transformers/models/lasr/feature_extraction_lasr.pylinear_to_mel_weight_matrixr. sJ ZZN M #%MS-9MUcderest'(:gNqRTR\R\}] KK%9%9q
E cr
2::q=!
a BJJM"
ab "**a- !/JN )>9j>>YZL"%99nz>YZLCL,)OP
66$
q'9Aq6&B C J J5 QQ )torch)backendsc e Zd ZdZddgZ d fd ZddZ ddej e e
z e ej z e e e
z de d e dz d
e
ez dz d e dz d e
dz d
e dz de dz de dz de
dz de dz defdZ xZS )LasrFeatureExtractora7
Constructs a LASR feature extractor.
This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains
most of the main methods. Users should refer to this superclass for more information regarding those methods.
This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the `Short Time
Fourier Transform` which should match pytorch's `torch.stft` equivalent.
Args:
feature_size (`int`, *optional*, defaults to 128):
The feature dimension of the extracted features.
sampling_rate (`int`, *optional*, defaults to 16000):
The sampling rate at which the audio files should be digitalized expressed in hertz (Hz).
hop_length (`int`, *optional*, defaults to 160):
Length of the overlapping windows for the STFT used to obtain the Mel Frequency coefficients.
n_fft (`int`, *optional*, defaults to 512):
Size of the Fourier transform.
win_length (`int`, *optional*, defaults to 400):
The window length for the STFT computation.
padding_value (`float`, *optional*, defaults to 0.0):
Padding value used to pad the audio. Should correspond to silences.
input_featuresattention_maskc
t | d|||d| || _ || _ || _ t j t ||dz dz |ddt j | _
y )N) feature_size
sampling_rate
padding_valuer r g @_@g L@)r
r r r
r r ) super__init__
hop_lengthn_fft
win_lengthr0
from_numpyr. r r mel_filters) selfr7 r8 r= r> r? r9 kwargs __class__s r- r< z LasrFeatureExtractor.__init__^ sn wl-_lwpvw$
$ ++ ' )%*aZ!^ )!&!'jj
r/ c t j | j d|t j }|j t j }|j d| j | j }t j j ||z | j }t j | dz }| j j | }t j ||z d }t j | }|S )NF)periodicdevicer r )nr gh㈵>)min)r0 hann_windowr? r tounfoldr= fftrfftr> absrA clamplog) rB waveformrG windowframesstft
power_specrA mel_specs r- _torch_extract_fbank_featuresz2LasrFeatureExtractor._torch_extract_fbank_featuresx s ""4??U6Y^YfYfg;;u}}-
T__dooFyy~~fvo~<YYt_)
&&))&1 ;;zK 7TB99X&r/ N
raw_speech
truncationpad_to_multiple_ofreturn_tensorsreturn_attention_maskpadding
max_lengthr8 do_normalizerG return_token_timestampsr c |O|| j k7 rmt d| j j d| j d| j d| d t j d| j j d t
|t j rt j | }nqt
|t t f r[t
|d t t j f r#|D
cg c] }
t j |
}}
nt j | }t
|t j xr t |j d
kD }|rVt |j d kD r>t j d | j j d
|j! d }t
|t t f }|r^|D ]Y }
t |
j d
kD s t j d | j j d
|
j! d }
[ |s|r4|D
cg c]( }
|
dddf j# t j$ * }}
n'|dddf j# t j$ g}t' d|i }| j) ||||||d }|j* j- d }| j/ ||
}d|j# t j$ i}|rO|j0 dd| j2 d
z
d| j4 f }|j# t j6 |d< t' || S c c}
w c c}
w )a
Main method to featurize and prepare for the model one or several sequence(s). Implementation uses PyTorch for
the STFT computation if available, otherwise a slower NumPy based one.
Args:
raw_speech (`np.ndarray`, `list[float]`, `list[np.ndarray]`, `list[list[float]]`):
The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float
values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not
stereo, i.e. single float per timestep.
truncation (`bool`, *optional*, default to `True`):
Activates truncation to cut input sequences longer than *max_length* to *max_length*.
pad_to_multiple_of (`int`, *optional*, defaults to None):
If set will pad the sequence to a multiple of the provided value.
This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
`>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128.
return_attention_mask (`bool`, *optional*):
Whether to return the attention mask. If left to the default, will return the attention mask according
to the specific feature_extractor's default.
[What are attention masks?](../glossary#attention-mask)
For Parakeet models, `attention_mask` should always be passed for batched inference, to avoid subtle
bugs.
return_tensors (`str` or [`~utils.TensorType`], *optional*):
If set, will return tensors instead of list of python integers. Acceptable values are:
- `'tf'`: Return TensorFlow `tf.constant` objects.
- `'pt'`: Return PyTorch `torch.Tensor` objects.
- `'np'`: Return Numpy `np.ndarray` objects.
sampling_rate (`int`, *optional*):
The sampling rate at which the `raw_speech` input was sampled. It is strongly recommended to pass
`sampling_rate` at the forward call to prevent silent errors and allow automatic speech recognition
pipeline.
padding_value (`float`, *optional*, defaults to 0.0):
The value that is used to fill the padding values / vectors.
do_normalize (`bool`, *optional*, defaults to `False`):
Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly
improve the performance of the model.
device (`str`, *optional*, defaults to `'cpu'`):
Specifies the device for computation of the log-mel spectrogram of audio signals in the
`_torch_extract_fbank_features` method. (e.g., "cpu", "cuda")
return_token_timestamps (`bool`, *optional*, defaults to `None`):
Deprecated. Use `return_attention_mask` instead from which the number of frames can be inferred.
Whether or not to return the number of frames of the input raw_speech.
These num_frames can be used by the model to compute word level timestamps.
Nz3The model corresponding to this feature extractor: z& was trained using a sampling rate of zI. Please make sure that the provided `raw_speech` input was sampled with z and not .zDIt is strongly recommended to pass the `sampling_rate` argument to `zN()`. Failing to do so can result in silent errors that might be hard to debug.r r r z2Only mono-channel audio is supported for input to z;. We will take the mean of the channels to convert to mono.r r4 pt)r^ r_ rZ r[ r] r\ r5 )data tensor_type) r8
ValueErrorrD __name__loggerwarning
isinstancer ndarrayr0 tensorlisttupleTensorlenshapemeanrK float32r r r4 squeezerX r5 r? r= bool)rB rY rZ r[ r\ r] r^ r_ r8 r` rG ra rC speechis_batched_torchis_batched_sequencebatched_speech
padded_inputsr4 re r5 s r- __call__z LasrFeatureExtractor.__call__ s! H $ 2 22 I$..JaJaIb c))-););(< =))-););(
! * 9 9!T__q=P=cTXTcTc=c:c dN%3%6%6uzz%BD! ">BB_ M. Vs , M*-M) i> i i r )cpu)
FNNNlongestNNNr N)rh
__module__ __qualname____doc__model_input_namesr< rX r rl rn floatrv intstrr r r|
__classcell__)rD s @r- r3 r3 B s< 0 *+;<
4, !)-26-1 '!%$($( "/3ECJJe,tBJJ/??$tE{BSSEC EC $J EC
j(4/ EC $d{
EC tEC $JEC TzEC TkEC d
EC "&EC
ECr/ r3 )numpyr r0 audio_utilsr !feature_extraction_sequence_utilsr feature_extraction_utilsr utilsr r utils.import_utilsr
get_loggerrh ri r r rl r. r3 __all__r: r/ r- r s ' I 4 ( *
H %"R"R "R "R "R
"R ZZ"RJ
: MC3 MC MC` "
"r/