| Server IP : 54.37.205.81 / Your IP : 216.73.216.76 Web Server : nginx/1.22.1 System : Linux vps-249481fa 6.1.0-50-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.176-1 (2026-07-02) x86_64 User : debian ( 1000) PHP Version : 8.2.32 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : OFF Directory : /lib/python3/dist-packages/lunr/__pycache__/ |
Upload File : |
�
WIbN � �< � d dl mZ d dlmZ d dlmZ dZd� Zdd�ZdS ) � )�deepcopy)�Token)� as_stringu
-c � � | o| t v S �N)�SEPARATOR_CHARS)�chars �0/usr/lib/python3/dist-packages/lunr/tokenizer.py�default_separatorr s � � � +�D�O�+�+� Nc �v ��� | �g S �pi �t | t t f� � r�fd�| D � � S ��t }nt �� � r�}n�fd�}t | � � � � � }t |� � }g }d}t |dz � � D ]�}||k r|| nd} ||z
}
|| � � s||k rt|
dk rii } ||
g| d< t |� � | d< | � �� � t ||� � } |� t || | � � � � |dz }��|S ) a� Splits a string into tokens ready to be inserted into the search index.
Args:
metadata (dict): Optional metadata can be passed to the tokenizer, this
metadata will be cloned and added as metadata to every token that is
created from the object to be tokenized.
separator (callable or compiled regex): This tokenizer will convert its
parameter to a string by calling `str` and then will split this
string on characters for which `separator` is True. Lists will have
their elements converted to strings and wrapped in a lunr `Token`.
Returns:
List of Token instances.
Nc � �� g | ]>}t t |� � � � � t �� � � � ��?S � )r r �lowerr )�.0�element�metadatas �r
�
<listcomp>z Tokenizer.<locals>.<listcomp>" sL �� �
�
�
�FM�E�)�G�$�$�*�*�,�,�h�x�.@�.@� A� A�
�
�
r c �. �� �� | � � S r )�match)�c� separators �r
�<lambda>zTokenizer.<locals>.<lambda>+ s �� �� � ��!3�!3� r r � � �position�index)
�
isinstance�list�tupler �callable�strr �len�range�update�slice�appendr )
�objr r � is_separator�string�length�tokens� slice_start� slice_endr � slice_length�token_metadata�sls
`` r
� Tokenizerr2
s� ��� � �{�� ��~�2�H��#��e�}�%�%�
�
�
�
�
�QT�
�
�
�
� ��(� � � �)� � � 4� � � �3�3�3�3� �
��X�X�^�^�
�
�F�
��[�[�F�
�F��K��6�A�:�&�&�
(�
(� �$-��$7�$7�v�i� � �R�� �;�.� � �<�� � �
(��f�!4�!4��a� � �!#��.9�<�-H��z�*�*-�f�+�+��w�'� �%�%�h�/�/�/��;� �2�2���
�
�e�F�2�J�� ?� ?�@�@�@�#�a�-�K�� �Mr )NN) �copyr �
lunr.tokenr �
lunr.utilsr r r r2 r r r
�<module>r6 sm �� � � � � � � � � � � � � � � � � � �$��,� ,� ,�3� 3� 3� 3� 3� 3r