| Server IP : 54.37.205.81 / Your IP : 216.73.216.76 Web Server : nginx/1.22.1 System : Linux vps-249481fa 6.1.0-50-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.176-1 (2026-07-02) x86_64 User : debian ( 1000) PHP Version : 8.2.32 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : OFF Directory : /lib/python3/dist-packages/nltk/__pycache__/ |
Upload File : |
�
T�c2 � �� � d dl Z d dlmZ d dlmZ d dlmZ eddg�� � Z e j e�� � e j � � d � � � � � Z
e
� d
� � e j d d d
d�� � e j dd ddd�� � e j dddd�� � e j dddd�� � e j d d d d �� � d � � � � � � � � � � � � � Z
dS )!� N)�tqdm)�
word_tokenize)�parallelize_preprocessz-hz--help)�help_option_names)�context_settingsc � � d S )N� r � �*/usr/lib/python3/dist-packages/nltk/cli.py�clir s � � �Dr
�tokenizez
--languagez-l�enz1The language for the Punkt sentence tokenization.)�default�helpz--preserve-lineTzIAn option to keep the preserve the sentence and not sentence tokenize it.)r �is_flagr z --processesz-j� zNo. of processes.z
--encodingz-e�utf8zSpecify encoding of file.z --delimiterz-d� z%Specify delimiter to join the tokens.c
� � t j d|�� � 5 }t j d|�� � 5 }|dk rWt |� � � � � D ]4}t |� t
|� � � � d|�� � �5nRt t |� � � |d�� � D ]'}t |� |� � d|�� � �(d d d � � n # 1 swxY w Y d d d � � d S # 1 swxY w Y d S )
z;This command tokenizes text stream using nltk.word_tokenize�stdin)�encoding�stdoutr �
)�end�fileT)� progress_barN)�click�get_text_streamr � readlines�print�joinr r ) �language�
preserve_line� processesr � delimiter�fin�fout�line�outlines r �
tokenize_filer* s� � �*
� �w�� :� :� :� H�c�
�
"�8�h�
?�
?�
?�
H�4� �A�~�~� �� � � � � 1� 1� T� T�D��)�.�.� �t�)<�)<�=�=�4�d�S�S�S�S�S�T� 6�!�3�=�=�?�?�I�D� � � � H� H�G� �)�.�.��1�1�t�$�G�G�G�G�G�
H�
H�
H�
H�
H�
H�
H�
H�
H�
H�
H����
H�
H�
H�
H� H� H� H� H� H� H� H� H� H� H� H� H���� H� H� H� H� H� Hs5 �D�B0C*� D�*C. �.D�1C. �2D�D� D)r r �nltkr � nltk.utilr �dict�CONTEXT_SETTINGS�group�version_optionr �command�optionr* r r
r �<module>r3 s� ��
� � � � � � � � � � � � � � � � ,� ,� ,� ,� ,� ,��4�4��*:�;�;�;� �
��� .�/�/�/������ � � �� 0�/� � ���Z���
�� ��� � <� � � � �� ��� � � T� � � � �� �m�T�1�3F�G�G�G�
�� �l�D�&�7R�S�S�S�
�� ��4��+R�� � �
H�
H�� � T�S� H�G�� �
� � ��&
H�
H�
Hr