| Server IP : 54.37.205.81 / Your IP : 216.73.216.76 Web Server : nginx/1.22.1 System : Linux vps-249481fa 6.1.0-50-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.176-1 (2026-07-02) x86_64 User : debian ( 1000) PHP Version : 8.2.32 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : OFF Directory : /lib/python3/dist-packages/nltk/__pycache__/ |
Upload File : |
�
T�c�8 � � � d Z ddlZddlmZmZmZmZ ddlm Z m
Z
ddlmZ ddl
mZ G d� d� � Z G d � d
e� � Z G d� de� � Z G d
� de� � Zdd�Zedk rfddlZddlmZ edej d z � � Zn
# e$ r dZY nw xY w edej d z � � Zn
# e$ r dZY nw xY w eee� � g d�ZdS )a�
Tools to identify collocations --- words that often appear consecutively
--- within corpora. They may also be used to find other associations between
word occurrences.
See Manning and Schutze ch. 5 at https://nlp.stanford.edu/fsnlp/promo/colloc.pdf
and the Text::NSP Perl package at http://ngram.sourceforge.net
Finding collocations requires first calculating the frequencies of words and
their appearance in the context of other words. Often the collection of words
will then requiring filtering to only retain useful content terms. Each ngram
of words may then be scored according to some association measure, in order
to determine the relative likelihood of each ngram being a collocation.
The ``BigramCollocationFinder`` and ``TrigramCollocationFinder`` classes provide
these functionalities, dependent on being provided a function which scores a
ngram given appropriate frequency counts. A number of standard association
measures are provided in bigram_measures and trigram_measures.
� N)�BigramAssocMeasures�ContingencyMeasures�QuadgramAssocMeasures�TrigramAssocMeasures)�ranks_from_scores�spearman_correlation)�FreqDist)�ngramsc � � e Zd ZdZd� Ze dd�� � Zed� � � Zed� � � Z d� fd �Z
d
� Zd� Zd� Z
d
� Zd� Zd� Zd� ZdS )�AbstractCollocationFindera�
An abstract base class for collocation finders whose purpose is to
collect collocation candidate frequencies, filter and rank them.
As a minimum, collocation finders require the frequencies of each
word in a corpus, and the joint frequency of word tuples. This data
should be provided through nltk.probability.FreqDist objects or an
identical interface.
c �T � || _ |� � � | _ || _ d S �N)�word_fd�N�ngram_fd)�selfr r s �3/usr/lib/python3/dist-packages/nltk/collocations.py�__init__z"AbstractCollocationFinder.__init__: s# � ����������� ��
�
�
� FNc �� �� |f|dz
z �|r+t j � �fd�|D � � � � S |r+t j � �fd�|D � � � � S dS )zU
Pad the document with the place holder according to the window_size
� c 3 �B �K � | ]}t j |�� � V � �d S r ��
_itertools�chain��.0�doc�paddings �r � <genexpr>zAAbstractCollocationFinder._build_new_documents.<locals>.<genexpr>H sA �� � � � 2� 2�36�
� ��g�.�.�2� 2� 2� 2� 2� 2r c 3 �B �K � | ]}t j �|� � V � �d S r r r s �r r zAAbstractCollocationFinder._build_new_documents.<locals>.<genexpr>L sA �� � � � 2� 2�36�
� ��#�.�.�2� 2� 2� 2� 2� 2r N)r r �
from_iterable)�cls� documents�window_size�pad_left� pad_right�
pad_symbolr s @r �_build_new_documentsz.AbstractCollocationFinder._build_new_documents? s� �� � �-�;��?�3��� ��#�1�1� 2� 2� 2� 2�:C�2� 2� 2� � �
� � ��#�1�1� 2� 2� 2� 2�:C�2� 2� 2� � �
� � r c �b � | � | � || j d�� � � � S )z�Constructs a collocation finder given a collection of documents,
each of which is a list (or iterable) of tokens.
T�r'