�
��abc @ sB d d l Z d d l Z d d l m Z d e f d � � YZ d S( i����Ni ( t ProbingStatet
CharSetProberc B s� e Z d Z d
d � Z d � Z e d � � Z d � Z e d � � Z d � Z
e d � � Z e d � � Z
e d � � Z RS( gffffff�?c C s( d | _ | | _ t j t � | _ d S( N( t Nonet _statet lang_filtert loggingt getLoggert __name__t logger( t selfR ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/chardet/charsetprober.pyt __init__' s c C s t j | _ d S( N( R t DETECTINGR ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/chardet/charsetprober.pyt reset, s c C s d S( N( R ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/chardet/charsetprober.pyt charset_name/ s c C s d S( N( ( R t buf( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/chardet/charsetprober.pyt feed3 s c C s | j S( N( R ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/chardet/charsetprober.pyt state6 s c C s d S( Ng ( ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/chardet/charsetprober.pyt get_confidence: s c C s t j d d | � } | S( Ns ([ -])+t ( t ret sub( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/chardet/charsetprober.pyt filter_high_byte_only= s c C sz t � } t j d | � } xX | D]P } | j | d � | d } | j � re | d k re d } n | j | � q" W| S( s5
We define three types of bytes:
alphabet: english alphabets [a-zA-Z]
international: international characters [�-�]
marker: everything else [^a-zA-Z�-�]
The input buffer can be thought to contain a series of words delimited
by markers. This function works to filter all words that contain at
least one international character. All contiguous sequences of markers
are replaced by a single space ascii character.
This filter applies to all scripts which do not use English characters.
s% [a-zA-Z]*[�-�]+[a-zA-Z]*[^a-zA-Z�-�]?i����s �R ( t bytearrayR t findallt extendt isalpha( R t filteredt wordst wordt last_char( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/chardet/charsetprober.pyt filter_international_wordsB s
c C s� t � } t } d } x� t t | � � D]� } | | | d !} | d k rT t } n | d k ri t } n | d k r( | j � r( | | k r� | r� | j | | | !� | j d � n | d } q( q( W| s� | j | | � n | S( s�
Returns a copy of ``buf`` that retains only the sequences of English
alphabet and high byte characters that are not between <> characters.
Also retains English alphabet and high byte characters immediately
before occurrences of >.
This filter can be applied to all scripts which contain both English
characters and extended ASCII characters, but is currently only used by
``Latin1Prober``.
i i t >t s