�
��abc ! @` s d d l m Z m Z m Z d d l m Z m Z d d l m Z m Z d d l
Z
d d l Z d d l m
Z
d d l m Z m Z m Z m Z d d l m Z d d l m Z d d
l m Z y d d l m Z Wn e k
r� e Z n Xe g e D] Z e j d � ^ q� � Z e g e D] Z e j d � ^ q"� Z e g e D] Z e j d � ^ qJ� Z e e d
d g � BZ d Z! e j" r�e! d d k r�e! j# d � d k s�t$ � e j% e! d e& d � d � Z' n e j% e! � Z' e( d d d d d d d d d d d d d d d! d" d# d$ d% d& d' d( d) d* d+ d, d- d. d/ d0 d1 d2 g � Z) e j% d3 � Z* i Z+ d4 e, f d5 � � YZ- d6 � Z. d7 e, f d8 � � YZ/ d9 e/ f d: � � YZ0 d; e1 f d< � � YZ2 d= e, f d> � � YZ3 d? e, f d@ � � YZ4 dA � Z5 d S(B i ( t absolute_importt divisiont unicode_literals( t text_typet binary_type( t http_clientt urllibN( t webencodingsi ( t EOFt spaceCharacterst asciiLetterst asciiUppercase( t ReparseException( t _utils( t StringIO( t BytesIOu asciit >t t | j | � | k rd | t | j | � 8} | d 7} q' W| | g | _ d S( Ni i ( t _bufferedBytest AssertionErrorR R R ( R R t offsett i( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt seekL s c C` sp | j s | j | � S| j d t | j � k r_ | j d t | j d � k r_ | j | � S| j | � Sd S( Ni i i����( R t _readStreamR R t _readFromBuffer( R t bytes( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt readU s
c C` s& t g | j D] } t | � ^ q
� S( N( t sumR R ( R t item( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR ^ s c C` sL | j j | � } | j j | � | j d c d 7 Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
u i i u utf-8u certainN(
R
t supports_lone_surrogatest Nonet reportCharacterErrorsR t characterErrorsUCS4t characterErrorsUCS2t newLinest lookupEncodingt charEncodingt
openStreamt
dataStreamt reset( R R? ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR � s c C` sC d | _ d | _ d | _ g | _ d | _ d | _ d | _ d S( Nu i ( R t chunkSizet chunkOffsett errorst prevNumLinest prevNumColsRF t _bufferedCharacter( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRO � s c C` s( t | d � r | } n t | � } | S( uv Produces a file object from source.
source can be either a file object, local filename or a string.
u read( R: R ( R R? R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRM � s c C` st | j } | j d d | � } | j | } | j d d | � } | d k r\ | j | } n | | d } | | f S( Nu
i i����i ( R t countRS t rfindRT ( R R R t nLinest positionLinet lastLinePost positionColumn( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt _position� s
c C` s& | j | j � \ } } | d | f S( u: Returns (line, col) of the current position in the stream.i ( R\ RQ ( R t linet col( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR � s c C` sL | j | j k r% | j � s% t Sn | j } | j | } | d | _ | S( uo Read one character from the stream or queue if available. Return
EOF when EOF is reached.
i ( RQ RP t readChunkR R ( R RQ t char( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR` � s
c C` sO | d k r | j } n | j | j � \ | _ | _ d | _ d | _ d | _ | j j | � } | j
r� | j
| } d | _
n
| s� t St | � d k r� t
| d � } | d k s� d | k o� d k n r� | d | _
| d } q� n | j r| j | � n | j d d � } | j d
d � } | | _ t | � | _ t S( Nu i i i����i
i � i�� u
u
u
( RF t _defaultChunkSizeR\ RP RS RT R RQ RN R$ RU R9 R t ordRG t replacet True( R RP R( t lastv( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR_ � s0
(
c C` s: x3 t t t j | � � � D] } | j j d � q Wd S( Nu invalid-codepoint( t rangeR t invalid_unicode_ret findallRR R' ( R R( t _( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRH % s "c C` s t } x� t j | � D]� } | r( q n t | j � � } | j � } t j | | | d !� r� t j | | | d !� } | t k r� | j
j d � n t } q | d k r� | d k r� | t
| � d k r� | j
j d � q t } | j
j d � q Wd S( Ni u invalid-codepointi � i�� i ( R9 Rg t finditerRb t groupt startR
t isSurrogatePairt surrogatePairToCodepointt non_bmp_invalid_codepointsRR R' Rd R ( R R( t skipt matcht codepointR t char_val( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRI ) s c
C` s} y t | | f } Wn� t k
r� x& | D] } t | � d k s+ t � q+ Wd j g | D] } d t | � ^ qZ � } | s� d | } n t j d | � } t | | f c B` sb e Z d Z d d d d d e d � Z d � Z d � Z e d � Z d � Z d � Z
d � Z RS(
u� Provides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
u windows-1252c C` s� | j | � | _ t j | | j � d | _ d | _ | | _ | | _ | | _ | | _ | | _
| j | � | _ | j d d k s� t � | j � d S( u� Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
i id i N( RM t rawStreamR= R t numBytesMetat numBytesChardett override_encodingt transport_encodingt same_origin_parent_encodingt likely_encodingt default_encodingt determineEncodingRL RF R RO ( R R? R� R� R� R� R� t
useChardet( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR � s c C` s3 | j d j j | j d � | _ t j | � d S( Ni u replace( RL t
codec_infot streamreaderR� RN R= RO ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRO � s "c C` sU t | d � r | } n t | � } y | j | j � � Wn t | � } n X| S( uv Produces a file object from source.
source can be either a file object, local filename or a string.
u read( R: R R R R ( R R? R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRM � s c C` s! | j � d f } | d d k r& | St | j � d f } | d d k rO | St | j � d f } | d d k rx | S| j � d f } | d d k r� | St | j � d f } | d d k r� | d j j d � r� | St | j � d f } | d d k r| S| r�y d d l
m } Wn t k
r4q�Xg } | � } x[ | j
s�| j j | j � } t | t � szt � | s�Pn | j | � | j | � qGW| j � t | j d � } | j j d � | d k r�| d f Sn t | j � d f } | d d k r| St d � d f S( Nu certaini u tentativeu utf-16( t UniversalDetectoru encodingu windows-1252( t detectBOMRF RK R� R� t detectEncodingMetaR� t namet
startswithR� t chardet.universaldetectorR� t ImportErrort doneR� R$ R� R4 R# R R' t feedt closet resultR R� ( R t chardetRL R� t bufferst detectorR t encoding( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� � sR '
c C` s� | j d d k s t � t | � } | d k r5 d S| j d k re t d � } | d k s� t � nr | | j d k r� | j d d f | _ nF | j j d � | d f | _ | j � t d | j d | f � � d S( Ni u certainu utf-16beu utf-16leu utf-8i u Encoding changed from %s to %s( u utf-16beu utf-16le( RL R RK RF R� R� R RO R ( R t newEncoding( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt changeEncoding s
c C` s� i d t j 6d t j 6d t j 6d t j 6d t j 6} | j j d � } t | t � s_ t
� | j | d � } d } | s� | j | � } d } | s� | j | d � } d } q� n | r� | j j | � t
| � S| j j d � d
Sd
S( u� Attempts to detect at BOM at the start of the stream. If
an encoding can be determined from the BOM return the name of the
encoding otherwise return Noneu utf-8u utf-16leu utf-16beu utf-32leu utf-32bei i i i N( t codecst BOM_UTF8t BOM_UTF16_LEt BOM_UTF16_BEt BOM_UTF32_LEt BOM_UTF32_BER� R$ R4 R# R t getR RK RF ( R t bomDictt stringR� R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� s&
c C` s� | j j | j � } t | t � s* t � t | � } | j j d � | j � } | d k r| | j
d k r| t d � } n | S( u9 Report the encoding declared by the meta element
i u utf-16beu utf-16leu utf-8N( u utf-16beu utf-16le( R� R$ R� R4 R# R t EncodingParserR t getEncodingRF R� RK ( R R t parserR� ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� 9 s N( R1 R2 R3 RF Rd R RO RM R� R� R� R� ( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR> � s ( > "t
EncodingBytesc B` s� e Z d Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z
e e
e � Z d � Z
e e
� Z e d
� Z d � Z d � Z d
� Z RS( u� String-like object with an associated position and various extra methods
If the position is ever greater than the string length then an exception is
raisedc C` s+ t | t � s t � t j | | j � � S( N( R4 R# R t __new__t lower( R t value( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� L s c C` s
d | _ d S( Ni����( R\ ( R R� ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR P s c C` s | S( N( ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt __iter__T s c C` sS | j d } | _ | t | � k r/ t � n | d k rD t � n | | | d !S( Ni i ( R\ R t
StopIterationR< ( R t p( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt __next__W s c C` s
| j � S( N( R� ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt next_ s c C` sY | j } | t | � k r$ t � n | d k r9 t � n | d | _ } | | | d !S( Ni i ( R\ R R� R< ( R R� ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt previousc s c C` s+ | j t | � k r t � n | | _ d S( N( R\ R R� ( R R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt setPositionl s c C` s< | j t | � k r t � n | j d k r4 | j Sd Sd S( Ni ( R\ R R� RF ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt getPositionq s
c C` s | | j | j d !S( Ni ( R ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt getCurrentByte{ s c C` sc | j } xJ | t | � k rU | | | d !} | | k rH | | _ | S| d 7} q W| | _ d S( u Skip past a list of charactersi N( R R R\ RF ( R R{ R� R| ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRp � s c C` sc | j } xJ | t | � k rU | | | d !} | | k rH | | _ | S| d 7} q W| | _ d S( Ni ( R R R\ RF ( R R{ R� R| ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt skipUntil� s c C` sQ | j } | | | t | � !} | j | � } | rM | j t | � 7_ n | S( u� Look for a sequence of bytes at the start of a string. If the bytes
are found return True and advance the position to the byte after the
match. Otherwise return False and leave the position alone( R R R� ( R R# R� R( R, ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt
matchBytes� s c C` sh | | j j | � } | d k r^ | j d k r= d | _ n | j | t | � d 7_ t St � d S( u� Look for the next sequence of bytes matching a given sequence. If
a match is found advance the position to the last byte of the matchi����i i N( R t findR\ R Rd R� ( R R# t newPosition( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt jumpTo� s ( R1 R2 R3 R� R R� R� R� R� R� R� t propertyR R� t currentBytet spaceCharactersBytesRp R� R� R� ( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� H s R� c B` s_ e Z d Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z
d � Z RS(
u? Mini parser for detecting character encoding from meta elementsc C` s t | � | _ d | _ d S( u3 string - the data to work on for encoding detectionN( R� R( RF R� ( R R( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR � s c C` s� d | j f d | j f d | j f d | j f d | j f d | j f f } xv | j D]k } t } xR | D]J \ } } | j j | � rk y | � } PWq� t k
r� t } Pq� Xqk qk W| sX PqX qX W| j
S( Ns ( R( R� ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� � s c C` sK | j j t k r t St } d } x"t rF| j � } | d k rG t S| d d k r� | d d k } | rC| d k rC| | _ t Sq% | d d k r� | d } t | � } | d k rC| | _ t Sq% | d d k r% t t
| d � � } | j � } | d k rCt | � } | d k r@| r4| | _ t S| } q@qCq% q% Wd S( Ni s
http-equivi s content-typet charsett content( R( R� R� Rd R9 RF t getAttributeR� RK t ContentAttrParserR� t parse( R t hasPragmat pendingEncodingt attrt tentativeEncodingt codect
contentParser( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� � s:
c C` s
| j t � S( N( t handlePossibleTagR9 ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� � s c C` s t | j � | j t � S( N( R� R( R� Rd ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� � s
c C` s� | j } | j t k r9 | r5 | j � | j � n t S| j t � } | d k ra | j � n+ | j � } x | d k r� | j � } qp Wt S( NR (
R( R� t asciiLettersBytesR� R� Rd R� t spacesAngleBracketsR� RF ( R t endTagR( R| R� ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� � s
c C` s | j j d � S( NR ( R( R� ( R ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� s c C` s� | j } | j t t d g � B� } | d k sI t | � d k sI t � | d k rY d Sg } g } x� t r
| d k r� | r� Pnz | t k r� | j � } Pn^ | d
k r� d j | � d f S| t k r� | j
| j � � n | d k r� d S| j
| � t | � } qh W| d k r7| j
� d j | � d f St | � | j � } | d k r�| } x� t r�t | � } | | k r�t | � d j | � d j | � f S| t k r�| j
| j � � qb| j
| � qbWn^ | d k r�d j | � d f S| t k r | j
| j � � n | d k r-d S| j
| � x} t r�t | � } | t k rwd j | � d j | � f S| t k r�| j
| j � � q=| d k r�d S| j
| � q=Wd S( u_ Return a name,value pair for the next attribute in the stream,
if one is found, or Nonet /i R t =R) t 't "N( R N( R� R ( R� R� ( R( Rp R� t frozensetRF R R Rd R* t asciiUppercaseBytesR' R� R� R� R� ( R R( R| t attrNamet attrValuet quoteChar( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� sh $
( R1 R2 R3 R R� R� R� R� R� R� R� R� ( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� � s $ R� c B` s e Z d � Z d � Z RS( c C` s" t | t � s t � | | _ d S( N( R4 R# R R( ( R R( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR f s c C` s: y!| j j d � | j j d 7_ | j j � | j j d k sH d S| j j d 7_ | j j � | j j d k r� | j j } | j j d 7_ | j j } | j j | � r� | j | | j j !Sd SnP | j j } y( | j j t � | j | | j j !SWn t k
r | j | SXWn t k
r5d SXd S( NR� i R� R� R� ( R� R� ( R( R� R Rp R� RF R� R� R� ( R t quoteMarkt oldPosition( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� j s.
( R1 R2 R R� ( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyR� e s c C` sw t | t � r: y | j d � } Wq: t k
r6 d SXn | d k ro y t j | � SWqs t k
rk d SXn d Sd S( u{ Return the python codec name corresponding to an encoding or None if the
string doesn't correspond to a valid encoding.u asciiN( R4 R t decodet UnicodeDecodeErrorRF R t lookupt AttributeError( R� ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyRK � s
(6 t
__future__R R R t pip._vendor.sixR R t pip._vendor.six.movesR R R� Rv t pip._vendorR t constantsR R R
R R R) R
t ioR R R� R� R& t encodeR� R� R� R� t invalid_unicode_no_surrogateRE RV R Rw t evalRg t setRo t ascii_punctuation_reRt t objectR RD R= R> R# R� R� R� RK ( ( ( sE /usr/lib/python2.7/site-packages/pip/_vendor/html5lib/_inputstream.pyt sR "
((( +
J ��h�'