3
�Pf$+ � @ s� d dl mZmZmZ d dlmZ d dlmZ ddl m
Z
ddl m Z ddl m Z m
Z
ddl mZmZmZ dd l mZmZ dd
l mZ dd lmZ dd lmZ ee �ZG d
d� de�ZdS )� )�absolute_import�division�unicode_literals)�unichr)�deque� )�spaceCharacters)�entities)� asciiLetters�asciiUpper2Lower)�digits� hexDigits�EOF)�
tokenTypes�
tagTokenTypes)�replacementCharacters)�HTMLInputStream)�Triec sd e Zd ZdZd�� fdd� Zdd� Zdd� Zd�d
d �Zd d
� Zdd� Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd� Zd d � Zd d � Zd d!� Zd"d#� Zd$d%� Zd&d'� Zd(d)� Zd*d+� Zd,d-� Zd.d/� Zd0d1� Zd2d3� Zd4d5� Z d6d7� Z d8d9� Z d:d;� Z dd?� Z!d@dA� Z"dBdC� Z#dDdE� Z$dFdG� Z%dHdI� Z&dJdK� Z'dLdM� Z(dNdO� Z)dPdQ� Z*dRdS� Z+dTdU� Z,dVdW� Z-dXdY� Z.dZd[� Z/d\d]� Z0d^d_� Z1d`da� Z2dbdc� Z3ddde� Z4dfdg� Z5dhdi� Z6djdk� Z7dldm� Z8dndo� Z9dpdq� Z:drds� Z;dtdu� Zdzd{� Z?d|d}� Z@d~d� ZAd�d�� ZBd�d�� ZCd�d�� ZDd�d�� ZEd�d�� ZFd�d�� ZGd�d�� ZHd�d�� ZId�d�� ZJd�d�� ZKd�d�� ZL� ZMS )��
HTMLTokenizera This class takes care of tokenizing HTML.
* self.currentToken
Holds the token that is currently being processed.
* self.state
Holds a reference to the method to be invoked... XXX
* self.stream
Points to HTMLInputStream object.
Nc sF t |f|�| _|| _d| _g | _| j| _d| _d | _t t
| �j � d S )NF) r �stream�parserZ
escapeFlagZ
lastFourChars� dataState�state�escape� currentToken�superr �__init__)�selfr r �kwargs)� __class__� � /usr/lib/python3.6/_tokenizer.pyr " s zHTMLTokenizer.__init__c c s\ t g �| _xL| j� rVx&| jjr:td | jjjd�d�V qW x| jrR| jj� V q>W q W dS )z� This is where the magic happens.
We do our usually processing through the states and when we have a token
to return we yield the token which pauses processing until the next token
is requested.
�
ParseErrorr )�type�dataN)r �
tokenQueuer r �errorsr �pop�popleft)r r r r! �__iter__1 s
zHTMLTokenizer.__iter__c % C s( t }d}|rt}d}g }| jj� }x(||krJ|tk rJ|j|� | jj� }q$W tdj|�|�}|tkr�t| }| j jt
d dd|id�� �nld| ko�d kn s�|d
kr�d }| j jt
d dd|id�� �n(d | ko�d
kn �s�d| ko�dkn �s�d| k�odkn �s�d| k�o4dkn �s�|t ddddddddd d d d d d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d
g#�k�r�| j jt
d dd|id�� y t |�}W n> t
k
�r� |d6 }t d|d? B �t d7|d8@ B � }Y nX |d9k�r$| j jt
d d:d;�� | jj|� |S )r" z'expected-tag-name-but-got-right-bracket)r# r$ rJ z<>�?z'expected-tag-name-but-got-question-markzexpected-tag-namerD T)r r6 �markupDeclarationOpenStater �closeTagOpenStater
r r � tagNameStater% r7 r r= �bogusCommentState)r r$ r r r! r] i s6
zHTMLTokenizer.tagOpenStatec C s� | j j� }|tkr0td |g dd�| _| j| _n�|dkrX| jjtd dd�� | j | _nn|t
kr�| jjtd dd�� | jjtd d
d�� | j | _n0| jjtd d d |id
�� | j j |� | j | _dS )NrV F)r# rU r$ rW rj r" z*expected-closing-tag-but-got-right-bracket)r# r$ z expected-closing-tag-but-got-eofrJ z|tkr�| jjtd d d�� | j | _n| jjtd |d�� d
S ) Nr| rJ )r# r$ rD r[ r" zinvalid-codepointu �zeof-in-script-in-scriptT)
r r6 r% r7 r � scriptDataDoubleEscapedDashStater �(scriptDataDoubleEscapedLessThanSignStater r )r r$ r r r! r� � s$
z*HTMLTokenizer.scriptDataDoubleEscapedStatec C s� | j j� }|dkr2| jjtd dd�� | j| _n�|dkrZ| jjtd dd�� | j| _n�|dkr�| jjtd dd�� | jjtd dd�� | j| _nF|t kr�| jjtd d d�� | j
| _n | jjtd |d�� | j| _d
S ) Nr| rJ )r# r$ rD r[ r" zinvalid-codepointu �zeof-in-script-in-scriptT) r r6 r% r7 r �$scriptDataDoubleEscapedDashDashStater r� r� r r )r r$ r r r! r� s(
z.HTMLTokenizer.scriptDataDoubleEscapedDashStatec C s | j j� }|dkr*| jjtd dd�� n�|dkrR| jjtd dd�� | j| _n�|dkrz| jjtd dd�� | j| _n�|dkr�| jjtd dd�� | jjtd d d�� | j| _nF|t kr�| jjtd d
d�� | j
| _n | jjtd |d�� | j| _d S ) Nr| rJ )r# r$ rD rj r[ r" zinvalid-codepointu �zeof-in-script-in-scriptT) r r6 r% r7 r r� r rf r� r r )r r$ r r r! r� s,
z2HTMLTokenizer.scriptDataDoubleEscapedDashDashStatec C sP | j j� }|dkr8| jjtd dd�� d| _| j| _n| j j|� | j | _dS )Nri rJ )r# r$ r, T)
r r6 r% r7 r rr � scriptDataDoubleEscapeEndStater r= r� )r r$ r r r! r� 0 s
z6HTMLTokenizer.scriptDataDoubleEscapedLessThanSignStatec C s� | j j� }|ttd�B krR| jjtd |d�� | jj� dkrH| j | _
q�| j | _
nB|t kr�| jjtd |d�� | j|7 _n| j j
|� | j | _
dS )Nri rj rJ )r# r$ r� T)ri rj )r r6 r r: r% r7 r rr ru r� r r� r
r= )r r$ r r r! r� ; s
z,HTMLTokenizer.scriptDataDoubleEscapeEndStatec C s0 | j j� }|tkr$| j jtd� �n|tkrJ| jd j|dg� | j| _n�|dkr\| j � n�|dkrn| j
| _n�|dkr�| j jt d
d d �� | jd j|dg� | j| _n�|d
kr�| j jt d
dd �� | jd jddg� | j| _nF|t
k�r| j jt d
dd �� | j| _n | jd j|dg� | j| _dS )NTr$ r, rj ri �'�"rH rD r" z#invalid-character-in-attribute-name)r# r$ r[ zinvalid-codepointu �z#expected-attribute-name-but-got-eof)r� r� rH rD )r r6 r r^ r
r r7 �attributeNameStater rZ rq r% r r r )r r$ r r r! rp K s6
z&HTMLTokenizer.beforeAttributeNameStatec C s� | j j� }d}d}|dkr&| j| _�n0|tkr^| jd d d || j jtd� 7 < d}� n�|dkrld}n�|tkr~| j| _n�|dkr�| j | _n�|d kr�| j
j t d
d d �� | jd d d d
7 < d}n�|dk�r| j
j t d
dd �� | jd d d |7 < d}nH|t
k�r8| j
j t d
dd �� | j| _n | jd d d |7 < d}|�r�| jd d d jt�| jd d d< xP| jd d d� D ]:\}}| jd d d |k�r�| j
j t d
dd �� P �q�W |�r�| j� dS ) NTFrH r$ r r rj ri r[ r" zinvalid-codepoint)r# r$ u �r� r� rD z#invalid-character-in-attribute-namezeof-in-attribute-namezduplicate-attributerK rK )r� r� rD rK rK rK rK rK rK )r r6 �beforeAttributeValueStater r
r r^ r �afterAttributeNameStaterq r% r7 r r r rX r rZ )r r$ ZleavingThisStateZ emitTokenrU �_r r r! r� i sR
&
z HTMLTokenizer.attributeNameStatec C sF | j j� }|tkr$| j jtd� �n |dkr8| j| _�n
|dkrJ| j� n�|tkrp| jd j |dg� | j
| _n�|dkr�| j | _n�|dkr�| j j t
d d d
�� | jd j d dg� | j
| _n�|dk� r�| j j t
d dd
�� | jd j |dg� | j
| _nF|tk�r&| j j t
d dd
�� | j| _n | jd j |dg� | j
| _dS )NTrH rj r$ r, ri r[ r" zinvalid-codepoint)r# r$ u �r� r� rD z&invalid-character-after-attribute-namez expected-end-of-tag-but-got-eof)r� r� rD )r r6 r r^ r� r rZ r
r r7 r� rq r% r r r )r r$ r r r! r� � s:
z%HTMLTokenizer.afterAttributeNameStatec C sj | j j� }|tkr$| j jtd� �nB|dkr8| j| _�n.|dkrX| j| _| j j|� �n|dkrl| j| _� n�|dkr�| j j
t d dd�� | j � n�|d kr�| j j
t d d
d�� | j
d d d d
7 < | j| _n�|dk�r| j j
t d dd�� | j
d d d |7 < | j| _nL|tk�rD| j j
t d dd�� | j| _n"| j
d d d |7 < | j| _dS )NTr� rC r� rj r" z.expected-attribute-value-but-got-right-bracket)r# r$ r[ zinvalid-codepointr$ r u �rH rD �`z"equals-in-unquoted-attribute-valuez$expected-attribute-value-but-got-eofrK )rH rD r� rK rK )r r6 r r^ � attributeValueDoubleQuotedStater �attributeValueUnQuotedStater= � attributeValueSingleQuotedStater% r7 r rZ r r r )r r$ r r r! r� � s>
z'HTMLTokenizer.beforeAttributeValueStatec C s� | j j� }|dkr | j| _n�|dkr0| jd� n�|dkrj| jjtd dd�� | jd d d d 7 < nN|t kr�| jjtd d
d�� | j
| _n&| jd d
d || j j d� 7 < d S )Nr� rC r[ r" zinvalid-codepoint)r# r$ r$ r u �z#eof-in-attribute-value-double-quoteTrK rK )r� rC r[ ) r r6 �afterAttributeValueStater rT r% r7 r r r r r^ )r r$ r r r! r� � s
z-HTMLTokenizer.attributeValueDoubleQuotedStatec C s� | j j� }|dkr | j| _n�|dkr0| jd� n�|dkrj| jjtd dd�� | jd d d d 7 < nN|t kr�| jjtd d
d�� | j
| _n&| jd d
d || j j d� 7 < d S )Nr� rC r[ r" zinvalid-codepoint)r# r$ r$ r u �z#eof-in-attribute-value-single-quoteTrK rK )r� rC r[ ) r r6 r� r rT r% r7 r r r r r^ )r r$ r r r! r� � s
z-HTMLTokenizer.attributeValueSingleQuotedStatec C s | j j� }|tkr | j| _� n�|dkr2| jd� n�|dkrD| j� n�|dkr~| jjt d d d
�� | j
d d d |7 < n�|d
kr�| jjt d dd
�� | j
d d d d7 < nV|t kr�| jjt d dd
�� | j | _n.| j
d d d || j j
td�tB � 7 < dS )NrC rj r� r� rH rD r� r" z0unexpected-character-in-unquoted-attribute-value)r# r$ r$ r r[ zinvalid-codepointu �z eof-in-attribute-value-no-quotesT)r� r� rH rD r� rK rK rK )rC rj r� r� rH rD r� r[ )r r6 r rp r rT rZ r% r7 r r r r r^ r: )r r$ r r r! r� s,
z)HTMLTokenizer.attributeValueUnQuotedStatec C s� | j j� }|tkr | j| _n�|dkr.| j� np|dkr@| j| _n^|tkrt| jj t
d dd�� | j j |� | j | _n*| jj t
d dd�� | j j |� | j| _dS )Nrj ri r" z$unexpected-EOF-after-attribute-value)r# r$ z*unexpected-character-after-attribute-valueT)
r r6 r rp r rZ rq r r% r7 r r= r )r r$ r r r! r� s"
z&HTMLTokenizer.afterAttributeValueStatec C s� | j j� }|dkr&d| jd< | j� n^|tkrZ| jjtd dd�� | j j|� | j | _
n*| jjtd dd�� | j j|� | j | _
dS )Nrj TrW r" z#unexpected-EOF-after-solidus-in-tag)r# r$ z)unexpected-character-after-solidus-in-tag) r r6 r rZ r r% r7 r r= r r rp )r r$ r r r! rq 4 s
z&HTMLTokenizer.selfClosingStartTagStatec C sD | j jd�}|jdd�}| jjtd |d�� | j j� | j| _dS )Nrj r[ u ��Comment)r# r$ T) r r^ �replacer% r7 r r6 r r )r r$ r r r! ro F s
z HTMLTokenizer.bogusCommentStatec C s� | j j� g}|d dkrT|j| j j� � |d dkrPtd dd�| _| j| _dS � n�|d d kr�d}x.d&D ]&}|j| j j� � |d' |krjd}P qjW |r�td dd d dd�| _| j| _dS n�|d( dk�rH| jd k �rH| jj j
�rH| jj j
d) j | jj j k�rHd}x2d*D ]*}|j| j j� � |d+ |k�rd}P �qW |�rH| j
| _dS | jjtd dd�� x |�rz| j j|j� � �q`W | j| _dS ),Nr r| r� r, )r# r$ T�d�D�o�Or@ �C�t�T�y�Y�p�P�e�EFZDoctype)r# rU �publicId�systemId�correct�[�Ar" zexpected-dashes-or-doctyperK rK rK )r� r� �r� r� �r@ r� �r� r� �r� r� �r� r� �r� r� )r� r� r� r� r� r� rK rK rK )r� r� r� r� r� r� rK )r r6 r7 r r �commentStartStater � doctypeStater ZtreeZ openElements� namespaceZdefaultNamespace�cdataSectionStater% r= r' ro )r r? �matched�expectedr r r! rl U sR
z(HTMLTokenizer.markupDeclarationOpenStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� | jj| j� | j| _nP|t kr�| jjtd d
d�� | jj| j� | j| _n| jd |7 < | j
| _d S ) Nr| r[ r" zinvalid-codepoint)r# r$ r$ u �rj zincorrect-commentzeof-in-commentT) r r6 �commentStartDashStater r% r7 r r r r � commentState)r r$ r r r! r� � s(
z HTMLTokenizer.commentStartStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� | jj| j� | j| _nT|t kr�| jjtd d
d�� | jj| j� | j| _n | jd d| 7 < | j
| _d S ) Nr| r[ r" zinvalid-codepoint)r# r$ r$ u -�rj zincorrect-commentzeof-in-commentT) r r6 �commentEndStater r% r7 r r r r r� )r r$ r r r! r� � s(
z#HTMLTokenizer.commentStartDashStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < nT|tkr�| jjtd dd�� | jj| j� | j | _n | jd || j j
d
� 7 < d S ) Nr| r[ r" zinvalid-codepoint)r# r$ r$ u �zeof-in-commentT)r| r[ ) r r6 �commentEndDashStater r% r7 r r r r r^ )r r$ r r r! r� � s
zHTMLTokenizer.commentStatec C s� | j j� }|dkr | j| _n�|dkrV| jjtd dd�� | jd d7 < | j| _nT|t kr�| jjtd dd�� | jj| j� | j
| _n | jd d| 7 < | j| _d S )
Nr| r[ r" zinvalid-codepoint)r# r$ r$ u -�zeof-in-comment-end-dashT) r r6 r� r r% r7 r r r� r r )r r$ r r r! r� � s
z!HTMLTokenizer.commentEndDashStatec C s, | j j� }|dkr*| jj| j� | j| _n�|dkrd| jjtd dd�� | jd d7 < | j| _n�|dkr�| jjtd d d�� | j | _n�|d
kr�| jjtd d d�� | jd |7 < nj|t
kr�| jjtd d d�� | jj| j� | j| _n4| jjtd d
d�� | jd d| 7 < | j| _dS )Nrj r[ r" zinvalid-codepoint)r# r$ r$ u --�rh z,unexpected-bang-after-double-dash-in-commentr| z,unexpected-dash-after-double-dash-in-commentzeof-in-comment-double-dashzunexpected-char-in-commentz--T) r r6 r% r7 r r r r r� �commentEndBangStater )r r$ r r r! r� � s6
z HTMLTokenizer.commentEndStatec C s� | j j� }|dkr*| jj| j� | j| _n�|dkrN| jd d7 < | j| _n�|dkr�| jjtd dd�� | jd d 7 < | j | _nT|t
kr�| jjtd d
d�� | jj| j� | j| _n | jd d| 7 < | j | _d S ) Nrj r| r$ z--!r[ r" zinvalid-codepoint)r# r$ u --!�z eof-in-comment-end-bang-stateT) r r6 r% r7 r r r r� r r� r )r r$ r r r! r� � s(
z!HTMLTokenizer.commentEndBangStatec C s� | j j� }|tkr | j| _nj|tkr\| jjtd dd�� d| j d< | jj| j � | j
| _n*| jjtd dd�� | j j |� | j| _dS )Nr" z!expected-doctype-name-but-got-eof)r# r$ Fr� zneed-space-after-doctypeT) r r6 r �beforeDoctypeNameStater r r% r7 r r r r= )r r$ r r r! r� s
zHTMLTokenizer.doctypeStatec C s� | j j� }|tkrn�|dkrT| jjtd dd�� d| jd< | jj| j� | j| _n�|dkr�| jjtd dd�� d | jd
< | j | _nR|t
kr�| jjtd d d�� d| jd< | jj| j� | j| _n|| jd
< | j | _d S )
Nrj r" z+expected-doctype-name-but-got-right-bracket)r# r$ Fr� r[ zinvalid-codepointu �rU z!expected-doctype-name-but-got-eofT) r r6 r r% r7 r r r r �doctypeNameStater )r r$ r r r! r� s.
z$HTMLTokenizer.beforeDoctypeNameStatec C s | j j� }|tkr2| jd jt�| jd< | j| _n�|dkrh| jd jt�| jd< | jj | j� | j
| _n�|dkr�| jj t d dd�� | jd d7 < | j | _nh|t
kr�| jj t d dd�� d | jd
< | jd jt�| jd< | jj | j� | j
| _n| jd |7 < d S ) NrU rj r[ r" zinvalid-codepoint)r# r$ u �zeof-in-doctype-nameFr� T)r r6 r r rX r �afterDoctypeNameStater r% r7 r r r� r )r r$ r r r! r� 6 s,
z HTMLTokenizer.doctypeNameStatec C sR | j j� }|tkr�n8|dkr8| jj| j� | j| _�n|tkr�d| jd< | j j |� | jjt
d dd�� | jj| j� | j| _� n�|d!kr�d }x$d'D ] }| j j� }||kr�d}P q�W |r�| j | _d S nJ|d(k�rd }x(d.D ] }| j j� }||k� r�d}P � q�W |�r| j | _d S | j j |� | jjt
d d d |id �� d| jd< | j
| _d S )/Nrj Fr� r" zeof-in-doctype)r# r$ r� r� T�u�U�b�B�l�L�i�Ir@ r� �s�Sr� r� r� r� r� r� �m�Mz*expected-space-or-right-bracket-in-doctyper$ )r# r$ r. )r� r� �r� r� �r� r� �r� r� �r� r� �r@ r� )r� r� r� r� r� )r� r� �r� r� �r� r� �r� r� �r� r� �r� r� )r� r� r� r� r� )r r6 r r% r7 r r r r r= r � afterDoctypePublicKeywordState� afterDoctypeSystemKeywordState�bogusDoctypeState)r r$ r� r� r r r! r� O sT
z#HTMLTokenizer.afterDoctypeNameStatec C s� | j j� }|tkr | j| _n�|d
krP| jjtd dd�� | j j|� | j| _nT|t kr�| jjtd dd�� d| j
d< | jj| j
� | j | _n| j j|� | j| _d S ) Nr� r� r" zunexpected-char-in-doctype)r# r$ zeof-in-doctypeFr� T)r� r� ) r r6 r �"beforeDoctypePublicIdentifierStater r% r7 r r= r r r )r r$ r r r! r� � s"
z,HTMLTokenizer.afterDoctypePublicKeywordStatec C s� | j j� }|tkrn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| jjt d dd�� d | jd
< | jj| j� | j
| _nh|t kr�| jjt d d d�� d | jd
< | jj| j� | j
| _n(| jjt d d d�� d | jd
< | j | _d
S )Nr� r, r� r� rj r" zunexpected-end-of-doctype)r# r$ Fr� zeof-in-doctypezunexpected-char-in-doctypeT)
r r6 r r �(doctypePublicIdentifierDoubleQuotedStater �(doctypePublicIdentifierSingleQuotedStater% r7 r r r r� )r r$ r r r! r� � s4
z0HTMLTokenizer.beforeDoctypePublicIdentifierStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� d
| jd < | jj| j� | j| _nR|t kr�| jjtd d d�� d
| jd < | jj| j� | j| _n| jd |7 < d
S )Nr� r[ r" zinvalid-codepoint)r# r$ r� u �rj zunexpected-end-of-doctypeFr� zeof-in-doctypeT)
r r6 �!afterDoctypePublicIdentifierStater r% r7 r r r r )r r$ r r r! r� � s*
z6HTMLTokenizer.doctypePublicIdentifierDoubleQuotedStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� d
| jd < | jj| j� | j| _nR|t kr�| jjtd d d�� d
| jd < | jj| j� | j| _n| jd |7 < d
S )Nr� r[ r" zinvalid-codepoint)r# r$ r� u �rj zunexpected-end-of-doctypeFr� zeof-in-doctypeT)
r r6 r� r r% r7 r r r r )r r$ r r r! r� � s*
z6HTMLTokenizer.doctypePublicIdentifierSingleQuotedStatec C s | j j� }|tkr | j| _n�|dkr<| jj| j� | j| _n�|dkrn| jjt d dd�� d| jd< | j
| _n�|dkr�| jjt d dd�� d| jd< | j | _nh|t kr�| jjt d d d�� d
| jd < | jj| j� | j| _n(| jjt d dd�� d
| jd < | j
| _d S )
Nrj r� r" zunexpected-char-in-doctype)r# r$ r, r� r� zeof-in-doctypeFr� T)r r6 r �-betweenDoctypePublicAndSystemIdentifiersStater r% r7 r r r �(doctypeSystemIdentifierDoubleQuotedState�(doctypeSystemIdentifierSingleQuotedStater r� )r r$ r r r! r� � s6
z/HTMLTokenizer.afterDoctypePublicIdentifierStatec C s� | j j� }|tkrn�|dkr4| jj| j� | j| _n�|dkrPd| jd< | j| _n�|dkrld| jd< | j | _nh|t
kr�| jjt d dd�� d | jd
< | jj| j� | j| _n(| jjt d d d�� d | jd
< | j | _d S )
Nrj r� r, r� r� r" zeof-in-doctype)r# r$ Fr� zunexpected-char-in-doctypeT)
r r6 r r% r7 r r r r� r� r r r� )r r$ r r r! r� s.
z;HTMLTokenizer.betweenDoctypePublicAndSystemIdentifiersStatec C s� | j j� }|tkr | j| _n�|d
krP| jjtd dd�� | j j|� | j| _nT|t kr�| jjtd dd�� d| j
d< | jj| j
� | j | _n| j j|� | j| _d S ) Nr� r� r" zunexpected-char-in-doctype)r# r$ zeof-in-doctypeFr� T)r� r� ) r r6 r �"beforeDoctypeSystemIdentifierStater r% r7 r r= r r r )r r$ r r r! r� s"
z,HTMLTokenizer.afterDoctypeSystemKeywordStatec C s� | j j� }|tkrn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| jjt d dd�� d | jd
< | jj| j� | j
| _nh|t kr�| jjt d d d�� d | jd
< | jj| j� | j
| _n(| jjt d dd�� d | jd
< | j | _d S )
Nr� r, r� r� rj r" zunexpected-char-in-doctype)r# r$ Fr� zeof-in-doctypeT)
r r6 r r r� r r� r% r7 r r r r� )r r$ r r r! r� / s4
z0HTMLTokenizer.beforeDoctypeSystemIdentifierStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� d
| jd < | jj| j� | j| _nR|t kr�| jjtd d d�� d
| jd < | jj| j� | j| _n| jd |7 < d
S )Nr� r[ r" zinvalid-codepoint)r# r$ r� u �rj zunexpected-end-of-doctypeFr� zeof-in-doctypeT)
r r6 �!afterDoctypeSystemIdentifierStater r% r7 r r r r )r r$ r r r! r� L s*
z6HTMLTokenizer.doctypeSystemIdentifierDoubleQuotedStatec C s� | j j� }|dkr | j| _n�|dkrN| jjtd dd�� | jd d7 < n�|dkr�| jjtd d d�� d
| jd < | jj| j� | j| _nR|t kr�| jjtd d d�� d
| jd < | jj| j� | j| _n| jd |7 < d
S )Nr� r[ r" zinvalid-codepoint)r# r$ r� u �rj zunexpected-end-of-doctypeFr� zeof-in-doctypeT)
r r6 r� r r% r7 r r r r )r r$ r r r! r� d s*
z6HTMLTokenizer.doctypeSystemIdentifierSingleQuotedStatec C s� | j j� }|tkrn~|dkr4| jj| j� | j| _n^|tkrt| jjt d dd�� d| jd< | jj| j� | j| _n | jjt d dd�� | j
| _dS ) Nrj r" zeof-in-doctype)r# r$ Fr� zunexpected-char-in-doctypeT) r r6 r r% r7 r r r r r r� )r r$ r r r! r� | s
z/HTMLTokenizer.afterDoctypeSystemIdentifierStatec C sZ | j j� }|dkr*| jj| j� | j| _n,|tkrV| j j|� | jj| j� | j| _n dS )Nrj T) r r6 r% r7 r r r r r= )r r$ r r r! r� � s
z HTMLTokenizer.bogusDoctypeStatec C s� g }xt|j | jjd�� |j | jjd�� | jj� }|tkr@P q|d dd � dkrl|d d d� |d< P q|j |� qW dj|�}|jd�}|dkr�x&t|�D ]}| jj t d d
d �� q�W |j
dd �}|r�| jj t d
|d �� | j | _ dS )N�]rj r � z]]r, r[ r r" zinvalid-codepoint)r# r$ u �rJ TrK �����rK r� rK )
r7 r r^ r6 r r9 �count�ranger% r r� r r )r r$ r6 Z nullCountr� r r r! r� � s.
z HTMLTokenizer.cdataSectionState)N)NF)N�__name__�
__module__� __qualname__�__doc__r r) rB rS rT rZ r r\ rb r` rd rf rg r] rm rn ra rs rt rc rw rx re ry r{ rz r} r� r r~ r� r� r� r� r� r� r� r� r� rp r� r� r� r� r� r� r� rq ro rl r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� �
__classcell__r r )r r! r s� H
P #
6 "- 3 r N)Z
__future__r r r Zpip._vendor.sixr r; � collectionsr Z constantsr r r
r r r
r r r r Z _inputstreamr Z_trier rL �objectr r r r r! � s