� c��j�$��� ddlZddlZddlZddlZdgZej dd�ZGd�d�ZGd�d�Z Gd�d �Z y) �N�RobotFileParser� RequestRatezrequests secondsc�X�eZdZ dd�Zd�Zd�Zd�Zd�Zd�Zd�Z d�Z d �Z d �Z d �Z d �Zy )rc�z�g|_g|_d|_d|_d|_|j |�d|_y)NFr)�entries�sitemaps� default_entry� disallow_all� allow_all�set_url� last_checked��self�urls �+/usr/lib64/python3.12/urllib/robotparser.py�__init__zRobotFileParser.__init__s;���� ��� �!���!������ � � �S�����c�� |jS�N)r �rs r�mtimezRobotFileParser.mtime&s�� � � � � rc�8� ddl}|j�|_y)Nr)�timer )rrs r�modifiedzRobotFileParser.modified/s�� � � �I�I�K��rc�r� ||_tjj|�dd\|_|_y)N��)r�urllib�parse�urlparse�host�pathrs rr zRobotFileParser.set_url7s0��:����%�|�|�4�4�S�9�!�A�>��� �4�9rc��� tjj|j�}|j �}|j |j d�j��y#tjj$rU}|jdvrd|_ n%|jdk\r|jdkrd|_ |j�Yd}~yd}~wwxYw)Nzutf-8)i�i�Ti�i�)r�request�urlopenr�readr�decode� splitlines�error� HTTPError�coder r �close)r�f�raw�errs rr&zRobotFileParser.read<s���B� 9����&�&�t�x�x�0�A��&�&�(�C� �J�J�s�z�z�'�*�5�5�7� 8���|�|�%�%� ��x�x�:�%�$(��!����S��S�X�X��^�!%��� �I�I�K�K��  �s�)A+�+C�A C�Cc��d|jvr|j�||_yy|jj|�y�N�*)� useragentsr r�append)r�entrys r� _add_entryzRobotFileParser._add_entryJs=�� �%�"�"� "��!�!�)�%*��"�*� �L�L� � �� &rc��� d}t�}|j�|D�]�}|s4|dk(r t�}d}n"|dk(r|j|�t�}d}|jd�}|dk\r|d|}|j �}|s�h|j dd�}t |�dk(s��|dj �j�|d<tjj|dj ��|d<|ddk(rB|dk(r|j|�t�}|jj|d�d}��*|ddk(r3|dk7s��9|jjt|dd��d}��e|dd k(r3|dk7s��t|jjt|dd ��d}���|dd k(r?|dk7s���|dj �j�rt!|d�|_d}���|dd k(r�|dk7s���|dj d �}t |�dk(rk|dj �j�rJ|dj �j�r)t%t!|d�t!|d��|_d}���|ddk(s���|j(j|d����|dk(r|j|�yy)Nrr��#�:z user-agent�disallowF�allowTz crawl-delayz request-rate�/�sitemap)�Entryrr6�find�strip�split�len�lowerrr�unquoter3r4� rulelines�RuleLine�isdigit�int�delayr�req_rater)r�lines�stater5�line�i�numberss rrzRobotFileParser.parseSs��� ������ � � ���D���A�:�!�G�E��E��a�Z��O�O�E�*�!�G�E��E�� � �#��A��A�v��B�Q�x���:�:�<�D����:�:�c�1�%�D��4�y�A�~��q�'�-�-�/�/�/�1��Q�� �,�,�.�.�t�A�w�}�}��?��Q����7�l�*���z�����.� %����$�$�+�+�D��G�4��E��!�W� �*���z����.�.�x��Q���/G�H� !���!�W��'���z����.�.�x��Q���/F�G� !���!�W� �-���z� ��7�=�=�?�2�2�4�*-�d�1�g�,�E�K� !���!�W��.���z�"&�q�'�-�-��"4����L�A�-�'�!�*�2B�2B�2D�2L�2L�2N� '�� � 0� 0� 2� :� :� <�-8��W�Q�Z��#�g�VW�j�/�-Z�E�N� !���!�W� �)� �M�M�(�(��a��1�o�p �A�:� �O�O�E� "� rc�d� |jry|jry|jsytjj tjj |��}tjjdd|j|j|j|jf�}tjj|�}|sd}|jD]&}|j|�s�|j|�cS|j r|j j|�Sy)NFT�r=)r r r rrr rE� urlunparser"�params�query�fragment�quoter� applies_to� allowancer )r� useragentr� parsed_urlr5s r� can_fetchzRobotFileParser.can_fetch�s���K� � � �� �>�>�� � � ���\�\�*�*�6�<�<�+?�+?��+D�E� ��l�l�%�%�r�"�Z�_�_� � � �j�.�.� �0C�0C�'E�F���l�l� � ��%����C��\�\�E���� �*����s�+�+�"� � � ��%�%�/�/��4� 4�rc���|j�sy|jD]!}|j|�s�|jcS|jr|jjSyr)rrrXrJr �rrZr5s r� crawl_delayzRobotFileParser.crawl_delay�sT���z�z�|���\�\�E���� �*��{�{�"�"� � � ��%�%�+�+� +�rc���|j�sy|jD]!}|j|�s�|jcS|jr|jjSyr)rrrXrKr r^s r� request_ratezRobotFileParser.request_rate�sT���z�z�|���\�\�E���� �*��~�~�%�"� � � ��%�%�.�.� .�rc�4�|jsy|jSr)rrs r� site_mapszRobotFileParser.site_maps�s���}�}���}�}�rc��|j}|j�||jgz}djtt|��S)Nz )rr �join�map�str)rrs r�__str__zRobotFileParser.__str__�s@���,�,�� � � � )���!3�!3� 4�4�G��{�{�3�s�G�,�-�-rN)rR)�__name__� __module__� __qualname__rrrr r&r6rr\r_rarcrh�rrrrsE��� �!�(�?� 9�'�G#�R�:��� .rc� �eZdZ d�Zd�Zd�Zy)rGc���|dk(r|sd}tjjtjj|��}tjj |�|_||_y)NrRT)rrrSr rWr"rY)rr"rYs rrzRuleLine.__init__�sP�� �2�:�i��I��|�|�&�&�v�|�|�'<�'<�T�'B�C���L�L�&�&�t�,�� �"��rc�Z�|jdk(xs|j|j�Sr1)r"� startswith)r�filenames rrXzRuleLine.applies_to�s%���y�y�C��A�8�#6�#6�t�y�y�#A�Arc�B�|jrdnddz|jzS)N�Allow�Disallowz: )rYr"rs rrhzRuleLine.__str__�s���>�>��z�T�A�D�I�I�M�MrN)rirjrkrrXrhrlrrrGrG�s��1�#�B�NrrGc�&�eZdZ d�Zd�Zd�Zd�Zy)r?c�<�g|_g|_d|_d|_yr)r3rFrJrKrs rrzEntry.__init__�s���������� ��� rc��g}|jD]}|jd|����|j�|jd|j���|j�7|j}|jd|j�d|j ���|j tt|j��dj|�S)Nz User-agent: z Crawl-delay: zRequest-rate: r=� ) r3r4rJrK�requests�seconds�extendrfrgrFre)r�ret�agent�rates rrhz Entry.__str__�s������_�_�E� �J�J��e�W�-� .�%� �:�:� !� �J�J��t�z�z�l�3� 4� �=�=� $��=�=�D� �J�J��� � ��a�� � �~�F� G� � � �3�s�D�N�N�+�,��y�y��~�rc�� |jd�dj�}|jD]}|dk(ry|j�}||vs�yy)Nr=rr2TF)rBrDr3)rrZr}s rrXzEntry.applies_to�sT��@��O�O�C�(��+�1�1�3� ��_�_�E���|���K�K�M�E�� �!�� %�rc�f� |jD]!}|j|�s�|jcSy)NT)rFrXrY)rrqrNs rrYzEntry.allowance s2�� %��N�N�D����x�(��~�~�%�#�rN)rirjrkrrhrXrYrlrrr?r?�s��I�� � �rr?) � collections� urllib.errorr� urllib.parse�urllib.request�__all__� namedtuplerrrGr?rlrr�r�sX�� ����� � ��$�k�$�$�]�4F�G� �.�.�DN�N�$(�(r