3
\=m � @ s< d Z dZddlZddlZddlZddlZddlZddlZddlm Z ddl
m Z m Z m
Z
ddlmZ ddlmZmZ dd lmZ d&d d �ZG d
d� de�Zdd� Zdd� Zdd� Zdd� Zej d'k r�ddl Z e j Z dd� Z dd� Z!n e Z eZ eZ!d d � Z"G d d � d e�Z#G d d!� d!e$�Z%G d"d#� d#e�Z&G d$d%� d%e%�Z'dS )(z�Refactoring framework.
Used as a main program, this can refactor any number of files and/or
recursively descend down directories. Imported as a module, this
provides infrastructure to write your own refactoring tool.
z#Guido van Rossum � N)�chain� )�driver�tokenize�token)� find_root)�pytree�pygram)� btm_matcherTc C st t | g g dg�}tjj|j�}g }xLttj|��D ]:}|jd�r2|jd�r2|rZ|dd� }|j |dd� � q2W |S )zEReturn a sorted list of all available fix names in the given package.�*�fix_z.py� N� �����)
�
__import__�os�path�dirname�__file__�sorted�listdir�
startswith�endswith�append)Z fixer_pkgZ
remove_prefixZpkgZ fixer_dirZ fix_names�name� r �(/usr/lib64/python3.6/lib2to3/refactor.py�get_all_fix_names s r c @ s e Zd ZdS )�
_EveryNodeN)�__name__�
__module__� __qualname__r r r r r + s r c C s� t | tjtjf�r(| jdkr t�| jhS t | tj�rH| jrDt| j�S t�t | tj �r�t
� }x*| jD ] }x|D ]}|j t|�� qlW qbW |S t d| ��dS )zf Accepts a pytree Pattern Node and returns a set
of the pattern types which will match first. Nz$Oh no! I don't understand pattern %s)
�
isinstancer Z NodePatternZ LeafPattern�typer ZNegatedPatternZcontent�_get_head_typesZWildcardPattern�set�update� Exception)Zpat�r�p�xr r r r$ / s
r$ c C s� t jt�}g }x�| D ]|}|jrjyt|j�}W n tk
rJ |j|� Y q�X xB|D ]}|| j|� qRW q|jdk r�||j j|� q|j|� qW x,tt j
j j � t j
j
�D ]}|| j|� q�W t|�S )z^ Accepts a list of fixers and returns a dictionary
of head node type --> fixer list. N)� collections� defaultdict�list�patternr$ r r Z _accept_typer r �python_grammarZ
symbol2number�values�tokens�extend�dict)Z
fixer_listZ
head_nodesZevery�fixerZheadsZ node_typer r r �_get_headnode_dictK s"
r5 c s � fdd�t � d�D �S )zN
Return the fully qualified names for fixers in the package pkg_name.
c s g | ]}� d | �qS )�.r )�.0�fix_name)�pkg_namer r �
h s z+get_fixers_from_package..F)r )r9 r )r9 r �get_fixers_from_packaged s
r; c C s | S )Nr )�objr r r � _identityk s r= r c C s | j dd�S )Nz
�
)�replace)�inputr r r �_from_system_newlinesr s rA c C s t jdkr| jdt j�S | S d S )Nr> )r �linesepr? )r@ r r r �_to_system_newlinest s
rC c sT d}t jtj| �j�� � fdd�}ttjt jtj h�}t
� }y�x�|� \}}||krVq@q@|tj krl|rfP d}q@|tj ko||dk�r,|� \}}|tj ks�|dkr�P |� \}}|tj ks�|dkr�P |� \}}|tj
kr�|dkr�|� \}}xJ|tj k�r(|j|� |� \}}|tj
k�s|d k�r P |� \}}q�W q@P q@W W n tk
�rJ Y nX t|�S )
NFc s t � �} | d | d fS )Nr r )�next)�tok)�genr r �advance� s z(_detect_future_features..advanceT�fromZ
__future__�import�(�,)r �generate_tokens�io�StringIO�readline� frozensetr �NEWLINE�NL�COMMENTr% �STRING�NAME�OP�add�
StopIteration)�sourceZhave_docstringrG �ignore�features�tp�valuer )rF r �_detect_future_features sD
r^ c @ s e Zd ZdZdS )�
FixerErrorz A fixer could not be loaded.N)r r r! �__doc__r r r r r_ � s r_ c @ s� e Zd Zddd�ZdZdZd4dd�Zdd � Zd
d � Zd d
� Z dd� Z
dd� Z d5dd�Z d6dd�Z
dd� Zd7dd�Zdd� Zd8d d �Zd d � Zd d!� Zd9d"d#�Zd:d$d%�Zd&Zd'Zd(d)� Zd*d+� Zd,d-� Zd.d/� Zd0d1� Z d2d3� Z dS );�RefactoringToolF)�print_function�write_unchanged_filesZFixr Nc C s2 || _ |p g | _| jj� | _|dk r0| jj|� | jd rDtj| _ntj | _| jj
d�| _ g | _ t
jd�| _g | _d| _tj| jtj| jd�| _| j� \| _| _g | _tj� | _ g | _ g | _ xXt | j| j�D ]F}|j r�| j j!|� q�|| jkr�| j j"|� q�|| jkr�| j j"|� q�W t#| j �| _$t#| j �| _%dS )z�Initializer.
Args:
fixer_names: a list of fixers to import
options: a dict with configuration.
explicit: a list of fixers to run even if they are explicit.
Nrb rc ra F)�convert�logger)&�fixers�explicit�_default_options�copy�optionsr& r �!python_grammar_no_print_statement�grammarr/ �getrc �errors�loggingZ getLoggerre � fixer_log�wroter ZDriverr rd �
get_fixers� pre_order�
post_order�files�bmZ
BottomMatcher�BMZ
bmi_pre_orderZbmi_post_orderr Z
BM_compatibleZ add_fixerr r5 �bmi_pre_order_heads�bmi_post_order_heads)�selfZ fixer_namesrj rg r4 r r r �__init__� s<
zRefactoringTool.__init__c C s\ g }g }�x&| j D �]}t|i i dg�}|jdd�d }|j| j�rV|t| j�d� }|jd�}| jdjdd� |D �� }yt ||�}W n$ t
k
r� t d ||f ��Y nX || j | j
�} | jr�| jd
k r�|| jkr�| jd |� q| jd |� | jd
k�r|j| � q| jdk�r |j| � qt d| j ��qW tjd�}
|j|
d� |j|
d� ||fS )a Inspects the options to load the requested patterns and handlers.
Returns:
(pre_order, post_order), where pre_order is the list of fixers that
want a pre-order AST traversal, and post_order is the list that want
post-order traversal.
r r6 r N�_� c S s g | ] }|j � �qS r )�title)r7 r) r r r r: � s z.RefactoringTool.get_fixers..zCan't find %s.%sTzSkipping optional fixer: %szAdding transformation: %sZpreZpostzIllegal fixer order: %rZ run_order)�key�����)rf r �rsplitr � FILE_PREFIX�len�split� CLASS_PREFIX�join�getattr�AttributeErrorr_ rj rp rg � log_message� log_debug�orderr �operator�
attrgetter�sort) rz Zpre_order_fixersZpost_order_fixersZ fix_mod_path�modr8 �parts�
class_nameZ fix_classr4 Zkey_funcr r r rr � s8
zRefactoringTool.get_fixersc O s � dS )z Called when an error occurs.Nr )rz �msg�args�kwdsr r r � log_error s zRefactoringTool.log_errorc G s |r || }| j j|� dS )zHook to log a message.N)re �info)rz r� r� r r r r� s zRefactoringTool.log_messagec G s |r || }| j j|� d S )N)re �debug)rz r� r� r r r r� s zRefactoringTool.log_debugc C s dS )zTCalled with the old version, new version, and filename of a
refactored file.Nr )rz �old_text�new_text�filename�equalr r r � print_output s z RefactoringTool.print_outputc C s< x6|D ].}t jj|�r&| j|||� q| j|||� qW dS )z)Refactor a list of files and directories.N)r r �isdir� refactor_dir�
refactor_file)rz �items�write�
doctests_onlyZ dir_or_filer r r �refactor# s
zRefactoringTool.refactorc
C s� t jd }x�t j|�D ]�\}}}| jd|� |j� |j� xH|D ]@}|jd� rBt jj|�d |krBt jj||�} | j | ||� qBW dd� |D �|dd�< qW dS )z�Descends down a directory and refactor every Python file found.
Python files are assumed to have a .py extension.
Files and subdirectories starting with '.' are skipped.
�pyzDescending into %sr6 r c S s g | ]}|j d �s|�qS )r6 )r )r7 Zdnr r r r: > s z0RefactoringTool.refactor_dir..N)
r �extsep�walkr� r� r r �splitextr� r� )
rz Zdir_namer� r� Zpy_ext�dirpathZdirnames� filenamesr �fullnamer r r r� , s
z RefactoringTool.refactor_dirc C s� yt |d�}W n. tk
r< } z| jd||� dS d}~X nX ztj|j�d }W d|j� X t|d|d��}t|j � �|fS Q R X dS )zG
Do our best to decode a Python source file correctly.
�rbzCan't open %s: %sNr r( )�encoding)NN)
�open�OSErrorr� r �detect_encodingrO �close�_open_with_encodingrA �read)rz r� �f�errr� r r r �_read_python_source@ s
z#RefactoringTool._read_python_sourcec C s� | j |�\}}|dkrdS |d7 }|rn| jd|� | j||�}| jsL||kr`| j|||||� q�| jd|� nH| j||�}| js�|r�|jr�| jt|�dd� |||d� n | jd|� dS ) zRefactors a file.Nr> zRefactoring doctests in %szNo doctest changes in %sr )r� r� zNo changes in %sr� )r� r� �refactor_docstringrc �processed_file�refactor_string� was_changed�str)rz r� r� r� r@ r� �output�treer r r r� P s z RefactoringTool.refactor_filec C s� t |�}d|krtj| j_zJy| jj|�}W n4 tk
r` } z| jd||jj |� dS d}~X nX W d| j| j_X ||_
| j d|� | j ||� |S )aF Refactor a given input string.
Args:
data: a string holding the code to be refactored.
name: a human-readable name for use in error/log messages.
Returns:
An AST corresponding to the refactored input stream; None if
there were errors during the parse.
rb zCan't parse %s: %s: %sNzRefactoring %s)
r^ r rk r rl Z parse_stringr' r� � __class__r �future_featuresr� �
refactor_tree)rz �datar r[ r� r� r r r r� g s
z RefactoringTool.refactor_stringc C s� t jj� }|rN| jd� | j|d�}| js2||krB| j|d|� q�| jd� n:| j|d�}| jsj|r~|jr~| jt |�d|� n
| jd� d S )Nz Refactoring doctests in stdinzzNo doctest changes in stdinzNo changes in stdin)
�sys�stdinr� r� r� rc r� r� r� r� )rz r� r@ r� r� r r r �refactor_stdin� s
z RefactoringTool.refactor_stdinc
C s� x"t | j| j�D ]}|j||� qW | j| j|j� � | j| j|j� � | jj|j � �}�xvt
|j � ��rАx`| jj D �]R}||ko�|| rv|| j
tjjdd� |jr�|| j
tjjd� �x
t|| �D � ]�}||| kr�|| j|� y t|� W n tk
�r w�Y nX |j�r(||jk�r(q�|j|�}|r�|j||�}|dk r�|j|� x,|j� D ] }|j�spg |_|jj|� �q^W | jj|j � �}x2|D ]*} | |k�r�g || < || j || � �q�W q�W qvW q\W x$t | j| j�D ]}|j ||� �q�W |j S )a� Refactors a parse tree (modifying the tree in place).
For compatible patterns the bottom matcher module is
used. Otherwise the tree is traversed node-to-node for
matches.
Args:
tree: a pytree.Node instance representing the root of the tree
to be refactored.
name: a human-readable name for this tree.
Returns:
True if the tree was modified, False otherwise.
T)r �reverse)r N) r rs rt Z
start_tree� traverse_byrx ry rw ZrunZleaves�anyr0 rf r� r ZBaseZdepthZkeep_line_orderZ
get_linenor- �remover �
ValueErrorZfixers_applied�match� transformr? r r2 Z finish_treer� )
rz r� r r4 Z match_set�node�results�newZ new_matchesZfxrr r r r� � sJ
$z RefactoringTool.refactor_treec C s^ |sdS xP|D ]H}xB||j D ]4}|j|�}|r |j||�}|dk r |j|� |}q W qW dS )a Traverse an AST, applying a set of fixers to each node.
This is a helper method for refactor_tree().
Args:
fixers: a list of fixer instances.
traversal: a generator that yields AST nodes.
Returns:
None
N)r# r� r� r? )rz rf Z traversalr� r4 r� r� r r r r� � s
zRefactoringTool.traverse_byc C s� | j j|� |dkr.| j|�d }|dkr.dS ||k}| j||||� |r`| jd|� | js`dS |rv| j||||� n | jd|� dS )zR
Called when a file has been refactored and there may be changes.
Nr zNo changes to %szNot writing changes to %s)ru r r� r� r� rc �
write_file)rz r� r� r� r� r� r� r r r r� � s z RefactoringTool.processed_filec % C s� yt |d|d�}W n. tk
r@ } z| jd||� dS d}~X nX zHy|jt|�� W n0 tk
r� } z| jd||� W Y dd}~X nX W d|j� X | jd|� d| _dS )z�Writes a string to a file.
It first shows a unified diff between the old text and the new text, and
then rewrites the file; the latter is only done if the write option is
set.
�w)r� zCan't create %s: %sNzCan't write %s: %szWrote changes to %sT)r� r� r� r� rC r� r� rq )rz r� r� r� r� r� r� r r r r�
s $
zRefactoringTool.write_filez>>> z... c
C s g }d}d}d}d}x�|j dd�D ]�}|d7 }|j� j| j�r�|dk r\|j| j||||�� |}|g}|j| j�} |d| � }q"|dk r�|j|| j �s�||| jj� d kr�|j |� q"|dk r�|j| j||||�� d}d}|j |� q"W |dk �r|j| j||||�� dj
|�S )a� Refactors a docstring, looking for doctests.
This returns a modified version of the input string. It looks
for doctests, which start with a ">>>" prompt, and may be
continued with "..." prompts, as long as the "..." is indented
the same as the ">>>".
(Unfortunately we can't use the doctest module's parser,
since, like most parsers, it is not geared towards preserving
the original source.)
Nr T)�keependsr r> r} ) �
splitlines�lstripr �PS1r2 �refactor_doctest�find�PS2�rstripr r� )
rz r@ r� �result�blockZ block_lineno�indent�lineno�line�ir r r r� % s:
z"RefactoringTool.refactor_docstringc
s y�j ||� �}W nd tk
rv } zH�jjtj�rRx |D ]}�jd|jd�� q8W �jd|||j j
|� |S d}~X nX �j ||��r
t |�j
dd�}|d|d � ||d d� } }|d jd�s�|d d7 < � �j |jd� g}|�r
|� �fd d
�|D �7 }|S )
z�Refactors one doctest.
A doctest is given as a block of lines, the first of which starts
with ">>>" (possibly indented), while the remaining lines start
with "..." (identically indented).
z
Source: %sr> z+Can't parse docstring in %s line %s: %s: %sNT)r� r r c s g | ]}� �j | �qS r )r� )r7 r� )r� rz r r r: j s z4RefactoringTool.refactor_doctest..r� r� )� parse_blockr' re Z isEnabledForro �DEBUGr� r� r� r� r r� r� r� r r� �pop)
rz r� r� r� r� r� r� r� r� Zclippedr )r� rz r r� P s$
"z RefactoringTool.refactor_doctestc C s� | j r d}nd}| js$| jd|� n&| jd|� x| jD ]}| j|� q8W | jrt| jd� x| jD ]}| j|� qbW | jr�t| j�dkr�| jd� n| jdt| j�� x&| jD ] \}}}| j|f|�|� q�W d S ) N�werez
need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:r zThere was 1 error:zThere were %d errors:)rq ru r� rp rn r� )rz r� �file�messager� r� r� r r r � summarizem s$
zRefactoringTool.summarizec C s" | j j| j|||��}t� |_|S )z�Parses a block into a tree.
This is necessary to get correct line number / offset information
in the parser diagnostics and embedded into the parse tree.
)r Z parse_tokens� wrap_toksrP r� )rz r� r� r� r� r r r r� � s zRefactoringTool.parse_blockc c sh t j| j||�j�}xN|D ]F\}}\}}\} }
} ||d 7 }| |d 7 } ||||f| |
f| fV qW dS )z;Wraps a tokenize stream to systematically modify start/end.r N)r rL � gen_lines�__next__) rz r� r� r� r1 r# r] Zline0Zcol0Zline1Zcol1Z line_textr r r r� � s
zRefactoringTool.wrap_toksc c s� || j }|| j }|}xV|D ]N}|j|�r@|t|�d� V n(||j� d krXdV ntd||f ��|}q W x
dV qrW dS )z�Generates lines as expected by tokenize from a list of lines.
This strips the first len(indent + self.PS1) characters off each line.
Nr> zline=%r, prefix=%rr} )r� r� r r� r� �AssertionError)rz r� r� �prefix1Zprefix2�prefixr� r r r r� � s
zRefactoringTool.gen_lines)NN)FF)FF)FF)F)NFN)N) r r r! rh r� r� r{ rr r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r r r ra � s:
4(
O
+
ra c @ s e Zd ZdS )�MultiprocessingUnsupportedN)r r r! r r r r r� � s r� c sB e Zd Z� fdd�Zd � fdd� Z� fdd�Z� fd d
�Z� ZS ) �MultiprocessRefactoringToolc s" t t| �j||� d | _d | _d S )N)�superr� r{ �queue� output_lock)rz r� �kwargs)r� r r r{ � s z$MultiprocessRefactoringTool.__init__Fr c s |dkr t t��j|||�S y dd l� W n tk
r@ t�Y nX �jd k rTtd��� j� �_� j � �_
� �fdd�t |�D �}z.x|D ] }|j � q�W t t��j|||� W d �jj
� x t |�D ]}�jjd � q�W x |D ]}|j� r�|j
� q�W d �_X d S )Nr r z already doing multiple processesc s g | ]}� j �jd ��qS ))�target)ZProcess�_child)r7 r� )�multiprocessingrz r r r: � s z8MultiprocessRefactoringTool.refactor..)r� r� r� r� � ImportErrorr� r� � RuntimeErrorZ
JoinableQueueZLockr� �range�startr� �putZis_alive)rz r� r� r� Z
num_processesZ processesr) r� )r� )r� rz r r� � s2
z$MultiprocessRefactoringTool.refactorc
sR | j j� }xB|d k rL|\}}ztt| �j||� W d | j j� X | j j� }q W d S )N)r� rm r� r� r� Z task_done)rz Ztaskr� r� )r� r r r� � s
z"MultiprocessRefactoringTool._childc s2 | j d k r | j j||f� ntt| �j||�S d S )N)r� r� r� r� r� )rz r� r� )r� r r r� � s
z)MultiprocessRefactoringTool.refactor_file)FFr )r r r! r{ r� r� r� �
__classcell__r r )r� r r� � s
r� )T)r r )(r` �
__author__r r� ro r� r+ rM � itertoolsr Zpgen2r r r Z
fixer_utilr r} r r r
rv r r' r r$ r5 r; r= � version_info�codecsr� r� rA rC r^ r_ �objectra r� r� r r r r � sF
(