U Id: @sdZddlZddlZddlZddlmZmZddlmZddl m Z m Z m Z m Z mZddlmZmZmZmZmZmZddlmZdd d d d d dddddddg ZedZdddddgZeddZGdddeZGddded Z Gd!d d e Z!Gd"d d e"Z#Gd#d$d$Z$e$Z%Gd%d&d&e&Z'Gd'd(d(Z(d)dZ)Gd*d+d+Z*e*Z+d,dZ,Gd-ddZ-Gd.ddeZ.Gd/d0d0eZ/Gd1d d e e/d Z0Gd2d d Z1Gd3d d e0Z2d4d5Z3Gd6d7d7e/Z4Gd8d9d9e0e4d Z5dS):z pygments.lexer ~~~~~~~~~~~~~~ Base lexer classes. :copyright: Copyright 2006-2023 by the Pygments team, see AUTHORS. :license: BSD, see LICENSE for details. N) apply_filtersFilter)get_filter_by_name)ErrorTextOther Whitespace _TokenType) get_bool_opt get_int_opt get_list_optmake_analysatorFuture guess_decode) regex_optLexer RegexLexerExtendedRegexLexerDelegatingLexer LexerContextincludeinheritbygroupsusingthisdefaultwordsline_rez.*? )sutf-8)szutf-32)szutf-32be)szutf-16)szutf-16becCsdS)Nxr r S/home/sam/Atlas/atlas_env/lib/python3.8/site-packages/pip/_vendor/pygments/lexer.py"r$c@seZdZdZddZdS) LexerMetaz This metaclass automagically converts ``analyse_text`` methods into static methods which always return float values. cCs(d|krt|d|d<t||||S)N analyse_text)r type__new__)mcsnamebasesdr r r#r)+szLexerMeta.__new__N)__name__ __module__ __qualname____doc__r)r r r r#r&%sr&c@s^eZdZdZdZgZgZgZgZdZ dZ ddZ ddZ dd Z d d Zdd dZddZdS)rau Lexer for a specific language. See also :doc:`lexerdevelopment`, a high-level guide to writing lexers. Lexer classes have attributes used for choosing the most appropriate lexer based on various criteria. .. autoattribute:: name :no-value: .. autoattribute:: aliases :no-value: .. autoattribute:: filenames :no-value: .. autoattribute:: alias_filenames .. autoattribute:: mimetypes :no-value: .. autoattribute:: priority Lexers included in Pygments should have an additional attribute: .. autoattribute:: url :no-value: You can pass options to the constructor. The basic options recognized by all lexers and processed by the base `Lexer` class are: ``stripnl`` Strip leading and trailing newlines from the input (default: True). ``stripall`` Strip all leading and trailing whitespace from the input (default: False). ``ensurenl`` Make sure that the input ends with a newline (default: True). This is required for some lexers that consume input linewise. .. versionadded:: 1.3 ``tabsize`` If given and greater than 0, expand tabs in the input (default: 0). ``encoding`` If given, must be an encoding name. This encoding will be used to convert the input string to Unicode, if it is not already a Unicode string (default: ``'guess'``, which uses a simple UTF-8 / Locale / Latin1 detection. Can also be ``'chardet'`` to use the chardet library, if it is installed. ``inencoding`` Overrides the ``encoding`` if given. NrcKs||_t|dd|_t|dd|_t|dd|_t|dd|_|dd |_|d pZ|j|_g|_ t |d d D]}| |qpd S)a This constructor takes arbitrary options as keyword arguments. Every subclass must first process its own options and then call the `Lexer` constructor, since it processes the basic options like `stripnl`. An example looks like this: .. sourcecode:: python def __init__(self, **options): self.compress = options.get('compress', '') Lexer.__init__(self, **options) As these options must all be specifiable as strings (due to the command line usage), there are various utility functions available to help with that, see `Utilities`_. stripnlTstripallFensurenltabsizerencodingguess inencodingfiltersr N) optionsr r2r3r4r r5getr6r9r add_filter)selfr:filter_r r r#__init__szLexer.__init__cCs(|jrd|jj|jfSd|jjSdS)Nzz)r: __class__r.r=r r r#__repr__s zLexer.__repr__cKs&t|tst|f|}|j|dS)z8 Add a new stream filter to this lexer. N) isinstancerrr9append)r=r>r:r r r#r<s  zLexer.add_filtercCsdS)a A static method which is called for lexer guessing. It should analyse the text and return a float in the range from ``0.0`` to ``1.0``. If it returns ``0.0``, the lexer will not be selected as the most probable one, if it returns ``1.0``, it will be selected immediately. This is used by `guess_lexer`. The `LexerMeta` metaclass automatically wraps this function so that it works like a static method (no ``self`` or ``cls`` parameter) and the return value is automatically converted to `float`. If the return value is an object that is boolean `False` it's the same as if the return values was ``0.0``. Nr )textr r r#r'szLexer.analyse_textFc sttsjdkr$t\}nވjdkrzddlm}Wn,tk rj}ztd|W5d}~XYnXd}tD].\}}|rtt |d |d}qqt|dkr| dd}  | d pd d}|n( jd r t d dnd r t d d d d  dd jrJnjr\d jdkrtjjrd sd 7fdd} | } |st| j} | S)ae This method is the basic interface of a lexer. It is called by the `highlight()` function. It must process the text and return an iterable of ``(tokentype, value)`` pairs from `text`. Normally, you don't need to override this method. The default implementation processes the options recognized by all lexers (`stripnl`, `stripall` and so on), and then yields all tokens from `get_tokens_unprocessed()`, with the ``index`` dropped. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. r7chardetr)rFzkTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/Nreplaceir6ruz   c3s$D]\}}}||fVq dSN)get_tokens_unprocessed)_tvr=rEr r#streamersz"Lexer.get_tokens..streamer)rCstrr6r pip._vendorrF ImportError _encoding_map startswithlendecodedetectr;rGr3stripr2r5 expandtabsr4endswithrr9) r=rE unfilteredrLrFedecodedbomr6encrPstreamr rOr# get_tokenssR              zLexer.get_tokenscCstdS)aS This method should process the text and return an iterable of ``(index, tokentype, value)`` tuples where ``index`` is the starting position of the token within the input text. It must be overridden by subclasses. It is recommended to implement it as a generator to maximize effectiveness. N)NotImplementedErrorrOr r r#rKs zLexer.get_tokens_unprocessed)F)r.r/r0r1r+aliases filenamesalias_filenames mimetypespriorityurlr?rBr<r'rbrKr r r r#r1s4 @) metaclassc@s$eZdZdZefddZddZdS)ra  This lexer takes two lexer as arguments. A root lexer and a language lexer. First everything is scanned using the language lexer, afterwards all ``Other`` tokens are lexed using the root lexer. The lexers from the ``template`` lexer package use this base lexer. cKs0|f||_|f||_||_tj|f|dSrJ) root_lexerlanguage_lexerneedlerr?)r= _root_lexer_language_lexer_needler:r r r#r?s  zDelegatingLexer.__init__cCsd}g}g}|j|D]H\}}}||jkrP|rF|t||fg}||7}q||||fq|rx|t||ft||j|S)N)rlrKrmrDrV do_insertionsrk)r=rEbuffered insertions lng_bufferirMrNr r r#rKs   z&DelegatingLexer.get_tokens_unprocessedN)r.r/r0r1rr?rKr r r r#r s c@seZdZdZdS)rzI Indicates that a state should include rules from another state. Nr.r/r0r1r r r r#r4sc@seZdZdZddZdS)_inheritzC Indicates the a state should inherit from its superclass. cCsdS)Nrr rAr r r#rB?sz_inherit.__repr__N)r.r/r0r1rBr r r r#rx;srxc@s eZdZdZddZddZdS)combinedz: Indicates a state combined from multiple states. cGs t||SrJ)tupler))clsargsr r r#r)Jszcombined.__new__cGsdSrJr )r=r|r r r#r?Mszcombined.__init__N)r.r/r0r1r)r?r r r r#ryEsryc@sFeZdZdZddZdddZdddZdd d Zd d Zd dZ dS) _PseudoMatchz: A pseudo match object constructed from a string. cCs||_||_dSrJ)_text_start)r=startrEr r r#r?Wsz_PseudoMatch.__init__NcCs|jSrJ)rr=argr r r#r[sz_PseudoMatch.startcCs|jt|jSrJ)rrVr~rr r r#end^sz_PseudoMatch.endcCs|r td|jS)Nz No such group) IndexErrorr~rr r r#groupasz_PseudoMatch.groupcCs|jfSrJ)r~rAr r r#groupsfsz_PseudoMatch.groupscCsiSrJr rAr r r# groupdictisz_PseudoMatch.groupdict)N)N)N) r.r/r0r1r?rrrrrr r r r#r}Rs   r}csdfdd }|S)zL Callback that yields multiple actions for each group in the match. Nc3stD]\}}|dkrqqt|tkrR||d}|r||d||fVq||d}|dk r|r|||d|_||t||d||D]}|r|Vqq|r||_dS)N) enumerater(r rrposr}r)lexermatchctxrvactiondataitemr|r r#callbackqs&  zbygroups..callback)Nr )r|rr rr#rmsc@seZdZdZdS)_ThiszX Special singleton used for indicating the caller class. Used by ``using``. Nrwr r r r#rsrc sjidkr:d}t|ttfr.|d<n d|fd<tkrTdfdd }nd fdd }|S) a Callback that processes the match with a different lexer. The keyword arguments are forwarded to the lexer, except `state` which is handled separately. `state` specifies the state that the new lexer will start in, and can be an enumerable such as ('root', 'inline', 'string') or a simple string which is assumed to be on top of the root state. Note: For that to work, `_other` must not be an `ExtendedRegexLexer`. statestackrootNc3sjr|j|jf}n|}|}|j|fD]\}}}||||fVq<|rf||_dSrJ)updater:r@rrKrrrrrrlxsrvrMrN) gt_kwargskwargsr r#rs zusing..callbackc3s^|jf}|}|j|fD]\}}}||||fVq0|rZ||_dSrJ)rr:rrKrrrr_otherrrr r#rs  )N)N)poprClistrzr)rrrrr rr#rs     c@seZdZdZddZdS)rz Indicates a state or state action (e.g. #pop) to apply. For example default('#pop') is equivalent to ('', Token, '#pop') Note that state tuples may be used as well. .. versionadded:: 2.0 cCs ||_dSrJ)r)r=rr r r#r?szdefault.__init__N)r.r/r0r1r?r r r r#rsc@s"eZdZdZdddZddZdS) rz Indicates a list of literal words that is transformed into an optimized regex that matches any of the words. .. versionadded:: 2.0 rqcCs||_||_||_dSrJ)rprefixsuffix)r=rrrr r r#r?szwords.__init__cCst|j|j|jdS)Nrr)rrrrrAr r r#r;sz words.getN)rqrq)r.r/r0r1r?r;r r r r#rs c@sJeZdZdZddZddZddZdd Zdd d Zd dZ ddZ d S)RegexLexerMetazw Metaclass for RegexLexer, creates the self._tokens attribute from self.tokens on the first instantiation. cCs t|tr|}t||jS)zBPreprocess the regular expression component of a token definition.)rCrr;recompiler)r{regexrflagsrr r r#_process_regexs zRegexLexerMeta._process_regexcCs&t|tks"t|s"td|f|S)z5Preprocess the token component of a token definition.z2token type must be simple type or callable, not %r)r(r callableAssertionError)r{tokenr r r#_process_tokenszRegexLexerMeta._process_tokencCst|trd|dkrdS||kr$|fS|dkr0|S|dddkrRt|dd Sdsbtd|nt|trd |j}|jd 7_g}|D],}||kstd ||||||q|||<|fSt|tr|D] }||ks|d kstd |q|Sdstd|dS)z=Preprocess the state transition action of a token definition.#pop#pushNz#pop:Fzunknown new state %rz_tmp_%drzcircular state ref %r)rrzunknown new state zunknown new state def %r) rCrQintrry_tmpnameextend_process_staterz)r{ new_state unprocessed processed tmp_stateitokensistater r r#_process_new_states>     z!RegexLexerMeta._process_new_statec Cst|tkstd||ddks0td|||kr@||Sg}||<|j}||D].}t|tr||ks~td|||||t|qZt|trqZt|t r| |j ||}| t djd|fqZt|tkstd|z||d||}Wn>tk rD} ztd |d||| f| W5d} ~ XYnX||d } t|d krhd}n| |d ||}| || |fqZ|S) z%Preprocess a single state definition.zwrong state name %rr#zinvalid state name %rzcircular state reference %rrqNzwrong rule def %rz+uncompilable regex %r in state %r of %r: %sr)r(rQrflagsrCrrrrxrrrrDrrrrzr Exception ValueErrorrrV) r{rrrtokensrtdefrrexerrrr r r#rsJ      zRegexLexerMeta._process_stateNcCs<i}|j|<|p|j|}t|D]}||||q$|S)z-Preprocess a dictionary of token definitions.) _all_tokensrrr)r{r+ tokendefsrrr r r#process_tokendef?s  zRegexLexerMeta.process_tokendefc Csi}i}|jD]}|jdi}|D]\}}||}|dkr||||<z|t}Wntk rpYq(YnX|||<q(||d}|dkrq(||||d<z|t} Wntk rYq(X|| ||<q(q|S)a Merge tokens from superclasses in MRO order, returning a single tokendef dictionary. Any state that is not defined by a subclass will be inherited automatically. States that *are* defined by subclasses will, by default, override that state in the superclass. If a subclass wishes to inherit definitions from a superclass, it can use the special value "inherit", which will cause the superclass' state definition to be included at that point in the state. rNr)__mro____dict__r;itemsindexrrr) r{r inheritablectoksrrcuritems inherit_ndx new_inh_ndxr r r# get_tokendefsGs0     zRegexLexerMeta.get_tokendefscOsLd|jkr:i|_d|_t|dr(|jr(n|d||_tj |f||S)z:Instantiate cls after preprocessing its token definitions._tokensrtoken_variantsrq) rrrhasattrrrrrr(__call__)r{r|kwdsr r r#rxs zRegexLexerMeta.__call__)N) r.r/r0r1rrrrrrrr r r r#rs#, 1rc@s$eZdZdZejZiZdddZdS)rz Base for simple stateful regular expression-based lexers. Simplifies the lexing process so that you need only provide a list of states and regular expressions. rc csd}|j}t|}||d}|D](\}}} |||} | r"|dk rrt|tkrb||| fVn||| EdH| }| dk rHt| tr| D]D} | dkrt|dkr| q| dkr| |dq| | qnbt| t rt | t|kr|dd=n || d=n,| dkr*| |dndsMetaclass for ProfilingRegexLexer, collects regex timing info.csLt|tr t|j|j|jdn|t|tjffdd }|S)Nrcs`jdfddg}t}|||}t}|dd7<|d||7<|S)Nrrrr) _prof_data setdefaulttimer)rErendposinfot0rest1r{compiledrrr r# match_funcsz:ProfilingRegexLexerMeta._process_regex..match_func) rCrrrrrrsysmaxsize)r{rrrrr rr#rs   z&ProfilingRegexLexerMeta._process_regexN)r.r/r0r1rr r r r#rsrc@s"eZdZdZgZdZdddZdS)ProfilingRegexLexerzFDrop-in replacement for RegexLexer that does profiling of its regexes.rc#sjjit||EdHjj}tdd|Dfdddd}tdd|D}t t djj t ||ft d t d d t d |D]}t d |qt d dS)NcssN|]F\\}}\}}|t|ddddd|d|d||fVqdS)zu'z\\\NAi)reprrYrG).0rrnrMr r r# s  z=ProfilingRegexLexer.get_tokens_unprocessed..cs |jSrJ)_prof_sort_indexr!rAr r#r$r%z.T)keyreversecss|]}|dVqdS)Nr )rr"r r r#rsz2Profiling result for %s lexing %d chars in %.3f mszn==============================================================================================================z$%-20s %-64s ncalls tottime percall)rrzn--------------------------------------------------------------------------------------------------------------z%-20s %-65s %5d %8.4f %8.4f) r@rrDrrKrsortedrsumprintr.rV)r=rErrawdatar sum_totalr-r rAr#rKs(   z*ProfilingRegexLexer.get_tokens_unprocessedN)r)r.r/r0r1rrrKr r r r#rsr)6r1rrrpip._vendor.pygments.filterrrpip._vendor.pygments.filtersrpip._vendor.pygments.tokenrrrrr pip._vendor.pygments.utilr r r r rrpip._vendor.pygments.regexoptr__all__rrrT staticmethod_default_analyser(r&rrrQrrxrrzryr}rrrrrrrrrrrrrrr r r r#sd       ]' 2 )aH@