← Back
Editing: lexer.cpython-311.pyc
� �e}_� � �\ � d Z ddlmZ ddlmZ ddlmZ ddlmZ G d� d� � Z d d �Z dS ) z SQL Lexer� )� TextIOBase)�tokens)� SQL_REGEX)�consumec �* � e Zd ZdZedd�� � ZdS )�Lexerz?Lexer Empty class. Leaving for backwards-compatibility Nc # �R K � t | t � � r| � � � } t | t � � rn�t | t � � rT|r| � |� � } nk | � d� � } nT# t $ r | � d� � } Y n3w xY wt d� t | � � � � � � �t | � � }|D ]�\ }}t D ]�\ }} || |� � }|s�t |t j � � r||� � � fV � n.t |� � r ||� � � � � V � t! ||� � � |z dz � � nt j |fV � ��dS )a� Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. Also preprocess the text, i.e. expand tabs and strip it if wanted and applies registered filters. Split ``text`` into (tokentype, text) pairs. ``stack`` is the initial stack (default: ``['root']``) zutf-8zunicode-escapez+Expected text or file-like object, got {!r}� N)� isinstancer �read�str�bytes�decode�UnicodeDecodeError� TypeError�format�type� enumerater r � _TokenType�group�callabler �end�Error)�text�encoding�iterable�pos�char�rexmatch�action�ms �0/usr/lib/python3/dist-packages/sqlparse/lexer.py� get_tokenszLexer.get_tokens s� � � � � �d�J� '� '� ��9�9�;�;�D� �d�C� � � 0� � ��e� $� $� 0�� 9��{�{�8�,�,���9� �;�;�w�/�/�D�D��)� 9� 9� 9� �;�;�'7�8�8�D�D�D�9���� � I� "�F�4��:�:� .� .�0� 0� 0� �T�?�?��!� )� )�I�C��$-� )� )� ��&� �H�T�3�'�'��� ,� � ���(9�:�:� ,� �!�'�'�)�)�+�+�+�+�+� �f�%�%� ,� �&������+�+�+�+�+���!�%�%�'�'�C�-�!�"3�4�4�4��� �l�D�(�(�(�(�� )� )s �0B � B(�'B(�N)�__name__� __module__� __qualname__�__doc__� staticmethodr# � � r" r r s> � � � � � �� � �-)� -)� -)� �\�-)� -)� -)r+ r Nc �F � t � � � | |� � S )z�Tokenize sql. Tokenize *sql* using the :class:`Lexer` and return a 2-tuple stream of ``(token type, value)`` items. )r r# )�sqlr s r" �tokenizer. L s � � �7�7� � �c�8� ,� ,�,r+ r$ ) r( �ior �sqlparser �sqlparse.keywordsr �sqlparse.utilsr r r. r* r+ r" �<module>r3 s� �� �� � � � � � � � � � � � � '� '� '� '� '� '� "� "� "� "� "� "�3)� 3)� 3)� 3)� 3)� 3)� 3)� 3)�l-� -� -� -� -� -r+
Save File
Cancel