Parsing and tokenization

Understand the idea

Parsing converts text to structure using a grammar. Tokenization identifies units; conversion interprets them. Specify delimiters, signs, escaping, whitespace, and invalid input before splitting.

Work through an example

'12, -3, 5' can be split on commas, stripped, then converted with int. '12,,5' contains an invalid empty numeric token unless explicitly permitted.

Time and space complexity

A scan is O(n), with O(n) token/output storage; integer conversion depends on digit length.

Why it works

Every accepted token satisfies the chosen grammar.

Common mistakes

• split(' ') differs from split() on repeated whitespace.
• Quoted delimiters need a real parser.

Chapter lab · Python implementation

These examples operate on Python Unicode code points. A user-visible character can span several code points; normalization and grapheme segmentation need separate policies. The lab compares mirror positions without allocating a reversed copy. For every text problem, state whether case, punctuation, and whitespace matter.

This shared lab connects the chapter's topics. Follow the current lesson's worked example first, then trace the complete implementation. Additional algorithms described in topic exercises are variations you can implement independently.

def palindrome(text):
    left, right = 0, len(text)-1
    while left < right:
        if text[left] != text[right]:
            return False
        left += 1
        right -= 1
    return True

def frequencies(text):
    counts = {}
    for char in text:
        counts[char] = counts.get(char,0)+1
    return counts

Run and test the lab

Copy the implementation and these assertions into a Python 3 file and run it. An assertion failure means the actual result differs from the expectation. Add the current topic's exercise and boundary tests to deepen your understanding.

assert palindrome('level') and palindrome('')
assert not palindrome('lever')
assert frequencies('abaca')=={'a':3,'b':1,'c':1}