Tokenizer doubt

strings-04-designing-string-tokeniser

kaniket9811_62464638c1639a8f (Kaniket9811 62464638c1639a8f) January 27, 2024, 1:59pm #1

I am not able to understand this implementation , can you provide simpler explanation