Hash :1736f4aa Author : Date :
2017-02-08T01:36:27
Fix corner cases for the greedy flag
By refactoring the tokenize() method into two separate methods, it is
possible to recursively clean up any broken tokens left by the greedy
feature.
This should fix the issue 1075.