public abstract class AbstractTokenizer extends Object implements Tokenizer
| Modifier and Type | Field and Description |
|---|---|
protected List<String> |
ignorableCharacter |
protected List<String> |
ignorableStmt |
protected char |
oneLineCommentChar |
protected boolean |
spanMultipleLinesString |
protected List<String> |
stringToken |
| Constructor and Description |
|---|
AbstractTokenizer() |
protected char oneLineCommentChar
protected boolean spanMultipleLinesString
public void tokenize(SourceCode tokens, Tokens tokenEntries)
Copyright © 2002-2014 InfoEther. All Rights Reserved.