Class SAXDrivenTokenStream

All Implemented Interfaces:

public class SAXDrivenTokenStream
extends Parser2XMLTokenStreamAdapter

This class is a token stream parser driven by an underlying SAX parser.

This parser is runs in a thread of its own and delivers its events to a BlockingLinkedListTokenQueue. If the queue is full the SAX parser will be blocked until a token gets fetched.
When nextToken() is called by an ANTLR generated parser, it tries to get a token from the queue. If there is none the ANTLR parser will be blocked until one is available.

Although this may seem a bit overcomplicated it is necessary as SAX parsers have the habbit to push information by call backs. Unfortunately ANTLR parsers also like to take control and decided when they want to have the next token. So, as neither SAX parser nor ANTLR generated parser wait natively, we need to synchronize the communication between them.

Even though we could do this without a queue, it has been used to make things as concurrend as possible to avoid the somewhat sluggish performance of earlier version doing without a queue. Besides BlockingLinkedListTokenQueue is very convenient as it encapsulates all that nasty synchronization stuff.

Field Summary
          Maximum amount of tokens to queue.
Fields inherited from class de.zeigermann.xpa.tokenParser.Parser2XMLTokenStreamAdapter
Constructor Summary
SAXDrivenTokenStream(XMLTokenTypeManager tokenManager, InputSource inputSource)
          Creates a new token stream parser.
Method Summary
protected  void applyLocatorInfo(Token token)
          Apply line, column and filename information from SAX events to token.
 Token nextToken()
          Gets next token from token queue.
protected  void setNextToken(Token token)
          Adds a new token to the token queue.
Methods inherited from class de.zeigermann.xpa.tokenParser.Parser2XMLTokenStreamAdapter
getIgnoreAllWhiteSpace, getTreatValidWhiteSpaceAsIgnorable, setIgnoreAllWhiteSpace, setTreatValidWhiteSpaceAsIgnorable, tellEndDocument, tellEndElement, tellPCDATA, tellStartElement, tellWhiteSpace
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail


public static int TOKEN_QUEUE_SIZE
Maximum amount of tokens to queue. Should not be too large, as every buffered token uses memory.

Constructor Detail


public SAXDrivenTokenStream(XMLTokenTypeManager tokenManager,
                            InputSource inputSource)
                     throws ParserConfigurationException,
Creates a new token stream parser.

tokenManager - the token manager to determine token types for token
inputSource - the inputSource to parse XML from
Method Detail


public Token nextToken()
                throws TokenStreamException
Gets next token from token queue.

Specified by:
nextToken in interface TokenStream
Specified by:
nextToken in class Parser2XMLTokenStreamAdapter
token generated by parser that feeds this adapter
See Also:


protected void applyLocatorInfo(Token token)
Apply line, column and filename information from SAX events to token.

Specified by:
applyLocatorInfo in class Parser2XMLTokenStreamAdapter


protected void setNextToken(Token token)
Adds a new token to the token queue.

Specified by:
setNextToken in class Parser2XMLTokenStreamAdapter
See Also:

Copyright 1999-2003 Oliver Zeigermann. All Rights Reserved.