de.zeigermann.xpa.tokenParser
Class SAXDrivenTokenStream

java.lang.Object
  |
  +--de.zeigermann.xpa.tokenParser.Parser2XMLTokenStreamAdapter
        |
        +--de.zeigermann.xpa.tokenParser.SAXDrivenTokenStream

All Implemented Interfaces:

TokenStream

public class SAXDrivenTokenStream

extends Parser2XMLTokenStreamAdapter

This class is a token stream parser driven by an underlying SAX parser.

This parser is runs in a thread of its own and delivers its events to a BlockingLinkedListTokenQueue. If the queue is full the SAX parser will be blocked until a token gets fetched.
When nextToken() is called by an ANTLR generated parser, it tries to get a token from the queue. If there is none the ANTLR parser will be blocked until one is available.

Although this may seem a bit overcomplicated it is necessary as SAX parsers have the habbit to push information by call backs. Unfortunately ANTLR parsers also like to take control and decided when they want to have the next token. So, as neither SAX parser nor ANTLR generated parser wait natively, we need to synchronize the communication between them.

Even though we could do this without a queue, it has been used to make things as concurrend as possible to avoid the somewhat sluggish performance of earlier version doing without a queue. Besides BlockingLinkedListTokenQueue is very convenient as it encapsulates all that nasty synchronization stuff.

Field Summary

static int

TOKEN_QUEUE_SIZE Maximum amount of tokens to queue.

Fields inherited from class de.zeigermann.xpa.tokenParser.Parser2XMLTokenStreamAdapter

tokenManager

Constructor Summary

SAXDrivenTokenStream(XMLTokenTypeManager tokenManager, InputSource inputSource)
Creates a new token stream parser.

Method Summary

protected void	applyLocatorInfo(Token token) Apply line, column and filename information from SAX events to token.
Token	nextToken() Gets next token from token queue.
protected void	setNextToken(Token token) Adds a new token to the token queue.

Methods inherited from class de.zeigermann.xpa.tokenParser.Parser2XMLTokenStreamAdapter

getIgnoreAllWhiteSpace, getTreatValidWhiteSpaceAsIgnorable, setIgnoreAllWhiteSpace, setTreatValidWhiteSpaceAsIgnorable, tellEndDocument, tellEndElement, tellPCDATA, tellStartElement, tellWhiteSpace

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TOKEN_QUEUE_SIZE

public static int TOKEN_QUEUE_SIZE

Maximum amount of tokens to queue. Should not be too large, as every buffered token uses memory.

Constructor Detail

SAXDrivenTokenStream

public SAXDrivenTokenStream(XMLTokenTypeManager tokenManager,
                            InputSource inputSource)
                     throws ParserConfigurationException,
                            SAXException

Creates a new token stream parser.

Parameters:

tokenManager - the token manager to determine token types for token

inputSource - the inputSource to parse XML from

Method Detail

nextToken

public Token nextToken()
                throws TokenStreamException

Gets next token from token queue.

Specified by:

nextToken in interface TokenStream

Specified by:

nextToken in class Parser2XMLTokenStreamAdapter

Returns:

token generated by parser that feeds this adapter

TokenStreamException

See Also:

Parser2XMLTokenStreamAdapter.setNextToken(antlr.Token)

applyLocatorInfo

protected void applyLocatorInfo(Token token)

Apply line, column and filename information from SAX events to token.

Specified by:

applyLocatorInfo in class Parser2XMLTokenStreamAdapter

setNextToken

protected void setNextToken(Token token)

Adds a new token to the token queue.

Specified by:

setNextToken in class Parser2XMLTokenStreamAdapter

See Also:

Parser2XMLTokenStreamAdapter.nextToken()