org.apache.derby.impl.services.daemon
Class IndexStatisticsDaemonImpl

java.lang.Object
  org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl

All Implemented Interfaces:

java.lang.Runnable, IndexStatisticsDaemon

public class IndexStatisticsDaemonImpl
extends java.lang.Object
implements IndexStatisticsDaemon, java.lang.Runnable

Daemon acting as a coordinator for creating and updating index cardinality statistics.

The need for updated statistics is currently determined when compiling a SELECT query. The unit of work is then scheduled with this daemon, and the work itself will be carried out in a separate thread. If the worker thread doesn't exist it is created, if it is idle the unit of work will be processed immediately, and if it is busy the unit of work has to wait in the queue.

The daemon code has a notion of a background task. If the update is run as a background task, it will try to affect other activity in the Derby database as little as possible. As far as possible, it will not set locks on the conglomerates it scans, and if it needs to take locks it will give up immediately if the locks cannot be obtained. In some cases it will also roll back to release locks already taken, ad then retry. Since we are accessing shared structures the background work may still interfere with the user activity in the database due to locking, but all such operations carried out by the daemon are of short duration.

The high level flow of an update to index statistics is:

1. schedule update (the only action carried out by the user thread)
3. for each index:
1. scan index
2. invalidate statements dependent on current statistics
3. drop existing statistics
4. add new statistics

List of possible improvements:

1. Reduce potential impact of multiple invalidations (per table), probably by finding a way to invalidate only once after all indexes for a table have had their statistics updated. So far invalidation has proven to be the most difficult piece of the puzzle due to the interaction with the data dictionary and sensitivity to concurrent activity for the table.

Implementation notes: List of potential cleanups before going into a release:

1. Consider removing all tracing code. May involve improving logging if parts of the trace output is valuable enough.

Nested Class Summary

private static class

IndexStatisticsDaemonImpl.KeyComparator Support class used to compare keys when scanning indexes.

Field Summary

private static boolean	AS_BACKGROUND_TASK
private static boolean	AS_EXPLICIT_TASK
private ContextManager	ctxMgr The context manager for the worker thread.
private boolean	daemonDisabled Tells if the daemon has been disabled.
private LanguageConnectionContext	daemonLCC The language connection context for the worker thread.
private java.lang.String	databaseName
private Database	db The database object for the database we are handling automatic index statistics update for.
private java.lang.String	dbOwner The name of the database owner.
private boolean	doLog Tells if logging is enabled.
private boolean	doTrace Tells if tracing is enabled.
private int	errorsConsecutive Number of consecutive errors, used as a metric to decide if the damoen should be automatically shut down.
private long	errorsKnown
private long	errorsUnknown
private HeaderPrintWriter	logStream
private static int	MAX_QUEUE_LENGTH Maximum number of work units allowed in the queue.
private java.util.ArrayList	queue A list of tables that shall have their index statistics updated.
private java.lang.Thread	runningThread The thread in which the index statistics refresh operation is being executed, if any.
private long	runTime The period of time (ms) for which the daemon has been doing active work.
boolean	skipDisposableStats Tells if disposable stats should be generated, which will happen in soft-upgrade mode or when the user asks us to revert to the old behavior.
private long	timeOfCreation Specifies when the daemon was created.
private boolean	traceToDerbyLog Tells if traces are written to the Derby log file.
private boolean	traceToStdOut Tells if traces are written to standard out.
private java.lang.StringBuffer	tsb
private long	wuProcessed
private long	wuRejectedDup
private long	wuRejectedFQ
private long	wuRejectedOther
private long	wuScheduled

Constructor Summary

IndexStatisticsDaemonImpl(HeaderPrintWriter log, boolean doLog, java.lang.String traceLevel, Database db, java.lang.String userName, java.lang.String databaseName)
Creates a new daemon.

Method Summary

private boolean	acceptWork(TableDescriptor td) Determines if the given work can be accepted.
private void	appendRunStats(java.lang.StringBuffer sb) Appends runtime statistics to the given string buffer.
private static java.lang.String	cardToStr(long[] cardinality) Produces a textual representation of the cardinality numbers.
private boolean	dbAtLeast10_9(Database db) Tells if the database is 10.9 or newer.
private static java.lang.String	extractIstatInfo(java.lang.Throwable t) Purely for debugging, to avoid printing too much info.
private static java.lang.String	fmtScanTimes(long[][] timings) Format array of scan durations as a string.
private void	generateStatistics(LanguageConnectionContext lcc, TableDescriptor td) Generates index statistics for all indexes associated with the given table descriptor.
private boolean	handleExpectedErrors(TableDescriptor td, StandardException se) Handles expected errors.
private boolean	handleFatalErrors(ContextManager cm, StandardException se) Handles fatal errors that will cause the daemon to be shut down.
private boolean	handleUnexpectedErrors(TableDescriptor td, StandardException se) Handles unexpected errors.
private void	invalidateStatements(LanguageConnectionContext lcc, TableDescriptor td, boolean asBackgroundTask) Performs an invalidation action for the given table (the event being statistics update).
private boolean	isShuttingDown() Return true if we are being shutdown
private void	log(boolean asBackgroundTask, TableDescriptor td, java.lang.String msg)
private void	log(boolean asBackgroundTask, TableDescriptor td, java.lang.Throwable t, java.lang.String msg) Logs the information given.
private void	logAlways(TableDescriptor td, java.lang.Throwable t, java.lang.String msg) Logs the information given.
private void	processingLoop() Main processing loop which will compute statistics until the queue of scheduled work units has been drained.
void	run() Drives the statistics generation.
void	runExplicitly(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, java.lang.String runContext) Runs the statistics update sequence explicitly as requested by the user.
void	schedule(TableDescriptor td) Schedules an update of the index statistics for the specified table.
private void	setHeapRowEstimate(TransactionController tc, long tableId, long rowEstimate) Sets the row estimate for the heap conglomerate.
private static void	sleep(long ms) Puts the current thread to sleep for maximum ms milliseconds.
void	stop() Stops the daemon.
private void	trace(int indentLevel, java.lang.String msg)
private void	updateIndexStatsMinion(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, boolean asBackgroundTask) Updates the index statistics for the given table and the specified indexes.
private void	writeUpdatedStats(LanguageConnectionContext lcc, TableDescriptor td, UUID index, long numRows, long[] cardinality, boolean asBackgroundTask) Writes updated statistics for the specified index to the data dictionary.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

AS_BACKGROUND_TASK

private static final boolean AS_BACKGROUND_TASK

See Also:

Constant Field Values

AS_EXPLICIT_TASK

private static final boolean AS_EXPLICIT_TASK

See Also:

Constant Field Values

MAX_QUEUE_LENGTH

private static final int MAX_QUEUE_LENGTH

Maximum number of work units allowed in the queue.

logStream

private final HeaderPrintWriter logStream

doLog

private final boolean doLog

Tells if logging is enabled.

doTrace

private final boolean doTrace

Tells if tracing is enabled.

traceToDerbyLog

private final boolean traceToDerbyLog

Tells if traces are written to the Derby log file.

traceToStdOut

private final boolean traceToStdOut

Tells if traces are written to standard out.

daemonDisabled

private boolean daemonDisabled

Tells if the daemon has been disabled.

ctxMgr

private final ContextManager ctxMgr

The context manager for the worker thread.

skipDisposableStats

public final boolean skipDisposableStats

Tells if disposable stats should be generated, which will happen in soft-upgrade mode or when the user asks us to revert to the old behavior.

Made public to allow access for CreateIndexConstantAction and FromBaseTable, but this is no longer necessary when the debug property to keep disposable statistics is removed.

daemonLCC

private LanguageConnectionContext daemonLCC

The language connection context for the worker thread.

db

private final Database db

The database object for the database we are handling automatic index statistics update for.

dbOwner

private final java.lang.String dbOwner

The name of the database owner.

databaseName

private final java.lang.String databaseName

queue

private final java.util.ArrayList queue

A list of tables that shall have their index statistics updated. Note that the descriptor isn't removed before the work has been completed.

runningThread

private java.lang.Thread runningThread

The thread in which the index statistics refresh operation is being executed, if any. Created as needed, but there will only be one thread doing the work. The thread is allowed to die since it is assumed that index statistics regeneration is rather infrequent.

errorsConsecutive

private int errorsConsecutive

Number of consecutive errors, used as a metric to decide if the damoen should be automatically shut down.

errorsUnknown

private long errorsUnknown

errorsKnown

private long errorsKnown

wuProcessed

private long wuProcessed

wuScheduled

private long wuScheduled

wuRejectedDup

private long wuRejectedDup

wuRejectedFQ

private long wuRejectedFQ

wuRejectedOther

private long wuRejectedOther

timeOfCreation

private final long timeOfCreation

Specifies when the daemon was created.

runTime

private long runTime

The period of time (ms) for which the daemon has been doing active work.

tsb

private final java.lang.StringBuffer tsb

Constructor Detail

IndexStatisticsDaemonImpl

public IndexStatisticsDaemonImpl(HeaderPrintWriter log,
                                 boolean doLog,
                                 java.lang.String traceLevel,
                                 Database db,
                                 java.lang.String userName,
                                 java.lang.String databaseName)

Creates a new daemon.

Parameters:

log - the log to write to

doLog - whether to log activity information

traceLevel - whether, and to where, trace information should be written ("off|log|stdout|both")

db - the database ("off|log|stdout|both")

userName - the name of the database owner

databaseName - the name of the database (not stored in the db obj)

Method Detail

dbAtLeast10_9

private boolean dbAtLeast10_9(Database db)

Tells if the database is 10.9 or newer.

schedule

public void schedule(TableDescriptor td)

Schedules an update of the index statistics for the specified table.

Assume the descriptor will be valid until we get around to generate the statistics. If it turns out to be invalid, it will be discarded.

Specified by:

schedule in interface IndexStatisticsDaemon

Parameters:

td - base table descriptor to update index statistics for

acceptWork

private boolean acceptWork(TableDescriptor td)

Determines if the given work can be accepted.

Parameters:

td - the table descriptor to check

Returns:

true if work can be accepted, false if not.

generateStatistics

private void generateStatistics(LanguageConnectionContext lcc,
                                TableDescriptor td)
                         throws StandardException

Generates index statistics for all indexes associated with the given table descriptor.

This method is run as a background task.

Parameters:

lcc - connection context to use to perform the work

td - target base table descriptor

Throws:

StandardException - if accessing the conglomerates fail

isShuttingDown

private boolean isShuttingDown()

Return true if we are being shutdown

updateIndexStatsMinion

private void updateIndexStatsMinion(LanguageConnectionContext lcc,
                                    TableDescriptor td,
                                    ConglomerateDescriptor[] cds,
                                    boolean asBackgroundTask)
                             throws StandardException

Updates the index statistics for the given table and the specified indexes.

API note: Using null to update the statistics for all conglomerates is preferred over explicitly passing an array with all the conglomerates for the table. Doing so allows for some optimizations, and will cause a disposable statistics check to be performed.

Parameters:

lcc - language connection context used to perform the work

td - the table to update index stats for

cds - the conglomerates to update statistics for (non-index conglomerates will be ignored), null means all indexes

asBackgroundTask - whether the updates are done automatically as part of a background task or if explicitly invoked by the user

Throws:

StandardException - if something goes wrong

writeUpdatedStats

private void writeUpdatedStats(LanguageConnectionContext lcc,
                               TableDescriptor td,
                               UUID index,
                               long numRows,
                               long[] cardinality,
                               boolean asBackgroundTask)
                        throws StandardException

Writes updated statistics for the specified index to the data dictionary.

Parameters:

lcc - connection context to use to perform the work

td - the base table

index - the index of the base table

numRows - number of rows in the base table

cardinality - the number of unique values in the index (per number of leading columns)

asBackgroundTask - whether the update is done automatically as part of a background task or if explicitly invoked by the user

Throws:

StandardException - if updating the data dictionary fails

invalidateStatements

private void invalidateStatements(LanguageConnectionContext lcc,
                                  TableDescriptor td,
                                  boolean asBackgroundTask)
                           throws StandardException

Performs an invalidation action for the given table (the event being statistics update).

Parameters:

lcc - connection context to use to perform the work

td - the table to invalidate for

asBackgroundTask - whether the update is done automatically as part of a background task or if explicitly invoked by the user

Throws:

StandardException - if the invalidation request fails

setHeapRowEstimate

private void setHeapRowEstimate(TransactionController tc,
                                long tableId,
                                long rowEstimate)
                         throws StandardException

Sets the row estimate for the heap conglomerate.

Parameters:

tc - transaction to use

tableId - the heap table

rowEstimate - estimate of number of rows in the table

Throws:

StandardException - if accessing the table fails

run

public void run()

Drives the statistics generation.

This method will be run in a separate thread, and it will keep working as long as there is work to do. When the queue is exhausted, the method will exit (the thread dies).

Specified by:

run in interface java.lang.Runnable

processingLoop

private void processingLoop()

Main processing loop which will compute statistics until the queue of scheduled work units has been drained.

runExplicitly

public void runExplicitly(LanguageConnectionContext lcc,
                          TableDescriptor td,
                          ConglomerateDescriptor[] cds,
                          java.lang.String runContext)
                   throws StandardException

Runs the statistics update sequence explicitly as requested by the user.

Specified by:

runExplicitly in interface IndexStatisticsDaemon

Parameters:

lcc - connection context to use to perform the work

td - the base table

cds - the indexes to update (non-index conglomerates are ignored)

runContext - the context in which the operation is run (i.e. 'ALTER TABLE', may be null)

Throws:

StandardException - if updating the index statistics fails

stop

public void stop()

Stops the daemon.

Will also clear the queue and print runtime statistics to the log the first time the method is invoked.

Specified by:

stop in interface IndexStatisticsDaemon

handleFatalErrors

private boolean handleFatalErrors(ContextManager cm,
                                  StandardException se)

Handles fatal errors that will cause the daemon to be shut down.

Parameters:

cm - context manager

se - the exception to handle

Returns:

true if the error was handled, false otherwise

handleExpectedErrors

private boolean handleExpectedErrors(TableDescriptor td,
                                     StandardException se)

Handles expected errors.

The logging of expected errors is for observability purposes only. The daemon is capable of dealing with these errors, and no interaction from the user is expected.

Parameters:

se - the exception to handle

Returns:

true if the error was handled, false otherwise

handleUnexpectedErrors

private boolean handleUnexpectedErrors(TableDescriptor td,
                                       StandardException se)

Handles unexpected errors.

Unexpected errors are error conditions the daemon isn't set up to handle specifically. For this reason the stack trace will be logged to allow for later investigation.

In general it is expected that the daemon will be able to recover by dropping the current unit of work and move on to the next one (if any).

Parameters:

se - the exception to handle

Returns:

true if the error was handled, false otherwise

sleep

private static void sleep(long ms)

Puts the current thread to sleep for maximum ms milliseconds.

No guarantee is provided for the minimum amount of time slept. If interrupted, the interrupt flag will be set again.

Parameters:

ms - target sleep time

fmtScanTimes

private static java.lang.String fmtScanTimes(long[][] timings)

Format array of scan durations as a string.

log

private void log(boolean asBackgroundTask,
                 TableDescriptor td,
                 java.lang.String msg)

See Also:

log(boolean, TableDescriptor, Throwable, String)

log

private void log(boolean asBackgroundTask,
                 TableDescriptor td,
                 java.lang.Throwable t,
                 java.lang.String msg)

Logs the information given.

Note that if asBackgroundTask is false, nothing will be logged currently.

Parameters:

asBackgroundTask - true if logging for the background daemon automatically updating stats, false if not

td - current table descriptor being worked on, may be null

t - raised error, may be null

msg - the message to log

logAlways

private void logAlways(TableDescriptor td,
                       java.lang.Throwable t,
                       java.lang.String msg)

Logs the information given.

Parameters:

td - current table descriptor being worked on, may be null

t - raised error, may be null

msg - the message to log

trace

private void trace(int indentLevel,
                   java.lang.String msg)

appendRunStats

private void appendRunStats(java.lang.StringBuffer sb)

Appends runtime statistics to the given string buffer.

Parameters:

sb - the string buffer to append to

cardToStr

private static java.lang.String cardToStr(long[] cardinality)

Produces a textual representation of the cardinality numbers.

Parameters:

cardinality - index cardinality

Returns:

A string.

extractIstatInfo

private static java.lang.String extractIstatInfo(java.lang.Throwable t)

Purely for debugging, to avoid printing too much info.