DatedFileAppender for Log4j and Tomcat

DatedFileAppender for Log4j and Tomcat

Written by Geoff Mottram (geoff at minaret dot biz).

Placed in the public domain on August 28, 2004 by the author.

Last updated: January 27, 2005.

This document and all associated software is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. In no event shall the author be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with this document and associated software or the use or other dealings in same.

Contents
Introduction
File Loggers
Installing DatedFileAppender
Compiling DatedFileAppender
Configuring DatedFileAppender
Log File Directory
Change History

Introduction
DatedFileAppender is an "appender" object designed for use with the Apache Log4j logging system. It provides an Apache Tomcat style FileLogger implementation that differs from the file loggers provided with Log4j. While the DatedFileAppender was written with Apache Tomcat in mind, it does not depend on Tomcat and may be used in any other environment.

Please note: The new alpha version of Log4j 1.3 has added support for a daily rolling log file. See Log4j Version 1.3 and Apache Tomcat for instructions on how to install and configure this new feature.

A separate technical tip describes how to configure Log4J to be the default system Logger for Tomcat. In brief, you must:

Shutdown Tomcat if it is currently running.

Download the Commons Logging package from the Apache web site (unless you already have it).

Copy the commons-logging.jar file from the distribution into your Tomcat common/lib directory.

Download the Log4j package from the Apache web site (unless you already have it).

Copy the log4j.jar file from the distribution into your Tomcat common/lib directory.

To use DatedFileAppender as the file logger in Tomcat, install the datedFileAppender-1.0.2.jar file into your Tomcat common/lib directory.

Install a log4j.properties file (a sample file comes with the DatedFileAppender distribution) into your Tomcat common/classes directory.

Restart Tomcat.

The DatedFileAppender source code, binaries and associated documentation have all been placed in the public domain.

File Loggers
The standard Log4J DailyRollingFileAppender writes to a log file (say tomcat.log) until some pre-established criteria has been met (i.e. the current day has changed). At that point, the existing log file is closed and renamed using some sort of file name extension (i.e. moved to tomcat.log.2004-08-28). The original log file is then re-created and the system continues to chug along. In order to trigger a log rotation, a message must be logged to the log file. This creates a problem if you want to be able to run a cron job at night to analyze, compress or remove your old log files. This is because you can't be sure that the log files have been rotated (if you are logging lots of data, the logs will probably have been rotated, but if you are only logging ERROR level messages, you have no guarantee).

The FileLogger implementation that comes with Tomcat to log Servlet messages avoids this problem by always using dates in the log file names. Any time after midnight, you know that Tomcat's Servlet logs will not get any more information. There is no guarantee that a previous log file has been closed by the Tomcat FileLogger because it too relies on logging a new message in order to trigger a log file change. However, for your purposes it doesn't matter since the old log files will not be receiving any more messages.

The DatedFileAppender, described here, is a drop in replacement for the Log4J DailyRollingFileAppender but which operates in a manner similar to Tomcat's FileLogger.
Installing DatedFileAppender
DatedFileAppender is available as a zip file that contains this documentation, a sample log4j.properties file for Tomcat, a jar file and the source code for the DatedFileAppender class. The directions and supporting shell scripts were written for a Unix-like system. However, users of other operating systems should have no problem making the appropriate adjustments.

Stand alone installation instructions:

Download the datedFileAppender-1.0.2.zip file to your computer.

Unzip the contents of the zip file. This will create a datedFileAppender-1.0.2 subdirectory in your current directory.

Change to the datedFileAppender-1.0.2 subdirectory.

Copy the datedFileAppender-1.0.2.jar to the same directory as your other application jar files and add the jar file name to your CLASSPATH.

Modify your log4j.properties file to use the DatedFileAppender class (see below for configuration options).

Installation instructions for use as the Tomcat system logger:

Shutdown Tomcat if it is currently running.

Download the datedFileAppender-1.0.2.zip file to your computer.

Unzip the contents of the zip file. This will create a datedFileAppender-1.0.2 subdirectory in your current directory.

Change to the datedFileAppender-1.0.2 subdirectory.

Copy the datedFileAppender-1.0.2.jar to your Tomcat common/lib directory.
Copy the log4j.properties to your Tomcat common/classes directory. If you already have a properties file, edit it as appropriate to use the DatedFileAppender class (see below for configuration options).

Restart Tomcat.

If you use the log4j.properties file that is provided, three log files will be created in your Tomcat logs directory: catalina.out, servlet.yyyy-mm-dd.log and tomcat.yyyy-mm-dd.log; where yyyy-mm-dd will be replaced with today's date.

The servlet log contains output from calls to the Java servlet log() method. The tomcat log will contain messages that have been output by applications using the Apache Commons Logging interface and catalina.out contains any spurious output from standard output and standard error.

Compiling DatedFileAppender
The source code for DatedFileAppender can be found in the src subdirectory. If you are inclined to recompile the DatedFileAppender source code, a shell script has been provided (compile.sh) to assist you. Before running the script, you must edit it and change the CLASSPATH environment variable to point to your log4j jar file.

You run the script by typing:
sh compile.sh
Once run, the compiled class file and associated jar file can be found in the build subdirectory.

Configuring DatedFileAppender
The full name of the DatedFileAppender class is:
biz.minaret.log4j.DatedFileAppender
DatedFileAppender supports buffered writes to its log file. By default, no buffering is used and log entries are written to the log file as soon as they are received. This insures that no new entries will be added to a log file after midnight of any given day. On a heavily loaded system there could be cases where the logger is preempted while writing an entry just before midnight such that the log file would not be physically updated until a little after midnight. Any log file related cron jobs should be run a reasonable time after midnight to be on the safe side.

If you enable buffering by setting the BufferedIO=true property in your log4j.properties file, there is no guarantee that DatedFileAppender has finished writing to a daily log file after midnight. A new message must be logged after midnight to force the previous day's log file to be flushed and closed.

If you are logging so much data that you require write buffering, you will probably log a message soon after midnight, forcing a log file roll-over. To be absolutely sure, any cron based external log file processing that is done could wait for the new day's log file to appear before starting its work on what is now yesterday's log file.

The log4j.properties file included with the distribution can be used to configure a DatedFileAppender as the Tomcat system logger. This file also illustrates all of the configuration options and default values for this class. The file is reproduced here:
#
# log4j.properties
#
# Configures Log4j as the Tomcat system logger
# using a DatedFileAppender.
#

#
# Configure the logger to output ERROR or INFO level messages into
# a Tomcat-style rolling, dated log file ("tomcat.DATE.log").
#
#log4j.rootLogger=ERROR, T
log4j.rootLogger=INFO, T

#
# Configure the appender "T".
#
# Note that the default file name prefix is being overridden
# to be "tomcat." instead of "application.".
#
log4j.appender.T=biz.minaret.log4j.DatedFileAppender
log4j.appender.T.layout=org.apache.log4j.PatternLayout
log4j.appender.T.layout.ConversionPattern=%d [%t] %-5p %c - %m%n
#
# If you don't want to use the DatedFileAppender default values
# seen below, uncomment that line and change the setting.
#
# Directory: If the directory name does not start with a leading slash,
# the directory name will be relative to your Tomcat home directory.
#log4j.appender.T.Directory=logs
#
# Prefix: The log file name prefix.
#log4j.appender.T.Prefix=application.
log4j.appender.T.Prefix=tomcat.
#
# Suffix: The log file name suffix.
#log4j.appender.T.Suffix=.log
#
# Append: true to append when opening a log file (good when restarting
# Tomcat) or false to truncate.
#log4j.appender.T.Append=true
#
# BufferedIO: true to use a buffered output stream to the log file (improves
# performance when logging a lot of data but not so good if the system
# crashes or you want to watch the logs in real time) or false to write
# flush each message out to the log file.
#
# The default behavior of using non-buffered writes insures that a day's
# log file will not be written to after midnight. When buffering is enabled,
# a new message must be written to a log after midnight to force the previous
# day's log file to be flushed and closed.
#
#log4j.appender.T.BufferedIO=false
#
# BufferSize: sets the size of the buffer to use if BufferedIO is true.
# The default size is 8K.
#log4j.appender.T.BufferSize=8192

#
# Application logging options
#
#log4j.logger.org.apache=DEBUG
#log4j.logger.org.apache=INFO
#log4j.logger.org.apache.struts=DEBUG
#log4j.logger.org.apache.struts=INFO
   
Log File Directory
You can set the directory that DatedFileAppender uses for its log files with the Directory property in the Log4J properties file. An example using the default directory is:
log4j.appender.T.Directory=logs
If the directory name is absolute, as determined by the Java File.isAbsolute() method (under Unix, this means having a leading slash), DatedFileAppender will use that directory to store its log files.

If a relative directory name is used, DatedFileAppender will look for the system property "catalina.base". This is the base directory for Tomcat and is set by Tomcat when it is running. If this property is set, DatedFileAppender will use this as its base directory, appending the name you provide in the Directory property to create the full path name of the logs directory. If there is no "catalina.base" system property, the current directory is used as the base directory.

If you are not using Tomcat but you want DatedFileAppender to use a base directory other than the directory you start your application in, set the "catalina.base" system property when you run your application. The best place to do this is when you start your application, like this:
java -Dcatalina.base=YOUR_BASE_DIRECTORY ...
If there are spaces in the directory name, you will need quotes, as such:
java "-Dcatalina.base=YOUR_BASE_DIRECTORY" ...
Change History

Version Date Description

1.0.2 September 15, 2004 Replaced calls to Calendar.getTimeInMillis() and Calendar.getTimeInMillis() for backwards compatibility with JVM's prior to 1.4 in which these methods are protected.

1.0.1 September 8, 2004 Documentation changes only.
Added documentation on the issue of I/O buffering.

Fixed example log4j.properties BufferSize entry.

1.0 August 28, 2004 Initial release.

Version	Date	Description
1.0.2	September 15, 2004	Replaced calls to Calendar.getTimeInMillis() and Calendar.getTimeInMillis() for backwards compatibility with JVM's prior to 1.4 in which these methods are protected.
1.0.1	September 8, 2004	Documentation changes only. Added documentation on the issue of I/O buffering. Fixed example log4j.properties BufferSize entry.
1.0	August 28, 2004	Initial release.