Creating Custom Web Statistics

Revised: August 11, 2004

Contents

• Overview
• Requirements
• Preparing Your Computer
• Download Analog
• Download Bzip2
• Download ReportMagic
• Downloading A Log File
• Configure Analog For Windows
• Configure ReportMagic For Windows
• Create A Report
• Where to Go From Here

MOREnet provides web statistics for every web site hosted by the kinetic Service. These statistics provide details about the operation of the web site and its visitors, such as the visitors' domains, the number of requests to the site, and the words visitors searched for on the site. A variety of statistical reports are available, including statistics by day, week, and month. For more information on the provided reports, see the Web Statistics page.

To create a customized report which perhaps details only a certain portion of the web site, or perhaps concentrates on only certain statistics, kinetic Service customers with web publishing permissions allowing them to view their site's logs directory may download these logs and generate their own custom reports.

A variety of programs are available for performing web server log file analysis. Two of these are Analog and ReportMagic. Both are available for free and run on multiple operating systems, including Windows. This page documents a step-by-step process for installing Analog and ReportMagic, downloading log files from a kinetic hosted web site, and generating custom web site statistics reports on a desktop computer running Windows.

• A web browser with a connection to the Internet
• NotePad
• A program which will unzip .zip files
• An FTP client
• An account with web publishing permissions for your web site

1. On your desktop, Double click on MyComputer.
2. Double click on the icon for your 'C' drive.
3. Create a folder called Website-Reporting
4. Open the Website-Reporting folder.
5. Create three folders - Analog , Logs , and Report

1. Open your web browser and enter the URL for the Analog web site - http://www.analog.cx/.
2. Click the "Download analog" link in the middle of the page.
3. Below the heading "Windows" you will find the download link for the current Windows version of Analog.
4. Download the most current version of analog_xyzw32.zip file, where xyz is the version number of the most current version.
5. Save this file in a temporary folder of your choice.
6. Unzip the downloaded analog_xyzw32.zip file.
7. If your zip/unzip utility has options to "use folder names", make sure to use them.
8. An "analogx.yz" folder will be created once you have unzipped. Double click on this folder to open it.
9. Select all files/folders EXCEPT the following:
   • the images folder
   • anlgform.html
   • anlgform.pl
10. Move or Copy the selected items to the C:\Website-Reporting\Analog folder you created earlier.
11. Open the C:\Website-Reporting\Report folder you also created earlier.
12. Move or Copy the images folder from the analogx.yz folder to the C:\Website-Reporting\Report folder.
13. You will be left at this point with two files in the analogx.yz folder. These two files, anlgform.html and anlgform.pl will not be needed for the reporting you will be doing. You may delete the analogx.yz folder and it's remaining two files at this point.

Bzip2 for Windows is a command line utility you will need to assist Analog in reading log files with .bz2 filename extensions. Bzip2 for Windows is an open-source, free utility available from RedHat (Linux).

1. Open your web browser and enter the url for the Sources.RedHat web site's section on Bzip2 for Windows - http://sources.redhat.com/bzip2/.
2. Scroll down to the section titled Getting an Older Version.
3. Find the link for PC, Windows 95/98/ME/NT/2000.
4. Right click on this link and select the option to 'Save Target As...' (IE) or 'Save Link As...' (Netscape).
5. You are downloading the bzip2-100-x86-win32.exe file. This file is 68kb in size.
6. Save this file in a temporary folder of your choice.
7. The file you have downloaded is a stand-alone executable. To make it easier to use (and in keeping with its normal usage), rename this file to bzip2.exe.
8. Move or copy the bzip2.exe file to the C:\Website-Reporting\Analog folder.

1. Open your web browser and enter the url for the ReportMagic web site - http://www.reportmagic.org/.
2. Near the top of the page, look for a link to Get Report Magic here. Click on this link.
3. Scroll down the page until you find the link for the most current "Version x.yz Windows Edition". The x.yz will be the version number.
4. Right click on this link and select the option to "Save Target As".
5. Download the rmagicx.yz.exe file.
6. Save this file to a temporary folder of your choice.
7. Open the folder in which you saved the rmagicx.yz.exe file.
8. Double click on the rmagicx.yz.exe file.
9. This file is a self-extracting zipped installer. After accepting the license agreement, a window will appear which asks you to select an installation location.
10. Navigate to and select the C:\Website-Reporting folder you created earlier.
11. Click Next.
12. The extractor will place all ReportMagic files in the folder you selected, and shortly thereafter will display a 'Finished' box indicating a successfull install.
13. Exit the installer.
14. You may now close your web browser and the folders you saved the downloaded files in.

Before proceeding with configuration, download an access_log from your web site to use for testing purposes.

1. Open your FTP client and log in to your kinetic hosted web site.
2. Find the logs folder for your web site. (It is at the same directory "level" as your 'www' folder).
3. From the logs directory, download one of the files named accesss_log.xx. Make sure that you download this file in ASCII mode.
   
   Note: The .xx is a number representing the week of the year. Any one of these will do for testing. (Access logs older than 5 weeks will have been bzipped. You will see them with filenames like access_log.xx.bz2 )
4. Save the access_log file in the C:\Website-Reporting\Logs folder on your computer.

1. Open the C:\Website-Reporting\Analog folder.
2. Copy the 'analog.cfg' file. Rename the copy to something like 'original-analog.cfg'.
3. Open the 'analog.cfg' file with NotePad.
4. Find the following line:

   LOGFILE logfile.log

   Change this line to match the following:

   LOGFILE C:\Website-Reporting\Logs\access_log*.*

5. Find the following line:

   OUTFILE Report.html

   Change this line to match the following:

   OUTFILE C:\Website-Reporting\Report\report.dat

6. Find the following line:

   HOSTNAME "[my organisation]"

   Change this line in a fashion similar to the following - use the name of your web site however:

   HOSTNAME "Name-of-Your Web Site"

7. Add the following line to the file (anywhere on a line by itself - placement doesn't matter)

   IMAGEDIR images/

8. Add the following line to the file (anywhere on a line by itself - a good place would be right after the HOSTNAME line). Change the 'your-site.mo.us' to the URL of your web site.

   HOSTURL http://your-site.mo.us/

9. Add the following line to the file (anywhere on a line by itself)

   OUTPUT COMPUTER

10. OPTIONAL : Add the following line to the file (anywhere on a line by itself)

    DEBUG ON

    This line will make Analog be more verbose in its status reporting in the errors.txt file. Use this if there are difficulties getting Analog to run correctly, or if you believe Analog is having trouble processing your log files.

1. Open the C:\Website-Reporting\ReportMagic folder.
2. Copy the 'rmagic.sample' file and rename it to 'rmagic.ini'.
3. Open the 'rmagic.ini' file in NotePad.
4. Find the following line in the '[statistics]' section:

   File_In = sample.dat

   Change this line to match the following:

   File_In = C:\Website-Reporting\Report\report.dat

5. Find the following line in the '[reports]' section:

   File_Out = reports\

   Change this line to match the following:

   File_Out = C:\Website-Reporting\Report\

6. Find the following 3 lines in the '[website]' section:

   Title = Web Statistics for my website
   Webmaster = webmaster@website.com
   Base_URL = http://www.website.com

   Change these lines in a fashion similar to the following - use appropriate information for your web site however:

   Title = Name-of-Your Web Site
   Webmaster = your@email.address.mo.us
   Base_URL = http://your.website.name.mo.us

1. For this first run you may use the log file downloaded previously. When you are ready to create more reports, use FTP to download the log files you need. Place these log files in the C:\Website-Reporting\Logs folder.
2. If you downloaded any web site log files which have .bz2 filename extensions, open an MS-DOS Command Prompt window.
   • The location varies, but on most Windows systems you can get to an MS-DOS Command Prompt by clicking on 'Start', 'Programs', 'Accessories', 'Command Prompt'.
   • At the command prompt (typically "C:\>"), type the following and press the Enter key:

     C:\Website-Reporting\Analog\bzip2.exe   -d   C:\Website-Reporting\Logs\access_log.*.bz2

   • Bzip2 will uncompress all log files with a .bz2 extension.
   • When it is finished you will see the command prompt again.
   • Close the MS-DOS Command Prompt window.
      
     
3. Open the C:\Website-Reporting\Analog folder.
4. Double click on the 'analog.exe' file.
5. You will briefly see an MS-DOS Command Prompt window open. The amount of time this window will stay open depends on the size of the log file(s) you are processing. You may probably see various messages scroll in this window.
6. Wait for the MS-DOS Command Prompt window in which Analog is running to close.
7. Open the C:\Website-Reporting\ReportMagic window.
8. Double click on the 'rmagic.exe' file.
9. You will briefly see an MS-DOS Command Prompt window open. The amount of time this window will stay open depends on the size of the log data file (report.dat) you are processing. You will see various messages scroll in this window.
10. Wait for the MS-DOS Command Prompt window in which ReportMagic is running to close.
11. Open the C:\Website-Reporting\Report folder.
12. Double click on the 'index.html' file.
13. Your web browser will open to display the report you have created.

• Documentation for Analog was included in the download you unzipped. The documentation can be found on your computer in the C:\Website-Reporting\Analog\docs folder.
• A good starting point in the Analog documentation would be the following: C:\Website-Reporting\Analog\docs\map.html.
• The multitude of options available in Analog primarily control what information is reported on, time and date ranges within which to report, plus "floor" and "ceiling" limits.
• One of the first options for configuring Analog which you may want to read about are the "FROM" and "TO" configuration options. These can be used to control the starting FROM date/time and the ending TO date/time of the report.
  
  
• Documentation for ReportMagic was included in the download you installed. The documentation can be found on your computer in the C:\Website-Reporting\ReportMagic\docs folder.
• A good starting point in the ReportMagic documentation would be the following: C:\Website-Reporting\ReportMagic\docs\map.html .
• The options available in ReportMagic primarily control the look and style of the report generated.
  
  
• The last 5 weeks worth of logs are in text format on the web server. These are the logs which do not have a .bz2 filename extension.
• Analog for Windows can process both unzipped (the ones with only numbers at the end of their filename) and .gz compressed log files. (.gz compressed logs are not used on the kinetic Service, but if you use Analog to analyze logs from other systems, you may run across one.)
• Logs from the server with the .bz2 filename extension are the ones you will need bzip2.exe to uncompress before Analog can process them. These are log files older than 5 weeks, which have been bzip compressed for easier storage.
• The log files are also numbered. The number toward the end of the file name corresponds to the week of the year for which that log contains information. The logged information for the 25th week of the year would be contained in 'access_log.25.bz2', for example.
  
  
• Since The weekly log files are re-used, the bzip2 compressed log file for a given week of the year replaces last years data for that week. If you want to retain log files for more than the last year, you will need to download the bzip2 compressed access logs you want to keep at some time before they rotate. A good time to do this would be before the end of the first week in January, at which time you could download all of the bzip compressed access logs for the previous year for your web site. Place them in an appropriately labelled directory, or perhaps burn them to CD.