The Ground Report

GROUND REPORT DATABASE

Assumptions

1.	This database assumes that for HTTP data that the scripts have been recorded using the HTTP Proxy. Therefore every test that ends with the numbers "00" is assumed to be an instrumented test (a "page"). To make use of the this db it is best to ensure that all element tests do not end in "00". Future enhancements to the db plan to make use of the assumption that there are a maximum of 99 elements per page.
2.	All test numbers are unique tests. Do not load data from multiple scripts with overlapping test numbers where each script uses the same test number for a different test.
3.	Data content only consists of pages and elements (instrumented and non instrumented tests) No provision is made for nested intrumented tests at present.

DB Tuning Tips

1. The following settings (away from defaults) are suggestions for postgresql.conf:

Variable	Value
shared_buffers	(set to 1/4 of RAM for dedicated machine) (Leave at 32MB on Windows)
work_mem	50MB
maintenance_work_mem	256MB
wal_buffers	1MB (for SMP systems)
checkpoint_segments	20
effective_cache_size	(set to 2/3 of RAM for dedicated machine)
fsync	off
synchronous_commit	off
autovacuum	off (turn on the autovacuum daemon to maintain the db tables)

2. Mount the following directory on a different disk/filesystem:

pg_xlog

Installation

1. create a db called "Ground_Stats" in your postgres implementation owned by postgres user.
   a) Create the db with UTF8 encoding.
   b) Ensure that the plpgsql language installed in the created database.
        If you come across errors about plpgsql not existing then you can install the language using the following command:

CREATE LANGUAGE plpgsql; (this SQL command should be run as the db superuser)
createlang plpgsql "Ground_Stats" (this commandline command should run as the db superuser)

2. edit the database.properties file in the etc directory and alter the following properties with the appropriate information:

   jdbc.server
   jdbc.port
   jdbc.database
   jdbc.user
   jdbc.pass
   jdbc.driver

3. run the databaseInterface script in the bin directory:

a) Select option 1 "install Database"

Uploading Files

1. edit the data.properties file in the etc directory and alter the following properties:

Option	Value	Comment
fileType	data \|\| descr	select the type of file you are uploading
dataType	http \|\| norm	select the type of data contained within the data files and therefore database
fileList	["/path/to/data/data.log"]	enter the list of files that you'd like to upload. Jython list.

NB : ensure that filenames are enclosed in quotes
         ensure that fileList list is encapsulated with a pair of [] braces
         separate filenames with a comma "," and no whitespaces
         ensure that you always use a "/" for a directory separator even on Windows

2. run the databaseInterface script in the bin directory:

a) Select option 4 "upload Files"

Descriptions Files

***LOADING DESCRIPTIONS FILES IS AN OPTIONAL STEP***

HTTP Descriptions file must be a CSV file and have a headers row with the headers:

"Test","Description","SorDorP"

Header	Comment
Test	Test identifiers that should match up with the Tests stored in the raw_data_http table.
Description	Field containing v.short description of Test (e.g GET grinder.gif).
SorDorP	Static or Dynamic or Page (this describes the type of content in the test) "s" or "d" or "p" are the allowable entries for this field.

A sample file called descr_http.txt has been provided in the samples directory:

$GROUNDREPORT_HOME/docs/www/samples/descr_http.txt

NORM Descriptions file must be a CSV file and have a headers row with the headers:

"Test","Description"

Header	Comment
Test	Test identifiers that should match up with the Tests stored in the raw_data_norm table.
Description	Field containing v.short description of Test (e.g GET dbConnection).

A sample file called descr_norm.txt has been provided in the samples directory:

$GROUNDREPORT_HOME/docs/www/samples/descr_norm.txt

In order to populate the materialised views for static and dynamic HTTP content within the DB it is necessary to either:

   a) load in the descriptions file BEFORE loading in the data files.
   or
   b) run the databaseInterface script in the bin directory AFTER loading in the data files:
        i) select option 5 "refresh Materialised Views"

Data Files

HTTP data files are raw data files which have been produced using the http plugin.

NORM data files are raw data files which have been produced without using the http plugin.

Each upload of data is considered a "load run" so remember to upload all files for a load run at the same time.

Drop/Delete Data

1. run the databaseInterface script in the bin directory:
a) Select option 6 "drop Data"

This option will delete all the data in the database.

Reset Database Commands

1. run the databaseInterface script in the bin directory:
a) Select option 3 "reset Database"

This option will delete the database and then create it again.

Uninstall/Delete Database

1. run the databaseInterface script in the bin directory:
a) Select option 2 "delete Database"

This option will delete the database.

Experimental

1. run the databaseInterface script in the bin directory:
    a) Select option 7 "cluster Database"

    The above option may help in the performance of option 5 "refresh Materialised Views"

    It is important to analyse the database tables after clustering. Having autovacuum turned on means you don't need to remember.

2. run the databaseInterface script in the bin directory:
a) Select option 8 "refresh Extended Stats Materialised Views"

The above option refreshes the extended stats materialised views.

Information

Schema information can be found in the docs folder:

$GROUND_HOME/docs/db_schema

The text for menu options is stored in the following file in the following location:

$GROUNDREPORT_HOME/includes/en/standardMenu.txt

GROUND REPORT 1.5 DOCUMENTATION