Configuration Options - Database Configuration
Veduta stores copies of the log files that it reads either in memory or in a SQL database (built into Veduta). Which one you choose will impact performance and maintenance issues.
Configuration options exist for database maintenance, dependant on the type of database selected. These are detailed below.
How To Choose Between the In-Memory and SQL Database
The simplest means to run Veduta is by using the in-memory database. Most of the examples come with this option preconfigured. When running with the in-memory database, the following applies:
- Veduta will store log contents within the running process. Consequently Veduta will run out of memory if not configured appropriately.
- The in-memory database can optionally write out the contents to a file on shutdown, and re-read them on startup.
If running with the SQL database option enabled, then the following applies:
- Veduta will write the logs to a database stored on your filesystem. Consequently Veduta will not run out of memory at any stage, and sizable logs can be read and processed.
- Veduta may require some additional maintenance when running with the SQL database. Because the SQL database stores log information, when you change consumers.xml, the database should be cleared down prior to starting Veduta (this is trivial to do, but you need to remember to do it).
We recommend running the in-memory database if you have the memory available on your machine.
For Veduta 1.0 it was recommended to run the SQL database. However the in-memory database is much more performant and this is especially noticeable when running Veduta in a webserver mode. If you intend to run Veduta in its original file-writing mode then we still recommend using the SQL database.
In-Memory Database Configuration
To configure the in-memory database, set the
<db/> section in
veduta.xml:
<system>
<db type="memory">
<file>db\db.ser</file>
</db>
</system>
On start-up, Veduta will report that it is using the in-memory database. The file
db\db.ser specifies the dump file to be used for writing results on shutdown.
SQL Database Configuration
Veduta provides its own SQL database, which runs as part of the Veduta process.
To configure the SQL database, set the <db/> section in veduta.xml:
<system>
<db type="sql">
<file>filename prefix</file>
</db>
</system>
and specify a filename prefix for the logs to be stored. Veduta will use this prefix to determine which filenames to write to. Veduta will write data into the following files:
${filename prefix}.data
${filename prefix}.log
${filename prefix}.properties
On start-up, Veduta will report that it is using the SQL database.
If the consumers.xml configuration change (affecting what log information is read into Veduta), it is advisable to clear this database down and let Veduta rebuild it. To do this, simply run Veduta with the -clearDb option:
> java -jar veduta-complete.jar .... -clearDb
This forces Veduta to destroy the contents of its database and exit. On the next run Veduta will re-read all the log files according to the
new configuration information, and rebuild the database appropriately.
Common Database Configurations
Veduta can be configured to erase database contents older than a given time. This means that you can control the database size, regardless of whether the database is configured to be SQL or in-memory. To configure the time limit, use the
oldest attribute:
<system>
<db type="sql" oldest="14">
<file>database</file>
</db>
</system>
The limit is configurable in days. The above will configure the database to erase entries older than 14 days. Veduta will monitor the database and perform a clean up operation every hour.
Veduta can be configured to maintain a maximum number of records per consumer in the database. This means that you can control the database size to a fixed maximum size per consumer. To configure the record limit, use the records attribute:
<system>
<db type="sql" records="10000">
<file>database</file>
</db>
</system>
The above will configure the database to erase the oldest records such that the total number per consumer is kept at 10000. Veduta will monitor the database and perform a clean up operation every hour.
Veduta will attempt to record every incoming event in the correct time order. However this can impose a load on the system for large databases. If you know that, for each consumer, events are going to be delivered in real time and each consumer corresponds to one log source / client, then you can enable latest-only recording. In this mode, the database will only record events newer than the latest stored in the database, and no attempt is made to insert delayed messages. Veduta will log the messages that have been rejected, so you can determine if this mode is suitable in your system.
The following configuration enables this behaviour:
<system>
<db type="sql" latest="true">
...
</db>
</system>