FAQ - Transactional Logging

Q: How does replication work with transaction logging enabled? Does the replicator read from the transactional log for any changes that need to be replicated from the Unified Buffer Manager (UBM) or are changes committed to the NSF first before replication occurs?

A: Replication is unaffected by Transactional Logging. Changes are available in the same manner as they are for un-logged databases. Changes are always read from the in-memory versions of databases when they are open and all cached changes are flushed when databases are closed.


Q: If a user is trying to read a note that has not been committed to the NSF with transactional logging enabled does the user session read the note from the transactional log, the UBM or is the note committed to the database first?

A: The note is read from the in-memory versions of databases when they are open and all cached changes are flushed when databases are closed.


Q: If something is done to change the DBIID of a database, are changes committed first before the DBIID gets changed?

A: Yes. Data is not lost. Things that change the DBIID, such as Fixup -J and Compact -B, change the DBIID because they are making database changes that are not logged. The order of operations is this:
(1) Flush all in-memory changes to the database
(2) Clear the DBIID and halt logging for the database
(3) Make the un-logged changes
(4) Flush all in-memory changes to the database
(5) Assign a new DBIID and start logging changes to the database again.

Note: None of these steps can be done manually; all of them are done automatically. The one manual step is to take a new backup of the database after you see the console message "Assigning New DBIID for DB xxxx."


Q: When is it beneficial to run fixup -j?

A: Hopefully never. It would be beneficial only if the database became corrupt and you did not have a backup to roll forward from.


Q: When fixup -j is run on the entire server I notice that it assigns a new DBIID to some databases but not others. Why is this?

A: Transactional logging should reassign the DBIID's for all transactionally logged databases. Fixup -j will run against unlogged as well as logged databases, so the logging status should be checked of the databases that didn't have a DBIID re-assigned to them and validate that it is not logged.


Q: If the DBIID of a database has changed for some reason, how is the database with the old DBIID and the new DBIID incorporated?

A: There is no intermixing. You must re-backup a database when the DBIID changes. Note: there is no relation to the REPLICAID or DBID.


Q: If a server panics and upon restart Domino automatically performs database recovery, will a large amount of data be lost or is the loss minimal?

A: All cached info will be re-applied and only partial API calls will be backed out. That is, if the server was in the middle of a NoteUpdate call, that would get undone. You will see EVERY NOTE you ever added (and the API call returned) with no loss of data. The Transactional Log is written directly, with no file cache (i.e. all writes are "committed" and writes are done to the Transactional Log at least after each transaction or API call).


Q: If only one database on a server becomes corrupt is it possible to restore only this one database from the transactional log or will the entire transactional log need to be replayed?

A: Transactional Logging restores only the database(s) that need to be. The whole log from the time of the backup must be replayed but only records pertaining to the databases being recovered will be processed.


Q: What is the control file used for?

A: It is used by the logger subsystem to know the state of the log files. The logging subsystem is all of the code in nsf\logger, where some of it runs in the "application" process, meaning in the process that makes logging calls (ex. the server, etc).


Q: If there are any databases or directory links on the server, will these databases be transactional logged as well?

A: Yes the actual database will be logged independent of links.


Q: If Compact changes the DBIID then what happens to a backup copy of the database? Will the log be able to recover in the event a database is replaced on-line?

A: It assumes you did not copy back an older version of the database while the server was crashed. Never "touch" databases after a server crash until you get through a clean restart. You can start a clean restart by simply entering the command "nfixup zzzz" before the Domino server is restarted (where zzzz is a database that does not exist). The initialization of fixup will trigger the restart and recover the logged databases to a clean state. This will clear the UBMand will harden all the information to the transactionally logged databases (the actual NSF), then the server can be restarted. Consistency check will run against all non transaction logged databases.

Q: When the server is live, how is it possible to monitor the health of the transactional log?

A: Any extra "monitoring" of the log at run-time is not recommended as it will disturb your system performance. The log is not read very much while the server runs so it is true that issues with the logs would not soon be noticed; however, the log is written to constantly and all hardware would give a bad return code and generate error events if write errors occurred.


Q: The customer uses FTP to move NSF files between "live" servers. Can the Transactional Log recover from finding a new database live while the server is running?

A: IMPORTANT: Never OS copy or FTP a live logged database. The reason is that a new replica creates a replica stub on the new server, then populates the database later, preserving the state of the transaction logging. An OS copied database will be subject to a consistency check when it is placed on the new server.

It is acceptable however to pull or create a new database replica of a live logged database. A new replica creates a replica stub on the new server, then populates the database later, preserving the state of the TL. An OS copied db will be subject to consistency check when it is placed on the new server.

There are no known problems adding new databases (new databases do not have a DBIID yet) to a server "live", but you must get a "clean" version and you should make it appear "instantaneously" so it does not show up as only half there, as it would when you FTP it.

The recommendation is as follows:

1. Use dbbackup from API Sample or a backup vendor's backup (dbbackup test tool takes a backup and "applies all pending changes" so it's a healthy backup). Make the extension of the database something non N* line .bak.
2. FTP or copy the bak file to the other server.
3. Rename the bak to nsf so it "appears" instantaneously.
4. The next open will assign a new DBIID and run fixup if it was logged before, to clear the log sequence numbers in the file. It may take a while if it was a large database, but then it will behave normally.

An existing database that is not open that needs to be moved from one server to another should be done using Notes\New Copy or New Replica. Prior to the move, the extension should be changed to something other than .nsf. The database can then be moved via new copy or new replica. When the move is complete, the extension should be changed back to .nsf. This method ensures that transactional logging begins logging the database only when it is complete and consistent. It is important to note that this will change the DBIID of the database so a backup must then be taken.


Q: A database on a Domino server is continuously becoming corrupted and is then being marked as read only. If transactional logging is enabled on the server will it mark the database as read only?

A: The only thing that transactional logging will do is mark the database corrupt. It will never mark the database as read only.


Q: How does transactional logging operate with SCOS (shared mail)?

A: The capability to have transactional logging for SCOS in R5.x does not exist. This capability is present in Notes/Domino 6.

Product documentation Abstract Domino supports transaction logging and recovery. With this feature enabled, the system captures database changes and writes them to the transaction log. Then if a system or media failure occurs, you can use the transaction log and a third-party backup utility to recover your databases

Content

Lotus Domino supports transaction logging and recovery. With this feature enabled, the system captures database changes and writes them to the transaction log. Then if a system or media failure occurs, you can use the transaction log and a third-party backup utility to recover your databases IMPORTANT: Enabling transaction logging can improve server performance in most cases. Transaction logging saves processing time because it allows Domino to defer database updates to disk during periods of high server activity. Transactions are recorded sequentially in the log files, which is much quicker than database updates to random, nonsequential parts of a disk. Because the transactions are already recorded, Domino can safely defer database updates until a period of low server activity. What is transaction logging? Transaction logging keeps a sequential record of every operation that occurs to data. If a database becomes corrupted, you can "roll back" the database to a point before it was corrupted and replay the changes from the transaction log. A single transaction is a series of changes made to a database on a server -- for example, a transaction might include opening a new document, adding text, and saving the document. Transaction logging provides three main benefits: - In most situations, you no longer need to run the Fixup task to recover databases following a system failure. Excluding Fixup results in quicker server restarts, since Fixup must check every document in each database, while transaction log recovery applies or undoes only those transactions not written to disk at the time of the system failure. - Transaction logging saves processing time because it allows Domino to defer database updates to disk during periods of high server activity. Transactions are recorded sequentially in the log files, which is much quicker than database updates to random, nonsequential parts of a disk. Because the transactions are already recorded, Domino can safely defer database updates until a period of low server activity. - Using transaction logging simplifies your daily backup procedure. You can use a third-party backup utility to perform daily incremental backups of the transaction logs, rather than perform full database backups. IMPORTANT: Transaction logging works with databases in format ODS 41 or higher but not with databases that use formats from earlier releases (ODS 20 will not work). After you enable transaction logging, all databases are automatically logged. To check database formats, use the Files tab in Domino Administrator. NOTE: To use all of the features of transaction logging and recovery, you need a third-party backup utility that supports Domino transaction logging. What is considered a transaction? A transaction is a single API call. It includes creating, modifying, reading (unread marks change) or deleting documents. A transaction is considered COMPLETE when the change has been saved to disk by the user. For example, if a user makes a change to the database, and does not save that change before the server crashes, that transaction is not considered COMPLETE. The transaction would have been COMPLETE only if the user had saved the change before the server had crashed. COMPLETE transactions are "commited" to the transactional log. What is a Transaction log? A transactional log is a binary file where transactions are written. The transactional log has a .txn file extension. These .txn files should never be deleted. The maximum size of each log extent (.txn file) is 64 MB. You can have several .txt logs based on the size specified in the Server document. The maximum total of .txn files is 4 GB. What is the Database Instance ID (DBIID) When you enable transaction logging, Domino assigns a Database Instance Identifier (DBIID) to each Domino database. When Domino records a transaction in the log, it includes the DBIID. During recovery, Domino uses the DBIID to match transactions to databases (it identifies which database the changes should be applied to). The DBIID is stored in the file header, along with the database ID and the Replica ID. Note: There is no relation to the Replica ID or the DBID. Some database maintenance activities, such as compaction with options, cause Domino to assign a new DBIID to a database. From that point forward, all new transactions recorded in the log use the new DBIID; however, any old transactions still have the old DBIID and no longer match the database's new DBIID. As a result, Domino cannot restore these old transactions to the database. To avoid losing data, you should immediately perform a full database backup whenever a database receives a new DBIID. When you perform this backup, you capture all the database transactions up until that point and ensure that Domino needs only the new transactions (with the new DBIID) to restore the database. If the DBIID changes and a backup is not taken after the fact, the database cannot be successfully restored (backup will have the old DBIID and the transactional log will not "know" the old DBIID. NOTE: The DBIID has no relation to the REPLICAID or DBID. Domino assigns a new DBIID to Domino databases when: You enable transaction logging for the first time. - System logging is disabled then re-enabled. - The database is compacted using copy-style compaction. - The database has had Fixup -J applied to it. IMPORTANT NOTES: If a database is logged, the default for Compact with no switches is -b (lowercase) If a database is un-logged, the default for Compact with no switches is -B (uppercase). Compact with no switches and Compact -b (lowercase b) are the only times Compact does not change the DBIID. The DBIID changes when a database is copy-style compacted because a copy-style essentially creates an entire new NSF with a new structure, which basically does not match the structure in the logs for the "old" NSF anymore. Note: -L, -c, and -i are switches that enable copy style compaction. -B at times uses copy style compaction. Compact -B may change the DBIID. This option uses in-place compaction unless there is a pending structural change in which case copy-style compacting occurs. So when using this option and transaction logging, do full database backups after compacting completes. Note: For additional information on the compact tasks switches refer to: Switches for COMPACT Server Task for Domino (#1084388) Fixup is forced on the database (fixup -j) You move a Notes database from one logged server to another logged server or from an unlogged server to a logged server. NOTE: Changing the log path or maximum log size (after initial set up and use) does not trigger a DBIID change. How to set up Transaction logging Ensure that all databases to be logged reside in the Domino data directory, either at the root or in subdirectories. From the Domino Administrator, click the Configuration tab. In the "Use Directory on" field, choose the server's Domino Directory. Click Server Configuration, and then click Current Server Document. Click the Transactional Logging tab. Complete these fields, and then save the document. Field Enter Transactional Logging Choose Enabled. The default is Disabled. Log path Path name location of the transaction log. The default path name is \LOGDIR in the Domino data directory, although it is strongly recommended to store the log on a separate, mirrored device, such as a RAID (Redundant Array of Independent Disks) level 0 or 1 device with a dedicated controller. The separate device should have at least 1GB of disk space for the transaction log. If you are using the device solely for storing the transaction log, set the "Use all available space on log device" field to Yes. Maximum log space The maximum size, in MB, for the transaction log. Default is 192MB. Maximum is 4096MB (4GB). Domino formats at least 3 and up to 64 log files, depending on the maximum log space you allocate. Use all available space on log device Choose one: Yes to use all available space on the device for the transaction log. This is recommended if you use a separate device dedicated to storing the log. If you choose Yes, you don't need to enter a value in the "Maximum log space" field. No to use the default or specified value in the "Maximum log space" field. Automatic fixup of corrupt databases Choose one: Enabled (default). If a database is corrupt and Domino cannot use the transaction log to recover it, Domino runs the Fixup task, assigns a new DBIID, and notifies the administrator that a new database backup is required. Disabled. Domino does not run the Fixup task automatically and notifies the administrator to run the Fixup task with the -J parameter on corrupt logged databases. Runtime/Restart performance This field controls how often Domino records a recovery checkpoint in the transaction log, which affects server performance. To record a recovery checkpoint, Domino evaluates each active logged database to determine how many transactions would be necessary to recover each database after a system failure. When Domino completes this evaluation, it: Creates a recovery checkpoint record in the transaction log, listing each open database and the starting point transaction needed for recovery. Forces database changes to be saved to disk if they have not been saved already. Choose one: Standard (default and recommended). Checkpoints occur regularly. Favor runtime. Domino records fewer checkpoints, which requires fewer system resources and improves server run time performance. Favor restart recovery time. Domino records more checkpoints, which improves restart recovery time because fewer transactions are required for recovery. Logging style Choose one: Circular (default) to continuously re-use the log files and overwrite old transactions. You are limited to restoring only the transactions stored in the transaction log. Archive (recommended) to not re-use the log files until they are archived. A log file can be archived when it is inactive, which means that it does not contain any transactions necessary for a restart recovery. Use a third-party backup utility to copy and archive the existing log. When Domino starts using the existing file again, it increments the log file name. If all the log files become inactive and are not archived, Domino creates additional log files. How to disable Transaction Logging for a specific database In most cases, disabling Transaction Logging (on a server or database level) is not recommended because you lose all of the benefits of transaction logging (there are no ill side effects of disabling, you simply lose the benefits). One of the benefits of transaction logging is fast server restart. Disabling transaction logging will cause Fixup to run on the database (or all databases on the server), creating the potential for slow restart. After you set up transaction logging, all databases that are in Domino Release 5 or higher format are logged. You can disable transaction logging of specific databases. Attachments are transactionally logged; however, attachments are logged redo only. Therefore, if the database is recovered using media recovery you will get back the last copy of the attachment (once they are done they stay done). If, however, the server crashes with uncommited attachment updates, they will not be undone since an undo record is never created for them . Views are not logged, so after media recovery, you will need to rebuild views. First, perform any of the following: When creating a new database, choose "Disable transaction logging" on the Advanced Databases Options dialog. For an existing database, choose "Disable transaction logging" on the Database Properties box, Beanies tab. In Domino Administrator, select a database on the Files tab, choose Tools - Database - Advanced Properties, then choose "Disable transaction logging" Use the Compact task with the -t parameter. Second, ensure that all users have closed the database. Next, use the DBCACHE command with the "flush" parameter to close the database in the database cache. Finally, open the database. How to schedule backups or Transaction logs and logged databases Backups are essential for recovering from a media failure, which is a failure of the server's disk or disks. If you have a third-party backup utility, you should: - Schedule daily incremental backups of the transaction log. Use the backup utility daily to back up the transaction log. - Schedule archiving of transaction log files. If you use the archive logging style, use a third-party backup utility to schedule archiving of log files. - Schedule weekly full database backups. Each week, it is recommended to run the Compact task with the option to reduce file size. Because this compaction style changes each database's DBIID, you should schedule compaction with a full database backup. How to fix corrupted databases Corrupted databases don't occur frequently when you use Release 5 or higher databases and transaction logging. When you use transaction logging to log changes to Release 5 or higher databases, a server automatically uses the transaction log to restore and recover databases after a system failure, for example after server failures or power failures. If a disk failure occurs, you use the transaction log along with a certified backup utility to restore and recover the databases. Using Transaction logging for recovery Transaction logging is an integral part of recovering from system and media failures. A system failure causes the server to stop and requires you to restart the server. During restart, Domino automatically performs database recovery. The system uses the transaction logs to apply or undo database transactions not flushed to disk for databases that were open during the system failure. Domino also runs the Fixup task on databases that use formats from earlier releases, databases that are in Release 5 or higher format but have transaction logging disabled, and on corrupt databases if you have the "Auto fixup of corrupt databases" field in the Server document set to Yes. Fixup -J Causes Fixup to run on databases that are enabled for transaction logging. Fixup -j should only be run if a database is corrupt and you have no backup of the database to roll forward from. Without this -j option, Fixup generally doesn't run on logged databases. The Fixup task interferes with the way transaction logging keeps track of databases. If you are using a backup utility certified for Domino, it's important that you schedule a full back up of the database as soon after Fixup finishes as possible. Notes.ini parameter: Translog_Status The TRANSLOG_Status NOTES.INI parameter is used to enable transaction logging for all databases on the server. "0" is disabled, "1" is enabled.