Understanding Lotus Notes Parent/Response document hierarchy

The way that Lotus Notes manages document relationships is quite flexible and powerful but it’s also an area where you might encounter some problems—we certainly get a lot of questions from customers related to the Notes document hierarchy.

With this in mind we decided to write a short backgrounder on how Notes deals with the Parent and Response document hierarchy, plus what to do when things go awry. And for the benefit those who use Ytria tools, we’ll give a rundown of some handy features related to Parent and Response documents.

First, some basics facts on Parent and Response documents in Notes:

Documents created in databases can have a relationship to facilitate easier categorization and/or grouping; this is usually called the ‘Parent/Response relationship.’

Documents created in databases can have a relationship to facilitate easier categorization and/or grouping; this is usually called the ‘Parent/Response relationship.’

You can describe documents as Parents or main documents; Response documents (children); or Response to Response documents (grand children). There is no practical limit to the number of descendant generations a document can have. However, views are limited to displaying a maximum of 31 levels of Responses to one main document (from R5 to R8.5.x).

The only thing that is responsible for the Parent/Response dependency is the presence of a $REF item in response document (populated with the UNID of the Parent document); this is what links the Response to its Parent. If the $REF item is missing or in an incorrect format (it must be a special Notes.Ref.List item type—more on this later) it can’t be a native Response document. (You technically could create a solution in a custom design, using text type fields to identify Parents, but this sort of thing is very complicated to set up and has the disadvantage of not supporting the native Parent/Response navigation scheme).

There are three ways to program Parents and Responses at the form level: Forms can be set as Document (i.e. first-level, main or parent), Response (second level) or Response to Response (third level). You should be aware of the following potential problems with form-level hierarchy:

Conflict documents are Response documents. However, once you edit and save them, their new form-level designation as a “main document” can and will override their document-level Response status. This can lead to duplicate documents in a database. There are several ways to deal with this:

Don’t let users save changes on Conflict documents (this would prevent them from breaking the Parent/Response link).

You can follow the example of the standard Document Library template and use the script in the Quyerymodechange event which prevents editing any Response document (Conflicts are Responses) with the 1st level form. Here is the code:

Dim doc As NotesDocument
Set doc=source.Document
If doc.IsResponse Then Msgbox GetString(52), 0, Getstring(40)
continue=False
End If End Sub

You could also add something to the Querymodechange or the QuerySave events to indicate that the user is accessing a Conflict document (i.e. if a $Conflict item is found).

Also note: If a document doesn’t have any Parent to reference and you’re attempting to open it in a Response or Response to Response form, you’ll going to encounter problems.

Parental reference to Responses: While Response documents have the aforementioned $REF item, there is also an important reference included from the Parent side as well; it’s called the Response ID Table. This table is not required for displaying the Parent/Response hierarchy in views (i.e. building the view index). We’ll discuss this table in greater detail later in this post.

On the View level there is a property called “Show Response Documents in Hierarchy.” This is enabled by default. As its name implies, it is responsible for displaying the Parent/Response hierarchy in view. You should be aware that orphan documents can only be shown in a flat view (i.e. when this property is unchecked).

The View Selection Formula, when set to display Response documents in hierarchy, can be set to display documents that use a given form. Additionally, by using a formula like in the example that follows, a view will show a document that matches a given form and will also show all its Responses, regardless of their forms: Form = MyDoc | @AllChildren

Using Columns for displaying Responses: If you set a Column to display ‘Responses only,’ it will display everything—right down to the deepest levels of document hierarchy. Note: You only use one Column to display these Response documents. This means that the Column Formula usually has to combine multiple items to display the most meaningful data (and in the event that different Forms are used for different levels, you’ll also need to differentiate those using something like: @If(Form=””;) etc.)

Here are some of the formulas to keep in mind when working with and displaying Parent/Response relationships:

♦ @AllChildren: View Selection/Selective Replication formula for including all Response documents at all levels for any Parent documents matching selection criteria

♦ @AllDescendants: View Selection/Selective Replication formula for including all Response and Response to Response documents for Parent documents matching selection criteria. (In other words, this will show descendant levels down to and including grandchildren/Responses to Responses but no further).

♦ @DocDescendants: View Title/Column formula for returning the number of all Descendants (regardless of level).

♦ @DocChildren: View Title/Column formula for returning the number of all direct Responses (and direct Responses only) for the current doc or category.

Displaying the Parent/Response hierarchy in Views is accomplished with the help of the View Index. Building and displaying the index and hierarchy does not require the Response ID Table support, it only uses the $REF items on documents.

Tip: the View Index can be seen in XML format on the Web using the ?ReadViewEntries tag; see screenshot below:

Showing the view index as an XML file; click to enlarge

There is another important but lesser-known contributor to management of Parent/Response hierarchies in Notes: It’s called the Response ID Table.

When the ‘Don’t support specialized response hierarchy’ Database property is un-checked (the default setting), Notes will keep a record of Response documents and store this information in something called the ‘Response ID Table.’ This table lists the Response document UNIDs for all Parents.

So, you might be wondering, “what’s the point of this Response ID Table if Notes is able to manage views using only $REF items (as discussed earlier in this post) in response docs? If this table takes up disk space (and it does), why do we need it?
Here’s where the Response ID Table is essential:

• In order to get to Responses from their Parents programatically, you must work with this table.
• This table is the source for the Responses property in the NotesDocument Class in LotusScript (and the closely related Java Class)
• For very advanced developers: The Notes C API NSFNoteOpen() Call (with the OPEN_RESPONSE_ID_TABLE flag enabled) will return a blank table if the Response ID Table is not present. (Note: scanEZ makes use of this C API Call when it lists Response document in the Parent/Response tab).

The ‘Don’t support specialized response hierarchy’ option in the Lotus Notes Application Properties dialog (Advanced tab)

In some older versions of Notes we would occasionally encounter issues where the table itself seemed to get corrupted; we haven’t personally observed such problems since R5, so we were inclined to think that the issue was fixed. (If any readers out there have seen invalid Response ID Tables, please let us know in the comments).

In addition, the IBM help documentation for the Responses property warns “In some cases, the collection returned by this property will include deletion stubs, ghosts of deleted documents that used to be responses to the document. Use the IsValid and IsDeleted properties to test all documents in the collection, before attempting any other operations with it.”

A deleted Response document as shown in scanEZ’s Parent/Response tab

The following is an example of how we protect our loops using the flags mentioned above. This might help give you some ideas on how to implement something like this in your own environment:

Dim ndR As NotesDocument
Dim ndcR As NotesDocumentCollection
Set ndcR = nd.Responses
If ndcR.Count > 0 Then Set ndR = ndcR.GetFirstDocument
Do While Not ndR Is Nothing If ndR.isvalid And Not (ndR.isdeleted) Then If ndR.ParentDocumentUnid = nd.UniversalID Then ( … )
End If End If Set ndR = ndcR.GetNextDocument( ndR )
Loop End If

The $REF item, as mentioned earlier, is not a text item; rather, it’s a special ‘Notes Ref. List‘-type item. This means that special rules govern how you can access its values. And here are those rules:

Reading $REF item values using formulas:

• Values can’t be read just by indicating the $REF Item name; we first need to convert it to Text using the @Text() tag. The same rules apply when using the @GetDocField() command.
• A special scenario: By using the @SetDocField, we can use the original item value (in the same way we can do this with the @DocumentUniqueID command) if it is used for the first parameter, identifying the target document (in this case, the Parent) by its UNID. For example: @SetDocField($REF;”DstDocItemName”;”NewItemValue”)

Reading $REF item values using LotusScript:

• Reading the $REF item using the ndoc.~$Ref or ndoc.GetFirstitem(“$Ref”) method will work and it will return an array of strings without having to cstr (convert to string).

Setting $REF item values:

• You cannot set values for these items through @formulas. While it’s true you could change the value using the @SetDocField command, this would have the severe side-effect of changing the Notes Ref. List item type into Text—and after this irreversible change, the document will no longer function as a Response.
• It’s also impossible to set values by manipulating the $REF item directly through LotusScript (the same unwanted item type switch will occur). Instead we would need to use the ndoc.MakeResponse() method, which then sets the right value for the $REF item. The downside here is that we need to go through a long and tedious list of declarations to do this properly. Note: If you’re using Ytria scanEZ, the ‘Set New Response to Parent’ option is a much quicker, much more convenient way of doing this (see the more details later in this post).

Dealing with orphan documents:

When a Response document is missing its Parent document, it’s sometimes referred to as an ‘orphan document.’ Orphan status for documents can create issues in Domino, so here are a few potential situations to help you understand missing Parents in databases:

♦ Parent Document is restricted due to lack of access rights: It’s possible for the current user and session to be forbidden from accessing a Parent due to Reader field restrictions. In such cases, a Response document would act like an orphan for that session.

♦ Parent Document deleted, UNID still in the database as Deletion Stub: This is actually OK for Domino, as there’s still a ‘”piece” of the Parent for the Response to hold on to, so problems shouldn’t arise.

♦ Parent Document deleted, Deletion Stub no longer in the database; This could go a couple different ways:

• Case 1: In the database where the deletion occurred, looking at the Parent using scanEZ gives us an “Invalid or Nonexistent Document” error message.
• Case 2: Copying or replicating this type of orphan Response document seems to prompt Domino to create a dummy ‘Ghost Note‘ that holds the appropriate parent UNID in the destination database, so that response has something to refer to. (Note: This document is currently not shown in scanEZ unless you attempt to access the parent from the Response document in question. This ghost does not contain any items, it’s basically just a UNID).

When a document is highlighted in scanEZ’s Selection Tree, the Parent/Response Tab will appear if applicable. In the screenshot below, we can see that the example document has a parent as well as two responses. The responses are listed in a grid (with the right-click options available for ‘Delete‘; ‘Remove response link with parent‘; or ‘Add to a new MySelection‘). There is also a ‘Go to Parent‘ button that lets you open the Parent document (this is only active if the document selected in the tree is a Response or a Response to Response). The Parent/Response tab will offer informative error messages if scanEZ encounters any problems (like the issues discussed earlier in this post).

Note: The Response ID Table is required for the Parent/Response tab to show all Responses and their statuses.

The Document Analyzer tool is designed to help you discover problematic documents that can’t be found easily using formulas. The following Document Analyzer scans can be helpful for troubleshooting Parent/Response documents:

Document has responses, but response is missing: Also known as ‘Parents with missing Children.’ You’ll encounter this issue when Response ID Table links point to a deleted document. And as with Parent documents, Reader field access issues can render Response documents inaccessible for a user/session, as shown here:

Document is a response, but parent is missing: Also known as ‘Orphan Documents.’ As mentioned earlier, a document may appear to be an orphan in cases where Reader type fields deny the current user/session access to the Parent. Documents with deleted Parents (whether or not a Deletion Stub exists) will also be flagged as orphans.

Document is a response, but is its own parent: This seems to be a rare occurrence but we have seen instances of such documents appearing. This is most likely caused by errant LotusScript code.

Using scanEZ’s Document Analyzer tool to track down documents with problematic hierarchical relationships

This option allows you to set a new parent to a response document or category of response documents highlighted in scanEZ’s Selection Tree. This is a unique, not to mention much faster and more efficient, method of setting a Response document’s parent. This command lets you set a new parent to one or many documents nearly instantly; you just need to select the document (or category/MySelection virtual folder of documents), use this command and enter the NoteID or UNID for the new ‘adoptive’ Parent (see the screenshots below).

Note 1: To break the Parent-Response relationship between two documents, you can either use the ‘remove response link with parent’ right-click menu item in the Parent/Response tab (see screenshot higher up in post) or simply delete the $REF item in scanEZ’s Item list window.

Note 2: ‘Set New Parent to Response(s)’ is also available in the Checkbox Selection menu. Checkbox selections are used for dealing with non-contiguous selections in scanEZ.

Conflicts are technically response documents. And scanEZ’s dedicated Conflict Solver tool provides a handy way to analyze and resolve these problem documents. A previous Ytria Tech Lab post gives a quick overview of this tool; read it here.

Note: Scans for documents with $Conflict items but no valid parent as well as documents that are their own conflicts are available in the Document Analyzer tool mentioned earlier.

‘MySelection’ virtual folders allow scanEZ users to create ad hoc groupings of documents. The following Parent/Response related options are available in the scanEZ right-click and MySelection drop-down menus:

Add Parent to MySelection
Add any Parents to MySelection
Add x Response(s) to MySelection
Add any Responses to MySelection

scanEZ will detect if the highlighted document has a parent and/or any responses and allow you to add them to a virtual folder

Tip: For a big picture view, Ytria databaseEZ lets you keep track of which databases have the ‘Don’t support specialized response hierarchy’ property set or un-set across a Domino server.