Hyper Files

Introduction

A hyper file is an updatable view onto a number of discrete data sections, hereafter referred to as hyper sections. Accessing a hyper file accesses its hyper sections as if they constituted one file, and their indexes as if they constituted one index.

Hyper files are primarily intended as a way to split a very large file into more manageable units. Significantly, item IDs must be unique across all hyper sections.

Hyper sections can be in the same dictionary as the hyper file or (when accessed via Q-pointers) anywhere in the database or even in remote databases. (However, a hyper file can be opened only if all of its hyper sections can be opened, so using a remote database to which the connection is unreliable is not recommended.)

In order to access the correct data section each item ID is processed by an attribute 8 code (also known as a pre-processor code or ENGLISH correlative) to extract a part of the ID known as the hyper key. The hyper key is looked up in a table to find the required data section.

The lookup table may use a simple one-to-one mapping (in which each individual hyper key maps to a specific hyper section) or a range mapping (in which each table entry implicitly denotes the upper limit of a range of hyper keys, each mapping to a specific hyper section).

The hyper file definition item is a modified D-pointer that contains all of the information needed to map item IDs to data sections, plus a list of indexes available for use on the data sections. Most of this information is normally hidden, but can be displayed by using the HYPER‐LIST command.

Hyper sections normally have an update lock applied to them, so that write operations can only be performed via the hyper file. This prevents items being written into the wrong hyper section, which would make the hyper file inconsistent. However a hyper file can be configured to not require this.

Hyper files are created and managed by a combination of TCL commands that operate exclusively on hyper files, commands that work with normal files and hyper files.

More about hyper keys and the lookup table

To determine in which hyper section a particular item belongs, the item ID must be processed to extract a hyper key which is then applied to a lookup table consisting of paired key value/section name entries.

When a hyper file is created, the user must specify a predefined data definition item that has the required attribute 8 code (also known as pre-processor code or an ENGLISH correlative) to extract key values. Notice that while item IDs must be unique across all hyper sections, their hyper key values typically will not be.

The definition of the lookup table is also part of creating a hyper file. Here is an example lookup table with three key/section entries:

2011  SALES_2011
2014  SALES_2014
2016  SALES_2016

This lookup table can be interpreted in one of two ways: as a simple lookup or a range lookup table. The type of lookup table is defined when the hyper file is created:

• A simple lookup table is a one-to-one mapping of key values to hyper sections. So in the example, all items with a hyper key value of 2014 belong in the SALES_2014 section. An item whose hyper key evaluated to 2013, for example, would be unidentified.

• In contrast, in a range lookup table each key value denotes the upper bound of a range of possible key values. So in the example, all items with hyper key values of 2011 or less belong in the SALES_2011 section, those between 2012 and 2014 in SALES_2014, and so on. An item whose hyper key evaluated to more than 2016 would be unidentified.

  Note

  The range lookup table is automatically kept sorted in ascending order of key values.

The data sections specified in the lookup table may be synonym file definition items (Q-pointers) to other accounts, or even other databases.

Optionally, the lookup table can identify a BIN section into which all unidentified items are swept. If no BIN section is defined, any attempt to read or write an unidentified item will fail, so a BIN section is recommended.

Because it may be necessary during lookup to say whether one key value is less than or greater than another, the hyper file definition also specifies whether right-aligned hyper keys are to be treated as numeric or alphanumeric (that is, including any leading zeros).

The process of ensuring that all items are in their correct sections is known as reconciliation. A hyper file is reconciled automatically whenever a new section is added to the hyper file by using the HYPER-ADD command, and on demand by using the HYPER-RECONCILE command. (In contrast, the HYPER-EDIT command does not force an automatic reconciliation.)

Sequential access

When listing or selecting, the hyper file item IDs are listed from each hyper section in turn; that is, all of the items from the first hyper section, followed by all of the items from the second hyper section, and so on.

Hyper file definition item

The hyper file definition item is a modified D-pointer, also referred to as an H-pointer or HPTR, and is denoted by a new Y file type in attribute 3.

• Attribute 1 has the standard D for a definition item, Y to emphasize that items in hyper files are not saved, and L to indicate that updates to the hyper file take part in transaction logging.

• Attribute 2 points to a single frame on disc that enables items locks to be uniquely taken against the hyper file.

• Attribute 3 summarises the file creation parameters, specifying the file type as Y3, giving the name of the data definition item used for key value extraction, and the number of sections in the lookup table.

• Attributes 2 and 4 may hold standard index information if indexes have been added to the hyper file.

Other attributes work as normal for a file definition item, although resize parameters have no effect.

Saving and restoring accounts that include hyper files

A hyper file behaves like a file synonym definition item (Q-pointer) during an account save. The H-pointer itself is saved, but the data is not. The data items are saved when the data sections that comprise the hyper file are saved. Note if some sections of a hyper file are pointing at a different account then saving an individual account containing the hyper file may result in inconsistencies in the hyper file.

During an account restore the hyper file H-pointer is reconstituted. However the hyper file cannot be opened unless all of its data sections can also be opened. If necessary, HYPER-VALIDATE can be used to identify missing data sections.

Transaction logging

Logging After images

A hyper file logs After images for updates written through the hyper file. These updates are logged as being made on the hyper file. Transaction logging of hyper sections opened via the hyper file is suppressed, even when the D-pointer is marked L. Logging after images for the hyper file ensures that if updates are applied after a hyper file configuration has been changed, the updates will affect the correct hyper sections.

Logging Before images

Updates to the hyper file inside a transaction cause Before images to be logged by the hyper sections themselves. This optimises performance when rolling back a transaction.

Item locks

Reality item locks are taken against a file handle. The file handle becomes the container for the locks.

By default, item locks on a hyper file will use the hyper file as the container. However this means that attempts to directly lock items in the hyper sections will not see the same locks.

During hyper file creation it is possible to choose the hyper section as the lock container; this is not recommended and should be chosen only if updates to individual hyper sections are to be allowed.

Update locks

Once a hyper file has been created it is recommended all updates are performed through the hyper file. This ensures that any new items are written to the correct hyper section and the consistency of the hyper file is maintained.

By default, an update lock is added to each hyper section. This update lock is silently bypassed when updating the hyper sections via the hyper file.

During hyper file creation it is possible to choose not to apply update locks to the hyper sections; as with item locks, this is not recommended and should be chosen only if updates to individual hyper sections are to be allowed.

Indexing hyper files

A hyper file does not maintain indexes. Indexes are maintained by the individual hyper sections. The hyper file makes indexes on the hyper sections visible to applications wishing to search the hyper file and to ENGLISH for automatic indexing. Only indexes that exist on all hyper sections can be added into the hyper file.

When indexes are read via the hyper file, index keys from all hyper sections are combined and presented in the required order to the querying application.

When setting position on an index through a hyper file it sets position on all the constituent hyper section indexes so that they are in the correct place ready for reading.

Converting files

A single large file can be converted to a hyper file with multiple data sections in one of two ways. If database space permits then hyper sections can be created, the hyper file definition set up and then the MOVE command used to move items into the hyper file, preserving update date and time. If there is insufficient space in the database then a series of ACCOUNT‐SAVE, DELETE-FILE, HYPER-CREATE, and SEL‐RESTORE (for individual hyper sections) commands must be used. (Although a T‐DUMP/T‐LOAD could be used this would not preserve update date- and time-stamps.)