Menai Corporation Gamelon File I/O Library White Paper

Gamelon Technology White Paper

The Gamelon Libraries

Gamelon was designed to serve the data storage needs of programs that support interactive data collection on the Internet and the data exchange requirements of loosely-coupled programs that communicate through the Internet. The libraries offer many of the facilities of a database, distinguished significantly by the fact that Gamelon requires no schema designed in advance of data storage, and it offers indexing and referencing capabilities that can provide hyperlinks for any kind of data.

There are two versions in current release:

Gamelon 2, a single-user persistent data library that supports single programs.
Gamelon 3, an extension of Objects that supports network data exchanges.

A multi-user Server edition of Gamelon 3 is available on a custom basis.

In Gamelon files, every type of data is an object and every object can be accessed by logical navigation calls, such as first, last, prior and next. Thus, a 30 megabyte BLOB, an integer, a program data structure and an encapsulated system file all reside logically adjacent to one another: they may be addressed as Next, Prior, etc., and the programmer has no obligation to keep track of object locations and sizes. File alterations also take place as logical object manipulations, such as Insert and Delete. The libraries include transaction management and they support indexing and encryption of any object. Programming of data management could hardly be easier.

These libraries and their supporting tools speed the design cycle for file layouts, preserve type safety of stored data, standardize the structure of arbitrarily complex files, permit the annotation of data objects without altering logical file structures and allow the creation of multi-dimensional file structures. Uses include multimedia programming (cross-referenced BLOBS), Internet workflow (self-descriptive platform-independent objects), configuration management (complex nested objects), CAD-CAM (free-form data structure variation) and medical records management (images mixed with structured records).

The technology was designed specifically for use as the persistence facility for mobile software agents, expected to operate in curcumstances of limited memory and limited storage. As a consequence, the libraries have small footprints, memory requirements are minimal, the overhead of the self-descriptive file structure is very modest, the libraries are supported on multiple platforms, and the API is interfaced to multiple computer languages.

A software magazine reviewer has described Menai’s achievement in data storage “an object lesson in how to do it” (EXE magazine, April 1997).

The Abstraction

Gamelon libraries provide an abstraction for the design and implementation of platform-independent, type-safe structured data files. The objects in these files can be exchanged among programs running on different platforms and can be re-used as models in cumulative programming efforts. Gamelon allows objects to be nested within other objects and allows objects to reference and cross-reference one another, even across files. Consistent with the requirements of distributed object stores, each library also provides unique object identifiers, indexing based on object identifiers, and transaction management.

The most commonly accepted definition of an object is that it is an entity that includes data, possesses an identifiable type, and exhibits at least one behavior. This definition encompasses elementary data types as well as complex data structures. The behavior associated with an object of a specific type may be supplied by a program that recognizes the data type, or by 'moveable' procedures that accompany the object.

In current practice, programmers use various methods to store and identify the type and location of any object within a file. For example, an absolute offset in a file bearing a particular name may be defined in hard code as the beginning of an object of a certain type; or certain byte values can be reserved to serve as markers for the beginning of certain types of object (which of course requires alternate means of encoding the marker value as a data value.) The limits of imagination are the only limits that constrain the possible ways of identifying the type and/or location of an object in persistent store.

Object behaviors are likewise subject to endless varieties of implementation. For example, in systems that interpret the object type of data structures in order to supply behaviors to them, a given data structure may have a type that is recognized differently or implemented differently by different programs. Such non-uniformity of behavior runs counter to the principles of object-oriented design, and it makes families of programs difficult to maintain. Systems that automatically associate functional code with the data of an object are meant to avoid this variability, but the delivery systems are complex and still evolving.

A major problem with this state of affairs is that the unique schemes that have been invented must be correctly coded and fully documented in order to work properly from the outset and on into the future of software maintenance. Further, the result of such efforts is likely to be a file structure that is specific to a particular program.

Menai's solution to this problem is to emphasize the abstraction of file structure rather than the abstraction of behavior-specific classes. The consequence of this approach is that the definition of application-specific class attributes and behaviors remains in the province of the application designer, while the definition of the persistent store characteristics of objects is reserved to the Gamelon library.

Data often is an aggregation of related sub-units, and thus the concept of 'aggregate' falls within the concept of objects. Gamelon implements consistent and comprehensive rules for nesting of objects within aggregate objects, the assignment and reading of type-specific data values, the annotation of objects and the means of navigating from one object to another. These rules permit the existence of generalized platform-independent file structures and support tools and they cause the files to document their own structures and contents, thus facilitating re-use of object models and concrete objects.

The abstraction is best illustrated by a documentation syntax that has been adopted by Menai. In this syntax, the left and right brace symbols { } represent the enclosure limits of an aggregate object, and the left and right brackets [ ] represent the enclosure limits of a data object. In this system (which may be extended), there is one generic aggregate object type, and there are index, reference and other data types, all shown by name and abbreviation as follows:

aggregate - AG
index - IN
reference - RE
signed integer - SI (with optional storage size suffix)
unsigned integer - UI (with optional storage size suffix)
float - FL (with optional storage size suffix)
character - CH
string - ST
Unicode character - UC
Unicode string - US
binary - BI
osfile - OS

Within any object there may be placed optional defined annotations. These are represented by abbreviations as follows:

class - CL
identity - ID
name - NA
comment - CO

Here, for example, is an object represented in the Gamelon syntax as used in specifications and documentation, followed by its interpretation:

{CO[ST : CL : NA] [SI : CO]}

This example of syntax represents a commented aggregate object that holds two data objects. The first data object is a string that is annotated with a class and a name; the second data object is a signed integer that is annotated with a comment.

The preceding declarations may be expanded by showing the values associated with a data type or with any annotation. This is done by enclosing the value within parentheses following the abbreviation. Since whitespace is ignored in this syntax, the above example with values included may be stated as follows:

{CO("This is an aggregate object.")[ST("Hello, world!") : CL(42) : NA("First Program")][SI2(11) : CO("This small integer with a value of 11 is stored in two bytes.")]}

Finally, with respect to data values, the syntax allows the declaration of an object that has a data type without an associated value; this is indicated by empty parentheses following the data type abbreviation, and it is used to indicate the placement and data type of an uninitialized (absent) value.

This syntax model becomes part of a software specification and documentation cycle. Any Gamelon file can be transformed by the Gamelon decompiler tool into a textual representation in the defined syntax; any correct textual representation can be transformed by the Gamelon compiler tool into a Gamelon file; and for certain computer languages the class definitions appearing in source file headers can be transformed into a textual representation of a Gamelon file which serves as a model for persistent storage of the class objects defined in those header files, and vice-versa.

The most important aspect of this software lifetime cycle is the standardization of file-related object behaviors. All objects generated through the Gamelon API can be documented in detail by the decompiler, although the objects may reside in files generated at different times by different programs that were designed by various analysts and programmers. Likewise, all source code that creates and accesses objects does so through a standard platform-independent API. This system reduces development cost and adds significant value to files and to source code in the development and maintenance phases of the software cycle.

As code which creates and updates data files is developed, the output can be examined with the Gamelon browser and file decompiler, supplied with the product, in order to assure correctness. Inversely, the browser or text compiler can be used to create a defined Gamelon file which can then be read and interpreted by a draft program to test the expected behavior of the program. At the conclusion of development of a program, the browser can be used to specially annotate a Gamelon file produced by the new program and the decompiler can then produce a textual representation of the annotated file for documentation purposes.

Using a different approach, a complex file structure may be annotated in the development shop for specification purposes and then used as a model for "stripped" copies that are produced by related programs, thus preventing distributed examples of the file from bearing the annotations. In cases where yet more security is required, encryption of any object (including any container object) can be made available. Likewise, compression of lengthy objects can be made available. Please contact Menai for information about encryption or compression.

Capabilities

Gamelon replaces standard low-level file I/O libraries, persistent store class libraries and database libraries. It provides both rigid and free-form data storage, and it allows the program developer to decide whether to allow modification of file layout at run time. The resulting files take the form of compound files that can contain any number of sub-components, each of which can have its own, nested internal structure.

Thus, a Gamelon file can hold tables, persistent store streams and unordered collections of objects, all in the same file, and all objects -- from characters to BLOBs -- are managed the same way. If the developer wishes, the file can be designed to have a header that describes the file layout, or the organization of the file layout can be hard-coded into the application, but all Gamelon files have complete internal structure, type, annotation and value information, so headers and hard-coded layouts are optional.

The libraries support high-level database functionality including logical navigation (e.g., 'Next', 'Prior'), logical alteration (e.g., ‘Insert’, Append’) automatic object locking (primarily for multi-threading), indexing and transaction management. In recognition of programming requirements, the Library offers differing transaction support for batch updates and for interactive edits.

The index and reference objects allow great flexibility in establishing random access and cross-linking. Random access is based on unique object identifiers (object annotations) which are the index keys to physical placement of the objects which they identify. Cross-linking is based on reference objects which hold as their values the object identifiers of the objects which they reference. Gamelon includes API calls which cause a cursor to jump from a reference object to the object which it references, using an index to secure the placement. Any aggregate (which can be the embodiment of a class object) can contain a virtually unlimited number of reference objects that provide links to any other object. References can cross files, and developers decide whether a supporting index will exist within the file that it supports or in another file.

The API

Important computer programs undergo an iterative source code review process. When the code is first written, often it is reviewed for correctness and consistency with shop coding and documentation standards; and time and again the same reviews occur in the bug-fix and functional enhancement sequences of program maintenance. In this review process there lie inherent difficulties that arise from the need for the reviewer to possess a high degree of familiarity with the syntax and semantics of code libraries. Not least among these difficulties that affect many software libraries are the variety of run-time error management systems and the common irregularities of style and naming. Much is to be said for consistency of naming conventions, call styles and error handling in high-level code libraries and their associated Application Program Interfaces (APIs). Menai has invested considerable effort in the resolution of these difficulties.

First of all, Gamelon uses an object-oriented approach in its code library, although the API is available in both structured and object-oriented language bindings. This approach is characterized (a) by the declaration of one or more cursors through which all access to a file is obtained, and (b) by the requirement that access to any object be secured through logical moves of a cursor, such as Next and Prior, instead of specification of offset positions within a file.

Second, the actions available through the API are labelled in a consistent naming system that references the common characteristics of similar or related actions. For example, all queries for embedded object attributes are made through calls that have the prefix 'ObjQuery', and the names of all functions follow the general pattern of reference-action-target.

Third, all actions available through the API return an integer error code of zero (0) on success and a non-zero encoded integer on failure. The non-zero integer contains information that assists programmers and, if necessary, the support technicians at Menai.

With respect to the second and third points, it should be noted that Menai is well aware of the preferences of programmers for use of functions that return meaningful values on success. These desires are met by alternative versions of the function calls that are available in the API.

The overall effect of using Gamelon is that the invention and implementation of file structures that support enterprise standards or families of programs occur much more easily and rapidly than without this facility, and the accomplishment is inherently well-documented and easily ported to other supported platforms.

The Library

Internally, Gamelon libraries are coded entirely in C++. All memory allocation required within the library is managed internally, so that all reserved memory is automatically released at the appropriate time. Whenever permitted by platform and language binding conditions, the library is supplied to developers as a run-time dynamically-linkable library (DLL); otherwise it may be available as a statically-linkable library supported where appropriate by independent server processes or TSRs. On any single node using a DLL, multiple threads may share a Gamelon file.

Menai has automated test programs that can be used to validate its library with any compiler or linker. As a practical matter, most compilers and linkers are likely to work. Menai expects to extend compiler/linker support in response to customer needs.

Language Bindings

Language Bindings are available for C, C++ and Visual Basic languages, together with header files appropriate for popular compilers. An interface module is supplied for Delphi and a class file for Java. On Windows platforms both a DLL and an ActiveX control are available.

Platforms

Gamelon libraries were first released on OS/2 version 2.x, and then on Warp, Microsoft Windows 3.1, Windows 95, Windows NT, Solaris and Irix. Near-term additional ports are expected to include other popular UNIX versions and Macintosh.

Documentation

The development kit includes a reference manual that describes the library design, provides pseudocode examples and documents the use of the included Compiler, Browser and Decompiler tools. There is separate documentation for each purchased language binding. Platform notes and README files are also supplied.

Customer Support

Menai Corporation maintains phone and FAX numbers, an Internet domain to receive queries and support requests. Menai Corporation also maintains an anonymous FTP site. Continuing support can be subscribed on a yearly basis. Replies to support requests are made by e-mail, telephone, and FAX.

Licensing

Gamelon libraries are available under product-line, site and enterprise licenses. Arrangements may include permission for free distribution in conjunction with evaluation copies of products. Inquire of Menai for prices, terms and conditions.

Caveat

Product availability, specifications and functionality, support arrangements and announced prices and license terms for new sales are subject to change without notice.

June 30, 1997

Can't wait to get this exciting product? Call 1-800-GAMELON to place your order today!

Menai Corporation, 1010 El Camino Real, Suite 300, Menlo Park, California 94025-4335
800-GAMELON (800-426-3566) (US only), 650-853-6450 (voice), 650-853-6453 (fax)
Questions about our products and services: info@menai.com.
Comments about this web site: webmaster@menai.com.

Copyright © 1996-98 Menai Corporation. All Rights Reserved. Menai, Gamelon and gamelon(stylized) are worldwide trademarks of Menai Corporation, registered in the United States of America, and the .[g] logo is a worldwide trademark of Menai Corporation. All other trademarks are owned by their respective owners.