The first article we wrote about the CTOS Caching Service was based upon the CTOS III 1.0  operating system.  With the release of CTOS III  1.1, there have been some additions to the picture, so we need a Part II article to investigate and understand these new features.

Types of Cache
     There are four of different terms describing cache that we need to understand.  Two are already familiar to us, and the other two are new.
The terms are:

1. File system cache.
   This type of cache has been available since CTOS/XE was released.
   It is a disk-block cache, as opposed to a file cache.  Each block in the cache represents
   eight sectors of disk.  This type of cache can (and often must) be configured on server
   workstations, and can also exist on any workstation which has a local file system.
   This type of cache is the only one that supports MemDisk.

2. Write-Behind cache.  While not a separate cache type, this is a new feature of the
   CTOS file system cache service.  It operates in a similar fashion to the cache used by
   the Unix file system.  With this type of cache, when a program issues a write request,
   the response from the OS to the application is immediate, but the disk sectors will not
   necessarily be immediately written to disk;  instead they are written to disk
   "in the background", via a "cleaning" task.

3. Workstation Agent cache.  This type of cache operates on a workstation, and is a
   file cache (not a disk-block cache).  The workstation agent cache contains files which
   have been opened in mode read.  As soon as a file is (re)opened in modify mode,
   that file is purged from the workstation agent cache (but it will remain in the server's
   cache).

4. Server Agent cache.  This type of cache is new to the multi-processor support in CTOS III.
   It resides on IOP boards (usually on Cluster IOP boards) and functions for the IOP board
   in the same way that a workstation agent cache functions on a diskless workstation.
   Files are cached while they are opened in read-only mode, but are purged whenever they
   are opened for modify.

     Caching, in most configurations, is a definite performance booster.  Any time that you can cause an operation to take place from within a computer's memory, as opposed to having to go outside the computer's memory (i.e. to an I/O device), the result is a large increase in performance.  The different types of cache that are available to CTOS customers allow the customer to make decisions regarding trading memory for performance improvements.  Nowadays, memory is relatively cheap and plentiful.

     CTOS III 1.1 is the first CTOS III release which supports multi-processor servers (CTOS/XE was a protected-mode OS, not demand-paged like CTOS III).  One of the more important restrictions of CTOS III in a multi-processor server is that you must define a file system cache on the motherboard (EP00).  The IOP boards will not boot if you do not have a cache defined on EP00.  All disk read/write operations which are required by services on the IOP boards must go through the cache, so if you do not define a cache for the motherboard, the IOP boards will not boot CTOS.

     Fortunately, since the presence of cache is so important on multi-processor servers, the default, if you specify nothing, is a 1/2 megabyte cache.  You must go out of your way to completely remove cache services from your system (you must specify 0 to both blockcount and blocksize).  Because cache is absolutely necessary for a multi-processor server to function under CTOS, you will never want to eliminate it.

     Now you have three different caches to deal with, and you can have all three operating at the same time.  This can be confusing.  How does one know when to define which type of cache and on what processor(s) ?  Hopefully, we will arrive at the correct answers to these questions together.

Write-Behind Cache
     Let's first discuss the new feature of the cache service that was not available previously.  It is called write-behind cache.  While new to CTOS, this type of cache has been an integral part of the Unix operating system for quite some time.

     The CTOS cache has always been a "Write-Through" cache, where a program issues a write operation, and while the cache is updated, the data are written to the disk before the response is sent to the application program.  Therefore, any CTOS programmer could be absolutely assured that when his program received ercOK from a write operation, that the data had successfully been written to the disk, without error.  Thus, the CTOS cache provided the same "guarantee" to the programmer that was there before the cache service was available.  It made for very reliable performance in mission-critical applications (it's nice to know for sure that something you thought you wrote was truly written to the disk -- like a typescript file for Editor or OFIS Document Designer).  You would almost certainly not want this type of cache for an ISAM file (but then again, you really should not cache ISAM files, anyway).

     The "Write-Behind" cache operates differently.  With this type of cache, when a program issues a write request, the data are copied to the cache block, but not immediately written to disk.  The cache service returns the request ercOK, even though nothing had been written.  As soon as memory (the cache block) is updated, the operation is deemed "successful".  Meanwhile, there is a background process, called the "cleaning process", which periodically issues writes to the disk of any cache blocks which have been updated (in cache memory) but have not yet been written to disk.

     This is "good news and bad news".  First the good news:  your program runs much faster, since it does not actually have to wait for physical disk I/O to be performed every time the program issues a write request.  The data is simply copied to memory and your program gets back ercOK.  Data are written to disk at a later time.

     The bad news:  if something happens to the system (crash, power failure, etc.) between the time your program issued the write and got back ercOK, and the time that the "cleaning process" gets around to actually writing the data to disk, then those updates are lost forever.  If they don't get written to disk, then it as if they never happened !   More bad news: if there is an actual problem (e.g. erc 301) writing the record to disk, the appllication will never know that a problem occurred, and can therefore take no corrective action to recover from the error.  Of course, the "cleaning process" will do its best to correct any problems, but the application program is totally unaware that a problem may have occurred (it gets ercOK immediately after issuing the Write request).

     So why use write-behind cache ?  For speed.  That's the answer, pure and simple.  If your system is stable, the disk writes will happen as the "cleaning process" is dispatched in the background.  Eventually all of the accumulated cache blocks that are "pending' will be written to the disk, but at a time when your program is waiting for some other event to occur; the "cleaning process" runs in the background.  You trade speed of operation for robustness.

     This may seem risky to a CTOSian, since we are accustomed to those disk writes actually taking place when we think they will.  And it does entail some risk, but the benefits of the additional speed of processing may be worth the risks.  Note also that the write-behind cache can only represent a problem for writing to disk; operations that take place for reading disk sectors are obviously immune to any problems, since they don't require the "cleaning process" to take any action at all.  You must make the decision as to which feature of the cache service you choose to use.  Remember, the default cache is a "write-through" cache, not the write-behind cache.

Shutting down a system with Write-Behind Cache
     There is one very important consideration when you are using a system with write-behind cache enabled.  Because the "dirty" (or "pending") cache blocks are not immediately written to disk, you cannot just turn off the power to your system when you wish to shut it down (maybe to go home for the weekend).  You have to have some way to force the system to write any cache blocks that are "pending" and have not been written back to the disk.  Fortunately, there is a command which lets you do this.

Logically, it is:

 Shutdown Cache

which causes the "cleaning process" to be dispatched and to write back all outstanding "dirty" cache blocks to disk.  When the command completes, you can be assured that all file updates have been completed by the Cache Service, and you can safely power down you processor.

     If you are using write-behind cache, then you must Shutdown Cache before powering off your processor or pressing the reset button.  If you use the "BootStrap" command, the command will automatically do the "shutdown cache" operation for you, ensuring that all blocks are written to disk, before performing the bootstrap operations.

Workstation Agent Cache
     The cache that you install on a cluster workstation is known as AgentCache.  This type of cache holds records for files which exist on the server's disk.  Whenever a file is opened for read access, it is cached in the Agent Cache.  As soon as the application program (re)opens the file for modify or write operation, the cache is purged of all cache blocks for that file.  The Agent Cache is strictly for speeding up read operations.

     Note that the file still remains in the server's file system cache, and caching still functions there, for input or output type files.  It is important to differentiate the read-only nature of the Agent Cache as opposed to the read/write nature of the file system cache  which operates only on the processor where the disks are physically attached.

     If you define a workstation Agent Cache, it can be present on client workstations which have local disk or on diskless client workstations.  If you have the memory available, the Agent Cache can result in a definite performance increase on diskless client workstations.
But you must remember, Agent Cache has the following restrictions:

 1.  It only caches "input" files which exist on a disk at the server.
 2.  It cannot contain a "cache memory disk".
 3.  You cannot "lock" files into the Agent Cache.

     If you boot a vClstrLfs operating system at your client workstation, you can define a file system cache at the client workstation.  This cache functions the same as the file system cache at the server workstation, but is functional only for local  disk files.  The Workstation Agent Cache handles caching of read-only files from the server, as described above.  You must have a vClstrLfs operating system booted in order to define cache memory disk at a client workstation, as cache memory disk requires the presence of the file system code within the CTOS operating system that is booted.

Server Agent Cache
     CTOS III 1.1 provides support for multi-processor servers, consisting of a processor with a motherboard and an EISA bus which can hold one or more "IOP" boards, each with its own CPU, memory, and copy of the CTOS III operating system.   These IOP boards come in several flavors:

 1.  Cluster IOP - equipped with RS-485 cluster communications hardware.
 2.  Comm IOP - equipped with hardware for external communcations.
 3.  Ethernet IOP - with controller for ethernet communcations.
 4.  ISDN IOP - equipped with hardware for ISDN communications.

Of course, a multi-processor server needs at least one Cluster IOP in order to be a server in the first place (there is no RS-485 port on the motherboard).  As previously mentioned, in order to boot any IOP board, there must be a cache defined for the motherboard (EP00, to use the correct terminology).

     There is also a new type of cache available for use with these multi-processor servers: Server Agent Cache.  This type of cache works on one of the IOP boards in the same fashion as the Workstation Agent Cache works for the workstation.  Files which are read-only are cached by the Server Agent Cache on the board where they are accessed.  As soon as a file is (re)opened for output, the cache is purged of all blocks relating to that file.

     Note that for Cluster IOP boards, this means that files which are opened by client workstation attached to the Cluster IOP are cached at the IOP, as long as the files are read-only.  They may also be cached at the client workstation, by the Workstation Agent Cache.  And they will be cached in the file system cache on the motherboard.  If you choose, you can activate Server Agent Cache on the IOP boards and not activate cache at the individual client workstations, and you will still enjoy some of the benefits of caching , as the read-only files accessed by client workstations will at least be cached at the Cluster IOP, and therefore may not require an actual disk access for the read operation to be satisfied.

     The current version of IOP boards have 4MB of on-board memory, and cannot expand beyond that point.  If there are not a lot of services running on the Cluster IOP, which is fairly typical, then you can use that extra memory for Server Agent Cache without sacrificing anything.  Note that a CTOS cache should be a minimum of 512K for any reasonable performance improvement (that's 128 blocks).

File System Cache
     We should probably do a mini-review of the file system cache, just for completeness.  The file system cache is, in reality, a disk-sector cache.  In fact, as the cache blocks are 4K in size, each cache block actually holds eight disk sectors.  Do not change the size of the cache block for the CTOS cache (any type).  Only systems which have local disks attached to them (or have booted a version of CTOS other than vClstr) can support the file cache service.  You can have a cache memory disk on a diskless workstation, but it must boot vClstrLfs, not vClstr, because the operating system must have the file system code resident to handle cache memory disk operations (below).

     The file system cache is also capable of supporting a cache memory disk "device", which operates exactly as if it were a real disk.  You can Format it, allocate new directories, create and delete files, do a Files or Volume Status command, etc.  On a workstation (vClstrLfs) you can have both Workstation Agent Cache and file system cache, and they share the same set of cache blocks.  The cache memory disk device "lives" in the memory where you have allocated the cache, and actually takes up cache blocks.  If your cache memory disk grows too large it can severely cut down on the number of remaining cache blocks; then the cache service can be rendered useless.  You should limit the size of the cache memory disk such that it cannot grow beyond a certain number of cache blocks, leaving you enough to make the cache still functional.

     Files can be "locked" into the file system cache only, not in the Workstation or Server Agent cache.  It is generally not a good idea to "lock" files into the cache; instead, let the cache service determine which are the most recently (and often) used blocks, as it probably "knows" more about the cache than you do.

Mixing type of Cache
     You can configure your system to take advantage of both types of cache at the same time, on a disk-by-disk basis.  What you need to do is run a utility program (Cache Action) to inform the cache service of your intentions.  The Cache Action command form looks like:

 Cache Action
     File list
     Action codes (E,D,W,N,R)
     [Print file]

and the method you use to set up different types of caching on the same system is to use the Cache Action command specifying the drive name in the "File list" parameter, and the value "W" to enable write-behind cache or the value "E" to enable write-through cache on that volume.  Put this in your SysInit.jcl file (at the beginning), and include an entry for each drive in your system.   You configure the cache service as if it is going to use write-behind cache (so the correct parameters are set), and the SysInit.jcl Cache Action commands take care of setting the necessary flags for the cache service so that it "knows" which volumes are to be handled write-behind, and which are to be handled write-through.

Controlling the Cache Service
     There is a byte in the file header of every file which controls the actions of the Cache Service for that file.  There are four bits in the cache control byte which are significant:

 01 Enable write-through caching on this file.
 02 Disable write-through caching on this file.
 80 Enable write-behind caching on this file.
 40 Disable write-behind caching on this file.

and the default for all files in the system is that none of these bits are turned on.  That is, the file is neither enabled nor disabled from either type of caching.  In fact, that's the reason for the "AgentCacheDefaultEnable" and similar configuration parameters.  They tell the Cache Service that even if a file is not specifically enabled, it should still be cached.

     The commands "Enable Caching" and "Disable Caching" are used to set the flag in the file header of individual files so that the Cache Service "knows" what to do with the particular file when it is opened.  In fact, these commands were released with the original Cache Service (before write-behind), and they are identical to using
"Cache Action" with the "E" or "D" option, respectively).  They pertain to the write-through cache, as opposed to write-behind.  If you run "Disable Caching" on a file, then whenever that file is opened, it will not be cached.  Similarly, if you run the "Enable Caching" on a file, then that file will be cached (write-through) whenever it is opened (assuming that the Cache Service is operational).
     The "Cache Action" command really supercedes the Enable and Disable commands discussed above.  It takes a list of file names, and has an option line which allows you to specify the setting to be applied to the file header of all files in the list.
The available options (letters) are:

 E Enable write-through caching.
 D Disable write-through caching.
 W Enable write-behind caching.
 N Disable write-behind caching.
 R Reset all caching flags in the file header.

The options are mutually exclusive.  When you choose one option, all other options are set to zero.  For example, if the file header currently specifies "enable write-through caching", and you select option "W", then the file header will be changed to "enable write-behind" caching, and the previous value is overwritten.

     As previously discussed, you can set any of the above options on a device but the setting only lasts until the processor is rebooted, when it must be set again.  That's why we recommended putting the command in SysInit.jcl when you wnat to set a type of caching on a volume.

     So, yes, you can have individual files set for different types of caching.  And yes, it works.  But you must set the file header value yourself -- the default is "reset", meaning that none of the options is set.

ISAM vs Cache - an Update
     Previously, we suggested that you disable ISAM files and indices from caching, using the "Disable Caching" command.  There is now another school of thought on this topic.

     A recent benchmark, in a live banking situation, showed that on a very high speed processor there can be a significant improvement in performance if ISAM files and indices are cached.  The key, it seems, is in the raw CPU speed of the processor, and in the amount of memory dedicated to the cache.

     In the specific instance, the processor was an SG-7000 server, which had 32MB memory available.  Of course, the SG-7000 contains a Pentium processor running at 60MHz, which is a very fast processor.  The ISAM records which were used in the banking application were around 350 bytes long.  This means that around 11 data records fit into one cache block.  To really evaluate the capabilities of the machine, 24MB of cache was allocated.

     With the caching of ISAM data and index files turned off, the average transaction took approximately 20 seconds to complete; with cache enabled, the transaction time averaged 10 seconds.  Furthermore, with the cache enabled, the number of transactions handled actually doubled in the same period of time.  These numbers combined resulted in a fourfold improvement over the non-cached configuration.  The values were even better when write-behind cache was employed in a test environment, but the bank decided (rightfully so) that write-behind cache was not a valid option for an on-line banking environment.

     The dramatic performance improvement would likely not be nearly as dramatic (and could be worse) if the processor was a 25MHz 80386, for example, with similar configuration of the cache.  And if the ISAM records were signficantly longer, for example, then the cache would not "pre-fetch" 10 or 11 records whenever a cache read took place.  You can see that a record size of 2K would result in pre-fetching only 2 records: the one you need and one adjacent to it.  In this case, the performance improvement would likely not be nearly as dramatic.

     What we can say, for now, is that on very high-speed processors with a lot of memory available, the cache can make a significant difference in overall application performance, even for an ISAM application.  We suggest that you perform tests in your own environment to determine what works best for you.  It appears that providing a lot of memory for the cache to use can significantly increase application performance in the right environment.

Configuring the Cache Service
     The previous article discussed most of the parameters for defining the cache.  This section will only discuss the new parameters,  available with CTOS III R1.1.0:

:FileCacheDefault: is a new parameter, which replaces the older parameter
                   :FileCacheDefaultEnable:.

     The older parameter will still be recognized and will work, but the new parameter has three subfields which allow for flexibility.  They are:

Enable=Yes/No         to tell the cache service whether it should by default enable caching
                      for all files or disable caching for all files which are not
                      specifically cache-enabled or cache-disabled.  The default, if not
                      specified, is YES.

WriteBehind=Yes/No    to tell the cache service to activate or not activate the write-behind
                      feature described above.  The default if not specified is NO.

CleanPriority=number  to define the priority of the "cleaning process" when write-behind
                      cache is enabled.  If you have not enabled write-behind cache, this
                      parameter is ignored.  The default value is 128.
 

:FileCacheService: or :CacheService: (either one can be used, as the parsing routines accept them both).

The parameters have not changed, but an additional parameter has been added:

DirtyBlockCount=number   to specify the maximum number of "dirty" blocks
                         (blocks modified, but not yet written to disk) that are allowed
                         in the cache before the "cleaning process" is dispatched.
                         There is no default.  Maximum is the number of blocks in the cache.
 

     A sample of the entries for different types of cache follows.  You will notice that you can define Agent Cache on either a client workstation or on an IOP board, using the same configuration parameters.  The different types of CTOS (vSrvr, vClstr, vClstrLfs) understand whether you mean Server or Client Agent Cache.

     Sample entries for a single-processor server  which defines a 4MB cache of the write-through variety.
 :CacheService: (Blockcount=1024, MinWorkingSetBlockCount=64)
 :FileCacheDefaultEnable:Yes

      Sample entries for a single-processor server  which defines a 6MB cache of the write-behind variety.
 :CacheService: (Blockcount=1536, DirtyBlockCount=300)
 :FileCacheDefault:(Enable=Yes, WriteBehind=Yes,CleanPriority=129)

     Sample entries for a multi-processor server  which defines write-behind cache (4MB) on the motherboard and Server Agent cache on each of two Cluster IOP boards (1MB each).
 :Processor:EP00
 :CacheService: (Blockcount=1024, DirtyBlockCount=300)
 :FileCacheDefault:(Enable=Yes, WriteBehind=Yes,CleanPriority=129)
 :Processor:EP01
 :CacheService: (Blockcount=256)
 :AgentCacheDefaultEnable:Yes
 :Processor:EP02
 :CacheService: (Blockcount=256)
 :AgentCacheDefaultEnable:Yes

     Sample entries for a vClstrLfs workstation  which defines write-through cache (8MB) for the local file system, a cache-memory disk, and workstation agent cache as well.  Note that the value for MaxSectors limits the size of the cache memory disk such that it cannot consume all of the cache blocks (8000 sectors is 1000 cache blocks).  In this configuration, one might even define the :ScratchVolumeName: on this workstation as [M0], to speed up operations on temporary files.  This might not gain much in performance if the application mix doesn't use many temporary files.
 :CacheService: (Blockcount=2048, MinWorkingSetBlockCount=96)
 :FileCacheDefaultEnable:Yes
 :AgentCacheDefaultEnable:Yes
 :MassStorage: (Class=CacheMemory,Unit=0,Device=M0,
                Volume=MemDisk1,Password=M0,MaxSectors=8000,
                MaxDirectories=6,MaxSysFiles=120,MaxFiles=200,MaxTempFiles=50)
 

Summary
     In summary, the new features available in CTOS III 1.1 provide you with the capability to trade memory for performance, and to trade performance for security.  You can easily configure the cache service, and have the ability to selectively control which files or drives are under the control of the cache service.

     The choices are available and it is up to you to determine what is best for your particular application mix and site requirements.