RealMem Memory Mode
-------------------
By Mark Feldman (pcgpe@oocities.com)
Disclaimer
----------
I assume no responsibility whatsoever for any effect that this file, the
information contained therein or the use thereof has on you, your sanity,
computer, spouse, children, pets or anything else related to you or your
existance. No warranty is provided nor implied with this information.
Introduction
------------
Intel's segmented memory scheme is a pain to use when you've got
large data structures in memory that you want to move around. Being limited
to 640K of base memory is also somewhat restrictive and if you're trying to
write a decent game you'll probably need more.
The most obvious way around segmented memory is to use protected mode.
Unfortunately this has it's problems. For starters it's notoriously
difficult to set up (unless you use an already made extender). You also
have to be extremely careful with your programming in protected mode, one
access to memory you're not supposed to access and up pops the notorious
"General Protection Fault" (remember your first Windows program?).
It'd be nice to be able to access all of memory, including xms, in a linear
fashion while still being able to keep the CPU in real mode, and there is!
If you are going to do any serious programming with realmem mode then
you'll have to be competent in assembly, but the C++ classes included at the
end of this article will get you started and could even be used for some
simple applications.
Let's say you are accessing memory by using the es:[ebx] pair. In normal
real mode the value of ebx can never be greater than 0xFFFF, if it is and
you try to access memory then a 0Dh interrupt is generated. On my 486
it sometimes worked ok even though the interrupts were being generated, on
the 386's I've tried it on it crashed the system every time. Real memory mode
is a way of simply setting up the CPU so that this interrupt isn't generated
and the correct memory is accessed.
How safe is it? To be honest I have no idea. I can tell you however that I
have successfully used it in one commercial project (da boss told me to)
and countless of my own experimental programs and I've never had a
problem with it. I believe Origin also uses it for their VOODOO memory
manager in Ultima 7.
I first learned about this technique from a text file written by the demo
group PROGREX. The file is called REALMEM and It's available on
x2ftp.oulu.fi in the /pub/msdos/programming/docs directory.
Technical Stuff
---------------
The CPU maintains a number of lists called "Descriptor Tables". These are
normally used in protected mode, the operating system (or dos extender)
fills this list with a bunch of 8-byte entries, each entry specifies one
block of memory that a program can access. An entry contains info such
as where the block can start and how long it is. There is one list called
the "Global Descriptor Table" which all programs can access. In a multi-
tasking situation each program running also has a "Local Descriptor Table"
which is a block specifying it's own memory area. If you try to access
memory outside the memory specified in these lists then boy are you in
trouble!!
In regular CPU real mode the segment limits are set at 64K, so you can
never access outside this area without generating a General Protection
Fault. Real memory mode simply switches to protected mode and sets up the
Global Descriptor Table so that there is one segment starting at address
0000:0000 and it is 4 gigabytes long. It then switches back to real mode
and the Global Descriptor Table holds. The only other thing we must do is
enable the A20 line so that we can access all the upper memory.
Fortunately HIMEM.SYS has a function to do this for us.
*** PENTIUM UPDATE ***
It would appear that REALMEM is not as attractive on the Pentium as it was
with earlier machines. In order to use 32-bit addressing with the CPU in
16-bit mode you need to prefix all relevent instructions with the address
size prefix (67h). In general, prefix bytes can only execute in the U pipe
and they cannot pair with anything in the V pipe. In other words, 1-cyle
instructions can take up to 4 *TIMES* as long to execute as they would in
32-bit mode.
The situation could in fact be a lot worse. In 16-bit mode a simple
instruction like MOV DS:[EDI],EAX contains 2 prefixes: an address size
prefix (67h) and a data size prefix (66h). Assuming each prefix takes up
both pipes for a full cycle, this simple instruction would take up to 6
*TIMES* as long to run on a Pentium in real mode than it does in protected
mode. I haven't managed to actually time this yet, but either way things
don't look good for REALMEM on Pentiums.
Advantages and Disadvantages
----------------------------
Real memory mode will not work if EMM386.EXE is installed in the system, and
there's probably a few other memory managers it won't work with either. (THIS
ARTICLE WAS ALSO WRITTEN BEFORE WIN95 WAS RELEASED. REALMEM WON'T RUN IN A DOS
SHELL, THUS FORCING YOUR USERS TO BOOT IN DOS MODE. YECH! I've heard a rumor
that there is a way to get it to work in a dos shell, but so far I haven't been
able to figure out how.) The reason for this is that EMM386 puts the system in
V86 mode. This is no big drama really, I never have it installed on my system
because it does cause problems with many protected mode programs, but you'll
have to make this clear in your user documentation.
All CPU instructions will work in their regular real mode ways. This means
that instructions such as movsb will move a byte from ds:[si] to es:[di],
NOT ds:[esi] to es:[edi]. If you want to use 32-bit addresses for string
instructions then you need to explicitly tell your assembler to add an
address size prefix byte to the front of them, eg:
rep movs word ptr es:[edi], ds:[esi]
This is equivelent to rep movs, but it adds the address size prefix
byte to the instruction.
The code supplied with this file must have HIMEM.SYS installed. HIMEM.SYS
contains a number of functions we need, such as finding out how much XMS
memory is present and available, and enabling the A20 line.
The realmem and XMS array C++ Classes
-------------------------------------
To help you get started with real memory mode I've written a simple Borland
C++ 3.1 class. The class sets up real memory mode when the object is created
and shuts it down again when it's destroyed. Make sure you only create one
instance of this class, it can be either global (blech) or local. The
realmem class allocates all available XMS, and trying to allocate another
instance will return an ERROR_NOXMS error.
The 'status' variable is used to determine whether an error has occurred.
These are the values it can contain:
There is also a range of classes you can use to access extended memory
from within your C++ program. They are extremely slow however, so if you
want to do any serious xms programming then I suggest writing your own
routines to access xms directly. The classes included are:
This is the class to set up real memory mode:
realmem
This class contains these variables:
int status; // holds any error messages if they occurr
unsigned XmsKBytes; // number of xms kilobytes allocated
unsigned long XmsBytes; // number of xms bytes allocated
unsigned long XmsStart; // the starting address of available xms memory
These classes are used to manipulate xms memory:
xmschar
xmsint
xmslong
The last 3 classes contain a variable called address which is the absolute
address of the variable. You can specify this address in the constructor
if you like, like this:
realmem rm;
xmschar buffer(rm.XmsStart); // set up a buffer at the start of xms
The classes can then be used as arrays:
unsigned i;
for (i=0; i<64000; i++) buffer[i] = 0; // clear first 64K in the buffer
Feel free to overload some of the other operators for these classes.
TEST.CPP
--------
Here's a very simple demo program to show you how to use the C++ classes.
First of all it attempts to set up real memory mode. If it can then it uses
the first 64000 bytes in extended memory as a virtual buffer for the
graphics screen. Next it switches over to graphics mode 13h, fills the
virtual bufer with random pixels and copies it to the screen. It then
waits for the user to press a key, switches back to normal mode and cleans
up. Compile this source along with realmem.cpp. Note that this program is
REALLY slow. Like I said, if you want to do anything serious you'll have to
write your own assembly routines to quickly access memory.
#include <stdio.h>
#include <stdlib.h>
#include <conio.h>
#include "realmem.h"
void main()
{
realmem xms;
xmschar buffer;
char far * screen;
long i;
// Make sure we switched over to xms ok
if (xms.status != ERROR_OK)
{
printf("REALMEM error # %i occurred! 'Moutta here...",xms.status);
}
else
{
// Point buffer to start of xms and set up screen pointer
buffer.address = xms.XmsStart;
screen = (char far *)0xA0000000;
// Tell the user to wait a tik
printf("Filling virtual buffer with random pixels, one tik...");
// Fill xms virtual buffer with random pixels
for (i=0; i<64000; i++) buffer[i] = random(256);
// Switch to graphics mode 13h
asm mov ax,0x13
asm int 0x10
// Now copy the entire virtual buffer to the graphics screen
for (i=0; i<64000; i++) screen[i] = buffer[i];
// Wait for a keypress then switch back to text mode
getch();
asm mov ax,3
asm int 0x10
}
}
REALMEM.H
---------
#ifndef __REALMEM__
#define __REALMEM__
#define ERROR_OK 0
#define ERROR_INV86 1
#define ERROR_NODRIVER 2
#define ERROR_WRONGVERSION 3
#define ERROR_NOFUNC 4
#define ERROR_VDISK 5
#define ERROR_A20 6
#define ERROR_NOXMS 7
#define ERROR_NOMEM 8
#define ERROR_NOHANDLES 9
#define ERROR_INVALIDHANDLE 10
#define ERROR_LOCKOVERFLOW 11
#define ERROR_LOCKFAIL 12
#define ERROR_UNKNOWN 13
class realmem
{
public:
realmem();
~realmem();
int status;
unsigned XmsKBytes;
unsigned long XmsBytes;
unsigned long XmsStart;
private:
unsigned XmsHandle;
unsigned long XmsDriver;
int InV86 ();
void LoadGDT ();
int DriverPresent ();
unsigned long GetDriverAddress ();
int WrongVersion ();
unsigned XmsAvail ();
int EnableA20 ();
int DisableA20 ();
unsigned XmsAlloc (unsigned kbytes);
void XmsFree (unsigned handle);
unsigned long XmsLock (unsigned handle);
void XmsUnlock (unsigned handle);
};
class xmschar
{
public:
xmschar();
xmschar(unsigned long addr);
xmschar operator [] (unsigned long offset);
operator = (char value);
operator char();
unsigned long address;
};
class xmsint
{
public:
xmsint();
xmsint(unsigned long addr);
xmsint operator [] (unsigned long offset);
operator = (int value);
operator int();
unsigned long address;
};
class xmslong
{
public:
xmslong();
xmslong(unsigned long addr);
xmslong operator [] (unsigned long offset);
operator = (long value);
operator long();
unsigned long address;
};
#endif // __REALMEM__
REALMEM.CPP
-----------
#include <dos.h>
#include "realmem.h"
#pragma inline
asm .386p
unsigned char MemoryCode[16]={0,0,0,0,0,0,0,0,0xFF,0xFF,0,0,0,0x92,0xCF,0xFF};
unsigned char Mem48[6];
// ----------------- realmem class ------------------------
realmem::realmem()
{
status = ERROR_OK;
// Make sure we are not in V86 mode
if (InV86()) return;
// Load the Global Descriptor Table
asm mov Mem48[0],16
asm mov eax,seg MemoryCode
asm shl eax,4
asm mov bx,offset MemoryCode
asm movzx ebx,bx
asm add eax,ebx
asm mov dword ptr Mem48[2],eax
asm lgdt pword ptr Mem48
asm mov bx,08h
asm push ds
asm cli
asm mov eax,cr0
asm or eax,1
asm mov cr0,eax
asm jmp protection_enabled
protection_enabled:
asm mov gs,bx
asm mov fs,bx
asm mov es,bx
asm mov ds,bx
asm and al,0FEh
asm mov cr0,eax
asm jmp protection_disabled
protection_disabled:
asm sti
asm pop ds
// Make sure an xms driver is present
if (!DriverPresent()) return;
// Get the driver function call address
XmsDriver = GetDriverAddress();
// Make sure the version number is at least 2
if (WrongVersion()) return;
// Find out how much xms is available
XmsKBytes = XmsAvail();
if (status != 0) return;
XmsBytes=(unsigned long)XmsKBytes * 1024L;
// Enable the A20 line
EnableA20();
if (status != 0) return;
// Allocate some xms
XmsHandle = XmsAlloc(XmsKBytes);
if (status != 0) return;
// Lock the block into place and get it's address
XmsStart = XmsLock(XmsHandle);
}
realmem::~realmem()
{
// Unlock the memory block
XmsUnlock(XmsHandle);
// Free the block
XmsFree(XmsHandle);
// Disable the A20 line
DisableA20();
}
// This function tests if the cpu is in V86 mode, if it is then a memory
// manager such as EMM386.EXE is probably running.
int realmem::InV86()
{
unsigned result;
asm mov eax,cr0
asm and ax,1
asm mov word ptr result,ax
if (result) status = ERROR_INV86;
return result;
}
// This function makes sure the HIMEM.SYS in installed.
int realmem::DriverPresent()
{
struct REGPACK regs;
regs.r_ax = 0x4300;
intr(0x2F,®s);
regs.r_ax &= 0xFF;
if (regs.r_ax != 0x80)
{
status = ERROR_NODRIVER;
return 0;
}
return 1;
}
// This function gets an address used to call HIMEM.SYS functions directly.
unsigned long realmem::GetDriverAddress()
{
unsigned long address;
asm mov ax,0x4310
asm int 0x2F
asm mov ax,es
asm shl eax,16
asm mov ax,bx
asm mov [address+0],bx
asm mov [address+2],es
return address;
}
// This function makes sure that the HIMEM.SYS version is at least 2.0
int realmem::WrongVersion()
{
unsigned long addr = XmsDriver;
char version;
asm mov ah,0
asm xor dx,dx
asm call dword ptr addr
asm mov version,ah
if (version < 2)
{
status = ERROR_WRONGVERSION;
return 1;
}
return 0;
}
// This function returns the amount of free XMS memory available
unsigned realmem::XmsAvail()
{
unsigned long addr = XmsDriver;
unsigned kbytes;
asm mov ah,8
asm xor dx,dx
asm call dword ptr addr
asm mov word ptr kbytes,dx
if (kbytes == 0) status = ERROR_NOXMS;
return kbytes;
}
// This function calls HIMEM.STS to enable the A20 line so we can access
// extended memory.
int realmem::EnableA20()
{
unsigned long addr = XmsDriver;
unsigned a,b;
asm mov ah,5
asm call dword ptr addr
asm mov a,ax
asm mov b,bx
if (a != 1)
{
b &= 0xFF;
switch (b)
{
case 0x80: status = ERROR_NOFUNC; return 0;
case 0x81: status = ERROR_VDISK; return 0;
default: status = ERROR_A20; return 0;
}
}
return 1;
}
// This function calls HIMEM.STS to disable the A20 line.
int realmem::DisableA20()
{
unsigned long addr = XmsDriver;
unsigned a,b;
asm mov ah,6
asm call dword ptr addr
asm mov a,ax
asm mov b,bx
if (a != 1)
{
b &= 0xFF;
switch (b)
{
case 0x80: status = ERROR_NOFUNC; return 0;
case 0x81: status = ERROR_VDISK; return 0;
default: status = ERROR_A20; return 0;
}
}
return 1;
}
// This function calls HIMEM.SYS to allocate a block of extended memory.
unsigned realmem::XmsAlloc(unsigned kbytes)
{
unsigned long addr = XmsDriver;
unsigned result,handle;
unsigned char error;
asm mov ah,9
asm mov dx,kbytes
asm call dword ptr addr
asm mov result,ax
asm mov error,bl
asm mov handle,dx
if (result != 1)
{
switch (error)
{
case 0x80: status = ERROR_NOFUNC; return 0;
case 0x81: status = ERROR_VDISK; return 0;
case 0xA0: status = ERROR_NOMEM; return 0;
case 0xA1: status = ERROR_NOHANDLES; return 0;
default: status = ERROR_UNKNOWN; return 0;
}
}
return handle;
}
// This function calls HIMEM.SYS to lock an allocated block of memory into
// place. This ensures that HIMEM.SYS won't go moving memory around on us.
unsigned long realmem::XmsLock(unsigned handle)
{
unsigned long addr = XmsDriver;
unsigned long address;
unsigned result;
unsigned char error;
asm mov ah,0Ch
asm mov dx,handle
asm call dword ptr addr
asm shl edx,16
asm mov dx,bx
asm mov address,edx
asm mov result,ax
asm mov error,bl
if (result != 1)
{
switch (error)
{
case 0x80: status = ERROR_NOFUNC; break;
case 0x81: status = ERROR_VDISK; break;
case 0xA2: status = ERROR_INVALIDHANDLE; break;
case 0xAC: status = ERROR_LOCKOVERFLOW; break;
case 0xAD: status = ERROR_LOCKFAIL; break;
default: status = ERROR_UNKNOWN; break;
}
}
return address;
}
// This function calls HIMEM.SYS to unlock our allocated memory block.
void realmem::XmsUnlock(unsigned handle)
{
unsigned long addr = XmsDriver;
asm mov ah,0Dh
asm mov dx,handle
asm call dword ptr addr
}
// This function frees our allocated memory block.
void realmem::XmsFree(unsigned handle)
{
unsigned long addr = XmsDriver;
asm mov ah,0Ah
asm mov dx,handle
asm call dword ptr addr
}
// ----------------- xmschar class ------------------------
xmschar::xmschar()
{
address = 0;
};
xmschar::xmschar(unsigned long addr)
{
address = addr;
};
xmschar xmschar::operator [] (unsigned long offset)
{
return xmschar(address + offset);
}
xmschar::operator = (char value)
{
unsigned long addr = address;
asm xor ax,ax
asm mov fs,ax
asm mov ebx,addr
asm mov al,value
asm mov fs:[ebx],al
return value;
};
xmschar::operator char()
{
unsigned long addr = address;
char result;
asm xor ax,ax
asm mov fs,ax
asm mov ebx,addr
asm mov al,fs:[ebx]
asm mov result,al
return result;
};
// ----------------- xmsint class ------------------------
xmsint::xmsint()
{
address = 0;
};
xmsint::xmsint(unsigned long addr)
{
address = addr;
};
xmsint xmsint::operator [] (unsigned long offset)
{
return xmsint(address + offset);
}
xmsint::operator = (int value)
{
unsigned long addr = address;
asm xor ax,ax
asm mov fs,ax
asm mov ebx,addr
asm mov ax,value
asm mov fs:[ebx],ax
return value;
};
xmsint::operator int()
{
unsigned long addr = address;
int result;
asm xor ax,ax
asm mov fs,ax
asm mov ebx,addr
asm mov ax,fs:[ebx]
asm mov result,ax
return result;
};
// ----------------- xmslong class ------------------------
xmslong::xmslong()
{
address = 0;
};
xmslong::xmslong(unsigned long addr)
{
address = addr;
};
xmslong xmslong::operator [] (unsigned long offset)
{
return xmslong(address + offset);
}
xmslong::operator = (long value)
{
unsigned long addr = address;
asm xor ax,ax
asm mov fs,ax
asm mov ebx,addr
asm mov eax,value
asm mov fs:[ebx],eax
return value;
};
xmslong::operator long()
{
unsigned long addr = address;
long result;
asm xor ax,ax
asm mov fs,ax
asm mov ebx,addr
asm mov eax,fs:[ebx]
asm mov result,eax
return result;
};