The hardware within the RCX must be controlled by software. There are basically 3 levels of software that exist on the RCX. 

The first level of software is the ROM image which is stored inside the H8 chip on the RCX circuit board. The ROM image is unchangeable(not flashable), however it can be viewed. You can obtain a copy of the ROM image by following these instructions provided by Kekoa Proudfoot. The ROM is what controls the RCX when no other software is available. For instance, when you first get the RCX, before you download the Firmware, the ROM image is what tells the RCX how to communicate with the IR Tower and controls some basic functions.

The second level of software is either the Firmware, or some equivalent that serves the same purpose such as the LegOS, TinyVM, etc... This software talks to the ROM image and controls the RCX through calls to it. It also accepts commands from your programs. 

The third level of software is a program. This would be the programs you create using the Lego software or alternative programming systems such as NQC.

So basically, when you upload a program that you have written and start the RCX, the program tells the Firmware what to do, and the Firmware executes those commands which includes sending some commands to the ROM image to be handled.

The IR protocol associated with sending a "message" to the RCX is pretty simple. 
Bit encoding is 2400 baud, NRZ, 1 start, 8 data, odd parity, 1 stop bit. A '0' is coded as a 417us pulse of 38kHz IR, a '1' bit is 417us of nothing. 

At the packet level, all packets look like this: 0x55 0xff 0x00 D1 ~D1 D2 ~D2 ... Dn ~Dn C ~C 

where D1...Dn are the message body, and C = D1 + D2 + ... Dn. 

The data for sending an IR message is F7 followed by the 8 bit "message". 

For example: 55 ff 00 f7 08 12 ed 09 f6 is a packet sending the message "0x12" to the RCX.

LSI Logic have extended this standard to include an S4-record containing an address and a symbol.

The extended S-record standard is as follows:

S<type><length><address><data....><checksum>
S4<length><address><name>,<checksum>

Where:

type

is the record type. Where:

0

starting record (optional)

1

data record with 16-bit address

2

data record with 24-bit address

3

data record with 32-bit address

4

symbol record (LSI extension)

5

number of data records in preceeding block

6

unused

7

ending record for S3 records

8

ending record for S2 records

9

ending record for S1 records

length

is two hex characters. This defines the length of the record in bytes (not characters). It includes the address field, the data field, and the checksum field.

address

is 4, 6, or 8 characters. Corresponding to a 16-, 24-, or 32-bit address. The address field for S4 records is always 32 bits.

data

Are the data bytes. Each pair of hex characters represent one byte in memory.

name

Is the symbol name. The symbol is terminated by a ','.

checksum

Is the one's complement of the 8-bit checksum.

Example

S0030000FC
.
.
S325000004403C0880018D08DD900000000011000026000000003C0880012508DC50C50000B401
S32500000460C50100B8C50200BCC50300C0C50400C4C50500C8C50600CCC50700D0C50800D4FA
S32500000480C50900D8C50A00DCC50B00E0C50C00E4C50D00E8C50E00ECC50F00F0C51000F49A
S325000004A0C51100F8C51200FCC5130100C5140104C5150108C516010CC5170110C518011434
.
.
S70500000000FA

The S0 record starts the file. The S3 records contain the data. The S7 record contains the entry address and terminates the download.

For more information on c programs to create and read s-record files, go here.

Somebody asked how to get a complete ROM image.

Supposing you wanted to do that, and that you knew all the risks that were involved and were willing to take them, here is one set of steps you might follow to get yourself an image:

1. Get and compile send.c. Get the new one if you have the old one.
2. Make a copy of your firmware file. Bring up the copy in an editor.

   Search for S11396D0, delete that line, and replace it with:

      S11396D0B2D6AEA0ACB6A7A49BB8BFD0B23CAE3055
   

   Search for S11396F0, delete that line, and replace it with:

      S11396F0A998BFF0AD6AA21CAF50AABCA38899FA7E
   

   Search for S113BFD0, delete that line, and the two that follow, and replace them with:

      S113BFD06E7E001C6E76001D6B86BFC06E7E001EDA
      S113BFE06E76001F6B86BFC2FECB5A00A1F600001E
      S113BFF06B06BFC26DF66B06BFC06DF65A00BB9AE6
   

   Save this file.

3. Download the new firmware file to the RCX. I assume you know how to do this.

   If you want a Unix firmware downloader with C source, get and compile firmdl.c.

4. Get and perhaps edit getrom.pl. As is, this program assumes you are running under some flavor of Unix and that you have Perl 5 installed. It also assumes that the send program is in your path.
5. Run getrom.pl and redirect its stdout to a file. When getrom.pl completes, this file should contain an image of the ROM in s-record format.
6. Don't forget to reinstall the normal firmware once you've got your ROM image.