Quick Update 1999/05/24: I've gotten an initial prototype started in Java. If you'd like to test this, send me mail and I'll point you to it.
This will most likely contain a growing spec, and notes on that spec. For now, here is a minimalist entry.
A while back I had started on a system to distribute renderings among different computers, mainly for POV-Ray. At the time I started it was a little slow going, as I only had the one computer at home. I had gotten my "R-Farm" system working with an initial prototype stage, but then shelved it for a while due to various demands on my time.
With my taking on Technical Director for the IMP and the needs brought up by that project, the distributed rendering problem was no longer just an acedemic excersise, but a real issue. Thus, I dusted things off and jumped back into the fray.
My initial R-Farm was intended as more of a local render-farm manager, but the needs of the IMP required something more like a WAN than a LAN, so I'm working out a protocol & set of formats to serve that purpose.
Get sources to people so that they can render things and return them.
For real, what we need to do is first determine WHAT to do, then determine HOW to do it. That means step back and look at things from a high level. Remember WHAT and WHY, then later do HOW.
Well, here we go with a little more info. Later things will get worked out into a draft, this is a more of just a preliminary description. (Also, please ignore any spelling or grammer mistakes in this version, as I am typing straight in late at night and just need to get something out for initial review)
Provide a system to facilitate distributed graphical rendering via render-casts (or R-Casts). In a high-level view, the needed renderings are set up into R-Casts, which will allocate and distribute the required assets to all particiapating clients. The clients will process the renderings and return the final results back to the designated final destination(s).
The process will be separated in to several distinct layers, with clearly defined communication and formats exchanged between layers. It is important that different implemenations of the different layers work correctly and interchangeably with each other
It is important to support a wide base of different platforms for the end clients, and to support different communication pathways to the end clients. An end-goal is to support all client platforms that have POV-Ray on them.
The end clients might be either individual users, or local render farms. Sending data to and from the end clients may take different forms, but initial gateways should be designed for Mail, HTTP and a custom TCP/IP protocol. Mail and HTTP support should be addressed first.
The system should provide for transfering a minimal ammount of redundant data. This will be facilitated by tracking assets required and matching with those already present at the end user. Only missing assets will need to be sent.
The system should also be very fault tollerant, assuming that any communication channel and any given end user mail be delayed or fail at any time.
The data format used to send assignments, list needed assets, list assets present at the end user, etc. should all be in a human readable format. It should also be in a format that is optimized for machine processing. All characters used should stay within the US-ASCII 7-bit range.
| Scene creation | Movie Creation | ||||||||||
| | V | ^ | |
||||||||||
| Scene storage | Frame storage | ||||||||||
| | V | ^ | |
||||||||||
| Package assignment | Frame return | ||||||||||
| | V | ^ | |
||||||||||
|
|||||||||||
One point to remember is that the rendered frames may not be expected to return to the same server that the R-Cast package was recieved from. This allows for different servers to be setup for different purposes, especially in the final stanges when the final frames may grow quite large and might need to be stored at several different locations.
This then implies the need for different servers implementing the same layer to coordinate with one another.
Initial development needs to focus on the final transfers from server to end client and from end client to server. Initially, the server can be setup manually. In the case of using mail for transport, the entire functionality of the servers can first be done completly by human effort, and then later automated.
This then comes to what data is sent over that channel of communication. The format of the data should be either similar to that used by majordomo, or in XML. For the mail transport, using the simple format would probably be the best for a first implementation.
The data sent out should be in a .ZIP file, using only the non-patented compression (and thus, be a valid JAR file also). Contained in the ZIP file should be a file named "r-cast.txt" that contains the actuall information on what assets are needed, what frames are to be rendered, etc. For the mail transport, a duplicate of this information may also be in the message body itself. The ZIP file should also contain a manifest file that follows the JAR spec and lists all files included in the.
All files should be indentified by a MD5 signature. Note: for text files, accomodations will have to be made for different line-end conventions.
The client, upon receiving the package, can then determine if any additional assets are needed, and request those if necesary.
Well, we're going to want to have a link button. For now I have one hacked together very quickly until I have more time for a little artistic effort.
People who want to be bugged about determining details:
Notes:
Model naming and model part naming. Include file versioning major/minor Declare version in calling code, check in include 8.3 naming issues Some scenes require 3.02, some require 3.1 Number frames based on scene. SMPTE times? * So, for now, we target 320x240, with the top and bottom 30 lines potentially cropped.