Scenes Problem Analysis
What am I working to fix?
- I need to be able to store collections of assets together with their component data included (aka a scene or a level). I also need to be able to
serialize that data to disk and load that data to disk.
Important: Don’t solve problems I don’t have. Trying to predict future problems and
trying to solve them just makes more problems and complicates things.
Context:
Why am I solving this problem?
- I need to solve this problem, so I can save my work to disk and load it at later times. This allows me to maintain my
progress on a project and come back to it to work on it more or load the scene to show someone. Also, games are usually
comprised of multiple different levels or areas to go to, so separating that data into collections to be loaded when
the player gets to that area (as opposed to loading the whole games assets at once) will help with keeping
memory lower and load times smaller.
Identify and Balance Constraints:
How much time to I have to make this?
- I am giving myself about two weeks to work on this. (It is about to be finals week, or it would be less)
How much research do I have to do on this?
- I have to do some research on how open world games stream in data vs loading scenes, so I can ease the transition when
I do need to change to streaming data in and out. Also, doing research on what a scene consists of typically would
give me a good baseline to work from when developing a scene concept within the context of Astral Engine.
How much memory can this take up?
- The scene will roughly take the same amount or a little bit more of memory compared to the ECS memory usage.
This is because the scene will be basically composed of an ECS for entity data and of asset handles for unloading assets
when the scene is unloaded. The asset handles though will take a smaller amount of memory compared to an ECS because
asset handles are only 4 bytes
What is the performance budget?
- The scene itself is just a container for data, but in terms of serializing and deserializing scenes, it should be quick.
I am going to give myself a budget 1 ms for scenes with less than 10 objects, 5 ms for scenes with less than 50 objects
and the performance budget will scale like that with adding 1 ms for every 10 objects. For how I came to that number, I just
mentally compared how long a typical frame takes to render (16.67 ms) for 60 fps and came up with that budget from how quick
I expect the scene to be serialized or deserialized to prevent stutters or latency of any kind. To be clear, most of the
budget will come from actually loading the assets from disk. Also, note that this budget is intended for release builds and
that the actual scene load time will depend on how big the assets are and how many assets need to be loaded in vs are already
loaded from previous scenes.
What is the latency/overhead requirements?
- The actual latency felt by the user should be relatively small. The performance budget was in the milliseconds, so
loading scenes should be done pretty quickly. Scene loading will block the main thread (for now), but it should be done
pretty quickly and the user will have to wait for the scene to be loaded anyways before doing anything.
Will this be handled differently in different builds (Debug vs Release vs Dist)?
- Loading and unloading scenes as well as how scenes store data won’t change from different builds. The only difference
would be that any logging related to scenes will be turned off.
What are the inputs and the outputs?
-
The inputs will be a scene file on disk that contains the filepaths of assets in this scene, their scene load ID, and an
ECS for the scene that comes populated with data that the user generated from the editor previously or an empty ECS if the
scene is new.
-
The outputs will be that the assets will be loaded into RAM and the scene will start to be rendered as well as ECS systems
start to act on the scene data. When the user is done, the scene data will be serialized onto disk.
What would the file format be?
-
For scene serialization to a file, I could do something human-readable like yaml or json while working with the editor
for development ease but when you cook the game into an executable, change the scene files to binary files for performance
because after that point, the scene file won’t be changing. For the actual format of the file will roughly the following:
- Header that contains the version of the scene file format and sanity flags/buffers to ensure that the file being loaded is in fact
a scene file generated by the engine
- a mapping of a scene ID to filepath of an asset
- an ECS with scene IDs filled in for asset IDs
Value:
Identify specific, concrete, measurable value.
- The scene system will allow users to split their game or application’s resources into groups of assets that are
only loaded when they are actually being used in the game or application. This helps by reducing the initial load time
when the game is first started as only the first area’s assets will be loaded as opposed to the whole games assets
being loaded up front. In addition to load time, this also allows for lower memory usage as only the assets being used
(which would be the first scene’s assets) will be loaded into RAM as opposed to the whole application or whole game’s
assets being loaded into RAM.
How much time is this problem worth?
- The scenes system is worth the 2 weeks that I am planning to take to implement it. It is actually probably worth more
than that because it is very important, but I think I will only need about 2 weeks to get it done.
What is the value of solving this problem?
- The value in solving this problem is the user will be able to save and load scenes/levels in addition to scenes allowing
for quicker initial load times and lower memory usage.
Cost:
What is the cost of solving the problem?
- Predicted development time cost: 2 weeks (hopefully)
- Maintenance cost: Potential bug fixes might be needed and eventually, the scene system might have to be adapted or adjusted to support open world things like
partitions and streaming data in and out later down the line.
Platform:
How will this handle the platform targets?
- There won’t be any OS specific code. The standard library handles file path differences between OSs.
How will this interact with shared resources (RAM, SSD)?
- Scenes can be serialized to disk as well as loaded from disk, but the actual scene file will be relatively small.
Data:
What is read and written over time? What is the access pattern?
- On the initial load of the scene, the file paths of the assets in the scene will be read in by the asset manager and
loaded from disk if needed. Over time however, there isn’t really an access pattern in the scene system itself. Once
a scene is active, the ECS which is contained within a scene will be read from when systems start like the physics
system or rendering system.
When is data accessed statistically relative to other data?
- Data from the scene is accessed when data from the ECS is needed by ECS systems that act on the entity data. Other than
that, only the asset filepaths are accessed on the initial loading of the scene.
What data values are common? Outliers?
- ECS data is commonly accessed, and that is going to be the hot section of scene data
Possible Future Features:
- Streaming data for open worlds