View source for Object Container Streaming

{{Review}}

'''Object Container Streaming''' ('''OCS''') is the umbrella term for all the technology in [[Star Engine]] that makes a vast seamless universe possible, by which ''[[Star Citizen]]'' (and to an extent ''[[Squadron 42]]'') can provide an extremely large virtual world through which the [[player]]s can move without seeing a loading screen.<ref name="jp">{{Cite Jump Point|title=Developer Feature: The Road to Object-Container-Streaming|volume=7|month=10|startpage=3|endpage=16|accessdate=2020-04-04|author1=Christopher Bolte}}</ref>

==Related technology==
=== Server-Side Object Container Streaming ===
Server-Side Object Container Streaming (SOCS) is a streaming technology needed to expand the scale of the universe on ''Star Citizen'' servers.<ref name="pt39">[https://youtu.be/tcMtPKSe2g4 Star Citizen: Pillar Talk - Alpha 3.9]</ref> The basic concept is that if no player is near an object, the game can "freeze" that object's state. And instead of keeping the frozen entity in memory (incurring a cost), the game can serialize (using Serialized Variables) its state and store the serialized state in a database. While a client is moving through the virtual world, the server updates its view into the database to restore entities now in proximity as well as free and store away no-longer-needed entities.<ref name="jp"/>

==== Entity Streaming Manager and StarHash ====
StarHash is used to store in-game entities in a way allowing efficient searches for all entities in an area of space by utilizing a data structure called a RadixTree. The Entity-StreamingManager is then the logic-driving the StarHash-RadixTree queries to trigger loading and unloading of entities on the server, based on the positions of all connected clients.<ref name="jp"/>

==== Cross Session Persistence ====
Cross Session Persistence allows the game to store the entity in a frozen state in an in-process database that can be used by different server or machine. It is an efficient network access layer to allow the storing of the entity in a database on a different machine. Before Cross Session Persistence, the state of an entity is lost when the server crashes or is restarted (besides the state the game already persist). With Cross Session Persistence, object state will persist over server restarts and crashes (until the persistence database is wiped).<ref name="jp"/>

===== I-cache =====
I-cache is a back-end system and service that provides and manages reliable persistence of all player and entity data.<ref name="pt39"/> First major part of I-cache was completed and tested internally in March 2019. It was introduced as a replacement of the existing pCache. It is a highly distributed and fault-tolerant storage/query engine that greatly out-performs pCache. It provides an indexing and query system that can be utilized by other services for specific and complex item queries.<ref name="mrmar19">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/17036-Star-Citizen-Monthly-Report-March-2019|text=Monthly Report - March 2019|int=Comm-Link:Monthly Report - March 2019|accessdate=2020-05-25}}</ref>

==== Server Meshing ====
Server Meshing is a technology that allows the game instance to utilize multiple servers. Instead of having a  single server manage all the views, the game will distribute the individual views over multiple servers. Doing this will reduce the load on each participating server. When those servers are placed on different machines on Amazon Web Services (AWS), the game can easily scale with the player count.<ref name="jp"/>

=== Network Bind Culling ===
Network Bind Culling is the process of deciding what entities to load and unload on any client. Specifically, it works on units of Entity Aggregates, using the information of the Entity Component Update Scheduler to decide what Entity Aggregates to load or unload on each client.<ref name="jp"/>

Network Bind Culling was first mentioned in November 2016 as entity bind culling.<ref name="mrnov16"/>

====  Entity Component Update Scheduler ====
Entity Component Update Scheduler is a system designed to control the update of entity components, based on how they are spatially placed relative to the player. It allows the game to skip updates of individual components which are too far away. The same information is utilized by Network Bind Culling.<ref name="jp"/>

It was first mentioned in April 2018 as being refactored and nearly finished.<ref name="mrapr18">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/16551-Monthly-Studio-Report-April-2018|text=Monthly Report - April 2018|int=Comm-Link:Monthly Report - April 2018|accessdate=2020-05-25}}</ref>

==== Entity Ownership Hierarchy and Entity Aggregates ====
Entity Ownership Hierarchy keeps track of entities that are related to each other.  If they are related, the game treats them as one group, or Entity Aggregate. Compared to Object Containers that split the static level geometry and objects into building blocks at level design time, Entity Aggregates are consisted of dynamic entities (e.g. players, ships, etc.) that can move around in the universe.<ref name="jp"/>

Entity Ownership Hierarchy was first mentioned in March 2017 with the development of Entity Owner Manager.<ref name="mrmar17">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15833-Monthly-Studio-Report-March-2017|text=Monthly Studio Report: March 2017|int=Comm-Link:Monthly Report - March 2017|accessdate=2020-05-25}}</ref>

==== Entity Spawn Batches and Entity Snapshots ====
Entity Spawn Batches represent a set of entities that should be spawned together and only made active when all spawned.  Entity Snapshots are the values of the Serialized Variables belonging to the entity. They are also used for serializing entity state to persistence, or for ''Squadron 42'' save games.<ref name="jp"/>

==== Serialized Variable Culling ====
Serialized Variable Culling is a system that would only send network updates to clients if that client is in proximity of the updated Entity-Aggregate. It provides a noticeable performance improvement on the clients. Additionally, it was the first real-life test of running our client game-simulation with only partial information of the whole universe.<ref name="jp"/>

It was originally planned for [[Star Citizen Alpha 3.0.0|Alpha 3.0]], but was delayed to [[Star Citizen Alpha 3.1.0|Alpha 3.1]].<ref name="jp"/>

===Groundwork technology===
====Zone System====
[[File:MonthlyReport-1507-F42F3.png|thumb|300px|Zone System in the Retaliator Multi-Crew setup in July 2015]]
Zone System is a spatial partitioning system that divides all objects in a level into smaller entities.<ref name="mrjan15">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/14481-Monthly-Report-January|text=Monthly Report - January 2015|int=Comm-Link:Monthly Report - January 2015|accessdate=2020-04-04}}</ref> It is required for moving large, complex, and interactive groups of physicalized objects efficiently.<ref name="mrjul15">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/14871-Monthly-Report|text=Monthly Studio Report: July 2015|int=Comm-Link:Monthly Report - July 2015|accessdate=2020-04-04}}</ref> It replaces the old CryEngine octree spatial partitioning scheme.<ref name="mrmay15">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/14758-Monthly-Report|text=Monthly Report: May 2015|int=Comm-Link:Monthly Report - May 2015|accessdate=2020-04-04}}</ref>

It was proposed in January 2015 in a tech summit with tech engineers from the [[CIG]] studios, [[Illfonic]], [[Behaviour Interactive]], and [[Wormbyte]].<ref name="mrjan15"/> The development began in May 2015 by the Frankfurt team.<ref name="mrmay15"/> It was deployed to the ''Star Citizen'' code mainline in June 2015.<ref name="mrjun15">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/14814-Monthly-Report|text=Monthly Studio Report: June 2015|int=Comm-Link:Monthly Report - June 2015|accessdate=2020-04-04}}</ref> It began to be integrated into various game systems by the UK, Frankfurt, and Santa Monica studios in July 2015, with [[Retaliator]] (used as the multi-crew testbed) and the Cover System were the first system being integrated.<ref name="mrjul15"/> Navigation mesh and characters' paths began to be integrated in August 2015.<ref name="mraug15">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/14937-Monthly-Studio-Report|text=Monthly Studio Report: August 2015|int=Comm-Link:Monthly Report - August 2015|accessdate=2020-04-04}}</ref> The Frankfurt team extended the Zone System to better support multiple AI objects in October 2015.<ref name="mroct15">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15043-Monthly-Studio-Report|text=Monthly Studio Report: October 2015|int=Comm-Link:Monthly Report - October 2015|accessdate=2020-04-04}}</ref> The Frankfurt team rewrote CryEngine's AreaManager (ambient sound effects and reverbs) to integrate with the Zone System in November 2015.<ref name="mrnov15">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15105-Monthly-Studio-Report|text=Monthly Studio Report: November 2015|int=Comm-Link:Monthly Report - November 2015|accessdate=2020-04-04}}</ref> The UK team began work on getting Object Containers to work with the Zone System in January 2016.<ref name="mrjan16">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15187-Monthly-Studio-Report-January-2016|text=Monthly Studio Report: January 2016|int=Comm-Link:Monthly Report - January 2016|accessdate=2020-04-04}}</ref> Hierarchical culling support was added in March 2016.<ref name="mrmar16">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15285-Monthly-Studio-Report-March-2016|text=Monthly Studio Report: March 2016|int=Comm-Link:Monthly Report - March 2016|accessdate=2020-04-07}}</ref> Support for tags (Tag System) was moved into the Zone System in May 2016.<ref name="mrmay16">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15364-Monthly-Studio-Report-May-2016|text=Monthly Studio Report: May 2016|int=Comm-Link:Monthly Report - May 2016|accessdate=2020-04-07}}</ref> All data in Zone System are converted into AABB tree structure in June 2016.<ref name="mrjun16">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15408-Monthly-Studio-Report-June-2016|text=Monthly Studio Report: June 2016|int=Comm-Link:Monthly Report - June 2016|accessdate=2020-04-07}}</ref> The Tag System went through a major refactoring and optimization pass in July 2016.<ref name="mrjul16">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15444-Monthly-Studio-Report-July-2016|text=Monthly Studio Report: July 2016|int=Comm-Link:Monthly Report - July 2016|accessdate=2020-04-07}}</ref> The Area system was moved into the Zone System in November 2016.<ref name="mrnov16">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15637-Monthly-Studio-Report-November-2016|text=Monthly Studio Report: November 2016|int=Comm-Link:Monthly Report - November 2016|accessdate=2020-04-07}}</ref>

====C++ entity logic====
C++ is a multi-thread safe scripting language which is used heavily for all kinds of entity logic. It allows resource loading in parallel to the game simulation without introducing very long wait times due to required mutual exclusion, in comparison to the old Lua code which is not multi-thread safe.<ref name="jp"/>

It is unknown when the conversion started, the first mention of the conversion was that the spawning logic of spaceships AI was converted from Lua to C++ in January 2016.<ref name="mrjan16"/> It was mentioned again as a work in progress in April 2016.<ref name="mrapr16">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15325-Monthly-Studio-Report-April-2016|text=Monthly Studio Report: April 2016|int=Comm-Link:Monthly Report - April 2016|accessdate=2020-04-07}}</ref> It was mentioned as being worked on by the UK team in January 2018.<ref name="mrjan18">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/16399-Monthly-Studio-Report-January-2018|text=Monthly Studio Report: January 2018|int=Comm-Link:Monthly Report - January 2018|accessdate=2020-05-25}}</ref>

====Entity component====
Entity component represents a part of specific game behaviour. With components, the behaviour of an entity is defined by the types of components it has. Without components, all kinds of different logic tends to be interleaved in one monolithic and very complex central logic block. Since they are smaller parts, it is much simpler to make them communicate efficiently with the game simulation while we load them concurrently. Additionally, they split the monolithic game logic into more manageable parts, which played a critical role in allowing a partial roll-out of concurrent entity initialization.<ref name="jp"/> 

Basic support of entity component was implemented in the main development branch in July 2015.<ref name="mrjul15"/>

====Object Container====
Object Container is a level building block. It replaces the old game level format. Instead of developing one large level, the content creators develop a small section. The final level (or universe) is then made out of many different Object Containers. It allows the developers to split the world into many smaller building blocks when they are building a level.<ref name="jp"/>

The concept and first prototype of Object Container was introduced in January 2016. The UK team began to convert prefabs into containers in the same month.<ref name="mrjan16"/> Basic support of loading levels with Object Container instead of as levels was implemented by the Frankfurt team in the development branch in February 2016.<ref name="mrfeb16">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15224-Monthly-Studio-Report-February-2016|text=Monthly Studio Report: February 2016|int=Comm-Link:Monthly Report - February 2016|accessdate=2020-04-05}}</ref> Support for LOD mesh merging and AI navigation meshes were added in August 2016.<ref name="mraug16">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15506-Monthly-Studio-Report-August-2016|text=Monthly Studio Report: August 2016|int=Comm-Link:Monthly Report - August 2016|accessdate=2020-04-07}}</ref> The old prefab system in hangars and shops were being replaced with an Object Container in February 2017.<ref name="mrfeb17">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15790-Monthly-Studio-Report-February-2017|text=Monthly Studio Report: February 2017|int=Comm-Link:Monthly Report - February 2017|accessdate=2020-05-25}}</ref>

==== Serialized Variables ====
Serialized Variables is a process to store entities state so that they can be transferred on the network and restored in the same state on a different machine. It take the parts of an entity and put it into a special wrapper object, and the wrapper object provides ways to serialize the entity state. It allows game code to be written in a uniform style, regardless of the serialized data will be transferred later.<ref name="jp"/> Compared to the old implementation in CryNetwork, the gameplay programmer only need to mark which variables they want to replicate to the server/clients. The underlying system can detect which have changed since the last time they were sent, and which haven't, and takes care of the rest in a nice and efficient manner.<ref name="mrjun16"/>

The concept of Serialized Variables was introduced in June 2016, with the development started in the same month.<ref name="mrjun16"/>

==== Mega Map ====
Mega Map is an empty map used to load and unload Object Containers. It is used to reduce the load time, and eliminate the need of load screens.<ref name="atvmegamap">[https://youtu.be/0B0QUFSCUgo?t=846 Star Citizen: Around the Verse - What is Mega Map?]</ref>

The concept of Mega Map was introduced to reduce load time of switching game modes in October 2016.<ref name="mroct16">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link/transmission/15587-Monthly-Studio-Report-October-2016|text=Monthly Studio Report: October 2016|int=Comm-Link:Monthly Report - October 2016|accessdate=2020-04-07}}</ref> Support for singleplayer game maps such as hangars and single player ''[[Arena Commander]]'' was implemented in [[Star Citizen Alpha 2.6.1|Alpha 2.6.1]] in February 2017.<ref name="sca261">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link//15743-Star-Citizen-Alpha-261|text=Star Citizen Alpha 2.6.1 patch notes}}</ref> Support for multiplayer maps such as Arena Commander and ''[[Star Marine]]'' was implemented in [[Star Citizen Alpha 2.6.2|Alpha 2.6.2]] in March 2017.<ref name="sca262">{{Cite RSI|url=https://robertsspaceindustries.com/comm-link//15810-Star-Citizen-Alpha-262|text=Star Citizen Alpha 2.6.2 patch notes}}</ref>

==Development history==
{{Note|'''Note''': The following list {{em|does not}} represent the full picture of the development process. Development status are often not accessible by the public so it only contains information that are released by CI to the public.}}<!-- 

Only updated to 2018-04 with Monthly Report info
Feel free to add onto it
- alistair3149

-->
{| class="wikitable timeline"
|-
|2015-01
|
*Zone System and entity streaming were proposed<ref name="mrjan15"/>
|-
|2015-05
|
*Zone System was being developed, and entity streaming was being investigated<ref name="mrmay15"/>
|-
|2015-06
|
*Deployed Zone System to ''Star Citizen'' main development branch<ref name="mrjun15"/>
|-
|2015-07
|
*Started implementing Zone System in the main development branch<ref name="mrjul15"/>
*Added basic support of the entity component system in the main development branch<ref name="mrjul15"/>
|-
|2016-01
|
*Introduced the concept of Object Container<ref name="mrjan16"/>
*Started converting prefabs into containers<ref name="mrjan16"/>
*Started getting containers working with the Zone System<ref name="mrjan16"/>
*Converted Spawning logic of spaceships AI from C++ to Lua<ref name="mrjan16"/>
|-
|2016-02
|
*Added basic support to load levels with Object Container instead of as levels<ref name="mrfeb16"/>
|-
|2016-03
|
*Added Hierarchical culling support to Zone System<ref name="mrmar16"/>
|-
|2016-05
|
*Added support for tags (Tag System) into Zone System<ref name="mrmay16"/>
*Added ability to reuse entity IDs<ref name="mrmay16"/>
|-
|2016-06
|
*Started work on Serialized Variables<ref name="mrjun16"/>
*Converted data structure to AABB tree in Zone System<ref name="mrjun16"/>
|-
|2016-07
|
*Major refactor and optimization of Tag System in Zone System<ref name="mrjul16"/>
|-
|2016-08
|
*Added Object Container support for LOD mesh merging and AI navigation meshes<ref name="mraug16"/>
|-
|2016-10
|
*Mega Map was being investigated<ref name="mroct16"/>
|-
|2016-11
|
*Moved Area system was moved into the Zone System<ref name="mrnov16"/>
*Started work on Network Bind Culling<ref name="mrnov16"/>
|-
|2017-02
|
*Implemented singleplayer maps support for Mega Map in [[Star Citizen Alpha 2.6.1|Alpha 2.6.1]]<ref name="sca261"/>
*Started work on replacing old prefab system in hangars and shops with Object Containers<ref name="mrfeb17"/>
|-
|2017-03
|
*Implemented multiplayer maps (not including PU) support for Mega Map in [[Star Citizen Alpha 2.6.2|Alpha 2.6.2]]<ref name="sca262"/>
*Started work on Entity Owner Manager<ref name="mrmar17"/>
|-
|2018-04
|
* Refactored Entity Component Update Scheduler<ref name="mrapr18"/>
|}
{{Timeline styles}}

==See also==
* {{Link RSI|url=https://robertsspaceindustries.com/spectrum/community/SC/forum/3/thread/road-to-dynamic-server-meshing-tech-overview-with-|text=Unofficial Road to Dynamic Server Meshing - Tech Overview with Explanations}} by unobtanium on Spectrum
* "Developer Feature: The Road to Object-Container-Streaming". ''[[Jump Point (magazine)|Jump Point]]''. Vol. 7 no. 10. pp.&nbsp;3–16
* [https://www.youtube.com/watch?v=hqXZhnrkBdo Star Citizen's Next-Gen Tech In-Depth: World Generation, Galactic Scaling + More!] by Digital Foundry
* [https://youtu.be/tcMtPKSe2g4 Star Citizen: Pillar Talk - Alpha 3.9]
* [https://youtu.be/yEwik6g0Lwo?t=309 Star Citizen: Pillar Talk - Alpha 3.8]
* {{Link RSI|url=https://robertsspaceindustries.com/spectrum/community/SC/forum/50259/thread/how-is-server-meshing-possible/2110241|text=Clive Johnson response of how is server-meshing possible}}
* [https://www.youtube.com/watch?v=EU0sMZbhqoU& Star Citizen: Around the Verse - Performance and Optimization] {{trx|url=around-the-verse-performance-and-optimization}}
* [https://youtu.be/E0YYKIjEC0E Star Citizen: Reverse the Verse LIVE - Object Container Streaming] {{trx|url=reverse-the-verse-7th-september-2018}}
* [https://youtu.be/0B0QUFSCUgo?t=844 Star Citizen: Around the Verse - What is Mega Map?] {{trx|url=around-the-verse-what-is-mega-map}}
* {{Link RSI|url=spectrum/community/SC/forum/3/thread/what-to-expect-from-ocs-abs-in-3-3-faq-guide|text=What to expect from OCS/ABS in 3.3 (FAQ/Guide) by @UrsaPolaris}}
* [https://www.youtube.com/watch?v=DKa4j-xxy2I Star Citizen 3.3 | OBJECT CONTAINER STREAMING - Major Core Update] by BoredGamer

==References==
{{Reflist}}


{{Short description|Technology behind Star Citizen that eliminate loading screen}}
[[Category:Development]]