Ogre Paging as a core feature - design notes(ogre paging 设计相关说明)

by sinbad » Tue Mar 03, 2009 7:39 pm
Hi folks,

Thanks to a co-operative project that I'm doing with a client, I'm going to have the opportunity to spend time doing 2 rather important things over the next couple of months:
1. Implementing a brand-new terrain system
2. Promoting the concept of paging to a core feature

This post is about #2 - I'm not going to talk about the specifics of the terrain here, but I mentioned it simply because I wanted to put it in context. It's going to be a paging-aware terrain, but I absolutely did not want to make the paging specific to terrain only. This deserves to be a core feature, so step 1 is defining what that paging system will look like.

Firstly, a few principles. As you know I like to keep things flexible, not make any assumptions I don't have to, and promote extensibility. That's why you'll probably look at the designs I'll post here and think they're overkill - but the intention is to provide a good base for the future. The kind of paging I want to see will:

1. Not assume that the system ever has complete knowledge of the entire world - new parts of the world must not just be capable of loading in based on a predefined structure, but of being 'discovered' dynamically too
2. Allows multiple strategies for decided the lifecycle of pages - as-the-crow-flies distance is an obvious one, but we must also allow for other approaches (particularly for dense, occluded scenes it might be region connectivity and occlusion stats)
3. Treats the paging strategy and content of the page as orthogonal concepts, ie you can mix and match them
4. Does not assume regular sized pages, or that pages occupy unique, non-overlapping regions in space
5. Supports varied structural page setups - grids for terrain perhaps, but 'snake' like areas for cavern systems.
6. Transitions must be possible between pages without type-specific knowledge (vital for supporting varied scenes, e.g transition from cave system, to 2D terrain, to space)
7. Pages must be able to be loaded in a separate thread, with the same modes as available for resources (full threading or semi-threading).
8. Pages should be able to have multiple LODs in which the physical content changes. Content within the page may have LOD of its own, but we should also allow the physical content to change too at a higher level
9. For now, no support for page-relative co-ordinate systems. To cope with precision in large worlds, double-precision coordinates and camera-relative rendering can be used. Page-relative co-ordinate systems require a larger scope change which will be reserved for a later version.
10. In general, paging support must not require major changes to SceneManager, those are still reserved for 2.0

So, this is pulling in a number of concepts from things that were scheduled for Ogre 2.0, but ring-fencing them so that they can be incorporated without requiring the wholesale changes to the SceneManager that will be required for that (hence no page-relative coordinates).

Here's the first design diagram I've come up with:


paging_design1.gif (18 KiB) Viewed 9692 times

PageManager
This primarily a central registration point for extension classes, such as the PageStrategy, PageContentFactory, and the place to access the currently loaded content (PagedWorld). I haven't decided yet whether I will hang this off Root in the core, or make it a separate library which a user references if they want.

PagedWorld
This is basically the data-driven part of the structure. A PagedWorld is a collection of world content which can be loaded from a file or defined in code. What's important is that it does not need to create everything that's underneath it at load time - minimal set up is creating the PagedWorldSections (see below), and optionally PageEntryPoints if there are 'spawn points' that need to be defined (this is important if it's not possible to derive what page needs loading simply from a camera world position). With these in place, requests can then dynamically load the rest of the parts that are needed on demand.

PagedWorldSection
This is a section of the world which conforms to a particular paging strategy. For example, an outdoor 2D terrain section (and by 'terrain' I means structurally 2D, not that it defines terrain content) might use one strategy, bounded in one way, a space section of the world might use another strategy, a complex indoor area might use another one. Only a high-level description is held here - a section ID within the world, general bounds information, an optional list of predefined entry points (with just a position and a page identifier - especially important if the strategy cannot translate a position directly into a page with no context).

PageStrategy
This class is responsible for determining how pages are loaded and purged, and optionally deriving a page just from a world position (and a PagedWorldSection). Whenever a camera moves, the PageStrategy is responsible for deciding how that affects page requests, within the context of a PagedWorldSection. This might be a grid system, it might be a case of using connections between pages to determine traversal depth. Whatever the case, given either a global position, an existing page, or a PageEntry point, this strategy must be able to determine what pages need to be requested or disposed of.

PageStrategy is designed to be user-extendable.

Page
This class collects together all representations of a page of data. It defines the page, but doesn't actually contain any data of its own. Rather, it just contains a list of LOD levels and a reference to a LOD strategy. A page has a unique ID within the world section.

PageLOD
This is to allow the content of a page to load more than once if need be. You may, for example, want to have a page which loads in multiple stages, as a very simply distant definition with basic content, and as more detailed definitions closer up. I intend to re-use the pluggable LodStrategy classes to represent the transition rules.

This now defines the uniqueness of our units of loading for pages: "WorldID:WorldSectionID:PageID:PageLODIndex". I will probably put a pluggable translator system in place so that takes these numbers and translates them into a unique resource to load from - which could be as simple as a file ("myworld_sec1_p1234_0.dat"), or could be translated into an offset in an existing packed resource file or something.

PageContent
While PageLOD is the unit of loading, you may still want to include multiple types of content. For example in a page you might have a chunk of terrain, some static geometry, some custom user data (like trigger regions, world objects etc). Therefore within a PageLOD you can have multiple PageContent instances, which all get loaded in together but each of which can have custom data formats and routines for bringing them online. At load time they will be instantiateed through a factory system based on named loaders.

Each of these PageContent types is free to do their own version of LOD too - for example if the content includes creating some Entity instances and they have Meshes which use LOD, then that will still apply. For the terrain, each page will have it's own subdivisions and LOD. But the assumption is that this is all in-memory LOD, suitable for lightweight dynamic adjustment, whilst the higher-level PageLOD is much more heavyweight and actually changes the content, changing the memory footprint etc (and also being potentially threaded). That's why there are 2 types of LOD here, it's not duplication.

PageContent and PageContentFactory are designed to be user-extendable.

Phew....
Ok, that's one of the largest posts I've ever made. I hope that looks good to people - it's still very much subject to refinement so nothing here is set in stone, but I've spent quite a bit of time thinking about the overall structure, what kind of functionality I want to support, and where the extension points are so I'm pretty happy with so far. Comments are welcome!


Things I forgot to mention:

1. Yes, this means I'll be defining a new 'world' file format, although actually it's a meta-format, since the data required to build the PageContent instance is designed to be variable. I will, however, be definining content implementations for terrain and probably a bunch of regular scene objects, so this could be seen as replacing dotscene to a degree, although there's no reason why dotscene couldn't still be used too. By nature the format I'm thinking of is a binary one, with possibly a new XML serializer (although the pluggable content data will limit how much that can be broken down beyond CDATA sections)
2. Paging will be entirely optional. PageContent will at the end of the day be inserting data into a SceneManager. People who don't need paging will not be forced to use any of this, which is why I'm considering making it a separate (but still core) library.
3. PageConnection is about knnowing about the neighbours of a given page - this is particularly important when going between PagedWorldSection instances. I'm still a bit wooly about the specifics, but it's a lot like the zone connections in PCZ, just much larger granularity
4. I probably need to think about PageLOD fading / smooth transitions. Not top priority for the moment, it's just important to have the concept of multiple PageLOD levels in there for now

Noman wrote:Is the terrain system going to be open source as well or is that private? If such a system is written, and no example paging geometry plugin is available on top of it, it will be hard for users to create their own paged systems, kinda like giving the SceneManager interface without specific implementations...

Everything will be open source, the same core Ogre license. The initial priority for concrete implementation will be the terrain-related classes, since that's my client's primary concern initially, but in order to maximise the investment in Ogre here I'm trying to make this a much more general, strategic chunk of functionality. I've reached a co-operative agreement with my client so that everyone can benefit long-term. It's great when you can find customers that appreciate the benefits of operating this way!


Having a bit of a hard time deciding whether I think paging should go into the core or as a plugin... Close call...

Well, I'm thinking a separate (but still in the Ogre core) utility library that users link to in addition to OgreMain, and tie the two together. So it's not really a plugin per se, since users will be using it directly.

(Which subclasses will the terrain system implement etc)

Take a look at the diagram for classes with 'Terrain' in them

Essentially the terrain will provide a connection to the paging system via TerrainPageContent primarily. Also GridPageStrategy is very closely related, although it can be used for more than just terrain so will not be directly associated.

lf3thn4d wrote:1. Would pages be able to load up and cached in memory before it is actually used?

Yes, the same kind of system as Resources is envisaged, with an ability to either fully load (fully threaded, including rendersystem) or simply page into memory (semi-threaded, via prepare() like functionality) in the background.

2. When does a page gets unloaded?

When the PageStrategy says it should. This is entirely customisable - it might be a simple grid distance, or it might be when the links from the page any camera is in is more than 'N' connections (plus any other metrics you might want to use) away.

3. Is it possible to specify strategies that works like portal?

That's precisely what PageStrategy is for. I put in a placeholder called 'ConnectivityPageStrategy' to show the ability to extend into this kind of area. To begin with I'll be starting with the grid-based functionality, but I wanted to make sure there was room to extend to other types.

KungFooMasta wrote:Very exciting! Also consider the case where there is paged terrain and physics involved. For example would the scenario where a large boulder on top of a hill (far from viewing distance) be able to roll down terrain into viewable regions? Maybe its not possible to do paging with this kind of situation. Just thought I'd throw this out there as an idea to think about.

This is why there is PageContent (which can include any content you need to load, might not even be graphical), and also PageLOD. At some point you're going to have to say that you'll freeze the physics at a certain distance, you can't simulate the entire world all the time. You could do that via a PageLOD. I'm actually thinking of moving the PageLOD below a PageContentType class or something like that so that multiple LOD strategies can be specified for different content. But anyway, all paging systems have to at some point deal with what happens with dynamic objects, and the PageLOD is designed to give you hooks to make that decision. At some point you will have to accept that you will lose track of a physics object once it gets a certain distance away - remember, we're talking pretty large distances here.

Would a paging solution worsen performance if there is no need to do paging? (ie a paging system has a world that is small enough that it fits on a page)

Well, you just wouldn't use it then. Like I say, the paging is supposed to be something you use if you want, it's not a required interface to use (hence why I'm increasingly thinking of making it a separate lib from OgreMain).

Praetor wrote:@sinbad This is really critical I think for the future. We were looking at a lot of this for 2.0, but it would have to be started some time and I think it's great that it is starting now. Having this functionality in the core puts Ogre up one more level on the power scale. It's definitely true that as we move further into general scene management we start to approach that blurry line of "are we trying to do too much?" But I wouldn't worry here, I think we're still well on the right side of that line with this.

Yes - this doesn't go to the extent of 2.0; there's no page-relative coordinate systems, I'm not attempting to tackle the fully threaded scene management yet, and essentially the entire paging system is just a 'feed' to the existing scene content structures. But the intention is to establish a base feature set which works with the way the SceneManagers currently operate but yet still delivers a lot of value. When the project came up and I really sat down to start thinking about it, I increasingly felt I could deliver a decent 'stepping stone' without having to undermine the 2.0 vision, but also without having to change the main interfaces too significantly. Being able to pull off a more manageable chunk like this and digest it is nice, and the fact that it's partially funded also means I can dedicate more time to it than I would be able to otherwise.

Praetor wrote:You didn't go over the PageRequestQueue but I'm guessing that's the point where applications "hook" into the system to intercept page load requests. If I were implementing this system in an application, what's the minimum I would need to get a paging system running? Meaning, what classes do I have to subclass, or listeners attach? I'm guessing the main components to interact with for control are the PageWorld and the PageContent. Am I getting the general idea?

The intention is that the PageStrategy pushes things onto the request queue. The idea is that this is mostly data-driven, since by nature all of the data is going to need to be 'discovered' on demand from previously saved data. Real-time construction will also be possible, but essentially in that situation you're overriding the PageStrategy because you're deciding when a new page gets constructed, and so this isn't the 'normal' usage. Assuming the data is already defined, then you would simply load a base world file, which would define the general world sections (which are defined by their type of paging - you could just have one here), which would establish a PageStrategy that should be used. You then have to place a camera, and the PageStrategy will decide, in conjunction with the world section which must include some strategy-specific instance data, what page(s) to request. So if your data is already created, you really only *need* to talk to the PageWorld and maybe the PageEntryPoint if you want a camera position suggestion. In practice however, you will probably have a need to drill down into the PageContent too if you want to do specific things like get a list of objects that are on the page, or perform terrain enquiries, or modify data in the case of authoring. All PageContent sublcasses will be required to be able to serialize themselves so that they can be saved / loaded as part of the binary page data.

@Jabberwocky:
Ok, firstly for global data like you describe, I wouldn't be using a paging system at all. Or, if you do still need some kind of paging, then you can use a different PagingStrategy to reflect how you want that data to be maintained. I made a specific design assertion that multiple PagedWorldSection instances can exist at one time, and that they do not need to be spatially separate. In addition, PageContent can be any type of content at all that needs to be paged in and out of memory, not necessarily graphical content. Finally, as I mentioned above I think I'm going to make PageLOD specific to content type so that per page, once you've decided that you're interested in the page as a whole, you can decide that each set of PageContent within it changes LOD based on different criteria. Essentially the PageStrategy is in charge of deciding when a Page is of interest at all, and therefore loads some base page definition which will include content types and LOD strategies for them. The LOD within the page (and now content type) is then required to decide when LODs of that content change, and what the effect of that will be.

I didn't make it clear but requirements for saving will be a part of the API at all levels.

When I talk about 'files', in regard to the world, the pages and the page content, I'm really talking about a load hook. I'm not sure where to put the implementation for this yet, but I the request will be placed on the PageRequestQueue and 'something' will process it. At the base level options for synchronous per-frame checks and a thread to do this are obviously on the table. When a request is being processed though, there must be a flexible way of mapping the request to an implementation - whether that's looking for a separate file for each request, or pulling data from a stream, or defining it procedurally. These implementations should be able to load / build the data then queue the result back for attachment to the world (which will be synchronous, and if we're using semi-threading then no GPU resources will be created in the separate thread).

Regarding dynamic state, I think it's important to think of the page system as a manager of 'snapshots' of data. Once a page is loaded, it may well create dynamic objects, which will just be regular MovableObjects as far as Ogre is concerned. KungFooMasta's point about a boulder rolling across a page boundary is illustrative - unless the page in which the boulder starts was already loaded, the boulder could not roll into an active page because it doesn't exist yet. This is an unavoidable simplification and I think pretty easy to understand, and people will structure their paging to make this as least noticeable as possible - what's more difficult is unloading. If your page spawned a dynamic object and the page get's unloaded, what should happen to the dynamic object? Well, essentially this depends on how mobile the object is. If it moves a bit, but generally stays in the local region, then destroying it when the page is removed probably isn't a problem. If you can carry the object all the way across the world with you (e.g. you drive a vehicle that was spawned in at a page), that could be more of a problem!

I had earmarked ObjectPageContent as a placeholder for spawning general MovableObject instances in a page. My suggestion would be for the default behaviour to be to destroy the objects when their page is unloaded, but to have the ability to detach an object (or perhaps its parent node) from a page, at which point it is essentially immortal until you manually destroy it, just like a regular Movable. I could also allow an object to be re-attached to a page (e.g. when you put it down and walked away) so that when that page (might be a different one) gets unloaded, the object then gets destroyed. Of course, this would require the user to be aware of pages, but that's how I envisage it being possible.

Essentially each PageContent subclass is responsible for how it manages the objects it creates in response to paging activities. Those objects do not necessarily have to remain attached to the paging system forever.

Hope that covers everything!

tuan kuranes wrote:1) would it be possible to colorize/emphasize user "extendable"/"mandatory extendable when paging" class in the graph ?

I can certainly colour code the extensible bits, I'm not sure what you mean by 'mandatory extendable' though - the idea is that if you use the content types and strategies already provided for (which for terrain, will already be there), then you don't have to extend anything if you don't want to.

if user provide page content, will he be able to tag mesh/renderables as world fragment ?

Essentially the PageContent implementation can do absolutely anything it likes, so long as it can implement load/unload/save. The intention is not to assume anything about what the 'content' might be.

just to be sure, we can have mutiple pagelod at the "same time" for the same page ?

That was what I was alluding to about 'page lod transition'. I haven't quite decided how to handle that yet (the base terrain won't need it, since only internal chunk LOD will be needed initially) but yes I would like to have the concept of an overlap between active LODs to allow for some kind of smooth transition.

Does "Camera" still belongs to scenemanager ?

Yes, it's important that nothing in the way SceneManager functions is changed here.

Will renderables be shared/shareable across a page world, or renderable will be per scenemanager/per pagecontent ?

Again, nothing will change from the current setup here. So standard MovableObject instances will be specific to SceneManagers, but users are free to create custom Movable/Renderable subtypes that are owned however they like, just like now.

would it be possible to have multiples pagemanagers?

In theory yes (since I intend that the user will instantiate PageManager themselves rather than it being a singleton, reinforcing it's 'utility' pattern), but in practice I'm not sure why you would need to. The PageManager is literally just a registry of strategy implementations, content factories, and loaded worlds.

5) for sure, all those "user extendable" introduces many caveats (a "lib", Root core)
Why not the just only the "Event" way ? I guess there will be PageContentEvents, PageLodEvents, etc. anyway ?

Mainly because these events cannot be predicted - because if you want to implement another way of deciding when a page is created / destroyed then the event that would trigger this is based on entirely different criteria, and therefore we cannot predict when the event should be raised. There will certainly still be events generated, but for full flexibility we need to provide pluggable ways of raising those events in the first place, hence the patterns.

Also, I was looking for a declarative approach rather than a procedural one. I think this fits with the most common requirements for paging. However, if a user wants to inject pages into the system themselves based on low-level information like camera movements instead of implementing a PageStrategy / PageContentFactory, then that should be possible too, but really they could be doing that already


http://www.ogre3d.org/forums/viewtopic.php?f=4&t=48320