Spatial maps are a critical part of the Magic Leap ecosystem as they help Magic Leap enable interactions between digital content and your actual physical surroundings. Device sensors continuously scan your environment, process that information, and use it to create three dimensional representations of your area (which we refer to as spatial maps). The end result is a mapped environment that enables apps to render digital media in your field of view as if it were really in front of you. Spatial maps can include the following three different levels of detail, based on the choices you make:
World Features
World Features are 3D representations (also called “point clouds”) of the world around you, and serve as the basic building blocks of a spatial map. World Feature points can be used as references to place and persist content (so you can come back to that content later, at the same place). Additional information about World Features can be found in the “More about World Features” section below.
World Models
World Models add more detail to your spatial maps, including dense mesh data and planes. The result is spatial maps that provide much more accurate representations of the world around you, enabling digital content to respond to the environment in a more natural way. Additional information about World Models can be found in the “More about World Models” section below.
World Understanding
World Understanding allows Magic Leap devices to recognize some types or classes of objects in your environment, such as the ability to identify chairs and posters, and include data about those objects in your spatial maps. Applications can then access object data in your spatial maps (with your permission), and use it in a variety of ways, including to make your digital content fit even more seamlessly into the world around you. Additional information about World Understanding can be found in the “More about World Understanding” section below.
More about World Features
World Features are 3D representations (also called “point clouds”) of the world around you, and serve as the basic building blocks of a spatial map. World Feature points can be used as references to place and persist content (so you can come back to that content later, at the same place).
How World Features Work
Computer vision algorithms use images collected from the peripheral cameras on your Magic Leap device to extract features from those images. These new features are then compared to previously stored features and merged to create one set of features that represent your surroundings. The final set of World Features is a spatial map at its most basic level. The processing to extract World Features from peripheral camera images is done locally on your Magic Leap device. If you choose to store your World Features in a Personal World or contribute them to Shared World, only the resulting World Features spatial map will be sent to Magic Leap’s cloud.
What About People in My Surroundings?
Spatial maps only reflect information from the parts of your environment that are stationary. If someone walks through the room while your device is spatially mapping the space, they will be excluded from the spatial maps. If someone is sitting in the room while your device is spatially mapping the space, that individual may be included in the spatial map created, but the spatial map is not designed to reflect a level of detail sufficient to identify that individual.
Your Choices
Magic Leap offers three different storage options for your World Features data. You can choose to store your World Features On Device or in a Personal World, or to contribute them to Shared World.
- If you choose to store your World Features On Device, World Features will still be available to you and your applications. However, there is a limit to the amount of spatial maps you can store on your device.
- If you elect to store your World Features in the cloud (either a Personal World or contributed to Shared World), you will reduce the mapping time required by some applications and will be able to store more spatial maps.
- Using Shared World means your device may not need to spatially map new areas you enter if other Shared World users have mapped them before you. In addition, if you contribute your World Features to Shared World, it will improve shared and other multi-user experiences by allowing both you and other users to utilize the same World Features - enabling you to see digital content in the same physical location.
You can change your selection around where your World Features are stored at any time in Settings. However, if you decide to contribute World Features to Shared World and then later decide to change your settings, the World Features you have already contributed to the collective spatial maps ordinarily will not be deleted. This is because once you contribute World Features to Shared World, they become part of the combined Shared World spatial maps, and are not directly associated with a particular device or account. If there are areas you do not want to be part of the collective spatial maps available to Shared World users, we encourage you to change your settings before entering those areas.
Also, please note that apps will be able to access the spatial maps of your surroundings, including World Features, while such apps are running on your device, even while in the background. Apps need this information to display within and interact with your environment.
More about World Models
World Models add more detail to your spatial maps, including dense mesh data and planes. The result is spatial maps that provide much more accurate representations of the world around you, enabling digital content to respond to the environment in a more natural way.
How World Models Work
World Models add additional components to your spatial maps, including:
Dense Mesh Data. Dense mesh data is a triangular geometry that closely approximates the structure of the visible walls and objects around you, such as furniture.
Planes. From dense mesh data, large flat surfaces or planes can be derived. This helps identify walls, floors, and optimal surfaces for placing content. Dense mesh data and planes additionally enable occlusion and collision consistent with your environment.
What About People in My Surroundings?
Spatial maps only reflect information from the parts of your environment that are stationary. If someone walks through the room while your device is spatially mapping the space, they will be excluded from the spatial maps. If someone is sitting in the room while your device is spatially mapping the space, that individual may be included in the spatial map created, but the spatial map is not designed to reflect a level of detail sufficient to identify that individual.
On Device Storage
World Models are designed to be processed and stored locally on your Device.
Also, please note that apps will be able to access the spatial maps of your surroundings, including World Models, while such apps are running on your device, even while in the background. Apps need this information to display within and interact with your environment.
More about World Understanding
World Understanding allows Magic Leap devices to recognize some types or classes of objects in your environment, such as the ability to identify chairs and posters, and include data about those objects in your spatial maps. Applications can then access object data in your spatial maps (with your permission), and use it in a variety of ways, including to make your digital content fit even more seamlessly into the world around you.
Magic Leap will continue to improve and develop new classes of objects for detection in upcoming releases. You can find details about new classes of objects in the release notes for each Lumin OS release.
World Understanding requires cloud processing using Shared World and is not currently available when you save spatial maps only on device or store your World Features in a Personal World.
How World Understanding Works
When you elect to use World Understanding, your Magic Leap device will send picture camera color images and depth sensor images to our World Understanding cloud service. The World Understanding service runs computer vision algorithms to identify objects in your space. Once objects are identified, the World Understanding service generates unique labels and other data about those objects, which are placed into your spatial maps (whether in a Personal World or contributed to Shared World).
Magic Leap retains the color images and depth sensor images collected by the World Understanding service for limited periods of time in order to train Magic Leap’s computer vision models and improve performance of this functionality.
What About People in My Surroundings?
Prior to running computer vision algorithms to identify objects, Magic Leap redacts the color images from your device using computer vision and data loss prevention algorithms, which are designed to obfuscate images that include faces or nudity. In addition, when World Understanding data capture is occurring, your device’s status indicator will display an orange LED pattern to let people in your environment know that color images are being transmitted to Magic Leap.
Your Choices
World Understanding requires cloud processing using Shared World and is not currently available when you save spatial maps only on device or store your World Features in a Personal World. If you have elected to contribute your World Features to Shared World, you can then decide if you would also like to enable World Understanding. If you enable World Understanding, other users who are contributing to Shared World may have access to the object data in the spatial maps you have contributed and you may have access to object data in the spatial maps contributed by other users.
You can change your selection around whether to enable World Understanding at any time in Settings. However, if you decide to enable World Understanding and contribute your spatial maps to Shared World and then later decide to change your settings, the World Understanding object data that you have already contributed to the collective spatial maps ordinarily will not be deleted. This is because once you contribute object data to Shared World, they become part of the combined Shared World spatial maps, and are not directly associated with a particular device or account. If there are areas you do not want to be part of the collective spatial maps available to Shared World users, we encourage you to change your settings before entering those areas.
Third-party apps require your permission to access object data that has been included in your spatial maps. You will be asked to allow a third-party app to access object data when such app tries to access object data for the first time. Note that you may be asked for this permission if you are contributing your spatial maps to Shared World, even if you have World Understanding disabled, as there may be object data on your spatial maps that was contributed by other users. If you wish to change which permissions you have granted a third-party app, you can do so through Settings at any time.