Localizing with VPS

To use Lightship VPS features such as Wayspot Anchors, your users need to first localize with a VPS-activated Wayspot or Private VPS Location. To localize to a VPS-activated Wayspot, you need to provide user location data and camera data of the user’s view of a VPS-activated Wayspot to Lightship VPS. Lightship VPS can then determine the user’s position and orientation relative to the VPS-activated Wayspot.

Note

If you just need to verify a VPS-activated Wayspot is localizable, you can do localization tests using the Niantic Wayfarer App.

Finding VPS-activated Wayspots to Localize Against

Before your user can localize against a VPS-activated Wayspot, they’ll need to find and physically move to the Wayspot location. Use the VPS Coverage API to show users nearby VPS coverage areas and localization targets that will help guide them to VPS-activated Wayspots and help them understand what to localize against. The Coverage API also includes “hint images” for VPS-activated Wayspots that you can use to instruct your user on what to point their device at during localization. For more information, see Using the VPS Coverage API.

Localizing with Private VPS Locations

For development and testing purposes only, you can localize with Private VPS Locations that you’ve created using the Niantic Wayfarer App. The process for localizing with Private VPS Locations is the same as localizing with VPS-activated Wayspots. Keep in mind the following when localizing with Private VPS Locations:

  • Your app needs to be configured with an API key created with the same lightship.dev account that you used with the Niantic Wayfarer app when you created the Private VPS Location. Lightship VPS checks the app’s API key against your lightship.dev developer account.

  • Private VPS Locations are not available from the VPS Coverage API.

Using Wayspot Anchor API to Localize

The Wayspot Anchor API attempts to localize with a VPS-activated Wayspot when you initialize VPS, either by creating a new WayspotAnchorService instance, or by using WayspotAnchorController.StartVps(). WayspotAnchorService provides a straight-forward “high-level” API for creating and managing Wayspot Anchors, whereas WayspotAnchorController provides a flexible, “low-level” API that you can use if you need more granular control.

Enable Location and Camera Permissions

The Wayspot Anchor API uses the device camera and location service to localize, so you’ll need to make sure the user has enabled camera and location permissions on their device. See Permissions for details on requesting device permissions.

Note

Use of features such as VPS localization or the VPS Wayspot Anchors API involves the collection of personal information from your end user. For more information, see the Lightship ARDK Data Privacy FAQ.

Using WayspotAnchorService

Create a WayspotAnchorService, providing your ARSession, a LocationService, and a WayspotAnchorsConfiguration . When starting your LocationService we recommend using a desired accuracy of 0.01 meters and a update distance of 0.01 meters for best results.

using Niantic.ARDK.AR.WayspotAnchors;
using Niantic.ARDK.LocationService;

var wayspotAnchorsConfiguration = WayspotAnchorsConfigurationFactory.Create();
var locationService = LocationServiceFactory.Create(_arSession.RuntimeEnvironment);
// For LocationService, we recommend using a desiredAccuracyInMeters of 0.01f
// and an updateDistanceInMeters of 0.01f for best results
locationService.Start(desiredAccuracyInMeters, updateDistanceInMeters);
WayspotAnchorService _wayspotAnchorService = new WayspotAnchorService(_arSession, locationService, wayspotAnchorsConfiguration);

// Set wayspotAnchorService.LocalizationStateUpdated event handler if we want to track localization state
wayspotAnchorService.LocalizationStateUpdated += LocalizationStateUpdated;

// VPS WayspotAnchorService begins localizing, provide instructions for your user on how to localize, etc

Instruct the user to start localizing by pointing their device camera at the VPS-activated Wayspot (as described in User Best Practices For Localizing). You can monitor the localization progress by adding an event handler for WayspotAnchorService.LocalizationStateUpdated. When the state equals LocalizationState.Localized, the session is localized to a VPS-activated Wayspot, and you can begin placing VPS Wayspot Anchors. Wayspot anchors restored prior to the session being localized will start to resolve their position once the session is localized.

private void LocalizationStateUpdated(LocalizationStateUpdatedArgs args)
{
    // Handle any changes in localization state as needed. For example,
    // if state is Localized with a FailureReason of None, you app can
    // start creating or restoring anchors
    // ...
}

If the session is localized, but VPS then detects possible intermittent network connectivity issues, a LocalizationStateUpdated event will trigger, but the state will remain as Localized, however the FailureReason will be set to CannotConnectToServer. Your app should check for this scenario and notify the user that the app is detecting poor network conditions.

Using WayspotAnchorController

Create a WayspotAnchorController, providing your ARSession and a LocationService. When starting your LocationService we recommend using a desired accuracy of 0.01 meters and a update distance of 0.01 meters for best results.

using Niantic.ARDK.AR.WayspotAnchors;
using Niantic.ARDK.LocationService;

var locationService = LocationServiceFactory.Create(_arSession.RuntimeEnvironment);
// For LocationService, we recommend using a desiredAccuracyInMeters of 0.01f
// and an updateDistanceInMeters of 0.01f for best results
locationService.Start(desiredAccuracyInMeters, updateDistanceInMeters);
_wayspotAnchorController = new WayspotAnchorController(_arSession, locationService);

Set an event handler for the controller’s LocalizationStateUpdated event:

_wayspotAnchorController.LocalizationStateUpdated += HandleLocalizationStateUpdated;

Create and configure a WayspotAnchorsConfiguration and call WayspotAnchorController.StartVps() with your configuration to start the localization process:

var wayspotAnchorsConfiguration = WayspotAnchorsConfigurationFactory.Create();

_wayspotAnchorController.StartVps(wayspotAnchorsConfiguration);

Instruct the user to start localizing by pointing their device camera at the VPS-activated Wayspot (as described in User Best Practices For Localizing). Monitor the localization status in your LocalizationStateUpdated event handler, check the LocalizationState for successful localization

private void HandleLocalizationStateUpdated(LocalizationStateUpdatedArgs anyLocalizationStateUpdatedArgs)
{
    if (anyLocalizationStateUpdatedArgs.State == LocalizationState.Localized) {
        // Client successfully localized with wayspot
    }
    if (anyLocalizationStateUpdatedArgs.State == LocalizationState.Failed) {
        // Localization failed, see anyLocalizationStateUpdatedArgs.FailureReason for reason
    }
}

When the state equals LocalizationState.Localized, the session is localized to a VPS-activated Wayspot, and you can begin placing VPS Wayspot Anchors. Wayspot anchors restored prior to the session being localized will start to resolve their position once the session is localized.

If the state equals LocalizationState.Failed, get more details on the failed localization from LocalizationStateUpdatedArgs.FailureReason and see Localization Failures & Recovery Steps for handling failure conditions.

If the session is localized, but VPS then detects possible intermittent network connectivity issues, a LocalizationStateUpdated event will trigger, but the state will remain as Localized, however the FailureReason will be set to CannotConnectToServer. Your app should check for this scenario and notify the user that the app is detecting poor network conditions.

Configuring the Localization Process

You can control some aspects of the localization by changing the default settings in the WayspotAnchorsConfiguration instance you use. In most scenarios, the default configuration should be sufficient. However, if you are encountering slow or failed localizations, you might need to adjust the localization timeouts. The WayspotAnchorsConfiguration properties to adjust include:

  • LocalizationTimeout: The timeout in seconds for the entire localization attempt. The default is 30 seconds.

  • RequestTimeLimit: The timeout in seconds for an individual request made during the overall localization attempt. The default is 10 seconds.

  • GoodTrackingWait: The number of seconds that VPS is required to wait after entering a good localization state before running Wayspot Anchor tracking. The default is 3 seconds. This ensures you’re not localizing before the local session is stable, and helps prevent drift after localization.

  • ContinuousLocalizationEnabled: Whether continuous localization is enabled (detailed in Continuous Localization). The default is False.

Understanding Localization Flow

Use the following information to understand how the Wayspot Anchors API localizes and how to handle localization failure conditions.

Before Localization

Your user first needs to find or be guided to a VPS-activated Wayspot, as described in Finding VPS-activated Wayspots to Localize Against. Once your user is located within the coverage area of a VPS-activated Wayspot, you can begin the VPS localization process.

VPS goes through different localization “states” during the localization process. When VPS is started, it attempts to localize to a VPS-activated Wayspot near the user. VPS is in a Localizing state. Camera frame and GPS information is being sent to the VPS backend service to try and localize to a VPS-activated Wayspot. During this state, the following things can happen:

  • VPS localizes successfully with a VPS-activated Wayspot. The VPS localization state moves to Localized.

  • VPS detects a possible intermittent network connectivity issue. The state will stay as Localizing, however a LocalizationStateUpdated event will trigger with a LocalizationFailureReason of CannotConnectToServer. VPS will continue to try to localize.

  • VPS fails to localize with a VPS-activated Wayspot for various reasons. Possible reasons and recovery steps are described in Localization Failures & Recovery Steps. The state moves to Failed.

  • VPS detects the ARSession tracking state has become unstable. VPS will wait for the tracking state to become stable and then continue to attempt to localize. The state remains in Localizing. See Dealing with ARSession Tracking Instability.

  • Your app can cancel the localization process. You might do this if your user has exited the portion of your app that uses VPS. The state moves to Failed. You can restart the localization process as described in Restarting VPS.

During Localization

Once VPS has successfully localized, VPS is in a Localized state, and you can start placing and resolving Wayspot Anchors. VPS will try to maintain this state, however, the following situations can trigger a re-localization or Failure state:

  • VPS detects a possible intermittent network connectivity issue. The state will stay as Localized, however a LocalizationStateUpdated event will trigger with a LocalizationFailureReason of CannotConnectToServer. Anchors can still be placed.

  • VPS detects an error condition (possible error conditions and recovery steps listed in Localization Failures & Recovery Steps). The state moves to Failed.

  • VPS detects that AR Session tracking state has become unstable. The state moves to Localizing. VPS will automatically try to localize again once it detects that the AR tracking state has become stable again. See Dealing with ARSession Tracking Instability.

Note

Once a session has successfully localized, any Wayspot anchors restored from payloads prior to localization will start to resolve the anchor position and orientation. Restored anchors are not immediately resolved when the session is localized. See Handle Anchor Tracking Updates for more information on tracking anchor position and orientation changes.

Localization Failures & Recovery Steps

VPS Failures

Localization can fail for a number of reasons. Your app should track localization state updates through your event handler for WayspotAnchorService.LocalizationStateUpdated or WayspotAnchorController.LocalizationStateUpdated, and examine the state and LocalizationFailureReason.

Possible reasons for localization failure include:

  • LocalizationFailureReason of Timeout. VPS was unable to localize with a VPS-activated Wayspot within WayspotAnchorsConfiguration.LocalizationTimeout seconds. One possible reason for this failure is that the user did not successfully scan the VPS-activated Wayspot. You can instruct the user to try again, with clear directions on what to point their device at, using the Localization Target “hint image” from the VPS Coverage API if you have it. You can also try increasing WayspotAnchorsConfiguration.LocalizationTimeout if the user is in an environment with poor network bandwidth.

  • LocalizationFailureReason of CannotConnectToServer. The user’s device has lost network connectivity, or VPS requests to the VPS backend are not getting a response within WayspotAnchorsConfiguration.RequestTimeLimit seconds, or your app has exceeded the VPS request rate limits (see VPS Localization Limitations). You can check if the user has disabled networking on their device, and ask the user to re-enable network connectivity. You can also try increasing WayspotAnchorsConfiguration.RequestTimeLimit if the user is in an environment with poor network bandwidth.

  • LocalizationFailureReason of LocationDataNotAvailable. VPS can’t access the user’s GPS location information. Check if the user has disabled GPS location permissions on their device, or the LocationService you used to initialize VPS is working as you expect.

  • LocalizationFailureReason of InvalidApiKey. Your application was configured with an invalid ARDK license key. Set a valid key in your app’s ArdkAuthConfig and rebuild.

  • LocalizationFailureReason of Canceled. Your application canceled the localization process.

VPS Recovery

To recover from VPS failure conditions you will need to restart VPS to retry the localization process. See Restarting VPS for how to do this. You do not need to restart your AR Session. In some situations (such as loss of network connectivity) you can continue to render virtual objects associated with Wayspot Anchors and ARDK will use local AR tracking to position virtual objects. You may encounter slight drift issues when doing this.

Dealing with ARSession Tracking Instability

If your app is localized and VPS detects that the ARSession tracking state has become unstable (AR camera frame data is invalid or corrupt), VPS will automatically attempt to re-localize, moving from Localized to Localizing state. If this happens, you may need to restore placed Wayspot Anchors and re-render virtual content once VPS has re-localized and moves back to Localized state.

You can inspect ARCamera.TrackingStateReason to determine the cause of the tracking state instability, and use this information to guide your users during the re-localization process. For example, if the ARCamera.TrackingStateReason is ExcessiveMotion, you can ask your users to reduce their movement speed while they scan.

You may also need to monitor device performance. If the device is running low on memory or CPU, the tracking system will begin to starve due to lack of system resources. Tracking performance can degrade significantly in this scenario.

Continuous Localization

Lightship VPS can be configured to do continuous localization by setting WayspotAnchorsConfiguration.ContinuousLocalizationEnabled to True when you start VPS. When continuous localization is enabled, VPS will continue to localize to the user’s current surroundings after the initial localization.

Continuous localization can help avoid drift in longer (10+ minutes) VPS sessions. As your app continuously localizes, the updated localization information helps correct for Wayspot Anchor drift.

Continuous localization can also improve the user experience if you anticipate your users will move to different VPS-activated Wayspots during the same VPS session. VPS will automatically localize with the new Wayspot the user has moved to. You can start placing Wayspot Anchors with the new VPS-activated Wayspot once you’ve determined the user has successfully localized with the new Wayspot.

Dealing with Drift

In some scenarios, you might encounter Wayspot Anchor tracking updates that introduce “drift”, where Anchor positions shift by small amounts. Localization error bounds can be within 50 cm, so Wayspot Anchors can appear to shift by that amount. This can occur when AR Tracking becomes unstable, or when continuous localization is enabled and VPS re-localizes.

To work around this issue, you can:

  • When notified of a Wayspot Anchor position update, if the position update is within 1m of the current position, you can skip re-positioning any child game objects to avoid small amounts of jitter.

  • When notified of a Wayspot Anchor position update, you can smoothly interpolate the position between where it was and where it needs to be.

  • You can opt to only apply small position updates when the Wayspot Anchor and associated game objects are outside the current field of view of the user.

Restarting VPS

If your app needs to restart the VPS session from scratch, you can use WayspotAnchorService.Restart(). You might need to do this if your app encounters issues with device location services or network connectivity, such as when a user enables and disables airplane mode. This will cancel any in-progress Wayspot Anchor creation or tracking and re-start VPS localization. Once your app has re-localized, WayspotAnchorService.Restart() will also automatically resume tracking on your Wayspot Anchors.

When you restart VPS you do not need to restart your AR Session. In some situations (such as restarting due to loss of network connectivity) you can continue to render virtual objects associated with Wayspot Anchors and ARDK will use local AR tracking to position virtual objects. You may encounter slight drift issues of virtual objects associated with anchors when doing this.

Note

If you are using WayspotAnchorController and restarting VPS via StopVps() and StartVps(), note that StartVps() does not automatically resume tracking on placed anchors, so you will need to call ResumeTracking() on your placed anchors after you’ve re-localized.

User Best Practices For Localizing

When users use their device to localize with a VPS-activated Wayspot, they should be instructed as follows:

  • Ensure that your environment conditions are good (and safe). This includes:

    • Localize during daylight hours

    • Localize during fair weather conditions (do not localize during rain storms, snowfall, or foggy conditions)

    • Localize relatively static environments. Avoid localizing if there are many moving people or vehicles in your camera view.

  • Stand roughly 3-5 meters / 10-15 feet away from the VPS-activated Wayspot.

  • Have your device camera looking straight at the VPS-activated Wayspot, not from a high or low angle. Avoid looking at the ground. We want to understand as much of the 3D scene as possible, and wide, flat surfaces don’t help.

  • Continue looking at the VPS-activated Wayspot until the process completes. This should take approximately 5 seconds, but may take up to 30 seconds depending on the complexity of the environment and the available network bandwidth of your device.

  • If the initial localization fails, the app will continue trying to localize.For subsequent localization attempts, keep your camera looking at the VPS-activated Wayspot and slowly move in a circular direction around the VPS-activated Wayspot to provide additional view-points to localize with.

If you use the Coverage API to guide your user to a wayspot, the localization target information the Coverage API provides will have a “hint image” photo of the VPS-activated Wayspot localization target. This picture can be used to help your users find the VPS-activated Wayspot and understand what they should be pointing their device camera at when localizing.

Mocking Localizations

When running in Mock Mode in the Unity editor, VPS will perform a mock localization and automatically move to the Localized state after 300 milliseconds to simulate the localization process. You can then place Wayspot Anchors in Mock Mode.

See Playing in Mock Mode for more details on running in Mock Mode. See “Wayspot Anchors and Mock Mode” in Using VPS Wayspot Anchors for details on placing and restoring Wayspot Anchors in Mock Mode.

VPS Localization Limitations

Keep in mind the following when localizing with VPS-activated Wayspots.

  • At this time, VPS localization cannot be used in conjunction with environment or marker-based localization.

  • When localizing, ARDK currently sends approximately 300K of data to the Niantic AR backend every 1-3 seconds. Keep this in mind if your AR experience is being used in bandwidth-limited situations.

  • Currently VPS localization requests are limited to 30 queries per second (QPS) per API license key. Niantic may adjust this limit in the future based on usage and traffic analysis. If your use cases need a higher limit, please reach out to our support team to request an increase.

VPS Localization Best Practices

We recommend only having the user localize with one VPS-activated Wayspot per VPS session. If the user needs to move to a different VPS-activated Wayspot, we recommend restarting VPS to localize to the new Wayspot for best results.

We recommend only placing Wayspot Anchors if your user is within 10 meters of the localized VPS-activated Wayspot.

VPS Support

For help with dealing with VPS localization issues, please visit our developer forums at https://community.lightship.dev/. To help Niantic investigate your issue, include the following information if possible:

  • The AR Session identifier for the session having the issue. You may need to log this using IARSession.StageIdentifier. We strongly recommend providing this if possible, as it helps us locate your localization requests.

  • GPS coordinates where the localization was attempted.

  • If you’re using the VPS Coverage API, LocalizationTarget.Identifier information of the VPS-activated Wayspot that was being localized with.

  • LocalizationState and LocalizationFailureReason if applicable.

  • Day and time the localization was attempted.