# Localization Guide for AR Apps

This tutorial shows how to send one camera image—or a coherent Rig.v2 frame set—to 01Spatial, read the returned camera pose, and use that pose to align AR content with the real world.

> **Scope:** Use your own API Key and Map Key (or Project Key) for single-frame or Rig.v2 localization, then apply the returned pose to AR content. Apart from the pose matrix and source frame required for AR alignment, this guide documents only `confidence`; all other response fields are outside the client integration contract.

## 1. Understand the result

Your AR framework and 01Spatial have different jobs:

1. Your AR framework tracks the device smoothly inside the current AR Session.
2. 01Spatial finds the captured camera inside a persistent Map.
3. Your app combines the two camera poses to align the Map with the AR Session.
4. All AR content saved in Map coordinates can then appear in the correct place.

```text
AR camera pose at capture time ─┐
                               ├─► Map-to-AR alignment ─► AR content
01Spatial camera pose in Map ──┘
```

01Spatial returns the camera pose. It does not choose where your AR content belongs. Each object must already have a position and orientation in the Map coordinate system.

## 2. Before you start

You need:

- An **API Key** from your Portal profile.
- A **Map Key** from a ready map in Portal, or a **Project Key** for multi-map localization.
- One camera image.
- The image intrinsics: `fx`, `fy`, `cx`, and `cy`.
- The AR camera transform saved at the same moment as the image.

Set `API_HOST` to the **origin of the Portal that contains your target and issued its key**. An origin contains only the scheme and host, with no path, query, or trailing slash.

Treat `API_HOST` as runtime configuration. Do not choose it from the device language or hardcode a regional hostname in shared application code. This lets the same application work with current and future regions without a code change.

Keep both keys private. Put them in the request body, not in a URL. Do not commit them to a public repository or write them to logs.

## 3. Capture one synchronized camera sample

Use the AR framework for the target device. The image, intrinsics, and AR camera transform must describe the **same camera frame**. Do not use a screenshot or a separately captured photo, because it cannot be matched reliably with the AR camera pose.

| Platform | Get the camera image | Save from the same frame |
| --- | --- | --- |
| iOS or iPadOS | ARKit `ARFrame.capturedImage` | `ARFrame.camera.intrinsics` and `ARFrame.camera.transform` |
| Android | ARCore `Frame.acquireCameraImage()` | `frame.getCamera().getImageIntrinsics()` and `frame.getCamera().getPose()` |
| WeChat Mini Program | XR-FRAME `scene.ar.getARRawData()` camera data | The same result's `intrinsics` and `viewMatrix`; derive and save its camera-to-session `trackerSpace` |
| Unity | AR Foundation `ARCameraManager.TryAcquireLatestCpuImage()` | `TryGetIntrinsics()` and the AR Camera transform from the same camera update |

If image encoding or conversion runs asynchronously, copy the intrinsics and camera transform before starting that work. Release ARCore `Image` and AR Foundation `XRCpuImage` objects after the pixels have been copied or encoded.

The intrinsics are pixel values for the exact image you upload:

| Field | Meaning |
| --- | --- |
| `fx` | Horizontal focal length |
| `fy` | Vertical focal length |
| `cx` | Horizontal principal point |
| `cy` | Vertical principal point |

If you resize the image, resize the intrinsics by the same amount. For example, when `1920 × 1080` becomes `960 × 540`:

```text
scale_x = 960 / 1920 = 0.5
scale_y = 540 / 1080 = 0.5

fx' = fx × scale_x
fy' = fy × scale_y
cx' = cx × scale_x
cy' = cy × scale_y
```

If you crop the image, subtract the crop's left offset from `cx` and its top offset from `cy`.

For the first integration, keep the uploaded pixels in the camera frame's native orientation. If you rotate the pixels before upload, changing the dimensions and intrinsics is not enough: the returned pose uses the rotated camera axes. You must also include the same rotation when you align the camera axes in section 8.

At capture time, also save the AR framework's camera-to-session transform. Do not read it after the network response arrives, because the device may have moved.

## 4. Send the localization request

Send a `multipart/form-data` request to `POST /localize`:

```bash
: "${API_HOST:?Set API_HOST to the Portal origin that contains your map}"

curl -X POST "${API_HOST%/}/localize" \
  -F "api_key=YOUR_API_KEY" \
  -F "map_key=YOUR_MAP_KEY" \
  -F "fx=1100" \
  -F "fy=1100" \
  -F "cx=960" \
  -F "cy=540" \
  -F "image_path=@frame.jpg"
```

In JavaScript, let `FormData` create the `Content-Type` header and multipart boundary. Do not set that header yourself.

```ts
type Intrinsics = {
  fx: number;
  fy: number;
  cx: number;
  cy: number;
};

async function localizeFrame(
  apiHost: string,
  apiKey: string,
  mapKey: string,
  image: Blob,
  intrinsics: Intrinsics,
) {
  const localizationUrl = new URL('/localize', new URL(apiHost).origin);
  const body = new FormData();
  body.set('api_key', apiKey);
  body.set('map_key', mapKey);
  body.set('fx', String(intrinsics.fx));
  body.set('fy', String(intrinsics.fy));
  body.set('cx', String(intrinsics.cx));
  body.set('cy', String(intrinsics.cy));
  body.set('image_path', image, 'frame.jpg');

  const response = await fetch(localizationUrl, {
    method: 'POST',
    body,
  });

  if (response.status === 422) return null;
  if (!response.ok) {
    throw new Error(`Localization request failed (${response.status})`);
  }
  return response.json();
}
```

## 5. Read the pose and confidence

A successful response contains one pose in `poses[0]`. The example below shows only the fields needed by an AR app:

```json
{
  "poses": [
    {
      "camera_to_world": {
        "matrix_column_major": [
          1, 0, 0, 0,
          0, 1, 0, 0,
          0, 0, 1, 0,
          1.2, 0.4, -3.1, 1
        ]
      },
      "metrics": {
        "confidence": 0.86
      }
    }
  ]
}
```

- `matrix_column_major` is the complete 4 × 4 camera-to-Map transform. Use this for AR alignment.
- `confidence` is a quality value from `0` to `1`. Higher is better.

Start with this simple policy:

```text
confidence >= 0.7  → apply the new pose
confidence < 0.7   → keep the current alignment and capture another frame
```

`confidence` is not a distance error and is not a probability guarantee.

HTTP `422` means that no reliable pose was found. Do not change the current alignment. Capture another sharp image with a clear view of the mapped area.

If single-frame results are repeatedly unstable, use Rig.v2 in the next section instead of applying several unrelated single-frame results in sequence.

## 6. Use Rig.v2 for multi-frame localization

`POST /localize_rig` accepts 2–6 nearby camera frames from one uninterrupted AR tracking epoch and returns the pose associated with one captured frame. Use it when single-frame localization is sensitive to viewpoint, occlusion, or limited scene detail.

Rig.v2 is useful when a single view may contain a blank wall, an occlusion, repetitive texture, glare, or too little of the mapped area. For a phone or tablet, a practical fallback is four sharp views collected while the user slowly turns through roughly half a circle. Do not send several nearly identical frames and expect the same geometric benefit.

### 6.1 Capture one coherent frame set

For every frame, save all of these values at the same capture time:

- The final JPEG pixels.
- The intrinsics for those exact pixels.
- The camera-to-AR-session transform.
- A monotonic capture timestamp.
- The current AR tracking state and tracking epoch.

All frames in one request must belong to the same tracking epoch. Cancel the request if the AR Session ends, its reference space resets, or tracking is lost. Use only frames captured while tracking is normal.

For `calibrated_rig`, the camera image, intrinsics, and tracking pose must describe the same physical camera. ARKit, ARCore, and AR Foundation can provide this when their camera APIs are sampled coherently. WebXR can provide it only when Raw Camera Access supplies an image associated with the same `XRView`; a separate `getUserMedia` stream must not be used for Rig localization.

Keep the encoded pixels in their native capture orientation for the first integration. Bake any EXIF orientation into the pixels and write Orientation `1` (or remove the tag); then set `encoded_image_rotation_cw_deg` to `0`. This makes the image, intrinsics, camera axes, and relative-pose convention unambiguous.

### 6.2 Compute the relative camera poses

Let `T_A_Ci_graphics` be frame `i`'s camera-to-AR-session transform saved at capture time. With unrotated encoded pixels, compute:

```text
T_frame0_from_frame_i = inverse(T_A_C0_graphics) × T_A_Ci_graphics
```

This transform maps camera coordinates from frame `i` into frame `0`. Encode its rotation as a normalized `quat_wxyz` and its translation as `tvec`:

- `frames[0].relative_pose` must be `null`.
- Every later frame in a `calibrated_rig` request must include `relative_pose`.
- `relative_pose_semantics` must be `frame0_from_frame`.
- `relative_pose_coordinate_system` must be `camera_gl_encoded`.

If you rotate the pixels before JPEG encoding, you must also rotate the intrinsics and conjugate each relative pose into the encoded image's graphics-camera axes. Do not send a calibrated request until that conversion has been tested. The server does not guess image rotations or coordinate conventions.

### 6.3 Build the Rig.v2 header

Use `rig_schema_version: 2`. The example below contains only the fields an AR client needs to construct a calibrated Rig request.

The example below contains two frames; production clients may send up to six. The declared width and height must exactly match each decoded JPEG, timestamps must be monotonic, frame IDs must be unique, and all frame epochs must match.

```json
{
  "map_key": "YOUR_MAP_KEY",
  "api_key": "YOUR_API_KEY",
  "rig_schema_version": 2,
  "rig_capability": "calibrated_rig",
  "client_platform": "ios",
  "client_build": "1.0.0",
  "operation_id": "rig-7f4d2c",
  "relative_pose_semantics": "frame0_from_frame",
  "relative_pose_coordinate_system": "camera_gl_encoded",
  "intrinsics_coordinate_space": "encoded_pixels",
  "frames": [
    {
      "frame_id": "frame-0",
      "fx": 1100.0,
      "fy": 1100.0,
      "cx": 960.0,
      "cy": 540.0,
      "image_width": 1920,
      "image_height": 1080,
      "capture_timestamp_s": 1234.000,
      "tracking_epoch": "ar-session-42",
      "tracking_state": "normal",
      "encoded_image_rotation_cw_deg": 0,
      "capture_quality": {},
      "relative_pose": null
    },
    {
      "frame_id": "frame-1",
      "fx": 1100.0,
      "fy": 1100.0,
      "cx": 960.0,
      "cy": 540.0,
      "image_width": 1920,
      "image_height": 1080,
      "capture_timestamp_s": 1234.620,
      "tracking_epoch": "ar-session-42",
      "tracking_state": "normal",
      "encoded_image_rotation_cw_deg": 0,
      "capture_quality": {},
      "relative_pose": {
        "quat_wxyz": [0.9998, 0.0, 0.0175, 0.0],
        "tvec": [0.05, 0.0, -0.02]
      }
    }
  ]
}
```

Use `project_key` instead of `map_key` for Project-wide localization. Do not send both targets. In either mode, the `api_key` must be allowed to access the target.

### 6.4 Send the length-prefixed body

The request uses `application/octet-stream`, not multipart form data. Its binary layout is:

```text
[4-byte little-endian JSON length]
[UTF-8 JSON header]
[4-byte little-endian JPEG 0 length]
[JPEG 0 bytes]
[4-byte little-endian JPEG 1 length]
[JPEG 1 bytes]
...
```

The JPEG order must match the `frames` array exactly. This browser example uses `Blob` parts so the JPEGs do not need to be copied into a second combined byte array:

```ts
function uint32LE(value: number): Uint8Array {
  const bytes = new Uint8Array(4);
  new DataView(bytes.buffer).setUint32(0, value, true);
  return bytes;
}

function buildRigBody(header: object, jpegFrames: Blob[]): Blob {
  const headerBytes = new TextEncoder().encode(JSON.stringify(header));
  const parts: BlobPart[] = [uint32LE(headerBytes.byteLength), headerBytes];

  for (const jpeg of jpegFrames) {
    parts.push(uint32LE(jpeg.size), jpeg);
  }
  return new Blob(parts, { type: 'application/octet-stream' });
}

async function localizeRig(
  apiHost: string,
  header: object,
  jpegFrames: Blob[],
  signal: AbortSignal,
) {
  const response = await fetch(
    new URL('/localize_rig', new URL(apiHost).origin),
    {
      method: 'POST',
      headers: { 'Content-Type': 'application/octet-stream' },
      body: buildRigBody(header, jpegFrames),
      signal,
    },
  );
  const result = await response.json();
  return { response, result };
}
```

Cancel an in-flight request and discard a late response as soon as the AR Session or tracking epoch is no longer current.

### 6.5 Apply the returned pose

In a successful Rig.v2 response, `poses[0].image` identifies the uploaded frame associated with the pose:

```json
{
  "poses": [
    {
      "image": "frame-1",
      "camera_to_world": {
        "matrix_column_major": [
          1, 0, 0, 0,
          0, 1, 0, 0,
          0, 0, 1, 0,
          1.2, 0.4, -3.1, 1
        ]
      },
      "metrics": {
        "confidence": 0.91
      }
    }
  ]
}
```

Apply the result only when the HTTP status is `200`, `poses[0]` exists, `confidence` passes your policy, and the AR Session has not changed. `poses[0].image` must belong to the submitted frame set.

The returned `camera_to_world` belongs to `poses[0].image`, which is not necessarily frame `0`. Use the saved camera-to-session transform from that frame in the alignment formula from section 8:

```text
T_A_M = T_A_Cselected_graphics × D_cv_to_graphics × inverse(T_M_Cselected)
```

If the returned pose belongs to another frame but you still use frame `0`'s AR pose, the alignment will be wrong. HTTP `422` means this request produced no reliable pose; keep the current alignment and capture a new set.

## 7. Understand the coordinate systems

Use these names:

- `M`: the Map coordinate system.
- `A`: the current AR Session coordinate system.
- `C`: the camera coordinate system.
- `T_M_C`: a transform from Camera to Map. This is the API's `camera_to_world` matrix.

The camera axes in `camera_to_world` use the computer-vision convention:

| Axis | Direction |
| --- | --- |
| `+X` | Right |
| `+Y` | Down |
| `+Z` | Forward |

WebXR and OpenGL-style AR cameras use `+X` right, `+Y` up, and `-Z` forward. Convert the camera axes once with:

```text
D_cv_to_graphics = diag(1, -1, -1, 1)
```

The Map part of the pose uses the map's public coordinate system. Store your persistent AR content in that same coordinate system. Do not assume that every map uses the same up axis. When a map has metric scale, translation values are in metres.

## 8. Align Map content with the AR Session

At image capture time, save the AR camera-to-session transform `T_A_C_graphics`. If the uploaded pixels were not rotated relative to that camera frame, compute:

```text
T_A_M = T_A_C_graphics × D_cv_to_graphics × inverse(T_M_C)
```

If you rotated the uploaded pixels, let `R_upload_from_capture` transform the original computer-vision camera axes into the uploaded image's camera axes. Use:

```text
T_A_M = T_A_C_graphics × D_cv_to_graphics × inverse(R_upload_from_capture) × inverse(T_M_C)
```

When the pixels were not rotated, `R_upload_from_capture` is the identity matrix and the two formulas are the same.

`T_A_M` transforms Map coordinates into the current AR Session. Apply it to one root node that contains all Map-authored content:

```text
AR Session
└── mapRoot                  apply T_A_M here
    ├── information marker  saved in Map coordinates
    ├── navigation path     saved in Map coordinates
    └── 3D object           saved in Map coordinates
```

For one object whose Map transform is `T_M_O`:

```text
T_A_O = T_A_M × T_M_O
```

Do not overwrite the AR framework's camera transform. Let local AR tracking continue to move the camera smoothly. Use a later successful localization to update `mapRoot` when you want to correct tracking drift.

## 9. Three.js and WebXR example

`THREE.Matrix4.fromArray()` accepts `matrix_column_major` directly. Do not transpose it again.

```ts
import * as THREE from 'three';

function readTrustedPose(response: any) {
  const pose = response?.poses?.[0];
  const matrix = pose?.camera_to_world?.matrix_column_major;
  const confidence = pose?.metrics?.confidence;

  if (!Array.isArray(matrix) || matrix.length !== 16) return null;
  if (!Number.isFinite(confidence) || confidence < 0.7) return null;
  return pose.camera_to_world;
}

function alignMapRoot(
  mapRoot: THREE.Object3D,
  arCameraAtCapture: ArrayLike<number>,
  cameraToWorld: { matrix_column_major: number[] },
) {
  const arFromCamera = new THREE.Matrix4().fromArray(arCameraAtCapture);
  const mapFromCamera = new THREE.Matrix4().fromArray(
    cameraToWorld.matrix_column_major,
  );
  const cvToGraphics = new THREE.Matrix4().makeScale(1, -1, -1);

  const arFromMap = arFromCamera
    .clone()
    .multiply(cvToGraphics)
    .multiply(mapFromCamera.clone().invert());

  mapRoot.matrixAutoUpdate = false;
  mapRoot.matrix.copy(arFromMap);
  mapRoot.matrixWorldNeedsUpdate = true;
}
```

Use the camera transform from the exact frame used to create the uploaded image:

- **WebXR:** `XRView.transform.matrix`.
- **ARKit, RealityKit, or App Clip:** `ARFrame.camera.transform`.
- **WeChat XR-FRAME:** the capture-time `trackerSpace` matrix.

The Three.js example assumes that the uploaded pixels were not rotated. If they were rotated, include `inverse(R_upload_from_capture)` in the multiplication shown above. Other AR engines follow the same transform order.

## 10. Common problems

| Result | What to check |
| --- | --- |
| HTTP `400` | The image field and all four intrinsics are present and valid. |
| HTTP `401` or `403` | The API Key and Map Key are correct and allowed to access the map. |
| HTTP `422` | Capture a sharper frame with more visible mapped detail. |
| HTTP `429` | Wait before sending another request. |
| HTTP `5xx` | Keep the current alignment and retry later. |
| A valid Map Key is rejected | Confirm that `API_HOST` is the same Portal origin that contains the map. |
| Content is mirrored or rotated | Apply the CV-to-graphics conversion exactly once. |
| Content moves after the response | Use the AR camera transform saved at image capture time. |
| Content has a constant offset | Confirm that object transforms and the localization pose use the same Map coordinates. |

## 11. Integration checklist

Before release, verify that your app:

- Gets the image, intrinsics, and AR camera transform from the same AR frame.
- Uses intrinsics for the exact uploaded pixels.
- Saves the AR camera transform at image capture time.
- Compensates the camera axes if the uploaded pixels were rotated.
- Reads `matrix_column_major` without another transpose.
- Applies the CV-to-graphics conversion exactly once.
- Applies only poses that pass the confidence policy.
- Ignores a delayed response if the AR Session was reset after capture.
- For Rig.v2, keeps every frame in one tracking epoch and cancels stale requests.
- For Rig.v2, aligns with the saved AR camera transform of `poses[0].image`, not always frame `0`.
- Stores persistent content in the same Map coordinate system as the pose.
- Applies alignment to a Map root instead of replacing the AR camera pose.
- Loads `API_HOST` from runtime configuration and uses the Portal that contains the map.
- Keeps the API Key and Map Key out of URLs, logs, and public source control.

Do not make client logic depend on response fields that this guide does not document.