# Building from Source

Full setup: Meta developer credentials, Universal Links, personal signing overlays,
in-app configuration, and troubleshooting. Start with the [Quick Start](../README.md#quick-start)
if you just want it running.

## Availability and first use

This guide documents **building from source on a Mac with Xcode 26+**, then installing on an
**iPhone running iOS 26+** with your signing configuration. If you only want to use the app, the
public beta is on [TestFlight](https://testflight.apple.com/join/1dcAytte) and needs no build. Any
other invitation or separately distributed build has its own access instructions; do not assume
that a Meta developer invitation also installs the iPhone app.

Compatible Meta glasses use Meta AI pairing, developer configuration, registration and camera
permissions. Choosing a different AI provider does not replace those steps. Phone-based features
and camera fallback let you explore before completing glasses setup.

After following the build steps below, check one thing at a time:

1. **Registration:** complete Meta setup and the callback into Avenkin. If Connect fails,
   check the credentials and the troubleshooting section before repeating registration.
2. **Permissions:** grant the microphone/camera and service permissions needed for the task;
   denied access needs a permission fix, not a different model.
3. **Fresh image:** request a new capture of a distinctive object. A held preview is not proof
   that the current camera session is delivering images.
4. **Audio:** ask a short question and check both microphone input and the intended spoken-output route.
5. **Useful task:** read a clearly printed sign, then ask about a calendar entry or save a simple note.
   Confirm the result in its destination. Native contact creation and the unified “handle this”
   transaction remain planned.

These are manual checks, not a claim that a guided setup checker has shipped. If using VoiceOver,
follow [the accessibility notes](CAPABILITIES.md#accessibility); complete independent setup and
recovery validation is still pending. For offline use, download the required models first and follow
[the offline voice checklist](CAPABILITIES.md#set-up-offline-voice), including a disconnected trial.

Cloud AI and speech credentials are configured separately. Verify the supported sign-in/API route
for the selected provider; a personal chat subscription is not itself an API key. For processing
destinations and the limits of the network monitor, see [Privacy and control](CAPABILITIES.md#privacy-and-control).

## Build and configure

### 1. Clone

```bash
git clone https://github.com/straff2002/OpenGlasses.git
cd OpenGlasses
```

### 2. Meta Developer Credentials

1. Go to [wearables.developer.meta.com](https://wearables.developer.meta.com/)
2. Create an account, organization, and app
3. Note your **Meta App ID** and **Client Token**
4. In Meta dashboard → iOS settings, enter your Apple Team ID, Bundle ID, and Universal Link URL

### 3. Configure Meta keys

Your Meta **App ID** and **Client Token** are build settings, substituted into the committed
`OpenGlasses/Info.plist` at build time. Put them in the gitignored `project.local.yml` — never
in the plist itself:

```yaml
targets:
  OpenGlasses:
    settings:
      base:
        MWDAT_META_APP_ID: "<your Meta app id>"
        MWDAT_CLIENT_TOKEN_HASH: "<hash after the second | of your client token>"
```

Then re-run `./Scripts/generate-xcodeproj.sh`. A client token looks like
`AR|<app id>|<hash>`; only the trailing hash goes in `MWDAT_CLIENT_TOKEN_HASH`, since the plist
composes the rest. `./Scripts/setup-local-dev.sh` preserves both values when it rewrites
`project.local.yml`.

Without them the app builds and launches normally, but DAT registration never completes — and
because the camera permission prompt is gated behind registration, the symptom is a Connect
button that appears to do nothing rather than an obvious credentials error. The app names this
at launch (`⚠️ Glasses config is still the committed placeholder`) and in the connect-failure
message.

> Don't override `INFOPLIST_FILE` to a personal plist copy. That mechanism was removed: the
> copy went stale and silently dropped newly added usage descriptions (App Store ITMS-90683),
> and when it was withdrawn it took the Meta credentials with it.

### 4. Universal Links

Host an `apple-app-site-association` file at `https://YOUR-DOMAIN/.well-known/apple-app-site-association`:

```json
{
  "applinks": {
    "details": [{
      "appID": "YOUR_TEAM_ID.YOUR_BUNDLE_ID",
      "paths": ["/YOUR-PATH/*"]
    }]
  }
}
```

**Both ends must match your build, or registration silently never completes** (#246):

- The AASA `appID` must be **your** Team ID + bundle ID — a copied AASA still declaring the upstream app makes iOS refuse the domain association.
- Your entitlements file must contain the **associated-domains** entitlement for the same domain (`applinks:YOUR-DOMAIN`). Without it, iOS never even attempts the link-back. If you use a personal entitlements overlay, check it has:

```xml
<key>com.apple.developer.associated-domains</key>
<array>
    <string>applinks:YOUR-DOMAIN</string>
</array>
```

The failure mode when either end is wrong: approval completes in the Meta AI app, but the approval callback (a Universal Link) never reaches Avenkin — registration never finalises, the devices listener never fires, and connecting fails. The in-app error names the stalled registration state and points here.

### 5. Enable Developer Mode

On iPhone: Meta AI app → Settings → About → tap version number **5 times** → toggle Developer Mode on.

With DAT 0.9 and later, also **apply/update the Developer Mode settings on the glasses** in
Meta AI. The phone's toggle alone is not sufficient. Update Meta AI first if that option is missing.
Bluetooth audio, a connected device and granted camera permission can all work while the
camera's device session fails before starting. Meta traced a matching `Device unavailable`
startup failure to settings that had not been pushed to the glasses
([Meta's explanation and confirmed recovery](https://github.com/facebook/meta-wearables-dat-ios/discussions/116)).

Camera transport also needs the Hotspot Configuration and Access Wi-Fi Information capabilities
in the **signed app**. Both keys are in the committed entitlements; existing personal overlays
can add them with `./Scripts/setup-local-dev.sh`. Allow Xcode to update provisioning profiles
for these capabilities, then rebuild and install. Accept the glasses Wi-Fi Join prompt and allow
Local Network access when asked. Meta's
[CameraAccess entitlements](https://github.com/facebook/meta-wearables-dat-ios/blob/main/samples/CameraAccess/CameraAccess/CameraAccess.entitlements)
show the required keys.

If startup instead reports `Session ended by device`, end any broadcast shown in Meta AI. If it
persists, close the glasses in their charging case for a minute and retry. This is a known
[device-side session recovery issue](https://github.com/facebook/meta-wearables-dat-ios/issues/231),
and repeated app restarts cannot reliably clear it. An `unexpectedError` without a recognized
reason does not identify either condition; export a new support report after trying again.

### 6. Build & Run

The repo ships [`project.base.yml`](../project.base.yml) plus an optional [`project.local.yml`](../project.local.yml.example) overlay; XcodeGen writes `OpenGlasses.xcodeproj` locally. Do not commit the generated project.

```bash
brew install xcodegen cmake
./Scripts/fetch-mediapipe-frameworks.sh    # vendored binaries, fetched not committed
./Scripts/fetch-llamacpp-framework.sh      # fetch or build the pinned local inference engine
./Scripts/generate-xcodeproj.sh
open OpenGlasses.xcodeproj
```

Two vendored packages keep their binaries out of the repository and obtain them with a script:
MediaPipe Tasks (downloaded, checksum-verified) and the llama.cpp engine (built from the exact
revision pinned in `Vendor/LlamaCpp/REVISION`, then recorded in `SHA256SUMS`). Both scripts are
idempotent — re-running them after the first time costs a second. The llama build needs `cmake`;
it will tell you rather than installing anything itself.

#### Continuous integration

Two pipelines build this repository, and they are deliberately built to agree with each other.

**XcodeGen is pinned and checksum-verified.** Both
[Xcode Cloud](https://developer.apple.com/documentation/xcode/xcode-cloud) (via
`ci_scripts/ci_post_clone.sh`) and GitHub Actions (`.github/workflows/`) install it with
`Scripts/install-xcodegen.sh`, which downloads the release named in `Scripts/xcodegen-pin.env`
and refuses to run it unless the sha256 matches. XcodeGen decides the contents of the project
file every build compiles, so it is verified to the same standard as the vendored MediaPipe and
llama.cpp binaries. Neither pipeline uses Homebrew — Xcode Cloud's network cannot resolve
`ghcr.io`, Homebrew's bottle host, and an unpinned tap version is not a build input anyone
reviewed.

Bumping XcodeGen means editing one file. Change `XCODEGEN_VERSION` in
`Scripts/xcodegen-pin.env` and recompute the digest:

```bash
curl -fsSL "https://github.com/yonaskolb/XcodeGen/releases/download/<version>/xcodegen.zip" | shasum -a 256
```

Local development still uses whatever `brew install xcodegen` gives you; the pin governs CI, where
nobody is watching the version go by.

**Package resolution is frozen to `ci_scripts/Package.resolved`.** Xcode Cloud cannot resolve
packages at all (automatic resolution is disabled there and `xcodebuild
-resolvePackageDependencies` fails with exit 74), so it copies that lockfile into the generated
project. GitHub Actions now does the same and passes `-onlyUsePackageVersionsFromResolvedFile`, so
a pin that cannot be satisfied fails the run rather than silently resolving to something else —
the two pipelines compile the same dependency graph from the same commit, or one of them goes red.

Run `./Scripts/update-package-resolved.sh` and commit the result whenever you add, remove or bump
an SPM dependency, or CI will build the old graph.

**Reviewing dependency updates.** `.github/dependabot.yml` automates one ecosystem and
deliberately leaves the other manual:

- **GitHub Actions** — automated, weekly. Dependabot reads the `# vX.Y.Z` comment beside each
  pinned commit SHA and moves both together, which is what keeps SHA pinning from quietly
  rotting. Minor and patch bumps arrive grouped; majors arrive one per PR.
- **SwiftPM** — manual, and it has to be. Dependabot's `swift` support reads a root
  `Package.swift` + `Package.resolved`. Both exist here, but neither builds the app: the project
  is generated by XcodeGen from `project.base.yml`, and the graph both CI paths compile is
  `ci_scripts/Package.resolved`. Dependabot has no XcodeGen support, so it would open PRs
  against a manifest nothing links while the authoritative pins stayed where they were.

  The manual path, run when a dependency is added, removed or bumped and otherwise on a
  scheduled review:

  ```bash
  # 1. Read the upstream release notes for anything you are about to move. The 0.x pins
  #    (meta-wearables-dat-ios, swift-huggingface) are pinned exactly, not `from:`, because
  #    those packages break API on minor bumps — see the comments in project.base.yml.
  # 2. Edit the version or revision in project.base.yml. That file is the authority.
  ./Scripts/generate-xcodeproj.sh
  xcodebuild -resolvePackageDependencies -project OpenGlasses.xcodeproj -scheme OpenGlasses
  ./Scripts/update-package-resolved.sh
  # 3. Commit project.base.yml and ci_scripts/Package.resolved together, so the graph that was
  #    reviewed is the graph that gets built.
  ```

  A PR that moves a pin without the matching lockfile line is a review of something CI will not
  build. The `sbom` job's component list is generated from `ci_scripts/Package.resolved`, so an
  unrefreshed lockfile also shows up as a stale SBOM.

**Pull requests get a dependency review.** `.github/workflows/security-scan.yml` runs GitHub's
`dependency-review-action` on every PR and fails on a high-severity advisory or a denied licence
in whatever the dependency graph does cover. It is a floor, not the SwiftPM review above: the
graph it sees is the root manifest's, so treat a clean dependency review as necessary and not
sufficient.

The unit workflow also writes a `provenance.txt` artifact (retained 90 days): commit, runner,
macOS build, Xcode and SDK versions, the XcodeGen pin, the lockfile digest and the digest of each
`Vendor/*/SHA256SUMS`. It is what lets a green run still mean something months later.

**Every run produces an SBOM.** `Scripts/generate-sbom.sh` writes a CycloneDX 1.5 document from
the pins already in the repository — `ci_scripts/Package.resolved`, the `Vendor/*/REVISION` and
`SHA256SUMS` files, the MediaPipe fetch script's archive digests, `Scripts/xcodegen-pin.env`,
`Scripts/gitleaks-pin.env` and the model catalogues — with a hash on every component that has
one. It needs no network, so an SBOM can be produced for any commit, including an old one.

It is deterministic on purpose: the timestamp comes from `SOURCE_DATE_EPOCH` or HEAD's committer
date and the serial number is derived from the component list by hash, so the same commit yields
a byte-identical document. The `sbom` job runs it twice and diffs, because a manifest nobody can
regenerate is a manifest nobody can check.

```bash
./Scripts/generate-sbom.sh                 # writes sbom.cdx.json
```

Extraction from Swift sources is anchored and asserted — if a pattern stops matching, the script
fails rather than emitting a smaller SBOM. Adding a dependency, a vendored binary or a model
means the generator either picks it up from the manifest it already reads, or it needs a new
input adding here. A manifest that silently loses a component is worse than none, because it is
believed.

**A release gets a dossier.** `Scripts/release-dossier.sh` writes one Markdown file for a commit
and a built artifact: the commit and its link, the CI run URL when `GITHUB_RUN_ID` is set, the
provenance record and SBOM digests, the artifact's sha256, its `CFBundleShortVersionString` and
`CFBundleVersion` read from the artifact's own `Info.plist` (an `.ipa`, `.app` or `.xcarchive`),
the signing identity from `codesign -dv --verbose=2`, and the list of gates that guard the branch.

```bash
./Scripts/release-dossier.sh --commit <sha> \
  --artifact build/OpenGlasses.ipa \
  --provenance provenance.txt      # downloaded from the CI run's artifacts
```

**When to run it: on a tag, or at App Store submission.** It is not wired into a release
workflow, because there is no release workflow — the store build runs in Xcode Cloud, whose
configuration lives in App Store Connect where the repository cannot see or version it. A job
that has never run is not evidence of anything. The `Release dossier` workflow
(`workflow_dispatch`) produces the source half on demand for any commit; the artifact half needs
running the script on the machine that holds the build.

It reads no secret. `codesign -dv` reports the certificate subject — an organisation name and a
team identifier, both printed on every App Store listing — and never touches a key.

Every field the script could not fill says so in the document, and the reason lands in a **Gaps**
section. A dossier with no artifact says it describes the source and not a release; a dossier
generated from a dirty tree says the artifact may not match the commit. An unstated absence reads
as a claim, so none of them are left unstated. One gap will not close from inside this repository:
Apple re-signs and re-packages what it distributes, so the digest that identifies the shipped
build is the one App Store Connect reports, and it has to be captured there.

**The published website is staged, not the checkout.** `Scripts/stage-pages-site.sh` copies an
explicit allowlist into `_site/` and then independently refuses to publish a tree containing a
denied path. New pages go in `site/`, which is staged to the site root as a tree; anything else
published from outside `site/` is added to that allowlist. `Scripts/check-pages-site.py` then checks
the staged tree: the paths shipped builds read are still there (`Scripts/site-legacy-paths.txt`), no
page loads anything from another site, and every internal link resolves. The vault guide page
(`site/field-assist/vault-guide/`) is converted from `docs/field-assist-vault-guide.md`: after
changing the guide, run `python3 Scripts/build-vault-guide-page.py --pdf` and commit the page and
the PDF, or the check fails (`--pdf` prints `docs/field-assist-vault-guide.pdf` with a local Chrome). To preview locally:

```bash
./Scripts/stage-pages-site.sh && python3 Scripts/check-pages-site.py _site
node Scripts/tests/site-auth-forward.test.js
python3 -m http.server 8000 --directory _site
```


Default generate includes **watch** and **unit tests**. To build a slimmer project locally (iPhone + widget only):

```bash
cp .openglasses-generate.env.example .openglasses-generate.env   # gitignored
./Scripts/generate-xcodeproj.sh
```

Or one-off: `OPENGLASSES_SKIP_WATCH=1 OPENGLASSES_SKIP_TESTS=1 ./Scripts/generate-xcodeproj.sh`

#### Optional: personal signing & Meta config

Team ID, entitlements, and Meta keys differ per developer. Those settings live in **gitignored** files (never committed), merged on top of the shared spec via `project.local.yml`:

| File (gitignored) | Purpose |
|-------------------|---------|
| `project.local.yml` | Team ID + `DEVELOPMENT_TEAM`, personal entitlements paths, and the Meta credentials `MWDAT_META_APP_ID` / `MWDAT_CLIENT_TOKEN_HASH` (see `project.local.yml.example`) |
| `Config/Entitlements/Personal/*.entitlements` | Capabilities your provisioning profile supports |

First-time setup from the templates:

```bash
./Scripts/setup-local-dev.sh
```

Edit `project.local.yml` (`developmentTeam`) and the files under `Config/` as needed, then run `./Scripts/generate-xcodeproj.sh` again.

If you only need Xcode’s automatic signing with the shared entitlements, skip the local overlay and set your team in Xcode after opening the generated project.

Select your iPhone, fix signing if prompted, and run (⌘R).

---


## Configuration

All settings are in-app — no source code editing needed.

### API Keys (Settings → Assistant → AI Models)

| Service | Purpose | Where to Get |
|---------|---------|--------------|
| Anthropic | Claude LLM | [console.anthropic.com](https://console.anthropic.com/) |
| OpenAI | GPT + Realtime | [platform.openai.com](https://platform.openai.com/) |
| Google Gemini | Gemini Live | [aistudio.google.com](https://aistudio.google.com/) |
| Groq | Fast inference | [console.groq.com](https://console.groq.com/) |
| DeepSeek | DeepSeek LLM | [platform.deepseek.com](https://platform.deepseek.com/) |
| Mistral AI | Mistral LLM | [admin.mistral.ai](https://admin.mistral.ai/plateforme/api-keys) |
| ElevenLabs | Natural TTS | [elevenlabs.io](https://elevenlabs.io/) |
| Perplexity | Web search | [perplexity.ai/settings/api](https://perplexity.ai/settings/api) |

### ChatGPT subscription sign-in

Choose **ChatGPT (Subscription)** in Settings → Assistant → AI Models to use the models enabled for your
ChatGPT workspace without adding an OpenAI API key. Normal browser sign-in is recommended on
iPhone. After connecting, **Fetch available models** loads the account-scoped Codex catalog and
uses the model marked as its current default.

For advanced or headless sign-in, expand **Other sign-in options** and choose **Use a device
code**. First enable **Device Code Authorization** in ChatGPT Settings → Security. Enter the
one-time code only at `auth.openai.com` and never share it; Avenkin keeps the device ceremony
in memory and stores only the resulting account credentials in Keychain. See the
[official authentication guidance](https://learn.chatgpt.com/docs/auth).

The existing **OpenAI (GPT)** provider remains separate: it uses an API key, usage-based API
billing, and is still required for OpenAI Realtime voice.

### xAI subscription sign-in

Choose **xAI (Grok)** in Settings → Assistant → AI Models and tap **Sign in with xAI** to use Grok models on a
SuperGrok subscription without an xAI API key. The sign-in token is accepted by the public xAI
API, so the model works like a keyed one — photos and tools included — and **Fetch available
models** lists the chat models the account can use. A key typed into the model is always used
instead of the sign-in. The account credentials are stored in Keychain on this device only.

### Services (Settings → Connections → Services & Integrations)

| Service | Settings |
|---------|----------|
| **ElevenLabs** | API key + voice selection (24 voices) |
| **Web Search** | Optional Perplexity / Tavily / Brave keys + SearXNG instance URL. Order: Perplexity → Tavily → Brave → SearXNG → Tavily free tier (keyless) → DuckDuckGo |
| **Live Streaming** | Platform + RTMP URL + stream key + chat read-aloud (Twitch channel, rate, mentions-only) |
| **OpenClaw** | Enable + connection mode + host/port + token |


Home Assistant's URL and Long-Lived Access Token are under Settings → Everyday → Home, Music & Messages.

---

## Troubleshooting

| Issue | Solution |
|-------|----------|
| Wake word not detecting | Tap mic button to restart; check Bluetooth audio routing |
| No audio through glasses | Verify Bluetooth connection in iOS Settings |
| Glasses not connecting | Tap "Connect to Glasses"; enable Developer Mode in Meta AI app |
| Camera says "Your glasses need an update" | The glasses-side DAT software lags the SDK this app builds against. Update the Meta AI app from the App Store, power-cycle the glasses in their case, and if it persists unlink/relink the app connection in Meta AI. If none of that helps, Meta hasn't shipped the matching glasses update yet — everything else works; camera/streaming resume automatically once the rollout lands |
| Glasses mic silent (beeps play, nothing transcribed) | Bluetooth *audio* pairing is broken while the app link still works — happens after a glasses reset. Forget the glasses in iOS Settings → Bluetooth, put them in the case with the lid open, hold the case button until the LED pulses blue, re-pair via the Meta AI app prompt, then restart the iPhone. Re-grant the app connection + camera permission after |
| HomeKit not finding devices | HomeKit initializes on first tool call — say "list smart home devices" and wait 10s |
| Local model won't load ("not enough memory") | Close other apps in the app switcher and use **Try again** in Download & Manage Models (live headroom shown there), or switch to a smaller model (0.5B–2B) |
| Model download stuck | Keep app in foreground; downloads continue if briefly backgrounded |
| "Untrusted Developer" | Settings → General → VPN & Device Management → Verify (requires internet) |

---
