Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Managing Complex OAuth Token Scopes in Multi-Purpose Email Automation Systems

What Was Done

During a recent session working with JADA's email and calendar automation infrastructure, we diagnosed and resolved critical OAuth token scope issues that were blocking the deployment of new email integration features. The core problem: our authentication system was attempting to use legacy, narrowly-scoped tokens for operations that required expanded Gmail API permissions, causing silent failures in draft creation and email modification workflows.

The solution involved:

  • Implementing a unified token file structure (/Users/cb/icloud-repos/tools/reauth_jada_all.py) that consolidates multiple OAuth scopes into a single, refreshable credential
  • Creating three companion automation scripts to manage the OAuth flow end-to-end
  • Building token-scope validation into the authentication pipeline to prevent silent degradation
  • Establishing a fallback mechanism for legacy token files while migrating to the unified approach

The Problem: Scope Fragmentation

The original infrastructure maintained separate token files for different use cases:

  • tokens/jada_google_tokens.json – "unified" token with broad Gmail scopes, but legacy and inconsistently used
  • tokens/jada_google_tokens_gmail_full.json – Gmail Full API access, but only for specific operations
  • Various API helper functions expecting different token sources

When jada_google.py attempted to create Gmail drafts via the compose API, it would:

  1. Load the wrong token file (narrow scope)
  2. Attempt the API call without the required gmail.compose scope
  3. Receive a 403 error that went unhandled in the calling code
  4. Fall back silently, leaving the user waiting for a draft that never arrived

Even worse, when we tried to refresh that narrow token, the OAuth provider would reject it with a 400 error because the token itself had been created with limited scopes and couldn't be "upgraded" via refresh.

Technical Implementation: Unified Token Strategy

Step 1: Re-authentication with Full Scopes

We created reauth_jada_all.py, which orchestrates the complete OAuth 2.0 authorization code flow with all required scopes bundled:

python reauth_jada_all.py

This script:

  • Constructs a Google OAuth URL with scopes: calendar, gmail.full, gmail.compose, gmail.modify
  • Spins up a local HTTP server on a configurable port (default 8080) to receive the OAuth redirect
  • Prints the authorization URL to the terminal
  • Polls for user consent, with a configurable timeout (we used 2-hour windows during testing to accommodate mobile workflows)
  • Exchanges the authorization code for tokens using stored OAuth client credentials
  • Writes the new tokens to the unified token file

Step 2: Post-Auth Finalization

After tokens are issued, finish_auth.py runs to:

  • Verify the token was written correctly
  • Test token validity by making a lightweight API call (list calendars)
  • Confirm all required scopes are present in the token metadata
  • Log success or detailed failure information for debugging

Step 3: Email-Based Auth Confirmation

For workflows where the operator isn't watching the terminal, watch_auth_email.py monitors the Gmail inbox for an OAuth confirmation email and:

  • Detects when Google sends the "new device sign-in" notification
  • Waits for manual confirmation by the account owner on their phone/browser
  • Polls the token file (checking modification time and scope metadata) to confirm the refresh succeeded
  • Signals completion or timeout after configurable duration

Infrastructure and Token File Organization

We consolidated the token file layout to a single source of truth at /Users/cb/icloud-repos/tools/tokens/:

tokens/
├── jada_google_tokens.json         # Unified, multi-scope token
├── .gitignore                       # Ensure tokens are never committed
└── README.md                        # Scope and refresh strategy docs

Each token file contains metadata that jada_google.py now validates at startup:

{
  "access_token": "...",
  "refresh_token": "...",
  "scopes": ["https://www.googleapis.com/auth/calendar",
             "https://www.googleapis.com/auth/gmail.full",
             "https://www.googleapis.com/auth/gmail.compose",
             "https://www.googleapis.com/auth/gmail.modify"],
  "token_type": "Bearer",
  "expiry": "2026-07-06T18:30:45Z"
}

The jada_google.py helper now includes a scope-checking function that runs before any API call:

def has_scope(token_path, required_scope):
    """Verify token has required scope before attempting API call."""
    with open(token_path) as f:
        token_data = json.load(f)
    return required_scope in token_data.get('scopes', [])

This prevents silent failures and provides clear error messages when the token needs re-authorization.

Key Decisions and Trade-offs

Decision 1: Single vs. Multi-File Token Strategy

We chose a single unified token file instead of maintaining separate tokens for each scope. Why: it simplifies token refresh (one API call vs. many), reduces the cognitive load on automation code, and eliminates the inconsistency that was causing the original bugs. The downside is that a compromise of the single token file grants broader access, but in this case, all operations (calendar, email, draft creation) are internal JADA business workflows.

Decision 2: Timeout Windows and Listener Duration

The OAuth server listener defaults to 2 hours instead of the typical 5-10 minutes. Why: operators often handle this workflow between other tasks and may not be immediately available to complete consent on their phone. A shorter window would cause the script to exit prematurely and require re-running. Two hours balances convenience against security (the listener port is only open locally).

Decision 3: Scope Validation Before API Calls

Rather than relying on API error codes, we check token scopes ahead of time. Why: this gives us fast, clear feedback at the point where the operator is making a decision (e.g., "create draft" or "add calendar event"), instead of letting the call fail silently downstream and having the operator wonder why nothing happened.

Monitoring and Future Work

The token file system is now monitored in two ways:

  • File modification time: watch_auth_email.py polls the mtime of the token file to detect when the OAuth process has completed and written new tokens
  • Scope metadata: On each API client initialization, we log the token's scope list and expiry time to catch silent downgrades or expirations

For future improvements, we plan to:

  • Add automatic token refresh when expiry is within 10 minutes (preventing API failures mid-operation)
  • Implement scope-specific sub-tokens if we need to limit blast radius further (though not yet required)
  • Build a status dashboard that shows current token state, last refresh time, and which operators have valid credentials

The result is a more robust, debuggable email automation layer that fails fast with clear error messages instead of silently degrading — critical for operational workflows where an operator may be waiting for a confirmation email that never arrives.

```