Debugging and Fixing Google OAuth Token Scope Issues in a Multi-Service Python Toolchain

What Was Done

During a development session focused on JADA operations tooling, we diagnosed and fixed a critical Google OAuth token refresh failure that was blocking the reauth_jada_all.py script from properly re-authorizing the application with Gmail API scopes. The root cause was a mismatch between legacy token files (which had only read-only Gmail scopes) and newer unified token files (which needed gmail.compose scope for draft creation). We implemented a polling-based token validation loop, proper timeout handling, and detailed scope inspection to ensure tokens were correctly refreshed with the full permission set needed by the application.

Technical Details: The Problem

The JADA toolchain stores Google auth tokens in two locations:

/Users/cb/icloud-repos/tools/token.json (unified multi-service token)
/Users/cb/icloud-repos/tools/token_gmail.json (legacy Gmail-only token with limited scopes)

When the application attempted to create Gmail drafts via the legacy token file, the Google API returned a 400 error during token refresh. The token had been granted only https://mail.google.com/ (read-only) and https://www.googleapis.com/auth/gmail.modify (label/mark operations), but lacked https://www.googleapis.com/auth/gmail.compose — the scope required to programmatically create drafts.

The script in /Users/cb/icloud-repos/tools/reauth_jada_all.py had a timeout issue as well: it waited only 3 minutes for the OAuth consent flow to complete before exiting, which wasn't long enough for all user interactions during the consent grant process.

Solution: Token Validation and Polling

We modified reauth_jada_all.py to:

Validate token scopes before use: Added inspection logic to read token files and check for required scopes before attempting API calls.
Implement a callback-server-based OAuth flow: The script now runs a local HTTP server on localhost:8080 to receive the OAuth callback, rather than relying on manual copy-paste of auth codes.
Increase and monitor timeout windows: Extended the callback wait from 3 minutes to 15 minutes (900 seconds), allowing sufficient time for the user to complete Google's consent flow in a browser.
Add polling-based token refresh verification: After re-auth completes, the script polls the unified token file every 2 seconds (max 4 hours of polling) to confirm that the new scopes have landed in the token file, rather than immediately trying to use the token.

Key code pattern in the validation loop:

def poll_for_scope(token_path, required_scope, poll_interval=2, max_duration=14400):
    """Poll token file until required_scope is present."""
    start = time.time()
    while (time.time() - start) < max_duration:
        try:
            with open(token_path, 'r') as f:
                token = json.load(f)
                scopes = token.get('scopes', [])
                if required_scope in scopes:
                    return True
        except (FileNotFoundError, json.JSONDecodeError):
            pass
        time.sleep(poll_interval)
    return False

Infrastructure and File Organization

We created supporting documentation files to ensure future maintainers understand the token strategy:

/Users/cb/icloud-repos/tools/CLAUDE.md — Documents project context, goals, and architectural decisions
/Users/cb/icloud-repos/tools/CONTEXT.md — Lists external dependencies, service integrations, and scope requirements

The unified token approach consolidates multiple Google service authentications into a single flow, reducing user friction (one consent grant instead of multiple). The token file structure includes a scopes array that we inspect programmatically to validate permissions before attempting operations.

Key Decisions and Rationale

Why polling instead of event-based notification? Google's OAuth response includes an expires_in field but does not guarantee immediate token file updates on disk. Polling the JSON file ensures we observe the actual state the application will use, avoiding race conditions between token issuance and persistence.

Why 15-minute timeout? The OAuth flow requires user interaction: Google may show account choosers, prompt for security verification, or display the consent screen. 15 minutes is long enough for interactive flows without leaving the server open indefinitely. The script can run in the background while the user handles these steps in their browser.

Why check scopes before using tokens? Attempting API calls with insufficient scopes produces cryptic 403/400 errors from Google. Pre-flight scope validation surfaces the issue clearly: "Token lacks gmail.compose scope" is far easier to debug than "Invalid Credentials" from a downstream API call.

Command-Line Usage

The re-auth script is invoked without arguments:

python3 /Users/cb/icloud-repos/tools/reauth_jada_all.py

This triggers:

Local HTTP callback server starts on localhost:8080
OAuth consent URL is printed to stderr
Browser opens (or user manually navigates) and completes consent
Token callback is received; server shuts down
Polling loop checks for scopes in /Users/cb/icloud-repos/tools/token.json
Once scopes are confirmed, the script exits (upstream automation can now proceed)

What's Next

Future improvements:

Centralize scope definitions (currently hardcoded; should be a config constant)
Log token refresh events to a monitoring endpoint so we catch future scope mismatches earlier
Implement graceful degradation: if a token lacks a scope, queue a re-auth request instead of failing the operation
Document the complete scope matrix (which scopes each operation requires) in CONTEXT.md