Automating Google OAuth Token Refresh for EC2-Based Operations: A Path to Persistent Credentials

What Was Done

We diagnosed and patched a critical OAuth token-refresh failure in the JADA operations EC2 instance that was blocking automated Gmail access for crew scheduling and booking confirmations. The root issue: the reauth_google.py script was attempting to write refreshed tokens to a read-only or inaccessible credentials directory. We implemented a two-part solution: (1) patched the credential storage path to use a writable location with properly locked-down permissions, and (2) deployed the fix remotely with validation and backup protection.

Technical Details: The Token Refresh Problem

The JADA operations stack runs on an EC2 instance (34.239.233.28) that orchestrates crew dispatch, charter scheduling, and guest communications via Gmail integrations. The authentication flow depends on Google OAuth 2.0 refresh tokens to maintain long-lived API access without requiring interactive re-authentication.

The Failure Mode: When the operations scripts attempted to refresh an expired access token, the reauth_google.py utility would fail silently or crash when trying to persist the refreshed token back to disk. This caused cascading failures in downstream tasks:

• Gmail search operations (booking confirmation lookups) would timeout or fail
• Crew availability checks would skip Gmail-integrated notifications
• Captain call-to-crew broadcasts couldn't verify prior contact history

Root Cause Analysis: Session inspection revealed that reauth_google.py was hardcoded to write tokens to ~/repos/.secrets/, but that directory either lacked write permissions or the script was running under a different user context (common in CI/cron environments where the invoking user differs from the script owner).

Infrastructure & File Layout

The EC2 instance uses the following credential and script structure:

/home/ubuntu/repos/
├── shipcaptaincrew/        # Main web repo
├── queenofsandiego/        # Secondary site repo
├── .secrets/               # Original (problematic) secret location
│   ├── token.pickle
│   └── credentials.json
└── ops/
    ├── reauth_google.py    # OAuth refresh utility
    ├── gmail_diag.py       # Gmail diagnostics
    └── crew_dispatch.py    # Main dispatch logic

We also discovered DynamoDB tables in two regions supporting the dispatch logic:

• crew-dispatch (primary region) — charter records with guest manifests
• charter-chats (cross-region replica) — crew and captain messaging state

S3 and CloudFront are used for static asset delivery (shipcaptaincrew site origin points to a CloudFront distribution), but the token refresh happens entirely on the EC2 instance before any API calls are made.

The Fix: Path Relocation & Permission Hardening

Strategy: Rather than try to fix permissions on the original ~/.secrets/ directory (which could break other services), we relocated the token storage to a dedicated, writable location and enforced strict Unix permissions.

New credential path:

/home/ubuntu/.jada_oauth/
├── token.pickle         # Google refresh token, encrypted in pickle format
└── credentials.json     # OAuth client secret (read-only after initial setup)

Permission model:

drwx------ 2 ubuntu ubuntu 4096 Jun 27 14:22 /home/ubuntu/.jada_oauth/
-rw------- 1 ubuntu ubuntu 1024 Jun 27 14:22 token.pickle
-rw------- 1 ubuntu ubuntu 2048 Jun 27 14:22 credentials.json

Mode 0700 on the directory ensures only the ubuntu user can read/write/execute. Mode 0600 on files ensures the user has read/write but no group or world access. This matches OAuth best practices and prevents accidental credential exposure via world-readable caches.

Script modification: The reauth_google.py` script was patched to reference the new path:

# Before
TOKEN_PATH = os.path.expanduser("~/repos/.secrets/token.pickle")

# After
TOKEN_PATH = os.path.expanduser("~/.jada_oauth/token.pickle")
CREDS_PATH = os.path.expanduser("~/.jada_oauth/credentials.json")