OAuth Token Refresh & Gmail Integration: Debugging Google Auth State on EC2

```html

During a recent development session, we encountered a critical failure in the Google OAuth token refresh pipeline that blocked automated Gmail operations on our shared EC2 instance. This post walks through the diagnosis, root-cause analysis, and the patch we deployed to restore the authentication flow.

The Problem: Token Refresh Deadlock

Our JADA operations box (EC2 instance ubuntu@34.239.233.28) runs a background job that sends crew notifications via Gmail. The job depends on a valid Google OAuth token stored in a credentials file within ~/repos/.secrets/. During a recent run, the token had expired, and the refresh mechanism failed silently.

When we attempted to trigger a Gmail search for a booking confirmation, the query returned no results—not because the email didn't exist, but because the authentication had degraded to an unauthenticated state. The service was failing open instead of loudly erroring.

Initial Diagnostics: Narrowing the Surface

We began by testing SSH connectivity and SSH key configuration on the box itself. The diagnostic sequence was:

Verify SSH agent was loaded and keys were available
Confirm the correct identity file (jada-key.pem) was being used
List all repositories in ~/repos/ to locate the reauth script
Inspect the secrets directory structure and file permissions

Once we had shell access, we found the main reauth script at ~/repos/reauth_google.py. This script is responsible for refreshing expired OAuth tokens by reading a stored refresh token, calling Google's token endpoint, and writing the new access token back to the credentials file.

Reading the Credentials Flow

The script's core logic reads from a JSON credentials file (path stored as an environment variable or hardcoded default) and uses the Google API client library to refresh the token. We inspected the file to understand the current state:

# Pseudocode structure of what we found:
# - Credentials file location: ~/repos/.secrets/jada_gmail_creds.json
# - Expected fields: client_id, client_secret, refresh_token, access_token, expires_in, token_expiry
# - Refresh method: google.oauth2.service_account or google.auth.transport.requests.Request

The credentials file had the correct structure, but the timestamp on the stored token showed it had been stale for several days. The reauth script should have run on a cron job or been triggered manually, but the refresh call was either not executing or was silently failing.

Root Cause: Path Resolution Issue

After examining the reauth script's code, we found the bug: the script was using a relative path to locate the secrets directory instead of an absolute path. When the cron job or background process invoked the script from a different working directory, the path resolution would fail, and the script would either:

Attempt to read from a non-existent file and catch the exception without logging
Write the refreshed token to the wrong location, leaving the original credentials stale
Fail to open the file due to permission issues (the secrets dir is restricted to 0700 permissions)

The error handling was insufficient—exceptions were being caught and swallowed without any logging to syslog or stderr.

The Fix: Absolute Paths & Better Error Handling

We created a patched version of the script with the following changes:

Use absolute path resolution: Replace all relative path logic with os.path.expanduser('~') or explicit /home/ubuntu/ prefix
Add logging: Import logging module and log all major steps (token read, API call, write success/failure) to a dedicated log file at ~/repos/logs/reauth_google.log
Fail fast: If the credentials file cannot be read or the refresh call fails, raise an exception and exit with a non-zero code so cron can detect the failure
Add a dry-run flag: Allow the script to be invoked with --dry-run to validate the path and file access without making any token changes

The key change to the path logic:

# Before (problematic):
creds_path = '.secrets/jada_gmail_creds.json'

# After (fixed):
creds_path = os.path.join(os.path.expanduser('~'), 'repos', '.secrets', 'jada_gmail_creds.json')
if not os.path.isfile(creds_path):
    logging.error(f"Credentials file not found: {creds_path}")
    raise FileNotFoundError(creds_path)

Deployment & Validation

We backed up the original script to ~/repos/reauth_google.py.bak and deployed the patched version. Then we:

Ran a syntax check with python3 -m py_compile reauth_google.py to ensure no parse errors
Executed the script with --dry-run to validate file paths and permissions
Checked that the log file was being written to ~/repos/logs/reauth_google.log
Ran the script in production mode and verified the access token was updated in the credentials file
Tested the Gmail integration by running a search query against the Gmail API using the refreshed token

Key Decisions & Rationale

Why absolute paths? Cron jobs and background services often run from unpredictable working directories. Hardcoding relative paths is a common source of "works on my machine" bugs. Absolute paths ensure the script finds the credentials file regardless of where it's invoked from.

Why logging instead of silent failure? The original script swallowed exceptions, making it impossible to diagnose problems without manually running it or checking the credentials file timestamp. Structured logging to a dedicated file allows us to run periodic audits and detect token refresh failures early.

Why a dry-run mode? This allows operators to validate that the script can find and read the credentials file without actually making API calls or modifying state. It's a safety net for deployment and troubleshooting.

What's Next

With token refresh working reliably, we can now:

Add the patched script to the EC2 instance's cron configuration to refresh tokens daily (or on a schedule matched to your token expiry window)
Set up CloudWatch Logs or syslog aggregation to monitor reauth_google.log for any failures
Extend this pattern to other OAuth-dependent services (Drive API, Calendar API, etc.) that may have the same path-resolution issue
Document the secrets directory structure and rotation procedures for team onboarding

The fix is minimal and surgical—just path handling and logging—but it eliminates an entire class of silent failures in our auth pipeline.

```