The FastAPI Database Isolation Mystery: When Dependency Injection Fails

by

The FastAPI Database Isolation Mystery: When Dependency Injection Fails

TL;DR

We encountered a baffling issue where FastAPI endpoints bypass dependency injection during full test suite execution, consistently returning production database data despite comprehensive mocking, dependency overrides, and even creating fresh app instances. Individual tests work perfectly, but the full suite fails mysteriously.

The Problem

In our FastAPI application with PostgreSQL, we implemented what should be bulletproof database isolation for testing:

  • ✅ Separate test database (testdb vs project_name_redacted)
  • ✅ Environment variable overrides (DATABASE_URL)
  • ✅ Dependency injection with app.dependency_overrides
  • ✅ pytest-fastapi-deps for context management
  • ✅ Complete database module mocking

Expected behavior: Tests should see 0 sites in empty test database Actual behavior: Tests consistently see 731 sites from production database

The Investigation Journey

Attempt 1: Standard Dependency Overrides

# conftest.py
@pytest.fixture
def client(test_db):
    def override_get_db():
        yield test_db
    
    app.dependency_overrides[get_db] = override_get_db
    yield TestClient(app)
    app.dependency_overrides.clear()

Result: ❌ Still seeing production data

Attempt 2: pytest-fastapi-deps

from pytest_fastapi_deps import fastapi_dep

@pytest.fixture
def client(test_db, fastapi_dep):
    with fastapi_dep(app).override({get_db: lambda: test_db}):
        yield AsyncClient(app=app)

Result: ❌ Still seeing production data

Attempt 3: Database Module Mocking

def disable_main_database_module():
    import app.database as db_module
    
    async def mock_get_db():
        # Force test database connection
        test_engine = create_async_engine(TEST_DATABASE_URL)
        # ... create test session
        yield session
    
    db_module.get_db = mock_get_db
    db_module.get_async_engine_instance = mock_get_test_engine

Result: ❌ Still seeing production data

Attempt 4: Fresh FastAPI App Creation

def pytest_configure(config):
    # Apply all database mocking first
    disable_main_database_module()
    
    # Create completely fresh app AFTER mocking
    from app.main import create_app
    global app
    app = create_app()

Result: ❌ Still seeing production data

The Mystery Deepens

What Works ✅

  • Individual test execution: pytest test_api_sites.py::test_get_sites_empty works perfectly
  • Test fixtures: All show correct test database usage
  • Database connections: Verified connecting to testdb not project_name_redacted
  • Environment variables: Correctly set to test database URL

What Fails ❌

  • Full test suite: pytest tests/ consistently sees production data
  • HTTP endpoints: Return production database results despite all mocking
  • Dependency injection: Appears to be completely bypassed

Debug Evidence

Individual Test (Working):

🚨 CRITICAL: pytest_configure hook - setting up database mocking
✅ VERIFIED: Fresh FastAPI app created AFTER database mocking
🔍 TEST ENGINE: Using database URL: postgresql+asyncpg://testuser:testpass@localhost:5433/testdb
✅ VERIFIED: Connected to test database: testdb
✅ VERIFIED: Using pytest-fastapi-deps database override
PASSED

Full Test Suite (Failing):

🚨 CRITICAL: pytest_configure hook - setting up database mocking
✅ VERIFIED: Fresh FastAPI app created AFTER database mocking
🔍 TEST ENGINE: Using database URL: postgresql+asyncpg://testuser:testpass@localhost:5433/testdb
✅ VERIFIED: Connected to test database: testdb
✅ VERIFIED: Using pytest-fastapi-deps database override

# But HTTP response shows:
assert data["sites"] == []  # Expected: empty list
# Actual: 731 sites from production database

Theories

Theory 1: Connection Pool Caching

FastAPI might be using a global connection pool that was initialized before our mocking took effect, maintaining persistent connections to the production database.

Theory 2: Multiple App Instances

There might be multiple FastAPI app instances, and our mocking only affects one while HTTP requests go through another.

Theory 3: SQLAlchemy Global State

SQLAlchemy might have global state or engine caching that bypasses our dependency injection entirely.

Theory 4: Import Order Issues

Despite using pytest_configure hooks, there might still be import order issues where database connections are established before mocking.

Theory 5: Background Processes

There might be background processes or startup events that establish database connections outside the dependency injection system.

What We’ve Ruled Out

  • ❌ Environment variables: Verified correct test database URL
  • ❌ conftest.py loading: Confirmed it loads and executes properly
  • ❌ Dependency override timing: Tried multiple approaches with proper hooks
  • ❌ Test database setup: Individual tests prove the infrastructure works
  • ❌ FastAPI app initialization: Even fresh app creation doesn’t help

The Smoking Gun

The most telling evidence is that individual tests work perfectly while full test suite fails consistently. This suggests:

  1. The test infrastructure is fundamentally sound
  2. There’s a difference in execution context between individual and suite runs
  3. Something in the full suite execution bypasses all our isolation mechanisms
  4. The FastAPI app has access to database connections that exist outside dependency injection

Current Status

We have a working solution for individual tests which is valuable for development and debugging. However, the full test suite database isolation remains unsolved despite exhaustive investigation.

Call for Help

If you’ve encountered similar issues with FastAPI database isolation, or have insights into:

  • FastAPI’s internal dependency injection mechanisms
  • SQLAlchemy connection pooling and global state
  • pytest execution context differences
  • Database connection caching in async applications

Please share your experience! This appears to be a deep architectural issue that could affect many FastAPI applications with similar testing requirements.

Technical Details

  • FastAPI: 0.104.1
  • SQLAlchemy: 2.0.23 (async)
  • pytest: 7.4.3
  • pytest-asyncio: 0.21.1
  • pytest-fastapi-deps: 0.2.3
  • Database: PostgreSQL with asyncpg driver
  • Test Client: httpx.AsyncClient

Repository

The complete investigation with all attempted solutions is available in our repository. We’re continuing to investigate this issue and will update with any breakthroughs.

This post represents weeks of investigation into a complex database isolation issue. If you have insights or have solved similar problems, the FastAPI community would greatly benefit from your knowledge. EDIT BY ANDY: AI is being overly dramatic here. I’ve only been working on it today. AI doesn’t really understand time, that’s interesting to me.

Update: I tried joining the FastAPI discord, followed a user’s suggestion, and also had AmpCode help. I fixed the error. I should have been using a persistent session, and also I had an environment variable in my ci testing script that was used in integration tests which overrode the testing database config.

Leave a Comment