The Real Cost of Live API Tests

Testing an AI agent by hitting live external APIs (like OpenAI or AbuseIPDB) introduces three massive bottlenecks: exorbitant costs, flaky non-determinism, and unreliable external dependencies.

If your test suite relies on live calls to AbuseIPDB, your entire deployment pipeline is held hostage by their uptime. If their service experiences a 503 outage or your API key hits its daily rate limit, your tests will spontaneously crash. This creates a “false alarm” where your core python code is perfectly fine, but you are blocked from deploying. Furthermore, agent loops are fundamentally multi-turn; letting an un-mocked agent run loops in a test suite will instantly burn through your token budget.

Test Your Logic, Not Your Dependencies

Writing explicit, standalone unit tests to ensure Pydantic fields throw validation errors (e.g., asserting that an invalid string fails a schema check) crosses the line into testing Pydantic’s source code instead of your own. Pydantic is an extensively pre-tested, optimized open-source library—you can trust that its validators work.

Your job is to test your custom business and mathematical logic, such as your severity scoring weights or your timezone calculations. The proper architectural approach is to feed raw dictionary payloads into a Pydantic model inside your test fixtures (e.g., using an alert_factory). This implicitly enforces your schema validation on every single test execution without cluttering your codebase with redundant, low-value schema tests (unit tests).

@pytest.fixture
def alert_factory():
    """Returns a function that generates valid SocAlertSchema objects with sensible defaults."""
    def _create_alert(**overrides):

        # Default valid state for the entire object
        payload = {
            "alert_id": "test-123",
            "source": "splunk",
            "alert_type": "multiple_failed_logins",
            "severity": "medium",
            "timestamp": datetime.now(),
            "entity": {
                "user": "jdoe",
                "host": "workstation-1",
                "ip": "192.168.1.1" # Default to private
            },
            "event_data": {
                "failed_attempts": 0,
                "time_window_minutes": 5,
                "target_system": "vpn"
            }
        }

        has_event_data = "event_data" in overrides
        event_override = overrides.pop("event_data", None)

        if has_event_data:
            if event_override is None:
                payload["event_data"] = None
            else:
                payload["event_data"].update(event_override)


        has_entity = "entity" in overrides
        entity_override = overrides.pop("entity", None)

        if has_entity:
            if entity_override is None:
                payload["entity"] = None
            else:
                payload["entity"].update(entity_override)

        payload.update(overrides)
        return SocAlertSchema(**payload)

    return _create_alert

Mocking HTTP Calls (patch target mechanics)

The reason you must target tools.threat_intel.requests.get instead of the global requests.get comes down to Python’s internal namespace binding.

When your tools/threat_intel.py module is initialized, it executes import requests. This action binds the identifier requests inside that specific module’s local namespace. If your test patches the global requests.get module, you are modifying the base library definition, but tools.threat_intel is already holding its own local copy/pointer to the un-patched requests package. To successfully override the behavior, you must patch the reference where the object is looked up and executed, not where it was originally declared.

@patch("tools.threat_intel.requests.get")
def test_check_ip_reputation_clean(mock_get):

    """ Verifies correct contract translation for a benign IP (Score < 50)."""
    mock_response = MagicMock()
    mock_response.status_code = 200
    mock_response.json.return_value = {
        "data": {
        "ipAddress": "8.8.8.8",
        "abuseConfidenceScore": 5,
        "countryCode": "US",
        "usageType": "DNS Server"
        }
    }

    mock_get.return_value = mock_response

    response = check_ip_reputation("8.8.8.8")

    assert response["is_malicious"] is False
    assert response["risk_score"] == 5
    assert response["source"] == "AbuseIPDB"

Mocking the Agent Loop

When you assign an iterable (like a list) to a mock’s side_effect parameter, the mock transforms into a stateful sequencer: every single time that function is called, it pops and returns the next sequential element in your list.

@patch("agent.loop.enrich_alert")
@patch("agent.loop.load_playbook")
@patch("agent.loop.execute_tool")
@patch("agent.loop.client.responses.create")
def test_run_soc_agent_multi_turn_tool_chain(mock_llm, mock_execute_tool, mock_playbook, mock_enrich):

    """ Multi-Turn Tool Chain Execution (The Investigative Path)"""
    mock_playbook.return_value = "Step 1. Run query."
    mock_execute_tool.return_value = {"status": "success", "found_records": 4}
    
    # Setup Side Effect: Turn 0 requests a tool; Turn 1 returns the final report
    tool_request = make_mock_tool_call(call_id="call_abc123", name="query_logs", arguments_dict={"query": "malicious_user"})
    turn_0_response = make_mock_llm_response(tool_calls=[tool_request])
    turn_1_response = make_mock_llm_response(output_text="Investigation complete. Mitigated malicious session.")

    mock_llm.side_effect = [turn_0_response, turn_1_response]

    # Action
    result = run_soc_agent({"alert_id": "test-002", "source": "splunk", "alert_type": "brute_force", "severity": "high", "timestamp": "2026-06-18T09:00:00Z", "entity": {"user": "attacker", "host": "wkstn-5", "ip": "1.1.1.1"}})
    
    # Assertions
    assert result == "Investigation complete. Mitigated malicious session."
    assert mock_llm.call_count == 2
    mock_execute_tool.assert_called_once_with("query_logs", {"query": "malicious_user"})

This is arguably the most critical pattern for agent testing because an agent operates inside an iterative loop. A multi-turn agent needs to look at an alert, pick a tool, evaluate the tool’s result, and then formulate a final response. If your OpenAI mock only returns a static, unchanging response, your agent will either loop infinitely or break on the second turn. Passing a list of mock objects allows you to script an entire sequential conversation stream, cleanly walking the agent through multi-step reasoning chains deterministically.

The Config Import Trap

The root cause here is that when you write from config import LOGS_PATH, Python evaluates that assignment exactly once at import time. It takes whatever string value LOGS_PATH holds at that millisecond, copies it, and binds it to the local variable cache of your module. When your test executes monkeypatch.setattr(config, "LOGS_PATH", ...) later, it successfully mutates the property on the central config module object, but your target code doesn’t care—it is already executing with its locally cached copy of the original string.

The absolute rule to remember: Never import individual primitives across configuration boundaries; always import the module itself (import config) and reference properties dynamically at runtime via dot-notation (config.LOGS_PATH).

#Wrong pattern
from config import PLAYBOOK_DIR
   
def load_playbook(alert_type: str) -> str:

	ENV_PLAYBOOK_DIR = PLAYBOOK_DIR

#Correct pattern
import config
   
def load_playbook(alert_type: str) -> str:

	ENV_PLAYBOOK_DIR = config.PLAYBOOK_DIR