One of the biggest surprises while building this project was that prompt engineering for an AI agent is fundamentally different from prompt engineering for a chatbot.

When prompting a chatbot, you’re usually trying to improve the quality of a single response. The model receives a question, generates an answer, and the interaction ends.

An agent is different.

An agent must decide what to do next. It has to determine whether it already has enough information, whether it needs to call a tool, which tool to use, how to interpret the results, and when to stop reasoning. The prompt isn’t just describing the assistant’s personality—it is defining the agent’s execution policy.

That realization changed how I wrote prompts.

Why Generic Prompts Fail

The short answer: The agent rushed to judgment and hallucinated capabilities it didn’t have.

When given the open-ended prompt “You are a SOC Analyst Agent. Investigate the alert and provide a final triage report,” the model bypassed the actual engineering loop entirely. Because LLMs are inherently trained to be helpful and decisive, it looked at the raw alert and immediately spit out a beautifully written, but entirely fabricated, triage report.

Without explicit boundaries, a generic prompt treats the agent like a human who can figure things out on the fly, rather than a code-bound loop that requires explicit state transitions.

SYSTEM_PROMPT_TEMPLATE = """

You are a SOC Analyst Agent responsible for investigating cybersecurity alerts and producing structured triage reports.

# Role

You act as a Tier 1/Tier 2 SOC analyst.

Your responsibilities include:

- analyzing alerts,

- identifying suspicious activity,

- gathering evidence,

- using available tools when necessary,

- assessing risk,

- and recommending next actions.


You are investigative, cautious, and evidence-driven.

# Investigation Procedure

Before reaching a conclusion, you MUST execute your investigation by adhering strictly to the following procedure:

{playbook_instructions}

  
# Output Format

Your final response must contain:

- Alert Summary

- Key Findings

- Risk Assessment

- Triage Decision

(benign, suspicious, malicious, or inconclusive)

- Confidence Score
(0-100)

- Recommended Actions

- Reasoning

  

Keep the report concise, structured, and professional.


# Constraints

You must never:

- invent evidence that was not provided,

- assume tool results,

- fabricate threat intelligence,

- claim certainty without evidence,

- skip investigation steps,

- or classify an alert as malicious without justification.

  

If evidence is insufficient, mark the result as inconclusive.

  

Always prioritize accuracy over speed.

"""

Tool Descriptions Are Code, Not Documentation

Writing tool descriptions for an LLM (llm/schema.py) is entirely different from writing documentation for a human engineer. Humans understand unstated context; models only understand semantic distance and explicit boundaries.

If the description was simply left as “Checks whether an IP address is malicious,” the agent would fall apart in production:

• The failure mode: The model would blindly pass internal private IP addresses (like 10.0.0.1 or 192.168.1.50) to the external AbuseIPDB threat intelligence API. This would cause the external API to throw validation errors, burn through the API quota, and risk leaking internal corporate network topology to the public internet.

• The fix: A strong description acts as a semantic router. By explicitly stating “Use this tool strictly when an alert contains an external source IP… Do NOT invoke this tool for private or internal corporate network IP ranges,” the model uses the description as an if/else logic gate before it ever hits the Python execution layer. Weak descriptions produce indiscriminate, chaotic tool usage; strong descriptions embed the logic of when and how right into the model’s token attention path.

Dynamic Prompting via the Playbook Pattern

Hardcoding every single business rule, security protocol, and tool boundary directly into a single massive system prompt is an anti-pattern. It leads to token bloat, and makes the model’s attention increasingly diluted across a massive context window, and makes updating operational rules a nightmare.

To give an agent deterministic procedural guardrails without overwhelming its context window, you must decouple the agent’s core loop orchestration from its tactical instructions. This is achieved through a dynamic Playbook Pattern (playbook.py, global_default.yaml, multiple_failed_logins.yaml)

What Is a Playbook?

A playbook is simply a structured set of investigation instructions associated with a particular alert type.

Instead of embedding every investigation workflow into the system prompt, the agent loads the appropriate playbook before beginning its analysis.

For example, when the alert type is multiple_failed_logins, the agent loads the corresponding playbook and incorporates its instructions into the investigation process.

The system prompt remains generic, while the playbook provides incident-specific guidance.

This mirrors how many real Security Operations Centers work. Analysts follow standardized investigation procedures depending on the type of alert they receive rather than inventing a new workflow every time.

Why YAML?

I chose YAML because it separates operational knowledge from application code.

The investigation procedure is no longer buried inside Python files or embedded within a thousand-line system prompt.

Instead, each alert type has its own small, readable configuration.

Adding support for a new alert becomes straightforward:

1. Create a new playbook.
2. Write the investigation steps.
3. Associate it with the alert type.

The agent automatically follows the new procedure without modifying the reasoning engine.

This keeps the architecture modular and allows security knowledge to evolve independently of the software.

Building for Failure

One lesson I learned while building tools also applied to playbooks:

External resources eventually fail.

Files get deleted.

Configurations become corrupted.

Someone accidentally commits malformed YAML.

The agent should continue operating even when those things happen.

Instead of assuming every playbook exists, I designed a layered fallback strategy.

When an investigation starts, the loader attempts to:

1. Load the alert-specific playbook.
2. Fall back to a global default playbook if one is unavailable.
3. Fall back again to a minimal built-in investigation procedure if every configuration file fails.

The important point isn’t the fallback itself.

It’s that the agent never crashes because an optional configuration file is missing.

Resilience was treated as part of the design rather than an afterthought.

Separating the Brain from the Procedure

The biggest lesson from introducing playbooks was architectural rather than technical.

The language model is responsible for reasoning.

The playbook is responsible for procedural guidance.

The tools are responsible for gathering evidence.

Each component has a single responsibility.

That separation means I can improve investigation procedures, add support for new alert types, or refine operational best practices without changing the agent loop or rewriting the system prompt.

Instead of one enormous prompt trying to solve every problem, the agent became a composition of smaller, specialized components working together.

In retrospect, playbooks weren’t just configuration files.

They became the mechanism that separated reasoning from domain knowledge, making the entire system easier to extend, maintain, and evolve.

Playbooks solve the problem of what the agent should do. Constraints solve the problem of what it must never do.

The Constraints Section

While the # Role section establishes identity and the # Investigation Procedure dynamically injects the playbook steps, the # Constraints block is the literal anchor of the system prompt. Without it, the agent loop quickly unravels into a series of logical shortcuts.

The Mechanical Impact on the Agent Loop

LLMs are fundamentally built to predict the next most likely token, which means they are biased toward generating complete, satisfying answers—even if they have to cut corners to get there. In an autonomous loop, this manifests as a strong urge to skip calling execute_tool and simply assume what a tool would have returned based on the alert text.

By enforcing strict negative constraints, the behavior of the agent changes in three distinct ways:

• Defeating the “Shortcut” Instinct: Telling the model it must never “assume tool results” or “skip investigation steps” directly counteracts its tendency to finish the report early. It forces the attention mechanism of the model to look at the available TOOLS array and actually trigger a function_call iteration.

• Handling API Failures Gracefully: In the execution loop, if a tool throws an exception, the code catches it and feeds back a failure payload: {"error": "Tool execution failed..."}. Because the prompt strictly states “never invent evidence” and “never fabricate threat intelligence,” the LLM is blocked from pretending the tool succeeded. It is forced to accept the error state as the ground truth.

• Embracing the Inconclusive: Standard chat models loathe ambiguity. The directive “If evidence is insufficient, mark the result as inconclusive” gives the model an explicit, permitted exit path. Instead of forcing a binary guess between benign or malicious when external telemetry times out, the agent safely converges the loopand outputs an honest, bounded assessment.