Learnings from SecAgent Development

Introduction

I am currently interviewing for a SecAI role. To be honest, I didn’t have prior InfoSec experience. While I’ve built various AI agents for different purposes, I hadn’t tackled cybersecurity before. To prepare, I built a specialized cybersecurity agent called SecAgent using pydantic-ai. Rather than attempting to cover every possible task, I focused on a concrete, high-value problem: alert triage analysis.

In this post, I want to share the practical lessons I learned throughout this process.

First, what is alert triage analysis, and why is it crucial? In modern cybersecurity operations, Security Operations Center (SOC) analysts are overwhelmed by the sheer volume of alerts generated daily by firewalls, IDS/IPS, endpoint security tools, SIEM platforms, etc. Analysts must prioritize these alerts and decide which ones require further investigation.

To simulate this environment, I set up Elasticsearch on Docker as a SIEM and generated realistic test alerts with Claude across 6 distinct scenarios:

1. Brute Force: The attacker used an automated Python script to perform a sustained brute-force attack against a single user’s password.
2. Password Spray: The attacker attempted a low-and-slow password spraying attack against 40 different accounts (generating 55 failures overall) from a single external IP.
3. Impossible Travel: The same account authenticated successfully from two geographically distant locations within a very short timeframe.
4. Off-hours Admin: A user authenticated successfully from an anomalous location (Germany) and time (10:45 PM) that fell outside their typical working patterns.
5. Dormant Account: A user account that had been inactive for 120 days was used to log into the network.
6. Service Account Misuse: A service account was used interactively to log in and access sensitive data.

I had Claude generate events for these scenarios and inject them into the Elasticsearch SIEM. You can inspect the raw alert dataset here: Alert Reports (CSV).

The important stuff here is how to design harness, which tools you should use, how to manage context and measure the best practices.

In particular, I want to focus on context compaction, as cybersecurity triage is inherently log-intensive.

Harness Design

As shown in the architecture above, the setup consists of 3 core tools, 5 message-level compaction methods, and tool output compaction.

Core Tools

1. search_events: Fetches authentication events, sorted newest or oldest first.
2. aggregate_events: Counts events grouped by a specified field over time.
3. entity_baseline: Profiles what an entity normally looks like prior to the alert window.

Note: I asked Claude to suggest practical tools for this task and chose these three. If you come from a cybersecurity background and have better recommendations, please feel free to reach out!

Compaction Methods

1. clamp: Uses ClampOversizedMessages from the pydantic-ai library. It takes two arguments (keep_head_chars and keep_tail_chars) and preserves only the first and last $N$ characters of oversized messages.
2. dedupe: As the documentation describes: “When the same file is read more than once, only the latest read keeps its content; earlier reads are blanked with a placeholder.” Since we aren’t reading files in this scenario, its utility is limited here.
3. clear_tools: Replaces older tool results with concise placeholders while keeping the most recent keep_pairs tool-call / tool-return pairs intact.
4. sliding: Retains only the most recent keep_messages messages and discards older ones.
5. summarize: Performs recursive LLM-based summarization of conversation history. This is the most promising method for retaining semantic context, though one must be mindful of added token costs and latency.

Tool Output Compaction

The most effective way to optimize an agent is to understand how it interacts with the environment—identifying what the outputs look like and where bottlenecks emerge.

I used Logfire for tracing and observing agent execution. While inspecting traces, I discovered that the search_events tool frequently returned repetitive log entries:

log 1 - user1 logged in
log 2 - user1 logged in
log 3 - user1 logged in
...
log N - user1 logged in

Instead of returning verbose individual entries, compacting them into something like user1 logged in (N times) saves a substantial number of tokens.

To quantify these bottlenecks, I measured the raw token consumption across different tool calls:

I got something unexpected I just thought search_events was the only tool output bottleneck but it wasn’t. As you can see from above aggregate_events is also very costly for the context window. So it’s best to properly benchmark and measure after something discovered.

After I implemented simple deterministic compaction I got those results in tool output token usage :