A product manager’s field guide to turning agent capabilities into useful products: understanding customer needs, working through messy data, and delivering features people can trust.
An analyst’s attention is finite. The data keeps growing.
These examples come from my work owning Illumio Insights Agent from inception through launch.
Before the agent, security analysts moved between country insights, malicious IPs, and risky traffic views. They applied filters and connected evidence around individual workloads.
Across hundreds of cloud accounts and hundreds of thousands of workloads, that workflow demanded more attention than teams could realistically provide.
The product opportunity
Have the agent perform that initial investigation and present the most interesting findings within minutes. Give analysts focused starting points for deeper investigation.
THE PRODUCT PRINCIPLE
Start with the customer’s workflow. Define the useful work an agent can take on.
A finding is a starting point for investigation. The analyst still evaluates its significance.
Early versions of the agent repeatedly surfaced expected behavior from load balancers, NAT gateways, scanners, and patching services. Bursts could resemble suspicious activity without the context needed to interpret them.
Listen where the work happens
I sat with early adopters and our own security team, reviewing reports together. The same false positives kept returning. I prioritized that adoption barrier and worked with design on two ways to respond: one-click “known behavior” feedback and free-text explanations.
Give the agent the missing context
Customer labels supplied environment, location, application, and role. Feedback added explanations that those labels couldn’t provide.
We stored the finding’s available context—including workloads, addresses, labels, and timing—as embeddings for later retrieval. Similar future activity could be rescored or suppressed using that feedback.
Keep meaningful changes visible
We tested new peers and different ports or services between existing peers. Prior feedback informed the assessment; changed behavior could still meet the reporting threshold.
WHAT I LEARNED
An AI feature must account for the work its outputs create.
In our experience, this reduced recurring noise and investigation friction. This is a qualitative observation.
Comparing adjacent daily periods gave the agent too narrow a view of normal activity. A weekend, quiet period, or temporary traffic spike could dominate the comparison.
SYNTHETIC TIMELINE / SAME ACTIVITY
Change the comparison
We moved toward week- and two-week comparisons so the agent could observe more recurring behavior across workloads, applications, and environments.
I think of it like spending: two days of shopping while on vacation can give a misleading impression of an ordinary month. A broader window supplies context.
WHAT I LEARNED
The observation window shapes the conclusions a product can responsibly draw.
In our experience, broader history helped reduce false positives. The useful window depends on the behavior being investigated.
Broader history improved interpretation, but increased retrieval work. We examined the size, grouping, and concurrency of the agent’s requests to our data sources.
Scope the work, then run it concurrently
An initial agent organized the investigation into clusters. Smaller, targeted requests retrieved relevant data for each cluster and period concurrently. The investigation brought those results together into findings.
CONCEPTUAL RETRIEVAL FLOW
Investigation→
Scoped requestScoped requestScoped request
→Findings
Bring the wait into the workflow
In the investigations I observed, waits that could reach roughly 30–60 minutes came down to minutes. That made deeper investigation more practical for customers.
WHAT I LEARNED
Investigation quality and turnaround time have to be designed together.
Timing reflects my reported experience, rather than a formal benchmark or service guarantee.