I got dropped into an environment with no documentation and no logging. Not thin documentation — none. No meaningful visibility into what was running, what was talking to what, or which identities were behind any of it.
The first job wasn’t securing anything. It was finding out what existed.
What I ended up building was not really a security system. It turned into the instrumentation layer for the entire organization, and the thing that made that possible wasn’t better data. It was that asking questions of the data got cheap. Once inquiry costs minutes instead of days, security telemetry stops being a security tool and starts being the fastest way to know anything about how the business actually operates.
That’s the whole argument. Here’s how I got there.
Start at the hub
I began with sign-in logs from Microsoft Entra.
Your identity provider is the hub of your environment. If you’re going to get solid visibility anywhere first, get it there. Everything authenticates somewhere, which makes the IdP the place you find out what’s actually on the grid versus what’s running in the shadows.
That one source answered more than I expected. Usage patterns. Suspicious users. Idle applications and dormant identities. And most usefully, the blind spots — the places where I could tell something was happening but couldn’t yet see it. That became the list of what to light up next.
After that it was the unglamorous grind. Add a data source, normalize it, work out what its timestamps actually mean, then do it again with the next one. Systems, SaaS applications, anything that emitted a log line. Month after month.
What “before” looked like
Most organizations have what we had: many disparate log sources, each with its own access control model, its own query language, and its own data store. Genuinely useful, all of it. But making sense of it required a smart human to sit down and correlate across five systems by hand.
That works. It works at human speed, which is the constraint that quietly shapes everything downstream of it.
The meeting
Three to six months into plugging away, the MCP revolution started kicking off. Elasticsearch — our SIEM — was very early in recognizing their position in the market and shipped an agent builder MCP server.
I minted myself a PAT and started experimenting.
The first real query I ran was during our annual planning meeting.
Our CISO was worried about our SaaS footprint — a reasonable thing to be worried about. He was working through how to move resources around to stand up a team of business analysts whose entire job would be picking the problem apart. Who’s behind SSO and who isn’t. Which applications are sanctioned, which aren’t. At the time, that genuinely was a team-and-a-quarter question.
In about ten minutes I handed him a list of applications that weren’t behind SSO, a full shadow IT report, and a prioritized list of problems to go solve.
Everyone in the room just sort of looked at each other.
Among the findings: three to five applications nobody knew existed. No active users, no identifiable owner, nobody who could say what they had been for. Six figures of derelict opex, auto-renewing quietly in the background.
Nobody had asked me to go looking for wasted spend. It fell out of a security review, because the security stack was the only thing instrumented well enough to notice. That’s security tooling paying for itself in a line item nobody would think to attribute to it.
The interesting part was never the speed. It was what the speed did to the questions. When an answer costs a day, you only ask what’s worth a day — and you never notice the questions you’ve quietly stopped asking. Drop the cost to minutes and that whole silent backlog opens up.
“Security data” is the wrong frame
Here’s where I’d push back on how most people talk about this.
The line between security and non-security is genuinely blurry now. What you actually have is the ability to answer a technical question with accurate, data-backed claims, queried at line speed. Whether that question is a security question is a statement about the use case — not about the data, and not about the systems involved.
Your authentication logs know who works odd hours. Your endpoint telemetry knows which machines are underprovisioned. Your sign-in data knows which SaaS tools nobody has opened in six months — a licensing question, right up until you notice that an application nobody is watching is also an application nobody is patching.
That’s the asset. Security tooling gets permission to instrument the whole environment; almost nothing else does. Treating the output as though it only answers security questions leaves most of its value sitting unused.
Two things got cheap, not one
Everything above is about knowing getting cheap. That’s half of it.
The other half is that doing got cheap at the same time, and the two compound in a way that’s easy to miss when you look at either one alone.
The underlying shift is unglamorous: we went from autocomplete to systems you can actually hold a conversation with. What that bought me was the ability to export my own knowledge and process into repeatable workflows — fleets of agents, running supervised or not — without the fragility and development overhead that automation always used to demand.
A system that can answer any question about your environment is interesting. A system that can answer the question and then act on the answer is a different category of thing. Cheap inquiry tells you the SaaS application has no owner. Cheap execution is what makes it plausible that something actually gets done about it, this week, without a project.
Agents aren’t a new identity problem — mostly
If you’re going to point agents at this data, the obvious question is how you secure the agents themselves. People tend to treat that as uncharted territory. From an IAM perspective it largely isn’t.
An agent is just another service account, or just another PAT, depending on the use case. A decade of zero trust architecture, modern authentication and cloud-native identity has been building toward exactly this, whether or not anyone framed it that way at the time. The primitives already exist.
Most of it comes down to two questions:
Is the agent an extension of you, or an autonomous system? These are different trust models and they shouldn’t share a credential strategy. Worth noting that the boundary drifts in practice — something that starts as an extension of you, then gets put on a schedule, has quietly become an autonomous system, and almost nobody re-reviews it at that point.
If it’s an extension of you — does it need your full access, or specific scopes? Almost always the latter. The failure mode is boring and predictable: scoping is work, handing over your own credentials is not, and you find out what “full access” actually meant at the worst possible moment.
Where the service account analogy breaks down is worth being honest about. A service account does what it was configured to do. An agent does what it was persuaded to do — by its instructions, and by whatever it happens to read along the way. The credential model transfers cleanly. The threat model doesn’t, entirely. Scope on the assumption that an agent’s effective permissions are whatever a motivated stranger could talk it into using.
Where it actually gets hard
Outside of IAM, this gets considerably more complex. The problems worth losing sleep over look like this:
- How do you route sessions to your preferred and controlled inference providers?
- How do you block the providers you haven’t approved?
- Dynamic tool access — what the agent can reach changes at runtime.
- Long-running workflows that outlive the approval that authorized them.
- Context and instruction manipulation.
- Prompt introspection and monitoring.
- Harness management and oversight.
- Remote versus local sandboxes.
Notice that most of those aren’t AI problems. Provider routing and egress control are network problems. Sandboxing is an endpoint problem. Harness management and oversight are developer experience problems. They’re challenges of their respective domains wearing a new hat, which is good news — you’re not starting from zero. The organizations that struggle here will mostly be the ones that already struggled with egress control and endpoint isolation, now discovering it at higher stakes.
Three of them, though, don’t decompose so neatly.
Dynamic tool access
Traditional authorization assumes you enumerate capabilities at design time. A service gets access to a defined set of APIs, someone reviews that grant, and the set stays put until a human files a change request.
Agents don’t work that way. The tool set is assembled at runtime, servers get added, and the agent chooses which capabilities to invoke and in what order. Your review gate assumed a human would ask permission. Now the requester is the thing you’re trying to constrain.
The sharper problem is composition. Give an agent a tool that reads from your secrets manager and a tool that posts to a chat channel. Neither is dangerous. Together they’re an exfiltration path, and nobody granted “exfiltrate” — it emerged from two reasonable approvals. Permission models built to evaluate individual grants have nothing useful to say about what those grants become in combination.
Worth adding: tool descriptions are themselves input the model reads. A tool registry is a supply chain, and it deserves the same suspicion you’d apply to any other dependency you didn’t write.
Long-running workflows
Session security assumes short-lived credentials and a transaction that starts and finishes. An agent working a task for six hours breaks both assumptions.
Blast radius scales with duration. A compromised request is an incident; a compromised long-running agent is persistence. And revocation gets genuinely awkward — killing a workflow mid-flight can leave state half-changed in three systems, so the safe move and the correct move stop being the same move.
The subtler issue is that approval decays. A human authorized the task at step one. By step four hundred the agent is doing things that approver never contemplated — not maliciously, just downstream of a chain of individually sensible decisions. Consent was granted once, at a granularity that made sense then, and it’s still being spent.
Context and instruction manipulation
This is the one with no clean answer, and I’d be suspicious of anyone who tells you otherwise.
We solved SQL injection by separating code from data — parameterize the query and the problem goes away structurally. There is no equivalent for a context window. Instructions and data arrive through the same channel, as the same tokens. The separation that fixed injection everywhere else isn’t available here.
Which means anything the agent reads can potentially instruct it. A web page. A ticket description. A filename. A log line.
That last one should give you pause, given everything I’ve argued above. If you point an agent at your security telemetry, you’ve built a pipeline where anyone who can write to a log you ingest can write into your agent’s context. Attacker-controlled strings land in logs constantly — that’s most of what a failed login is. Usually inert. Not inert if something downstream reads it with authority and a tool belt.
I don’t think that’s a reason not to do this. It is a reason to separate reading from acting: let the agent that queries your data lake summarize and recommend, and make the agent that changes anything take its instructions from a human, through a channel the data can’t reach. Assume retrieved content is hostile. Keep authority away from the components handling it. And on long-lived sessions, remember that something injected early sits in context for everything that follows.
The trap
The mistake most people are making right now: using AI to do the same work, with the same methods, they were already using.
Execution is now free. Features that once required thoughtful allocation of expensive developer resources can be built as a background task while you’re sitting in a meeting.
But build cost was never really the deterrent. The maintenance tax was.
Every experienced engineer has run this calculation, usually without saying it out loud: I could script this. It would take an afternoon. And then I would own it forever. It breaks when the vendor changes their API. It breaks when someone rotates a credential. It breaks when a log format shifts by one field and fails silently for three weeks before anybody notices. The afternoon was never the cost. The cost was signing up to be the person who fixes it, indefinitely, usually at an inconvenient moment.
So we didn’t script things. Not out of laziness, and not out of ignorance — out of discipline. There was an entire category of problems everyone knew was automatable and nobody automated, because doing it by hand every few weeks was predictable toil while owning a brittle script was an unbounded liability. Manual work has a known cost. Maintenance has a tail.
That calculus has inverted, and not because building got cheaper. Because repair did.
“Got this error, please fix” is now a complete bug report and, most of the time, a complete resolution. The thing that made automation expensive to own — that every shift in the environment demanded a human who still remembered how the script worked — largely isn’t true anymore. The script that would have rotted in a quarter now survives without becoming anyone’s second job.
Which is exactly why the trap is easy to fall into. The old judgment was correct. Engineers who refused to automate marginal work were right to refuse, and being right is what makes the instinct hard to notice and harder still to revisit. A decision that was sound in 2019 is quietly wrong now, and nothing in your day-to-day is going to announce it. You just keep not writing the script, for reasons that stopped applying a while ago.
Careful planning and requirements gathering existed to defend a precious commodity — developer time — from frivolous distraction. All that process was a rationing mechanism for a scarce resource. The resource isn’t scarce in the same way anymore, which means a lot of process is now guarding a vault that isn’t full.
The chokepoint moved. It’s focus. How many threads can you hold in your head at once? How many features can you thoughtfully define and sherpa through the agents building them for you? That’s a different skill than the one most of us spent our careers developing, and it isn’t obviously easier — but it’s where the constraint actually lives now, and optimizing the old one is wasted effort.
What to do with this
Start at the identity provider. It’s the hub. Everything authenticates somewhere, so it’s the fastest route to knowing what exists before you know anything else about it.
Instrument for questions you haven’t thought of yet. Retention and normalization decisions made purely against security use cases will fail you the moment someone asks an operational question. Storage is cheap. Regret is not.
Normalization is the real work. Every source you add is only as useful as your ability to correlate it with the others. This is the unglamorous part, and it’s where the value accrues.
Watch what your query interface costs. If answering a question requires a specialist and a free afternoon, you have a data archive. If it takes minutes, you have a data asset. The underlying data can be identical.
Separate reading from acting. The moment you point agents at this data, the thing that queries it should not be the thing that changes anything. Your logs contain strings an attacker chose.
The security stack isn’t a system that happens to collect a lot of data. It’s the instrumentation layer for the whole organization, and it’s already installed — you paid for it, it’s running, and its mandate to watch everything is a mandate nothing else in your environment has.
What you’re really building is an organization that can know itself. Quickly, accurately, and without convening a team every time someone asks a question. That capability was always latent in the data. What changed is that it finally got cheap enough to use.