Notes from the work.
Reliability, AI engineering, and the tooling in between. Agents and guardrails, RAG, HADR war stories, and model reviews. Written between incidents. Newest first.
LATEST · ON-CALLThe alert that cried wolf
Alert fatigue isn't a toughness problem. It's a math problem. Why single-threshold monitoring is a machine for manufacturing noise, how correlating independent signals fixes it, and the tool I built to do exactly that.
2026
The Mount Rushmore of Knowledge Transfer
Tribal knowledge is the raw granite your team is built on — valuable, and dangerous. Here are the four ways to carve it into something that outlasts any one person, and why it is a reliability problem, not an HR one.
Understand the Tools, Don't Marry the Methods
ITIL, SRE, the runbook, the framework — they are all tools, and a tool is only right for the job in front of you. A case against methodology dogma in operations, and for keeping the goal in view instead of the ritual.
Be Like Water: Adapting Your Incident Command Style
There is no single right way to run an incident. The best commanders read the room and change shape — directive when it is chaos, hands-off when the experts have it. A case for flexibility over dogma.
How to Run an Incident Without Losing Your Head
The job of an incident commander is not to fix the outage. It is to keep the room calm, restore service before chasing root cause, and make sure the right questions get asked. A field guide to staying level when everything is red.
Standing up open-source endpoint detection on a budget
Not every team has CrowdStrike money, and plenty who do still want to know what their EDR actually does. Building the stack from open-source parts teaches you the anatomy of detection in a way no dashboard can.
Three months in with the Spark: what I actually use it for
The novelty of local models wears off in weeks. Three months in, where the DGX Spark earned a permanent spot in the workflow, and where the cloud is still the right call. It's a portfolio, not a versus.
The RAG that never phones home
RAG over data too sensitive to ever upload. The whole pipeline running on a local DGX Spark. Embeddings, vector store, retrieval, generation. With nothing crossing the wire. A capability unlock, not a compromise.
Tuning local models on the Spark: what actually moved the needle
Quantization, context budgets, keeping the model resident. What actually changed the numbers running local models on 128GB of unified memory, and what turned out to be leaderboard-chasing distraction.
I brought a DGX Spark home to run AI that never leaves the building
A DGX Spark, 128GB of unified memory, and the question that drove it: what does it take to run useful AI on work that can never touch a cloud API? The start of a running series on local, secure inference.
I ran the new agent models against my actual workflow
A practitioner's review of Kimi K2, Claude Fable, and the current agent generation. Tool honesty, long-horizon coherence, what the benchmarks still miss, and why the Hacker News "AGI vs stochastic parrot" fight is arguing about the wrong thing.
An LLM in the incident loop. Without letting it touch prod
The self-healing system everyone pitches is a great demo and a bad system. Where I actually landed: the model reasons and compresses, the human stays on every action that can't be undone.
The log is full. Don't flip to SIMPLE.
Error 9002 shows up and someone always says "just switch to SIMPLE recovery." Please don't. Read log_reuse_wait_desc, address the real reason, and often you just let the log catch up. Chain intact.
A tool for when the only environment is production
A lot of real database work happens on live systems with no test bed that resembles them. So I built the opposite of the tools that "just fix it for you". One that reaches for nothing and can't take an action on the box you can't afford to break.
The log is full. Don't flip to SIMPLE.
Error 9002 shows up and someone always says "just switch it to SIMPLE recovery." Please don't. Read log_reuse_wait_desc, address the real reason, and often you just watch the queue drain or the transaction commit and let the log catch up. Chain intact.
Everything that doesn't fail over
The AG moved the databases. The app still died. A field guide to the four things that live outside the availability group. Logins, jobs, linked servers, credentials.
Least privilege for a thing that can improvise
You can't predict what a reasoning agent will do. You'll lose that race. You secure it by making the actions you fear structurally unavailable. Containment as the thing that makes autonomy safe to grant.
2025
RAG is a retrieval problem wearing a language-model costume
Nobody's RAG system fails because the model can't write. It fails because retrieval handed it the wrong evidence. Where the quality actually lives. Chunking, embeddings vs relevance, and logging what got retrieved.
Build for the network that fights you
No installs, no CDN, no backend, no outbound. Why the constraints of a locked-down environment made the tool better. And the one rule that keeps a field kit field-ready.
2024
Guardrails aren't prompts
A sentence in a system prompt is a wish, not a control. Real guardrails live in code the model can't talk its way past. Defense in depth, borrowed from security engineering.
My first agent did the wrong thing, confidently
The failure mode of an AI agent isn't that it breaks. It's that it succeeds at the wrong objective, with total confidence. The week that reframed how I build with these models.
2023
Agent harnesses for safe-level run triage
Building a harness that lets an AI agent do read-only triage and scoped diagnosis on production. With the guardrails, gates, and blast-radius limits that make it safe to run. Read-only observation, zero autonomy on mutation, a human on the trigger.
The redo queue that wouldn't drain
9 GB of unreplayed log, a green dashboard, and a secondary quietly drowning in PAGEIOLATCH_EX. Send-queue vs redo-queue. And why it changes who you call.