Wrestling with LLMs: The Developer's New Hats

In my previous post, I described how Claude confidently declared dokku-dns "production ready" after Phase 26. It wasn't. Real testing revealed catastrophic bugs - the kind that could delete all my DNS records.

I'm writing code differently, wearing new hats, adopting new skills, and building different muscles. The usual developer pitfalls (rubber stamping reviews, getting nerd-sniped into 100% test coverage) still works against me whether I'm directing an LLM or implementing myself.

All this raised a critical question: What's my role when working with LLMs effectively?

The Developer Problem

When I write code myself, my focus is writing the most maintainable code I can. To me, that means using variables only where needed, using variable and function names as code comments, organizing code into standard file structures. This requires knowing a lot of details about problem domains, as well as programming languages and syntax.

Working with an LLM flips this completely. My goal becomes iteration and direction, not being too prescriptive. I steer, direct, review, and course-correct. I keep my head when the LLM goes awry (see my post on zen and emotional regulation).

This creates tension - I want to understand complete solutions. I have worked for years to chisel myself into a machine that can turn feature requests and coffee into code, not describe what needs doing. That shifts when working with LLMs.

I need to focus on the bigger picture and let go of the details. Accept "good enough, write some tests" rather than complete perfection. Describing what's wrong often seems harder than just fixing it myself, but making the changes myself means I have to catch the LLM up after the fact.

Why the PM/QA Shift Happens

PM and QA roles already work iteratively. They're already focused on direction over implementation. More importantly, LLMs generate code too quickly for me to fully understand - I have to mentally blackbox at different abstractions. I can't keep everything in my head anymore, which means I need the skills of roles that never kept everything in their heads in the first place.

The Product Management Role

My workflow for dokku-dns centered on TODO.md and DONE.md files tracking "phases." Each phase became a small PR. Claude started generating these phases, though most needed heavy scoping.

What I actually did wasn't development - it was project planning, prioritizing work, and cutting scope. Saying "no" to unnecessary work. Bringing Claude back to focus on the larger task when it wandered into the weeds.

The Battle: Scope Creep

Claude loves generating work that sounds important but isn't.

What Claude wanted to build:

• Testing documentation
• Community support infrastructure
• Issue templates, discussion templates, contributing guidelines
• Performance monitoring
• Refactoring work that adds zero user value

What I actually needed: Get the plugin functional, run it on my server, and push to GitHub.

The PM skill here is constantly saying "no" to unnecessary phases. Cutting scope back to MVP. Rewriting vague tasks into concrete, minimal steps. Asking "does this actually matter for v1.0?" over and over. I felt like an Agile coach more than a developer.

The pattern is clear: LLMs optimize for "sounds professional and enterprise" not "is necessary." My job became relentlessly prioritizing real requirements over imagined ones.

The QA Engineering Role

LLMs don't use the code they write. I do. This makes QA unavoidable, just like when I wrote the code myself.

Phase 26: When "Production Ready" Wasn't

After Phase 26, Claude declared victory: all tests passing, "production ready," "enterprise grade," ready to ship. My own QAing revealed a different story. Catastrophic bugs everywhere. The plugin could delete all my DNS records. Critical functionality was silently broken. Tests passed, but they tested the wrong things.

The pattern became clear: LLMs optimize for "tests pass" not "works correctly." They're confident about successful execution. They have no intuition for "this seems wrong." They genuinely can't distinguish working from broken - they just see green checkmarks.

What This Actually Means

The shift is fundamental: I review every commit, run commands manually, check logs and verify behavior, inspect diffs carefully, and watch for silent failures - never trusting "all tests passing." The LLM writes code, I verify it works. It's a better TDD partner than lead dev, but I'm the only one who can actually test whether the code does what it's supposed to do.

What I Learned

This sounds like more work, and in some ways it is. But I gained something significant: I spend less time fighting with semicolons and brackets, not debugging typos and formatting. The LLM handles the mechanical parts while I focus on architecture, product decisions, and verifying functionality actually works.

Trust but verify became my mantra, especially when phases were marked "successful." I learned to start with abstractions before implementations because LLMs are better at filling in details than designing the overall structure. The trade is real: I gave up control over implementation details but gained the ability to ship alone what was insurmountable before.

If I embrace the bumpy parts - the new hats, the setup and enforcement of guardrails, the constant verification - I can build things I couldn't tackle solo before. Is it overall faster? Sometimes. Is it easy? Not necessarily. Does it broaden what I'm capable of? Absolutely.