![\"The](\"https://jonnonz.com/img/posts/pmf-gauntlet/gauntlet-loop.svg\")
Product market fit gets sold as a milestone. Find it and you're off to the\nraces. That's the bit nobody who's been through it actually believes.
\nPMF is a gauntlet. It eats teams, it bends founders, and it quietly poisons the\ntechnical decisions you're proud of at the time. Most of the damage I've watched\ndone to good companies wasn't from missing PMF. It was from how they behaved\nwhile they were looking for it.
\n
There's a particular kind of person who does well in the pre-PMF phase. High\ntolerance for ambiguity, low need for closure. The deck never feels finished,\nthe metric you're chasing changes every six weeks, and the answer to "what are\nwe doing in three months" is "depends."
\nPlenty of really good operators just cannot function in that environment. That's\nnot a character flaw — it's a stage mismatch. Some people thrive at zero to one.\nSome thrive at one to ten. Almost no one thrives at both, and the industry\npretending otherwise has cost a lot of careers and a lot of sanity.
\nNaming this honestly so people can self-select is one of the kindest things a\nfounder can do. You're not letting someone down by saying "this stage probably\nisn't for you, but the next one will be." You're saving them eighteen months of\nfeeling broken.
\nTiming, market, economy, what your one big regulator decides on a Tuesday — none\nof that is yours. You can have a sharp thesis and a great team and ship\nsomething nobody buys, because something three layers above you shifted while\nyou were heads-down.
\nThe only protection I've found against this is a vision the team is genuinely\nbought into. Not the slide. Not the wall poster. The actual reason you all got\nout of bed this morning. When the macro turns and the metric you were proud of\nlast quarter goes sideways, that vision is what stops the org from devouring\nitself.
\nI've watched startups where the thesis was right but the timing was a year early\nlose half their team in three months because nobody could explain why they were\nstill doing what they were doing. It wasn't a strategy problem. It was an\nalignment problem dressed up as a strategy problem.
\nThis is also where founders take the most damage personally. You can do\neverything well and still get hit by something nobody could have predicted. If\nyour sense of self is tied to PMF being a verdict on you, that breaks people.\nThe ones I've seen come through it healthy treated PMF like weather they were\nnavigating, not a test they were passing.
\nYour moat isn't the product. It isn't the tech. It definitely isn't the brand.\nYour moat is how fast the org can spot a shift in TAM or target market and\ntranslate it into a product move.
\nDays, ideally. Weeks if you have to. Not quarters.
\nThis is where the technical decisions made in the name of "scaling" quietly\ncripple you. The microservices you split out before you needed to. The custom\ninfrastructure someone stood up because their last job had it. The platform\nabstractions that mean a small UI change touches four repos. Each of those felt\ndisciplined at the time. Each is now a tax on the only thing you actually have —\nspeed.
\nAndreessen wrote the\noriginal PMF essay almost\ntwenty years ago, and the line that's aged best is the bit about doing whatever\nit takes — changing people, rewriting the product, moving markets. That's not a\nlicense to be chaotic. It's a reminder that the org needs to be physically\ncapable of those moves. If your architecture, process, or contracts make\nrewriting the product a six-month project, you've already lost the gauntlet\nwhether you know it yet or not.
\nI've got a strong opinion on this one: when in doubt, build it boring. Boring is\nfast to change.
\nThe customers who signed up first kept you alive. They also signed up for a\nslightly different company than the one you're now trying to become. That gap is\nwhere a lot of startups quietly die.
\nKeep them too happy and you slow your evolution. Push too hard toward the new\nvision and you churn the cohort that's funding your runway. The actual job is to\ndo both at once, which is why I sometimes call it internal schizophrenia. You're\na different company to them than you are to yourselves, and that's not a bug —\nthat's the mode you're operating in.
\nThe skill is being honest with the early cohort about where you're going without\nselling them something they didn't buy. The art is using their feedback to\nsharpen the bigger vision rather than letting yourself be pulled back into being\ntheir bespoke vendor. The dance is doing both of those without your team\nthinking you've gone off-piste, because the gap between "what we're shipping\ntoday" and "where we're going" looks weird from the inside.
\nThree patterns I keep seeing.
\n![\"Discipline](\"https://jonnonz.com/img/posts/pmf-gauntlet/discipline-vs-fragility.svg\")
Engineering over-scales the architecture. The team builds for the company they\nwant to be in two years instead of the company they need to be this quarter. By\nthe time PMF actually shows up, the org can't move. Worse, the engineers feel\nbusy and capable the whole time it's happening, which is why it's so hard to\nstop. Nobody is asking to slow down — everyone is shipping.
\nProduct holds the roadmap too tightly. The roadmap is the experiment at this\nstage. Treating it like a commitment is a category error. The product teams I've\nseen do this well treat the roadmap like a hypothesis with version numbers —\nlast month's was wrong, this month's is less wrong, and that's how it's supposed\nto feel. The ones who don't end up defending decisions they made when they knew\nless.
\nFounder comprehension debt builds up faster than anyone notices. The founder is\nheads-down on signal — every customer call, every dropped deal, every weird\npattern in the data lands in their head and gets metabolised on the spot. The\nteam is two beats behind, working from last week's mental model. Each individual\ndelay feels minor. The cumulative gap is the thing that kills decisions.
\nEach of these looks like discipline from the inside. Each of these is fragility\nwearing discipline's clothes.
\nMoats in the AI space are shifting quarter by quarter right now. Feature moats\nhave basically collapsed — anything you can describe in a screenshot can be\ncloned in a weekend with the current generation of tools. What's\nactually defensible has moved\ntoward proprietary data, deeply embedded workflows, distribution, trust, and\nregulatory positioning.
\nFor a founder in the PMF gauntlet that means the playbook is unreliable in a way\nit wasn't five years ago. You can't just lift what worked for the last cohort of\nSaaS winners and run it. You have to reason from first principles about where\nyour actual edge is going to come from over the next eighteen months, and place\nchips accordingly.
\nThe gauntlet itself hasn't changed. The chips you're placing have. That's harder\nthan it sounds, because most of us were trained in an era when the moat\nconversation was settled.
\nThe thing nobody tells you is that the founder or leader's most important job\nduring the PMF stretch isn't strategy or product or sales. It's getting the\ncontext that's in your head out into the org while everyone is running at five\nthousand miles an hour.
\nYou will not feel like you have time for this. You won't. You have to carve it\nout anyway. The teams I've seen come through PMF intact are the ones whose\nleaders forced themselves to stop, write things down, repeat themselves more\nthan felt necessary, and trust that the slowdown was the work.
\nThe teams that don't make it tend to look back and realise everyone was busy and\nnobody knew why.
\n","date_published":"Fri, 01 May 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/change-management/","url":"https://jonnonz.com/posts/change-management/","title":"Change management","content_html":"There's a whole business discipline called change management. Frameworks,\ncertifications, consultancies, the lot. Every big company has someone running it\nduring a restructure or a tech migration. Nobody runs it for you when your life\nturns over.
\nWhich is strange, because the personal version is the harder problem — and right\nnow, more people are facing it than at any point in recent memory.
\nMore than\n92,000 tech workers have been laid off in 2026 alone,\nbringing the total close to 900,000 since 2020. Meta cut 8,000 jobs last week.\nMicrosoft offered buyouts to 7% of its US workforce — the first time in its\n51-year history. Oracle has started cuts that could reach 30,000 by year end.\nCloser to home, Xero, Sharesies, Spark, One NZ and Eroad have all run their own\nrounds. AI is the headline reason, but the impact lands the same regardless of\nthe cause: hundreds of thousands of people closing a laptop and discovering\ntheir working identity has just been deleted.
\nThat's a lot of people being handed a forced version of personal change\nmanagement without ever signing up for the course.
\nThe business framing has the right insight buried in it.\nWilliam Bridges made a\ndistinction in the 90s that most people miss: change is external, transition is\ninternal. Change is the new org chart, the redundancy email, the merger.\nTransition is what happens inside people's heads while all that is going on.\nChange can happen overnight. Transition takes as long as it takes.
\nPersonal change management is just transition without the org chart.
\nI've been through a few years of it now. Not one big event — more like a slow\nstack of endings, some chosen, some not. Companies, relationships, versions of\nmyself I'd been building for a decade. The kind of stretch where you don't\nreally notice you're changing until you look up one day and the old you is gone.
\nThat's the part nobody warns you about. Real change isn't transformation. It's a\ncontrolled demolition followed by a slow rebuild, with a long, weird middle bit\nwhere neither the old you nor the new you is really there.
\nThe thing that goes is usually the organising self. Whatever the old you was\narranged around — a fear, a need for approval, a story about who you had to be,\nan ambition that was really a wound. When that goes, the structure it was\nholding up collapses. That's the death. It's real.
\nWhat survives is everything that wasn't load-bearing on the old arrangement.\nYour humour, your curiosity, the way you actually see people, the things you\ngenuinely care about. Those don't die because they weren't propping anything up.\nThey were just you, underneath.
\nThe disorienting part is feeling like a stranger to yourself and entirely\ncontinuous, at the same time. Both are true. The continuous parts are\ncontinuous. The organising self is gone. You're in between.
\nBridges calls this the neutral zone. The old reality has gone, the new one isn't\nthere yet. He says it's the hardest phase to manage, and most organisations rush\nthrough it because it looks unproductive. People do the same thing to\nthemselves.
\nThe temptation is to build a new identity fast, because the empty space is\nuncomfortable. Don't. Whatever you grab in a hurry will be made of whatever was\nlying around — which usually means the old patterns sneak back in wearing new\nclothes. Workaholism becomes "building my legacy". Approval-seeking becomes\n"being of service". Avoidance becomes "protecting my peace". Same machine, new\npaint.
\nThe test is always: is this coming from fear or from truth? You'll know. The\nbody knows before the mind does. Pay attention to the part of you that goes\nquiet around certain people, certain projects, certain decisions. That's the\nsignal.
\nYou don't get to fearless by trying. You get there by going through enough\nendings that the bluff stops working.
\nFear runs on a specific con: if this thing happens, you won't survive it. Not\nliterally die — but the you that exists now won't continue. You'll be broken,\nfinished, unrecognisable. The con works as long as it's untested. Then the thing\nhappens, and you go through it, and on the other side you notice you're still\nhere. Different, scarred, but continuous. The fear was lying about its hand.
\nAfter that, fear can still show up — it doesn't leave — but it can't run the\nsame con. You've seen the card it was holding. Next time it says you won't\nsurvive this, some quiet part of you knows: I already did.
\nThat's not the absence of fear. It's knowing you can act from what's true even\nwith the fear in the room.
\nHere's the bit that surprised me. Once the demolition is done and the rebuild\nstarts, what comes back isn't impressive. It's just real. Less reactive. Less\nnoise. Less performance. You stop needing to be seen a particular way, partly\nbecause you've watched a few of those selves die and you don't trust the next\none enough to stake everything on it.
\nThe goal of change management — the personal kind — isn't to become someone\nadmirable. It's to become someone who's the same alone as in public. Someone who\ndoes the next true thing without announcing it. Most of the depth of this stuff\nlives in the texture of regular days. How you handle a boring Tuesday. Whether\nyou rest when you're tired or push through to prove something to nobody.
\nBusiness change management has all this in it, and most people read it as a\nproject manager's manual. It's also a personal one. Endings, neutral zone, new\nbeginnings. Same shape, different blast radius.
\nThe seeds that grow through the demolition are the ones worth tending. The rest\nsorts itself out.
\n","date_published":"Sat, 25 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/three-ways-to-look-at-time/","url":"https://jonnonz.com/posts/three-ways-to-look-at-time/","title":"Three Ways to Look at Time","content_html":"ST-ResNet's core insight is that not all history is created equal.
\nWhen you're predicting crime in Auckland next month, three different kinds of\npast information matter. What happened in the last couple of months: the recent\ntrend. What happened at the same time last year: the seasonal pattern. And\nwhat's been happening over the longer term: whether crime is generally rising or\nfalling in an area.
\nConvLSTM treats all of this as one continuous sequence and hopes the network\nfigures out which parts matter. ST-ResNet\ntakes a more opinionated approach. It separates these three temporal scales\nexplicitly and gives each one its own dedicated neural network branch.
\nThe original paper by Zhang et al. was about predicting crowd flows in Beijing.\nPeople move through cities in patterns that look a lot like crime patterns:\ndaily rhythms, weekly cycles, long-term trends. The architecture\ntranslates well to crime data,\nwith some modifications.
\nThe three branches each look at different slices of history:
\nCloseness captures what's been happening recently. For our monthly data,\nthis means the last 3 months. If South Auckland has been trending upward over\nthe last quarter, the closeness branch sees that momentum.
\nPeriod captures seasonal patterns. It looks at the same month in previous\nyears. So to predict January 2026, it pulls in January 2025 and January 2024.\nThe assumption is that crime has an annual rhythm, and the same month tends to\nlook similar year to year.
\nTrend captures longer-term shifts. It uses quarterly averages from further\nback: broad strokes of whether an area is seeing more or less crime over time.\nThis is the slowest-moving signal.
\nEach branch independently processes its temporal slice through a stack of\nresidual convolutional blocks, then a learned fusion layer combines the three\noutputs:
\nprediction = W_c · closeness + W_p · period + W_t · trend + bias\nWhere W_c, W_p, and W_t are learned weights that vary by grid cell. This\nis a nice touch. It means the model can decide that the CBD's crime is mostly\ndriven by recent trends (closeness), while a residential suburb might be more\nseasonal (period). Different areas get different temporal recipes.
Each branch uses residual convolutional units, the building blocks that made\nResNet so successful in image recognition.
\nThe key idea: instead of learning the full output at each layer, the network\nlearns the residual, the difference between input and output. The identity\nshortcut connection means gradients flow cleanly through the network during\ntraining, which lets you stack more layers without the signal degrading.
\nResUnit(X) = ReLU(Conv(ReLU(Conv(X))) + X)\nThat + X at the end is the skip connection. If the layer has nothing useful to\nadd, it can learn weights near zero and just pass the input through. This makes\ndeeper networks stable, which matters when you're trying to learn spatial\nfeatures at multiple scales.
For our grid, I use 4 residual units per branch. Each unit has two 3×3\nconvolutional layers with 32 filters. That's deep enough to capture spatial\nrelationships across several kilometres without being so deep that the model\noverfits on 36 months of training data.
\nHere's where theory meets reality, and it gets a bit awkward.
\nST-ResNet was designed for dense, high-frequency data. The Beijing crowd flow\npaper used 30-minute intervals over months of data: thousands of timesteps. The\ncrime papers that report strong results typically use daily data over several\nyears.
\nWe have 48 monthly timesteps. Total. The period branch (which looks at the same\nmonth in previous years) has at most 3 data points per month (2022, 2023, 2024\nto predict 2025/2026). The trend branch is working with quarterly averages from\na four-year window. It's not a lot of temporal data for an architecture that's\nspecifically designed to decompose temporal patterns.
\nI had a feeling this would be the bottleneck, and it was.
\nCloseness branch:\n Input: last 3 months (3 × 6 channels = 18 input channels)\n → 4 ResUnits (32 filters, 3×3 kernels)\n → Output: 32 channels\n\nPeriod branch:\n Input: same month from 2 prior years (2 × 6 = 12 input channels)\n → 4 ResUnits (32 filters, 3×3 kernels)\n → Output: 32 channels\n\nTrend branch:\n Input: 2 quarterly averages (2 × 6 = 12 input channels)\n → 4 ResUnits (32 filters, 3×3 kernels)\n → Output: 32 channels\n\nFusion:\n → Learned weighted sum across branches\n → Conv2d(32, 6, 1×1) → 6 crime type predictions\nTotal parameters: roughly 180k. Slightly smaller than the ConvLSTM, which is\nfine. ST-ResNet's power is supposed to come from the temporal decomposition, not\nfrom model size.
\nTraining uses the same setup as ConvLSTM: Adam optimiser, learning rate 1e-4,\nMSE loss on log1p-transformed values, early stopping with patience of 15\nepochs. On CPU, each run takes about 35 minutes, a bit faster than ConvLSTM\nsince there's no sequential recurrence to deal with.
| Crime Type | \nHist. Avg MAE | \nConvLSTM MAE | \nST-ResNet MAE | \n
|---|---|---|---|
| Theft | \n1.28 | \n1.14 | \n1.18 | \n
| Burglary | \n0.35 | \n0.32 | \n0.33 | \n
| Assault | \n0.20 | \n0.19 | \n0.19 | \n
| Robbery | \n0.04 | \n0.04 | \n0.04 | \n
| Sexual | \n0.03 | \n0.03 | \n0.03 | \n
| Harm | \n0.01 | \n0.01 | \n0.01 | \n
| All types | \n0.39 | \n0.35 | \n0.36 | \n
ST-ResNet beats the historical average but doesn't quite match ConvLSTM. The\naggregate MAE of 0.36 is a 7.7% improvement over the baseline, compared to\nConvLSTM's 10.3%.
\nThat's not a terrible result, but it's not what I was hoping for.
\nWhen I dug into the learned fusion weights, the story became clear. The\ncloseness branch dominates. It gets 60–70% of the weight across most grid cells.\nThe period branch gets 20–25%, and the trend branch barely contributes at\n10–15%.
\nThe model is basically saying: "Recent months matter most, seasonal patterns\nhelp a bit, and long-term trends are mostly noise." That's not a failure of the\narchitecture. It's a fair assessment of what's in the data.
\nWith only 2–3 examples of each calendar month, the period branch can't reliably\nlearn seasonal patterns. It's overfitting to individual years rather than\nextracting a stable seasonal signal. ConvLSTM handles this better because it\nprocesses the full sequence and implicitly learns seasonality from the\ncontinuous flow of months, without needing to explicitly align calendar periods.
\nThe trend branch suffers even more. Quarterly averages over a four-year window\ndon't give it much to work with. In the original crowd flow papers with years of\nhalf-hourly data, the trend branch captures genuine long-term shifts in\npopulation movement. Here, it's essentially learning a constant.
\nDespite losing on aggregate, ST-ResNet has one clear advantage: it's better at\npredicting seasonal transitions.
\nThe months where crime shifts gears (the spring uptick in September/October and\nthe February dip) ST-ResNet handles more gracefully than ConvLSTM. The period\nbranch, sparse as its data is, does capture enough of the annual rhythm to\nanticipate these transitions a bit earlier.
\nConvLSTM tends to lag these transitions by about a month. It needs to "see" the\nuptick starting before it predicts continuation. ST-ResNet, by explicitly\nlooking at last year's same month, can anticipate the shift before it fully\nmaterialises in the recent sequence.
\nFor an operational forecasting tool, that one-month lead time on seasonal\ntransitions could be valuable. But in our test set metrics, it's a small\nadvantage that doesn't overcome ST-ResNet's overall weaker performance on\nmonth-to-month dynamics.
\n| Metric | \nHistorical Avg | \nConvLSTM | \nST-ResNet | \n
|---|---|---|---|
| Overall MAE | \n0.39 | \n0.35 | \n0.36 | \n
| Theft MAE | \n1.28 | \n1.14 | \n1.18 | \n
| Training time (CPU) | \nN/A | \n~40 min | \n~35 min | \n
| Parameters | \n0 | \n~200k | \n~180k | \n
| Seasonal transitions | \nPoor | \nLagging | \nBetter | \n
| Spatial dynamics | \nNone | \nGood | \nGood | \n
ConvLSTM is the better model for this specific dataset. Not by a lot. We're\ntalking about small differences on already-small error values. But consistently\nbetter on the main crime types that have enough signal to matter.
\nNeither model is a revelation. A 7–10% improvement over "just use the historical\naverage" is real but modest. Deep learning's strengths (learning complex\nnonlinear dynamics from huge datasets) are somewhat wasted on 48 monthly\ntimesteps over a relatively low-crime city.
\nIf I had daily data instead of monthly, or ten years instead of four, I'd expect\nST-ResNet to close the gap or pull ahead. Its architecture is fundamentally\nsound. The temporal decomposition is a genuinely good idea. It's just starved of\nthe data it needs to shine.
\nBoth models meaningfully beat the baselines. Both learn spatial patterns that\nsimple averages can't capture. And both are honest about the sparse crime types:\nthey predict near-zero and move on, which is the right call.
\nNext up: we'll take these predictions and build something you can actually look\nat. A 3D interactive dashboard where you can watch crime patterns evolve across\nAuckland over time. The modelling was the hard bit. Making it visual is the fun\nbit.
\n","date_published":"Thu, 23 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/what-an-hour-of-your-attention-is-worth/","url":"https://jonnonz.com/posts/what-an-hour-of-your-attention-is-worth/","title":"What an hour of your attention is worth","content_html":"I stood up a working social network for eight mates last weekend. Profile pages,\na shared feed, a photo wall, a jukebox bolted onto a spare domain. It took me a\nSaturday, about forty bucks in Claude credits, and exactly zero\nproduct-market-fit meetings.
\nThe same weekend, Meta earned about six bucks off me. Google made ten. LinkedIn,\nYouTube, TikTok, X — all quietly billing in the background, none of them sending\na receipt. If you add them all up for the average American, the annual total is\nnorth of $1,000. You just never see it, because no money changes hands and no\ninvoice arrives.
\nThe clever thing about "free" on the internet isn't that the trade doesn't\nexist. It's that it's been designed so you can't see it. No money moves. No\ninvoice lands. No app shows you the meter ticking as you scroll. The exchange is\nreal — your attention and your data in, Instagram and Google and LinkedIn out —\nbut by the time the numbers get tallied, they live in a quarterly earnings\nreport you'll never read. So the trade feels weightless.
\nIt isn't. You just can't see the price tag.
\nThe strange thing is the price tag has been public the whole time. Every\nplatform listed on a stock exchange tells you, four times a year, exactly what\nyou're worth to them. You've just never been shown how to read it — and until\nrecently, the only practical alternative to reading it was "live in a cabin."\nThat part has changed, and it's the part almost nobody is talking about.
\nThe invisibility isn't an accident either. If Meta had to send you a cheque\nevery month for the money they made off you, you'd treat the relationship very\ndifferently. You'd notice when the amount went up. You'd notice that the\nteenager version of the payment looks nothing like the adult version. You'd\nwonder why the Auckland cheque was ten times the Jakarta one for the exact same\nhour of scrolling. The whole edifice of "free" rests on keeping the accounting\none-sided — they measure you in basis points to three decimal places, you\nexperience the trade as a vague sense of having lost your afternoon.
\nThe number you want is called ARPU — average revenue per user. Every public\nplatform reports it, because investors demand it. The maths is blunt: take the\ncompany's annual revenue, divide by monthly active users. What comes out is what\nthe platform earns off the average human who shows up, per year.
\nFor Meta last year the global figure was about $52 per user. For YouTube's\nad-supported side, around $24. For\nLinkedIn it's $15 averaged across all 1.2B members,\nbut much higher once you strip out the dormant accounts.
\nThese aren't guesses from a watchdog group. They're from the companies\nthemselves, in the part of the earnings release where the whole purpose is to\nconvince shareholders each user is worth more than last quarter. The incentive\nis to talk the number up, not down.
\nWhatever ARPU says, the reality on the ground probably isn't lower. If anything,\nit's a floor.
\nRough figures, from the companies' own filings:
\nThe geographic skew is the part most people miss. Meta's figure in the US is\nroughly ten times what it is in Asia-Pacific. Europe sits in the middle at about\n$92. Same product, same features, same algorithm — different rate card, because\nad buyers pay more to reach wealthier audiences. You are literally worth more in\nAuckland than you are in Jakarta, and your feed is tuned accordingly.
\n![\"Meta](\"https://jonnonz.com/img/posts/arpu-meta-by-region.svg\")
The same skew shows up across every ad-funded platform. The US rate card is the\none the rest of the world gets compared to:
\n![\"US](\"https://jonnonz.com/img/posts/arpu-us-vs-global.svg\")
Marketplaces don't fit ARPU cleanly, but the extraction is still there if you\nlook for it. Uber and Lyft take around 20% of each fare. Airbnb combines host\nand guest fees for about 14–16%. DoorDash and Uber Eats take closer to 25%.\nShopify's card take is 2.9% plus 30 cents per transaction. Different mechanism,\nsame game — a percentage of every transaction, quietly skimmed, never itemised.
\nARPU is annual. Attention isn't spent in years though — it's spent in hours, in\nthe little windows between other things. So the honest conversion is to divide.
\nThe average US Meta user burns about 200 hours a year across Facebook and\nInstagram. $320 ÷ 200 = roughly $1.60 per hour of your attention. YouTube works\nout to about $0.27/hour. TikTok $0.22. Snapchat cheaper still. Do the same sum\non global averages and Meta drops to around 26 cents an hour, YouTube to 8.
\nThose rates are only what the platform earns this year, mind. They aren't what\nyour data is ultimately worth. Everything you click and hover and pause on\nfeeds ad targeting across the wider web, plus — now — AI training corpora. ARPU\nis the rent. The equity is bigger, and the equity compounds.
\nThe AI-training bit is genuinely new and worth pausing on. For fifteen years the\ndata you generated on these platforms powered one thing: better ad targeting on\nthose same platforms. It was a closed loop. You scrolled, they learned, they\nsold the targeting back to advertisers, the advertisers bought your attention\nagain. Bounded. Weird, but bounded.
\nThat loop isn't bounded anymore. Your posts and comments and DMs are now\ntraining data for models that will be sold, resold, and embedded into every\npiece of software you touch for the next decade. The $320 Meta earned off you in\nthe US last year is a rounding error next to what the underlying corpus is worth\nto the next generation of AI products. ARPU doesn't capture any of that. It's\nliterally last quarter's ad rent, with none of the capital gains on the asset.
\nEven the rent, laid out per hour, makes one thing obvious: you can see exactly\nwhy every platform is obsessed with "time spent" as a north-star metric. If one\nextra hour a week on Facebook is worth ~$83 a year per US user, multiplied\nacross three billion users, the maths for why the feed never stops scrolling is\nnot mysterious. The feed is a meter. Keeping it running is the business. Every\n"new feature" that shows up in your settings — reels, shorts, a nudge to open\nthe app on your commute — is a hand on that meter.
\nOnce you see it that way, a lot of product decisions stop looking like product\ndecisions.
\nThe point of making the numbers this concrete is that you can plug in your own\nusage and see what you personally throw into the machine each year. Drag the\nsliders for how much time goes into each platform and watch the ledger tally up.\nRates are global averages.
\nARPU is rent, not equity. What a platform earns this year isn't what the underlying data is worth across the wider web and AI training corpora.
Averages hide heavy users. Freemium smears free and paying users into one figure. If you're all-in, you're worth more than average.
Multi-product companies cheat the top line. Google's per-user number isn't all Search — it's Search plus Android plus YouTube plus Cloud.
The rates come from the earnings-report maths above — global ARPU divided by\naverage annual hours on the platform.
\nOnce the number has somewhere to sit, it's much harder to ignore.
\nMost people look at a total over $1,000/yr and go quiet for a second. Not\nbecause any one platform is egregious — on a per-hour basis they really aren't —\nbut because the aggregate is real, and it's been invisible until now. That's the\nfirst useful thing the exercise does. It makes a choice possible.
\nThe obvious next move is to look at alternatives. Signal instead of WhatsApp.\nKagi or Brave Search instead of Google. Paid Spotify instead of ad-supported\nSpotify. Bluesky or Mastodon instead of X. Fastmail instead of Gmail. None are\nperfect, and some cost actual money — but once you can price what you're\ncurrently "not paying", the paid alternative often looks less expensive than it\ndid five minutes ago. Fastmail at\n$5/month stops being a luxury when the honest comparison is "$60/yr vs being the\nproduct for an ad network that paid $500 for me last year."
\nThat's the defensive move. It's the one everyone talks about, every time one of\nthese pieces gets written. You switch to the more honest vendor, you feel\nslightly better, and the fundamental shape of the market doesn't move.
\nThe more interesting move is what's happened on the build side, and it's the\npart almost nobody has internalised yet.
\nStanding up a social app used to take a small team months. You needed a backend\nengineer, a frontend engineer, a designer, probably a DevOps person, and a spare\nthree months. That was the real moat — not the network effects, not the\nalgorithm, but the sheer human-hours required to put a working thing on the\ninternet. That's why the only viable answer for twenty years was to build\nsomething big enough to run ads against. Small social didn't exist because small\nsocial couldn't pay the salaries.
\nWith Claude Code, Cursor, v0, and Lovable, that equation has quietly inverted. A\nprofile page, a shared feed, a wall for photos, maybe a jukebox, a chat wall — a\nMySpace-sized thing for you and a dozen friends, on a domain you own, with none\nof it feeding anyone's ad platform — is a weekend. I know because I just did it.\nNot as some Silicon Valley startup trying to replace Facebook. As a Saturday\nproject for eight mates, on a domain that cost twelve bucks, running on a box\nthat costs ten a month.
\nThe bill of materials is embarrassingly short. A boring Postgres. A boring\nNext.js app. Auth via magic link. Storage for photos. An LLM for the fiddly bits\nnobody wants to write from scratch. All of it plumbed together in an afternoon\nof prompting, an evening of cleanup, and a Sunday of adding the jukebox because\nmy mate Hamish wouldn't stop asking.
\nIt is not good software. It is good enough software for eight humans who know\neach other.
\nThat qualifier is the whole thing. Facebook has to be good software at planet\nscale because Facebook is selling ad impressions at planet scale. A group of\neight doesn't need p99 latency and a content moderation policy. A group of eight\nneeds a place to put photos from the weekend where the photos don't end up\ntraining someone's image model in twelve months' time. Those are very different\nengineering problems, and the second one is much, much easier than the first.
\nA lot of things genuinely don't work on the weekend version. There's no\nrecommendation algorithm. There's no real search. The feed is\nreverse-chronological and that's it. When someone posts something at 3am nobody\nsees it until the morning. There's no cleverness about which photos get surfaced\nor which memories get resurrected. If you go on holiday for two weeks, you come\nback to a feed that's exactly what your eight mates posted, in the order they\nposted it.
\nThat sounds like a limitation until you notice the thing it is not doing is\noptimising for your engagement. Reverse-chronological across eight friends is\nnot a meter. It's a wall. You check it, you see what's there, you leave. There's\nno reason for the software to try to keep you around because there's nobody\npaying the software to keep you around. That inversion — from meter to wall — is\nthe entire point.
\nThe thing that would have been a VC round in 2015 is now a side quest you finish\nbefore the roast is in the oven. The tools genuinely got that much better in the\nlast eighteen months. We just haven't updated our intuitions yet about what that\nmeans.
\nWhat it means, specifically, is that the ad-supported social network is no\nlonger the only technically viable answer. For twenty years it was. That was the\nconstraint the whole "free web" was built around. The constraint is gone, and\nnobody has sent the memo.
\nThe cheapest social network in 2026 is the one you and six mates build on a\nSaturday afternoon. It doesn't scale. It doesn't need to. It costs less than a\nmonth of Netflix, produces no ad revenue for anyone, and feeds no one's training\nset. You own the domain. You own the data. You own the product decisions — which\nin practice means there are no product decisions, because nobody is trying to\nsqueeze another hour out of anyone's week.
\nNone of this replaces the platforms, to be clear. You still need Gmail for the\nrecruiter, LinkedIn for the job hunt, YouTube for the tutorial, WhatsApp for the\ngroup chat your family refuses to leave. The ad-supported internet isn't going\nanywhere and I'm not pretending it is. What's changed is that it's no longer the\nonly game in town. For the circle of people you actually care about — the eight\nmates, the cousins, the old uni flat — you don't have to hand them over to the\nad machine anymore. You can build them a room of their own, and the tools to\nbuild that room have become trivial in a way we haven't fully absorbed yet.
\nThe meter's been running your whole life. You just got the tools to turn it off.
\n","date_published":"Tue, 21 Apr 2026 12:00:00 GMT"},{"id":"https://jonnonz.com/posts/teaching-a-neural-network-to-watch-crime-like-video/","url":"https://jonnonz.com/posts/teaching-a-neural-network-to-watch-crime-like-video/","title":"Teaching a Neural Network to Watch Crime Like Video","content_html":"ConvLSTM was invented to predict rainstorms.
\nSpecifically,\nShi et al. at the Hong Kong Observatory\nneeded to forecast radar echo maps: 2D grids of rainfall intensity that evolve\nover time. They had sequences of spatial images and wanted to predict the next\nframes. Sound familiar?
\nThat's exactly what we built in Part 3. Crime on a 500m grid, one frame per\nmonth, six channels for crime types. The Auckland crime tensor is structurally\nidentical to a weather radar sequence. Same dimensionality, same prediction\ntask, just a very different domain.
\nStandard LSTM networks are fantastic at learning sequences. They're the backbone\nof a lot of time-series forecasting. But they have a fundamental problem with\nspatial data: they need flat vectors as input.
\nTo feed our 77×59 grid into a regular LSTM, we'd have to flatten it into a\nvector of 4,543 values per crime type. That's 27,258 values per timestep across\nall six channels. The network would process this as a sequence of big flat\nvectors, with no concept that cell (10, 5) is next to cell (10, 6).
\nAll the spatial structure (the fact that crime clusters, that hotspots have\nneighbourhoods, that the CBD is a contiguous area) gets thrown away. The model\nwould have to rediscover spatial relationships from scratch, purely from\ncorrelations in the flattened vector. With only 36 training months, that's not\nhappening.
\nConvLSTM's insight is elegant. Take the standard LSTM equations (the input gate,\nforget gate, output gate, cell state update) and replace every matrix\nmultiplication with a convolution operation.
\nIn a regular LSTM:
\ninput_gate = sigmoid(W_xi * x_t + W_hi * h_{t-1} + b_i)\nIn ConvLSTM:
\ninput_gate = sigmoid(W_xi ∗ X_t + W_hi ∗ H_{t-1} + b_i)\nThat ∗ is a convolution instead of a matrix multiply. X_t is the full 2D\ngrid at time t, and H_{t-1} is the previous hidden state, also a 2D grid.\nThe convolution kernel slides across the spatial dimensions, so each cell's gate\nvalues depend on its local neighbourhood.
This means the network naturally learns that a spike in cell (10, 5) might\naffect predictions for cell (10, 6). Spatial proximity is baked into the\narchitecture. It doesn't need to learn it from data.
\nThe kernel size controls how much spatial context each cell sees. A 3×3 kernel\nmeans each cell looks at its immediate 8 neighbours. Stack multiple ConvLSTM\nlayers and the effective receptive field grows. Deeper layers can capture\nrelationships between cells that are several kilometres apart.
\nHere's what I settled on after a fair bit of experimentation (which on CPU means\n"a lot of patient waiting"):
\nInput: (batch, 6, 6, 77, 59), 6 months, 6 crime types, 77×59 grid\n ↓\nConvLSTM2d(in=6, hidden=32, kernel=3×3, padding=1)\n ↓\nBatchNorm2d\n ↓\nConvLSTM2d(in=32, hidden=32, kernel=3×3, padding=1)\n ↓\nBatchNorm2d\n ↓\nConv2d(in=32, out=6, kernel=1×1), project to 6 crime type channels\n ↓\nOutput: (batch, 6, 77, 59), next month prediction\nTwo ConvLSTM layers with 32 hidden channels each. The 3×3 kernel gives each cell\na neighbourhood view, and stacking two layers means the effective receptive\nfield covers about 1–1.5 km. Enough to capture the spatial extent of most crime\nhotspots.
\nWhy only 32 hidden channels? This is where the CPU constraint actually helps. A\nbigger model would be tempting with a GPU, but on a Ryzen 5 we need to keep it\ntight. 32 channels gives us about 200k trainable parameters: small enough to\ntrain in under an hour, large enough to learn meaningful spatial-temporal\npatterns.
\nThe 1×1 convolution at the end is a channel projection. It maps the 32 learned\nfeatures back to 6 crime type predictions.
\nThe lookback window is six months. The model sees January through June and\npredicts July. Then February through July to predict August. And so on.
\nSix months captures one half of the seasonal cycle, which turned out to be the\nsweet spot. Shorter sequences (3 months) missed seasonal context. Longer\nsequences (12 months) didn't improve results, likely because the model doesn't\nhave enough data to learn year-long dependencies with only 36 training months\ntotal.
\nThe training set gives us 30 sequences (months 1–6 predict 7, months 2–7 predict\n8, all the way to months 30–35 predict 36). That's not a lot. Every sequence\ncounts.
\noptimiser = Adam(lr=1e-4)\nloss = MSE # on log1p-transformed values\nbatch_size = 4 # small because sequences are large\nepochs = 150 with early stopping (patience=15)\nThe log1p transformation from Part 3 is critical here. Raw crime counts range\nfrom 0 to 50+. After log1p, the range compresses to 0–4. Without this, the\nloss function would be dominated by the handful of high-count CBD cells, and the\nmodel would essentially ignore the rest of the grid.
Training on CPU takes about 40 minutes per run. Not fast, but manageable. I\ncould typically fit in 3–4 experimental runs per evening, which meant progress\nwas slow but steady. Each run I'd tweak one thing (kernel size, hidden channels,\nlearning rate) and compare validation MAE.
\nEarly stopping triggers around epoch 80–100 in most runs. The model converges\nrelatively quickly, which makes sense given the small dataset and architecture.
\nSo how does ConvLSTM stack up against the baselines from Part 5?
\n| Crime Type | \nHist. Avg MAE | \nConvLSTM MAE | \nImprovement | \n
|---|---|---|---|
| Theft | \n1.28 | \n1.14 | \n10.9% | \n
| Burglary | \n0.35 | \n0.32 | \n8.6% | \n
| Assault | \n0.20 | \n0.19 | \n5.0% | \n
| Robbery | \n0.04 | \n0.04 | \n2.5% | \n
| Sexual | \n0.03 | \n0.03 | \n~0% | \n
| Harm | \n0.01 | \n0.01 | \n~0% | \n
| All types | \n0.39 | \n0.35 | \n10.3% | \n
A 10% improvement on the aggregate MAE. Not earth-shattering, but real.
\nTheft gets the biggest lift because there's the most signal to work with. The\nmodel genuinely learns spatial dynamics that the historical average can't\ncapture. When a cluster of cells in South Auckland trends upward over several\nmonths, ConvLSTM picks up on that momentum and adjusts its predictions\naccordingly.
\nBurglary sees a decent improvement too, likely driven by the spatial correlation\nwith theft that we spotted in the EDA.
\nFor the sparse crime types (robbery, sexual offences, harm) ConvLSTM basically\nlearns to predict near-zero, same as the baseline. There simply isn't enough\nsignal at 500m monthly resolution for these types. The model is honest about\nwhat it doesn't know, which I actually respect.
\nThe improvement isn't uniform across the grid. ConvLSTM does best in the\ntransition zones: cells on the edges of established hotspots where crime counts\nfluctuate month to month. It learns that these boundary cells tend to follow the\ntrend of their neighbours, which is exactly the kind of spatial-temporal pattern\nit was designed to capture.
\nIn the stable hotspot cores (the CBD, Manukau) the model performs about the same\nas the baseline. Those cells are consistently high, and the historical average\nalready captures that well.
\nWhere it properly struggles is with sudden spikes in normally quiet areas. A\ncell that's been near-zero for months and then gets 5 thefts in one month: the\nmodel doesn't see that coming. Neither does any other model, to be fair. Those\nevents are closer to random noise than learnable signal.
\nA 10% MAE improvement is meaningful but modest.\nRecent ConvLSTM crime prediction papers\nreport larger gains, but they typically work with much more data: years of daily\nrecords across cities with higher crime density. Our setup is tougher. Monthly\nresolution limits temporal signal, Auckland is relatively low-crime by global\nstandards, and we only have four years.
\nThe model is also running on CPU with a deliberately small architecture. A\nbigger model on a GPU might squeeze out more performance. But the point of this\nproject was always to see how far you can push it with modest resources, and a\n10% beat over simple baselines feels like a real result.
\nThe question now is whether ST-ResNet's different approach to temporal modelling\ncan do better. ConvLSTM processes time as one continuous sequence. ST-ResNet\nbreaks it into three separate temporal scales: closeness, period, and trend.\nWith a seasonal dataset like crime, that decomposition might be exactly what's\nneeded.
\n","date_published":"Thu, 16 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/open-source-agent-that-teaches-claude-code-your-architecture/","url":"https://jonnonz.com/posts/open-source-agent-that-teaches-claude-code-your-architecture/","title":"Open-Source Agent That Teaches Claude Code Your Architecture","content_html":"AI has made building software cheap. A solo founder with Claude Code or Cursor\ncan ship an MVP in a weekend that would've taken a small team a month two years\nago. I've watched this happen across the NZ startup scene. Ideas that used to\ndie in the "can we afford to build it" phase now get built over a long weekend.
\nThis is mostly great. Velocity is what startups need. Cost of testing an idea is\nnow close to zero, and the business prioritises speed.
\nThe catch shows up when the idea works.
\nAI builds for right now. It optimises for the current prompt, the current\nfile, the current feature. It doesn't think about what happens when your billing\nservice needs to handle 10x the volume, or when your email notifications need to\nmove from inline calls to a queue. It doesn't plan for the evolutionary pressure\nyour system will face once it has users.
\nThat's the gap I've been thinking about, and it's what led me to build\ndomain-agents.
\nI want to be fair to the current generation of AI coding assistants. They're not\nstupid about finding code.
\nClaude Code runs an agentic search loop (grep, glob, file reads) iterating\nthrough your codebase to find what's relevant. Boris Cherny (who created Claude\nCode) has said they tried\nRAG with a local vector database early on and dropped it because agentic search\noutperformed it. Cursor takes a different approach: it\nchunks your codebase, generates embeddings,\nand stores them for semantic search so you can find code by concept rather than\nkeyword. Copilot combines semantic indexing with LSP-powered reference tracing\nfrom VS Code.
\nThe search works. If you ask Claude Code to find your billing service, it'll\nfind it. Ask Cursor for authentication logic and the embeddings will surface it\neven if the code never uses the word "authentication."
\nNone of them understand the architecture those files live in.
\nAll the information needed to understand domain relationships sits in the code:\nimport graphs, interface signatures, dependency patterns. These tools don't\nextract or structure it that way. They find files one at a time. They don't map\nout that your billing service depends on the email service, that\nBillingService is consumed by two other domains, or that changing its\ninterface is a cross-domain event. The information is in the codebase. Nobody's\npulling it together.
And every session starts from zero. The AI learned your architecture yesterday\nand forgot it today.
\nMy thesis: cheap AI-built MVPs plus expensive scaling problems point toward\nevolutionary architecture with domain-based boundaries.
\nThe idea isn't new. The reason it matters now is.
\nIn an evolutionary architecture, you focus on clean interfaces between business\ndomains. Your email service exposes a contract like\nsendEmail(to, subject, body), and the rest of the system calls that interface.\nBehind the interface, the implementation evolves through stages as your scaling\nneeds change:
graph LR\n A["Inline\\n(direct call)"] --> B["Async\\n(fire & forget)"]\n B --> C["Queued\\n(BullMQ/SQS)"]\n C --> D["Separate Service"]\n D --> E["Distributed"]\nDay one, sendEmail is a function that calls Resend directly. Inline,\nsynchronous, dead simple. When traffic picks up, you drop the await and let it\nrun in the background. Later, you introduce BullMQ or SQS. Eventually it becomes\nits own service. The interface stays put. Only the implementation behind it\nchanges.
This is the kind of evolution AI coding assistants are terrible at planning for.\nThey'll inline that email call because it works right now. They have no\nconcept of where this domain sits on its scaling trajectory.
\ndomain-agents is a CLI tool that\nruns static analysis on TypeScript codebases, discovers business domains, and\ngenerates AI agent context files for Claude Code and Cursor.
\ndomain-agents discover . # Analyse codebase → proposal.json\ndomain-agents init . # Generate agents/*.md + AGENTS.md\ndomain-agents hooks claude # Wire into Claude Code (rules + MCP server)\ndomain-agents hooks cursor # Wire into Cursor (.mdc rules)\nAfter setup, opening src/billing/invoice.ts in Claude Code loads the billing\ndomain agent into context. The AI now knows: billing depends on email (coupling\nscore 0.23), exposes BillingService consumed by 2 other domains, sits at the\n"inline" scaling stage with a path toward async queuing, and has 3 tracked tech\ndebt items.
It plans work accordingly. The context was loaded before the first prompt, no\nsearch required.
\nThe discovery engine runs 5 analysis passes because no single signal identifies\nbusiness domains on its own.
\nDirectory structure works for greenfield projects (src/auth/, src/billing/)\nbut fails for legacy MVC apps. Import graphs capture coupling but not business\nintent. Package dependencies hint at external integrations but miss internal\ndomains.
graph TD\n S["Structure Analysis"] --> O["Signal Orchestrator"]\n I["Import Graph\\n(TS Compiler API)"] --> O\n N["Naming Patterns"] --> O\n D["Dependency Mapping\\n(npm → domain hints)"] --> O\n IF["Interface Detection"] --> O\n O --> M["Merge Pipeline"]\n M --> R["Domain Proposal"]\nStructure detects whether the codebase is feature-organised,\nlayer-organised, mixed, or flat. Import graph uses the TypeScript Compiler\nAPI to parse each .ts file, resolve imports, and build a directed edge graph.\nType-only imports get weighted at 0.3 because they're a weaker coupling signal\nthan value imports. Naming patterns extract domain prefixes:\nauth.controller.ts → "auth". Dependency mapping maps npm packages to\ndomain hints (stripe → billing, @sendgrid/mail → email). Interface\ndetection identifies files imported across domain boundaries and calculates\ncoupling scores between domain pairs.
Each pass produces weighted signals. The orchestrator combines them with\nconfidence scoring: average signal strength plus a bonus for signal count,\ncapped at 0.99. Layer-organised codebases get an 0.85 multiplier because they're\nharder to discover.
\nFeature-organised codebases are easy. The directory structure is the domain.\nBut most real codebases look like this:
\nsrc/\n controllers/\n auth.controller.ts\n billing.controller.ts\n services/\n auth.service.ts\n billing.service.ts\n models/\n invoice.model.ts\n user.model.ts\nHere auth.controller.ts, auth.service.ts, and auth.routes.ts all belong to\nthe "auth" domain despite living in three different directories. domain-agents\nuses naming pattern extraction cross-referenced with import graph cohesion to\ncluster these. The auth.* files form a tight import cluster, which confirms\nthe naming signal.
Raw signals produce too many small, overlapping clusters. The orchestrator runs\na multi-phase normalisation pipeline.
\nPlurals merge: journals + journal → whichever has more files. Compound names\nconsolidate: bank-balance + bank-statement + bank-transaction →\nbank-accounts (the largest cluster). Small clusters merge into their strongest\nimport target, but only if they have a dominant dependency: more than 40% of\nimports from one target, and that target is at least 2x larger. This prevents\ncascading, where A merges into B, B gets bigger and attracts C, C pulls in D.
Files that import from 3+ domains get moved to "unassigned." These are coupling\nhotspots: middleware, orchestrators, shared handlers. Assigning them to one\ndomain would mislead the AI, so the tool surfaces them for a human decision.\nThat's the right call for architectural boundaries.
\nThe E2E test suite validates the complete pipeline against 3 fixture codebases\n(feature-organised, layer-organised, mixed). Current benchmark: 100% activation\naccuracy across all 3 patterns and all 3 activation levels (domain assignment,\nglob matching, MCP lookup).
\nThe integration into Claude Code and Cursor uses glob-based rule activation, the\nnative mechanism both tools already support.
\nEach domain gets a rule file with glob patterns in the frontmatter:
\n---\ndescription: billing domain\nglobs:\n - src/billing/**\n - **/billing.*\n - **/billing-*\n---\nWhen Claude Code opens a file matching those globs, the domain context loads. No\nMCP call, no background process, zero runtime overhead.
\nAn MCP server complements the rules with 4\non-demand tools: domain_lookup(file), domain_context(name),\ndomain_files(name), and list_domains(). A SessionStart hook prints a domain\nsummary at the start of every Claude Code session, so the AI has system-level\nawareness from the first prompt.
This is the bit I'm most keen on long-term.
\nAt Vend and Xero, teams owned domains. The billing team owned billing, the\nintegrations team owned integrations. Ownership meant knowing the interfaces,\nthe coupling points, the tech debt, and where things were headed. That knowledge\nlived in people's heads and got passed on through code reviews, architecture\nchats, and tribal memory.
\nDomain-specific AI agents formalise that same ownership model. An email agent\nloads the email domain's interface contract, its coupling to other domains, its\ncurrent scaling stage, and its tracked tech debt. A billing agent carries the\nsame for billing. They work within their boundaries and flag when a change\ncrosses a domain line.
\nYou don't need this from day one. Early on, one agent covers multiple areas. As\nthe product grows, agents split along the same lines engineering teams split: by\nbusiness domain. The operator (that's you) resolves conflicts where agents\ndisagree, the same way an engineering manager resolves cross-team dependencies.
\nThe analogy is rough, but it captures how AI-assisted development scales past a\nsingle person staring at a single context window.
\n","date_published":"Wed, 15 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/claude-code-can-now-spawn-copies-of-itself-in-isolated-vms/","url":"https://jonnonz.com/posts/claude-code-can-now-spawn-copies-of-itself-in-isolated-vms/","title":"Claude Code Can Now Spawn Copies of Itself in Isolated VMs","content_html":"The moment this project went from "fun weekend hack" to something I actually use\nevery day was when I got the MCP server working. Claude Code on my laptop sends\na prompt to the orchestrator sitting under my desk, which boots a VM, runs\nClaude Code inside it with full permissions, and streams the results back.\nClaude delegating work to Claude.
\nIt's a weird feeling watching it happen. You're in a conversation with Claude,\nit decides a task needs isolation, calls the MCP tool, and a few seconds later\nyou can see a fresh VM spinning up in the dashboard. Like having an intern who\ncan clone themselves.
\nPart 1\ncovered why I built this.\nPart 2 was the\nguts of it — rootfs, networking, the guest agent. This last post is about the\ninterfaces, the streaming pipeline, and what I'd change if this needed to work\nfor more than just me.
\nThe orchestrator exposes an MCP server with\neight tools. The main one is run_task — give it a prompt, optional config\n(RAM, vCPUs, timeout, max turns), and it blocks until the task completes.\nReturns the task ID, status, exit code, result files, cost, and the output\ntruncated to 4000 characters.
Two transport modes. Stdio for when Claude Code runs on the same machine:
\n{\n "mcpServers": {\n "orchestrator": {\n "command": "sudo",\n "args": ["/opt/firecracker/bin/orchestrator", "mcp"]\n }\n }\n}\nAnd Streamable HTTP for network access — Claude Code on any machine on the LAN\ncan use it:
\n{\n "mcpServers": {\n "orchestrator": {\n "type": "http",\n "url": "http://192.168.50.44:8081/mcp"\n }\n }\n}\nThe other tools are for poking around: get_task_status, list_vms,\nexec_in_vm (run a command in a still-running VM), read_vm_file,\ndestroy_vm, list_task_files, and get_task_file. That last one is smart\nabout content types — text files come back as plain text, images come back as\nbase64 MCP image content so Claude can actually see screenshots the VM took.
if isImageMime(mimeType) {\n encoded := base64.StdEncoding.EncodeToString(data)\n return mcplib.NewToolResultImage("Screenshot from task "+taskID, encoded, mimeType), nil\n}\nThis bit is worth telling because it'll save someone else the debugging time.
\nI originally built the MCP server with\nmcp-go v0.45.0 using SSE (Server-Sent\nEvents) transport. Worked great. Then Claude Code updated to expect the newer\nStreamable HTTP transport, and everything fell over.
\nThe failure mode was confusing. Claude Code would try to connect, attempt OAuth\ndiscovery against the /sse endpoint, get a 404 (my server doesn't do OAuth),\nand fail with:
Error: HTTP 404: Invalid OAuth error response: SyntaxError: JSON Parse error: Unable to parse JSON string\nNothing in my code changed. The client just started speaking a different\nprotocol.
\nThe fix was small once I understood it:
\n// Before — SSE transport\nfunc (s *Server) ServeSSE(addr string) error {\n sseServer := server.NewSSEServer(s.mcpServer,\n server.WithBaseURL("http://"+addr),\n )\n return sseServer.Start(addr)\n}\n\n// After — Streamable HTTP transport\nfunc (s *Server) ServeHTTP(addr string) error {\n httpServer := server.NewStreamableHTTPServer(s.mcpServer,\n server.WithEndpointPath("/mcp"),\n server.WithStateLess(true),\n )\n return httpServer.Start(addr)\n}\nBumped mcp-go from v0.45.0 to v0.46.0, swapped the server constructor, changed\nthe endpoint from /sse to /mcp, updated the client config. Done. But\ndiagnosing "OAuth error on a server that doesn't do OAuth" — that bit took a\nwhile.
When Claude Code runs inside a VM, its output needs to get from stdout inside\nthe guest all the way to a browser tab on my laptop. The path:
\nflowchart LR\n A["Claude Code stdout"] --> B["Guest agent\\nvsock frame"]\n B --> C["Host vsock client\\nExecStream"]\n C --> D["Task runner\\nOnEvent callback"]\n D --> E["Stream Hub\\nring buffer + fan-out"]\n E --> F["WebSocket\\nto browser"]\nThe stream hub (internal/stream/hub.go) is a per-task pub/sub system. Each\ntask gets a stream with a 1000-event ring buffer. When a WebSocket client\nconnects, it gets all the buffered history first, then live events as they\narrive.
Fan-out is non-blocking:
\nfor ch := range s.subscribers {\n select {\n case ch <- event:\n default:\n // Subscriber is slow, drop the event\n }\n}\nA slow WebSocket client can't block the task runner. If the browser can't keep\nup, it misses events. In practice this never happens because the bottleneck is\nalways Claude thinking, not the network.
\nThe React frontend is compiled to static files and embedded into the Go binary:
\n//go:embed all:web-dist\nvar webDistEmbed embed.FS\nSingle binary deployment. No nginx, no separate frontend server, no CORS\nheadaches in production. The API server falls through to index.html for\nunknown paths, which gives you SPA client-side routing.
The most interesting page is the task detail view. Claude Code's\n--output-format stream-json spits out one JSON object per line — thinking\nblocks, text responses, tool calls, tool results, cost summaries. The dashboard\nparses these into coloured blocks:
A useWebSocket hook connects when the task is running and disconnects when\nit's done. Green pulsing dot for live streaming. Auto-scroll to the bottom as\nevents arrive. Image files in the results get inline previews pointing at the\nAPI's file download endpoint — so when Claude takes a screenshot inside the VM,\nyou see it immediately.
Dark theme. Orange accents. Obviously.
\nThis runs on one box with no auth. It's a home lab project. But the gap between\n"works for me" and "works for a small team" isn't as big as it looks.
\nPersistence is the most obvious one. The task store is an in-memory Go map.\nOrchestrator restarts? All task history gone. VM metadata already persists to\ndisk and gets recovered on startup — tasks should too. SQLite or bbolt, a few\nhours of work. I just haven't needed it because I don't restart the process very\noften.
\nTask queue with backpressure. Right now tasks fire as goroutines with no\nconcurrency limit. Submit 20 tasks on a 30GB machine where each VM wants 2GB and\nthe last few fail because there's no memory left. A buffered channel or\nsemaphore would fix this. You could get fancier with priority queues — quick\ncode generation tasks ahead of long research tasks — but even a simple\nconcurrency cap would be enough.
\nAuthentication. The REST API and MCP server accept requests from anyone who\ncan reach the port. For a team: API keys at minimum, mTLS if you're serious\nabout it. The MCP spec supports auth flows now — that'd be the right way to do\nit for the MCP endpoint.
\nThe OnEvent callback race. This one's a latent bug. The task runner's\nOnEvent callback is stored on the runner struct, not passed per-task:
s.taskRunner.OnEvent = func(id string, event agent.StreamEvent) {\n taskStream.Publish(event)\n}\ns.taskRunner.Run(context.Background(), t)\nTwo simultaneous tasks overwrite each other's callbacks. It works today because\nMCP tasks block (one at a time) and the API handler sets up the stream before\nthe goroutine runs. But it's the kind of thing that works until it doesn't. Fix\nis trivial — pass the callback into Run() as a parameter.
Graceful shutdown. There's no signal handler. Ctrl-C kills the process,\nrunning VMs become orphans. They keep running as Firecracker processes — the\nrecoverState() function on next startup finds them and starts tracking them\nagain — but their tasks are lost. A proper signal handler would stop accepting\nnew tasks, wait for running ones to finish with a timeout, then tear everything\ndown cleanly.
For real multi-user you'd want result storage on S3 or R2 instead of local\ndisk. A web auth layer. Per-user credential vaults so different people's Claude\ntokens don't mix. Usage tracking and cost attribution.
\nWhat I wouldn't change: the single-binary deployment, vsock for host-guest\ncommunication, ephemeral VMs as the isolation model, the embedded frontend.\nThose are the right calls regardless of scale. The architecture is sound — it's\nthe operational bits around it that need work.
\nMost of these are a weekend each. The project is about 3,200 lines of Go and 860\nof TypeScript. It's not a big codebase. Adding persistence, auth, and a task\nqueue would maybe take it to 4,500 lines. Still fits in your head.
\nFor now, it sits under my desk and boots VMs when I ask it to. Claude delegating\nto Claude, in complete isolation, on hardware I own. That's enough.
\n","date_published":"Mon, 13 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/openhealth-chat-with-apple-health-data/","url":"https://jonnonz.com/posts/openhealth-chat-with-apple-health-data/","title":"OpenHealth – Chat with Apple Health Data, Anywhere","content_html":"For years I've worn an Apple Watch and let my iPhone quietly hoover up my\nresting heart rate, HRV, sleep stages, every workout, every nutrition log.\nMillions of data points. And for most of that time, when I wanted to actually\nask something about my training — "am I cooked this week?", "has my recovery\ngotten worse since Christmas?" — I'd open ChatGPT and get an answer that was\nbasically vibes, because it couldn't see any of the data.
\nSo I built openhealth. It turns your\nApple Health export into seven short markdown files any LLM can read. Drop the\nzip in your browser at\nopenhealth-axd.pages.dev, run the CLI, or\nbeam the zip straight from your iPhone over WebRTC. Paste the output into Claude\nor ChatGPT and start asking the questions you actually wanted to ask.
\n![\"openhealth's](\"https://jonnonz.com/img/posts/openhealth/hero.png\")
In January, Anthropic\nshipped an Apple Health connector\nfor Claude. OpenAI has one in ChatGPT. Both are US-only — if you're in New\nZealand like me, or the UK, EU, or Switzerland,\nthey're not available. That's\na lot of people locked out of the most natural way to use this data.
\nAnd even if you are in the US, you're letting Anthropic or OpenAI decide what\nthe model reads, how it's framed, and what tier unlocks it. I wanted control\nover the whole pipeline — including which LLM I feed it into.
\nopenhealth ships three ways.
\nA static web app. Drop export.zip, wait five seconds, download seven\nfiles. The browser does the parse. There's no upload endpoint because there's no\nserver — the Cloudflare Pages site is static HTML plus a tiny Web Worker. Open\nDevTools, watch the Network panel, nothing goes out.
A Bun-compiled CLI. openhealth ~/export.zip -o ./output gets you seven\nmarkdown files. --bundle concatenates them into one. --clipboard pushes that\nbundle straight to your system clipboard so you can paste it into any chat\nwindow. Zero deps beyond saxes for XML and fflate for unzip — even the\nargument parsing is node:util parseArgs, not Commander. One binary, put it\nwherever.
A phone-to-desktop handoff over WebRTC. The desktop site renders a QR code.\nPoint your iPhone camera at it, Safari opens a tiny receiver page, pick the zip,\nand it streams directly to your desktop browser over a DataChannel. The only\nbackend in the whole stack is a ~100-line Cloudflare Worker that relays the\nWebRTC handshake — it never sees a byte of your health data.
\n![\"Getting](\"https://jonnonz.com/img/posts/openhealth/walkthrough.png\")
Apple's export.xml is\nproperly huge.\nA long-term Watch user can easily have a 500MB–4GB file with millions of rows.\nMost XML parsers build a tree in memory, which OOMs before they finish.
openhealth uses saxes — a streaming SAX\nparser in pure TypeScript. It's isomorphic, so the same parser runs in Bun,\nNode, and the browser. I tested it against a synthetic 169MB / 1 million-record\nexport and it finished in about 5 seconds in Chrome, with the main-thread heap\nstaying around 5MB because the parse runs in a Web Worker.
\nThe rest of the core is a small pipeline: stream XML, accumulate\nper-record-type, roll up into weekly and monthly summaries, run each through a\nwriter that produces one markdown file. Every writer is snapshot-tested against\nbyte-for-byte expected output. 85 tests, TDD throughout.
\nEach one is deliberately small and shaped to be LLM-readable:
\nhealth_profile.md — baselines, data sources, long-term averagesweekly_summary.md — current week plus a 4-week rolling comparison with\nweek-over-week deltasworkouts.md — detailed log for the last 4 weeks: HR, duration, distance,\nenergybody_composition.md — weight trend, recent readings, nutrition averagessleep_recovery.md — nightly stages, 8-week averages, HRV, resting HR, SpO2\ntrendscardio_fitness.md — running log, HR-zone distribution, walking-speed trendsprompt.md — a ready-to-paste system prompt that frames the other six as\ncoaching inputDrop one file or all seven, depending on which chat model you're using.
\nFeeding real data to an LLM is a different experience from answering its\nquestions. When Claude can see that my resting HR has crept up 4bpm over the\nlast fortnight while my HRV has dropped and my training load stayed the same, it\ngives a real answer — "you're likely undercooked on recovery this week, here's\nwhat I'd change" — rather than a generic reminder to drink water.
\nIt's especially good if you've got multiple devices in the mix. I've got data\nfrom Apple Watch, the iPhone step counter, a Withings scale, and MyFitnessPal.\nThe parser picks the highest-trust source per metric — Apple Watch wins over\niPhone for steps, Watch sleep beats AutoSleep which beats Withings,\nduplicate-weight entries on the same day get deduped. You feed in one zip and\nget one coherent picture.
\nAsk it about your recovery, your training load, what you might be doing wrong,\nhow your sleep correlates with your long runs. It'll tell you — and it'll be\nright more often than not.
\nopenhealth itself never uploads your data. The web app parses in your browser\ntab. The CLI runs locally. The WebRTC handoff stays peer-to-peer — the\nCloudflare Worker that relays the handshake never sees a byte of the file. Clone\nthe repo, diff the build output, and confirm it yourself.
\nWhen you paste the seven files into ChatGPT or Claude, they see the data.\nThat's the trade most people will take for convenience, and it's fine. But if\nyou don't want to make that trade, you don't have to — run the CLI and pipe the\nbundle into a local model:
\nopenhealth ~/export.zip --bundle -o ./out\nollama run llama3 < ./out/openhealth.md\nOllama, llama.cpp, LM Studio, whatever you run. Your health data never leaves\nyour laptop. The output is just markdown — it doesn't care what reads it.
\nThat's why the shape is seven files and not an API. You pick what sees them.
\nI'm not a doctor. Neither is the model. Use this for thinking out loud about\nyour own training, not diagnosing anything.
\nMIT, source at\ngithub.com/jonnonz1/openhealth. Web\napp at openhealth-axd.pages.dev. If you've\nbeen sitting on a 200MB export.zip with nothing that'll open it, have a go.
Anthropic just dropped Project Glasswing\n— a big collaborative cybersecurity initiative with a shiny new model called\nClaude Mythos Preview that can find zero-day vulnerabilities at scale. Twelve\nmajor tech companies involved. $100M in credits. Found a 27-year-old flaw in\nOpenBSD. Impressive stuff.
\nBut let's be real about what's happening here. Anthropic trained a model so\ncapable at breaking into systems that they decided it was too dangerous to\nrelease publicly. So they wrapped the release in a collaborative security\ninitiative. The security work is genuinely valuable. But it's also a smart way\nto keep control of something they know is too powerful to let loose.
\nThe part that actually matters, though, is who benefits. Glasswing is for the\nbig players. The companies with security teams, budgets, and the kind of\ninfrastructure that gets invited to sit at the table with AWS, Microsoft, and\nPalo Alto Networks. What about the rest of us? The startups, the small SaaS\nshops, the indie developers running production systems on a shoestring?
\nThe internet is a\ndark forest.\nThat's not a metaphor anymore — it's becoming the literal reality. Bots,\nscrapers, automated exploit chains, credential stuffing, AI-generated phishing.\nA server goes up and within hours it's being scanned, fingerprinted, and probed\nby systems that don't sleep. Visibility equals vulnerability. And AI is making\nthe attackers faster, cheaper, and more autonomous every month.
\nThe\nISC2 put it plainly\n— both offence and defence now operate at speeds beyond human intervention. The\nthreats aren't people sitting at keyboards anymore. They're autonomous systems\nrunning campaigns end-to-end.
\nSo what do we do about it?
\nWhen I say offensive security, I don't mean red-teaming or penetration testing.\nI mean giving your systems the ability to fight back.
\nPicture an LLM that sits across your centralised logs — network traffic,\ndatabase queries, user interactions, access patterns — and builds an\nunderstanding of what normal looks like for your system over weeks and months.\nNot just pattern matching against known signatures. Actually understanding the\nshape of healthy behaviour.
\nWhen something breaks the pattern, it doesn't just alert. It acts.
\nDisable a compromised account. Kill a service that's behaving strangely. Block a\ndatabase connection that shouldn't exist. Create an incident with full context\nfor a human to review. The response is proportional and immediate — not waiting\nfor someone to check their phone at 3am.
\nThe architecture is pretty straightforward:
\ngraph TD\n A[Application Logs] --> D[Secure Isolated Log Store]\n B[Network Traffic] --> D\n C[Database Queries] --> D\n D --> F[Baseline Health Model]\n E[User Activity] --> D\n F -->|Anomaly Detected| G[LLM Analysis]\n G -->|Analyse & Plan| H{Threat Assessment}\n H -->|Low| I[Alert & Log]\n H -->|Medium| J[Restrict & Escalate]\n H -->|High| K[Disable & Isolate]\n I --> L[Human Review]\n J --> L\n K --> L\nThe key is that the logging and analysis layer has to be isolated and secured\nseparately from the systems it's watching. If an attacker can compromise the\nthing that's watching them, the whole model falls apart.
\nIn practice that means separate infrastructure with its own auth boundary.\nIngestion is write-only — your application services push logs in but can never\nread or modify what's already there. Append-only, immutable. The analysis layer\ngets scoped service accounts that can read logs, fire alerts, and pull specific\nemergency levers through a narrow API. Nothing else. If a compromised service\ntries to reach the log store directly, it hits a wall.
\nNone of this is exotic. Centralised logging, immutable storage, scoped IAM — the\nbuilding blocks exist. The hard part is wiring an LLM into that loop with the\nright constraints. Enough access to act, not enough to make things worse.
\nTraditional security tooling runs on signatures and static rules. Known bad\npatterns, blocklists, threshold alerts. That worked when threats were mostly\nhuman-paced. It doesn't work when you're up against autonomous systems that\nadapt faster than you can write rules.
\nThe alternative is a system that learns what normal looks like for your\nenvironment — not a generic baseline, but the actual shape of healthy behaviour\nin your specific infrastructure. Traffic patterns, query frequencies, access\ntiming, user behaviour. Weeks of observation before it starts making decisions.
\nWhen something breaks the pattern, the response is proportional. A sudden spike\nin unusual API calls might trigger deeper correlation — the system widens its\nsearch, pulls in more signals, lowers its threshold for flagging related\nactivity. Repeated failed auth attempts from new IPs tighten access controls\nautomatically. A database connection that shouldn't exist gets killed.
\nThis isn't a static ruleset you configure once and hope covers everything. It's\na system that develops behavioural intuition from running in your environment,\nresponding to your traffic. The difference matters — static rules are brittle\nagainst novel attacks, while adaptive systems can catch anomalies they've never\nseen before.
\nThe baseline isn't magic. It's watching five things:
\nThree layers of detection stack on top of each other. Layer one is simple\nthresholds — hard caps that trigger immediately. Layer two is statistical\ndeviation — standard deviations from the learned baseline. Layer three is\ncorrelation — looking across multiple signals simultaneously. A spike in rate\nalone might be fine. A spike in rate plus unusual composition plus new source\nIP? That's a pattern.
\nA pure anomaly detector would go nuts during deploys. New code paths, changed\nresponse times, config reloads — all of it looks unusual. Same with cron jobs.\nYour 3am batch job that hits the database hard every night would trigger alerts\nevery night.
\nTolerance patterns solve this. The system learns to recognise you.
\nMark a deploy event, and the system creates a tolerance window — elevated\nthresholds for the next 30 minutes. Register a recurring cron job, and the\nsystem expects that exact spike at that exact time. These aren't exceptions you\nconfigure manually. They're patterns the system learns from watching.
\nAfter a few weeks, it knows when your weekly cache warm-up runs, when your daily\nreports generate, when deploys happen. It stops bothering you about the things\nyou do on purpose.
\nCalling an LLM for every anomaly would be expensive. The trick is building\nimmune memory.
\nWhen the LLM analyses an anomaly and decides it's benign — say, a deploy spike\nor a legitimate traffic surge — that verdict gets stored. Next time the same\npattern appears, the system recognises it. No LLM call needed.
\nThis is how your security bill drops over the first few weeks. Early on,\neverything is novel. The LLM gets called constantly. A month in, most anomalies\nmatch patterns it's already seen. The LLM only gets called for genuinely new\nsituations.
\nThe more your system runs, the smarter it gets and the less it costs.
\nThe hardest part of any security tool is configuration. Getting thresholds\nright. Understanding your traffic patterns before you can tell the tool what's\nnormal.
\ndarkforest init flips this. Point it at a log sample — a day's worth of\ntraffic, a week if you've got it — and Claude reads it. Not just parsing,\nactually understanding the shape of your system. It figures out what your\nendpoints are, what normal request rates look like, what user agents show up,\nwhere your traffic comes from geographically.
Then it writes your config file for you.
\nYou review it, tweak anything that looks wrong, and you're running. No\nspreadsheets. No guesswork about what "normal" means for your specific stack.\nThe LLM that's going to watch your logs already understands them.
\nGlasswing is cool.\nOpen-source frameworks like CAI are\nmaking progress — but mostly on the offensive side, using LLMs for penetration\ntesting and vulnerability research. On the defensive side, the tooling barely\nexists. There's no open-source equivalent for the kind of adaptive monitoring\nand response I'm describing here.
\nThe building blocks are around. Centralised logging is a solved problem. Open\nstandards for security event formats are maturing. Smaller open models are more\nthan capable of pattern analysis on local infrastructure. What's missing is the\nglue — a framework that takes logs in, builds a baseline, detects anomalies, and\ncan actually respond. Something a small team can deploy without a six-figure\nsecurity budget.
\nThe threats don't discriminate by company size. The defences shouldn't either.\nThis can't be proprietary or locked behind enterprise contracts.
\nThe dark forest doesn't care how big your company is. The bots scanning your\ninfrastructure don't check your headcount before they attack. If the threats are\ngoing to be this accessible, the defences need to be too.
\nI'm building this. An open-source security agent — adaptive, autonomous, acts\nwhen something breaks the pattern. Small enough for a startup to run on their\nown infrastructure. Centralised logging, open LLMs, scoped response actions. The\npieces are all there. I'm wiring them together now.
\nFor v0.1, one real action working end-to-end: detect anomalous authentication\npatterns, call the LLM for analysis, and disable the compromised account via\nyour identity provider's API. Not just alerting — actually responding while\nyou're asleep. That's the proof of concept that matches the headline.
\nI'm actively working on this and looking for early testers. If you want alpha\naccess when it's ready, or just want to follow along,\ndrop your email below. I'll reach out when there's something to\ntry.
\n","date_published":"Sat, 11 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/29-hours-debugging-iptables-to-boot-vms-in-4-seconds/","url":"https://jonnonz.com/posts/29-hours-debugging-iptables-to-boot-vms-in-4-seconds/","title":"I Spent 29 Hours Debugging iptables to Boot VMs in 4 Seconds","content_html":"The first time I got a Firecracker VM to boot and respond to a vsock ping from\nthe host, I sat there grinning like an idiot. Typed a command on my machine, it\nreached through a kernel-level socket into a completely separate Linux system\nwith its own kernel, and got a reply. Under a second.
\nThat was about 30 hours into the project. The previous 29 were mostly fighting\nwith rootfs images and iptables rules.
\nPart 1\ncovered why I built this — Firecracker MicroVMs for running Claude Code in\nfull-permission isolation. This post is the actual build. Rootfs, networking,\nthe guest agent, and the streaming pipeline.
\nA Firecracker VM needs two things: an uncompressed Linux kernel (vmlinux, not\nbzImage — there's no bootloader) and an ext4 filesystem image to use as the\nroot disk.
The kernel is straightforward — grab a prebuilt 6.1 LTS vmlinux. The rootfs took\nmore work.
\nIt's a standard ext4 image with Debian Bookworm, and it needs everything Claude\nCode might want: Node.js 24, Python 3.11, Chromium for browser automation, git,\ncurl, jq, and the full Claude Code CLI installed globally via npm. The image\nends up at about 4GB.
\nThe guest agent — the Go binary that listens for commands from the host — lives\ninside the rootfs as a systemd service:
\nsudo mount /opt/firecracker/rootfs/base-rootfs.ext4 /mnt\nsudo cp bin/agent /mnt/usr/local/bin/agent\nsudo chmod +x /mnt/usr/local/bin/agent\n\nsudo tee /mnt/etc/systemd/system/agent.service <<'EOF'\n[Unit]\nDescription=Orchestrator Guest Agent\nAfter=network.target\n\n[Service]\nType=simple\nExecStart=/usr/local/bin/agent\nRestart=always\nRestartSec=1\n\n[Install]\nWantedBy=multi-user.target\nEOF\n\nsudo chroot /mnt systemctl enable agent.service\nsudo umount /mnt\nThat RestartSec=1 matters. If the agent crashes for any reason, systemd has it\nback up in a second. The orchestrator polls vsock every 500ms waiting for the\nagent, so even a crash during boot is barely noticeable.
You build this rootfs once, by hand. Every new VM gets a sparse copy of it.
\ninternal/vm/manager.go handles the whole lifecycle. It's sequential with\ncleanup at each step — if anything fails, it tears down what it already set up\nand returns the error.
flowchart TD\n A["Copy rootfs (sparse)"] --> B["Mount & inject network config"]\n B --> C["Create TAP device"]\n C --> D["Add iptables rules"]\n D --> E["Setup jailer chroot"]\n E --> F["Write Firecracker config JSON"]\n F --> G["Launch via jailer --daemonize"]\n G --> H["Find PID, save metadata"]\n H --> I["VM ready — poll vsock"]\nThe sparse copy is the first thing that happens:
\ncmd := exec.Command("cp", "--sparse=always", BaseRootfs, vm.RootfsPath)\n--sparse=always means zero blocks aren't allocated on disk. A 4GB image might\nonly use 2GB of actual disk space. Takes under a second on NVMe.
After copying, the rootfs gets mounted and three files are injected: a\nsystemd-networkd config with a static IP, /etc/resolv.conf for DNS, and\n/etc/hostname. Then it's unmounted and copied again into the jailer chroot.
Yeah, that's two copies of the rootfs per VM. The first for network injection,\nthe second because the jailer expects everything inside its chroot. I could\ncollapse this into one copy by injecting the network config directly into the\nchroot copy, but it's never been a bottleneck — sparse copy of 4GB takes less\ntime than Firecracker takes to boot. So I left it.
\nFirecracker's jailer is a separate binary that creates a chroot, sets up minimal\n/dev entries (kvm, net/tun, urandom), and runs the Firecracker process inside\nit. The VM config is a JSON file:
vmConfig := map[string]interface{}{\n "boot-source": map[string]interface{}{\n "kernel_image_path": "/vmlinux",\n "boot_args": "console=ttyS0 reboot=k panic=1 pci=off init=/sbin/init",\n },\n "drives": []map[string]interface{}{{\n "drive_id": "rootfs",\n "path_on_host": "/rootfs.ext4",\n "is_root_device": true,\n "is_read_only": false,\n }},\n "machine-config": map[string]interface{}{\n "vcpu_count": vm.VCPUs,\n "mem_size_mib": vm.RamMB,\n },\n "network-interfaces": []map[string]interface{}{{\n "iface_id": "eth0",\n "guest_mac": "06:00:AC:10:00:02",\n "host_dev_name": netCfg.TapDev,\n }},\n "vsock": map[string]interface{}{\n "guest_cid": vm.VsockCID,\n "uds_path": "/vsock.sock",\n },\n}\npci=off because Firecracker doesn't emulate PCI. Paths are relative to the\njailer chroot. The vsock entry creates a Unix domain socket at /vsock.sock\ninside the chroot — that's how the host talks to the guest.
Launch looks like this:
\ncmd := exec.Command(JailerBin,\n "--id", vm.JailID,\n "--exec-file", FCBin,\n "--uid", "0", "--gid", "0",\n "--cgroup-version", "2",\n "--daemonize",\n "--",\n "--config-file", "/vm-config.json",\n)\ncmd.Run()\nAfter launch there's a 2-second sleep — Firecracker needs a moment to start —\nthen the PID is found via pgrep and saved to a metadata file. If the\norchestrator restarts, it reads these metadata files and picks up where it left\noff. VMs survive orchestrator crashes.
This is where I burned the most time. Not because the concepts are hard, but\nbecause of one specific bug that had me questioning reality.
\nEach VM needs internet access for Claude Code to fetch packages, clone repos,\nand hit the Anthropic API. The approach: each VM gets a Linux TAP device on the\nhost, a dedicated /24 subnet, and iptables rules for NAT.
Subnets are deterministic, derived from the VM name using FNV-1a hashing:
\nfunc NetSlot(name string) int {\n h := fnv.New32a()\n h.Write([]byte(name))\n return int(h.Sum32()%253) + 1\n}\nVM named task-a3bfca80 might hash to slot 61, giving it subnet\n172.16.61.0/24, guest IP 172.16.61.2, TAP IP 172.16.61.1. No coordination\nneeded, no DHCP server, no IP pool to manage. The collision space is 253 slots —\nmore than enough for 12-13 concurrent VMs.
A TAP device is a virtual ethernet interface. Firecracker attaches the guest's\neth0 to it.
tap := &netlink.Tuntap{\n LinkAttrs: netlink.LinkAttrs{Name: cfg.TapDev},\n Mode: netlink.TUNTAP_MODE_TAP,\n}\nnetlink.LinkAdd(tap)\naddr, _ := netlink.ParseAddr(cfg.TapIP + "/24")\nlink, _ := netlink.LinkByName(cfg.TapDev)\nnetlink.AddrAdd(link, addr)\nnetlink.LinkSetUp(link)\nTAP names are fc-<vm-name>, truncated to 15 characters because Linux interface\nnames can't be longer. A fun constraint to discover at runtime.
Three rules per VM:
\n// NAT — rewrite source IP when traffic exits the host\nipt.AppendUnique("nat", "POSTROUTING",\n "-s", cfg.Subnet, "-o", cfg.HostIface, "-j", "MASQUERADE")\n\n// FORWARD — allow outbound from TAP\nipt.Insert("filter", "FORWARD", 1,\n "-i", cfg.TapDev, "-o", cfg.HostIface, "-j", "ACCEPT")\n\n// FORWARD — allow established/related inbound\nipt.Insert("filter", "FORWARD", 1,\n "-i", cfg.HostIface, "-o", cfg.TapDev,\n "-m", "state", "--state", "RELATED,ESTABLISHED", "-j", "ACCEPT")\nSee those Insert calls with position 1? That's the bug fix.
I originally used Append for the FORWARD rules. Traffic from the VM would\nleave the host fine (NAT worked), but return traffic got dropped. The VM could\nresolve DNS but couldn't complete TCP handshakes. I spent an embarrassing amount\nof time staring at tcpdump output before I figured it out.
Ubuntu's UFW adds a blanket DROP rule to the FORWARD chain. If you append your\nACCEPT rules, they land after UFW's DROP. They never match. The packets hit\nthe DROP rule first and get silently killed.
Insert at position 1 puts the rules before UFW's. Return traffic flows, VMs\nget internet access, everything works.
The traffic path through a working VM:
\nGuest (172.16.61.2) → eth0 → TAP (fc-task-xxx) → FORWARD ACCEPT\n→ NAT MASQUERADE (rewrite src to host IP) → host interface → internet\n→ response → RELATED,ESTABLISHED → TAP → guest eth0\nVMs can't reach each other. Each TAP device is point-to-point on its own /24.\nThere's no route between subnets.
cmd/agent/main.go — 420 lines of Go. It's a static binary that starts on boot,\nlistens on vsock port 9001, and handles five request types: ping, exec,\nwrite_files, read_file, and signal.
The interesting one is streaming exec.
\nWhen the orchestrator wants to run Claude Code, it sends an exec request with\nstream: true. The agent spawns the command, reads stdout and stderr line by\nline, and sends each line back as a framed event over the vsock connection. When\nthe process exits, it sends an exit event with the exit code.
Sounds straightforward. The tricky part is background processes.
\nClaude Code can start things that outlive the main command — dev servers, file\nwatchers, whatever it decides it needs. These child processes inherit the\nstdout/stderr pipes. If the agent waits for the pipes to close (the normal\napproach), it hangs forever because the children are still holding them open.
\nThe fix has three parts:
\n// 1. Process group isolation\ncmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}\n\n// 2. Wait for the main process, not the pipes\n<-waitDone\n\n// 3. Kill the entire process group\npgid, _ := syscall.Getpgid(cmd.Process.Pid)\nsyscall.Kill(-pgid, syscall.SIGTERM)\ntime.Sleep(500 * time.Millisecond)\nsyscall.Kill(-pgid, syscall.SIGKILL)\nSetpgid: true puts the command in its own process group. When the main process\nexits, kill the group (-pgid means "everything in this group"). SIGTERM first,\nwait half a second, then SIGKILL for anything that didn't listen.
Even after killing the group, there's a 3-second timeout waiting for the\npipe-reading goroutines to drain. If they're still stuck after that, move on and\nsend the exit event anyway. Can't let a hung pipe block the entire task.
\nThe line-by-line reader uses a 256KB buffer because Claude Code's\n--output-format stream-json can produce enormous single lines — tool results\nthat include the full contents of files it read.
Before Claude Code runs, the orchestrator writes five things into the VM via\nvsock:
\nOAuth credentials from the host's ~/.claude/.credentials.json (mode 0600). A\nsettings file that allows all tools. An environment script that sets\nCLAUDE_DANGEROUSLY_SKIP_PERMISSIONS=true. Task metadata. And a marker file to\ncreate the output directory.
The prompt itself gets written to a temp file inside the VM to avoid shell\nescaping nightmares, then referenced in the command:
\nclaudeArgs := fmt.Sprintf(\n "claude -p \\"$(cat %s)\\" --output-format stream-json --verbose",\n promptFile,\n)\ncmd := []string{"bash", "-c",\n "source /etc/profile.d/claude.sh && " + claudeArgs}\nWhen the VM is destroyed, the rootfs — containing the credentials — is deleted.\nCredentials only exist for the lifetime of the task.
\nAfter Claude Code finishes, the orchestrator searches for files it created:
\n// Anything in the output directory\nvsock.Exec(jailID, []string{"find", outputDir, "-type", "f", "-not", "-name", ".keep"}, nil, "/root")\n\n// Any new files under /root, created after the prompt was written\nvsock.Exec(jailID, []string{"find", "/root", "-maxdepth", "2", "-type", "f",\n "-newer", "/tmp/claude-prompt.txt"}, nil, "/root")\nEach file gets downloaded via vsock.ReadFile and saved to\n/opt/firecracker/results/<task-id>/. The runner also scans the accumulated\noutput for Claude's total_cost_usd field to record what the task cost in API\ncredits.
Then the VM is destroyed. Firecracker process killed, TAP device removed,\niptables rules deleted, jailer chroot deleted, VM state directory deleted. Clean\nslate.
\nThe whole cycle — boot, inject, run, collect, destroy — typically takes 30-120\nseconds depending on how complex the prompt is. The 4-second boot and ~1-second\nteardown are rounding errors compared to the time Claude actually spends\nthinking.
\nPart 3\ngets into the fun stuff — the MCP server that lets Claude delegate tasks to\nitself, the streaming architecture, the web dashboard, and what productionising\nthis would actually look like.
\n","date_published":"Fri, 10 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/can-you-beat-last-month/","url":"https://jonnonz.com/posts/can-you-beat-last-month/","title":"Can You Beat Last Month?","content_html":"Every machine learning project needs a reality check.
\nIt's tempting to jump straight to the neural network. That's the exciting bit,\nright? But if you don't establish what a dead-simple model can do first, you've\ngot no idea whether your fancy architecture is actually learning anything useful\nor just being expensive.
\nSo before ConvLSTM gets anywhere near this data, we're going to throw three\ngloriously simple baselines at it and see how they do.
\nThe dumbest possible model. To predict April, just use March's values. Every\ncell, every crime type. Carbon copy.
\nIt sounds ridiculous, but it works surprisingly well when patterns are stable.\nAnd as we saw in the EDA, Auckland's crime hotspots are remarkably persistent.\nThe CBD doesn't suddenly go quiet. South Auckland doesn't randomly calm down.
\nOn the six-month test set (August 2025 – January 2026):
\n| Crime Type | \nMAE | \nRMSE | \n
|---|---|---|
| Theft | \n1.42 | \n3.18 | \n
| Burglary | \n0.38 | \n0.91 | \n
| Assault | \n0.22 | \n0.64 | \n
| Robbery | \n0.04 | \n0.15 | \n
| Sexual | \n0.03 | \n0.12 | \n
| Harm | \n0.01 | \n0.04 | \n
Those MAE numbers for theft and burglary look small until you remember that most\ncells are zero. For the active cells (the ones we actually care about) the error\nis larger. A busy CBD cell might have 35 thefts in one month and 28 the next.\nPersistence would be off by 7 there, which is a 20% miss on an important\nprediction.
\nInstead of copying last month, copy the same month from the previous year.\nJanuary 2026 gets predicted from January 2025. This should capture seasonal\npatterns: the summer spike, the February dip.
\nThe catch? We only have four years of data. The test set months (August–January)\neach have at most three prior examples of the same month. That's not a lot of\nseasonal training data.
\n| Crime Type | \nMAE | \nRMSE | \n
|---|---|---|
| Theft | \n1.51 | \n3.42 | \n
| Burglary | \n0.41 | \n0.97 | \n
| Assault | \n0.24 | \n0.68 | \n
| Robbery | \n0.05 | \n0.17 | \n
| Sexual | \n0.04 | \n0.13 | \n
| Harm | \n0.01 | \n0.04 | \n
Slightly worse than persistence across the board. That surprised me initially.\nShouldn't capturing seasonality help?
\nThe issue is that the 2023-to-2025 decline we spotted in the EDA bites hard\nhere. If you predict January 2026 from January 2025, you're using data from a\nperiod when crime was higher. The seasonal pattern is real, but the\nyear-over-year trend works against it. With more years of data, seasonal naive\nwould likely pull ahead.
\nFor each cell and crime type, take the average across all 36 training months.\nThis smooths out month-to-month noise and gives you a "typical" value for each\nlocation.
\n| Crime Type | \nMAE | \nRMSE | \n
|---|---|---|
| Theft | \n1.28 | \n2.95 | \n
| Burglary | \n0.35 | \n0.84 | \n
| Assault | \n0.20 | \n0.58 | \n
| Robbery | \n0.04 | \n0.14 | \n
| Sexual | \n0.03 | \n0.11 | \n
| Harm | \n0.01 | \n0.04 | \n
The best baseline. By averaging over three years, it smooths out the\nmonth-to-month noise and the year-over-year trend simultaneously. It won't\ncapture seasonal peaks or sudden changes, but for the "typical month" prediction\nit's solid.
\nYou might wonder why I'm not reporting MAPE (Mean Absolute Percentage Error).\nIt's the standard metric in a lot of forecasting work. The reason: sparse data.
\nMAPE divides the error by the actual value. When the actual value is zero (which\nit is for 91.7% of our tensor) you get division by zero. Even for cells with\nsmall counts (1 or 2 crimes), a prediction of 0 gives you 100% MAPE while a\nprediction of 2 gives you 0–100%. The metric becomes wildly unstable.
\nMAE and RMSE are more honest here. They tell you the absolute magnitude of your\nerrors in actual crime counts, which is what we care about. A miss of 3\nvictimisations means the same thing whether the cell usually has 5 or 50.
\nHere's the scoreboard going forward. Any deep learning model needs to beat the\nhistorical average baseline to justify its existence:
\n| Crime Type | \nHistorical Avg MAE | \nHistorical Avg RMSE | \n
|---|---|---|
| Theft | \n1.28 | \n2.95 | \n
| Burglary | \n0.35 | \n0.84 | \n
| Assault | \n0.20 | \n0.58 | \n
| All types | \n0.39 | \n0.95 | \n
Theft is the easiest to beat because there's the most signal: high counts, clear\nspatial patterns, strong seasonality. Robbery, sexual offences, and harm are\nessentially noise at this resolution. The models will probably predict near-zero\nfor those types and be mostly correct.
\nThe real test will be the middle ground. Can ConvLSTM or ST-ResNet predict the\nchanges in theft and burglary better than a static average? Can they catch the\nmonths where a cell spikes or dips? That's where simple baselines fall flat,\nbecause they don't model dynamics at all.
\nIf the deep learning can't meaningfully beat "just use the average," then it's\nnot worth the CPU cycles. Or in my case, the many hours of a Ryzen 5 grinding\naway.
\n","date_published":"Thu, 09 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/claude-code-running-claude-code-in-4-second-disposable-vms/","url":"https://jonnonz.com/posts/claude-code-running-claude-code-in-4-second-disposable-vms/","title":"Claude Code Running Claude Code in 4-Second Disposable VMs","content_html":"Running Claude Code with full permissions inside a Docker container is a\nterrible idea. I did it anyway for about a week, then built something better.
\nAnthropic has an internal platform — people have been calling it\nAntspace\nsince it got reverse-engineered from the Claude Code source — that runs AI\ncoding tasks in isolated environments. It's part of a vertical stack they're\nbuilding internally: intent goes in, code comes out, and the agent never touches\nthe host machine.
\nI wanted that. Not the whole platform-as-a-service thing, just the core idea:\ngive Claude Code a prompt, let it run with zero permission restrictions, stream\nthe output back, grab any files it created, and destroy everything when it's\ndone. On a single Linux box sitting in my office.
\nThe result is about 3,200 lines of Go and 860 lines of TypeScript. It boots a\nfresh Linux VM in ~4 seconds, runs Claude Code inside it, and tears it down when\nthe task finishes. Three ways to use it: a CLI, a REST API with a web dashboard,\nand an MCP server so Claude Code on other machines can delegate tasks to it.
\nThis first post is about why I built it this way.\nPart 2 and\nPart 3 get\ninto the actual implementation.
\nCLAUDE_DANGEROUSLY_SKIP_PERMISSIONS=true — that's the environment variable\nthat tells Claude Code to stop asking before it runs shell commands or writes\nfiles. It just does whatever it thinks it needs to. For autonomous tasks, you\nneed this. Claude can't ask for confirmation when there's nobody watching.
The question is where you let it run.
\nDocker is the obvious first thought. Fast startup, everyone knows it, easy to\norchestrate. But containers share the host kernel. Every container on the\nmachine issues syscalls to the same Linux kernel, and a kernel vulnerability is\na vulnerability in every container on the host.\nThe isolation boundary is the container runtime,\nnot hardware — and that surface area is big.
\nFor most workloads this is fine. Running a web server in Docker? No worries. But\nrunning an AI agent that can execute arbitrary shell commands with root-level\npermissions? That's a different threat model. A container escape gives you the\nhost. And you've just given the thing inside the container permission to try\nanything.
\nAnthropic's own approach to\nsandboxing Claude Code\nuses OS-level primitives — bubblewrap on Linux, Seatbelt on macOS — for\nfilesystem and network isolation. They report an 84% reduction in permission\nprompts internally. That's smart for the normal use case where Claude is helping\nyou write code in your own project. But I wanted something more aggressive: full\nisolation where even a kernel exploit can't reach the host.
\nFirecracker is what AWS built for\nLambda and Fargate. Each MicroVM is a real KVM-backed virtual machine with its\nown guest kernel, its own memory space, and hardware-enforced isolation via\nIntel VT-x or AMD-V. The attack surface is the KVM hypervisor — which the kernel\nteam at AWS has spent years minimising.
\nThe trade-off is boot time. Containers start in under a second. Firecracker VMs\ntake about 4 seconds on my hardware once you account for the guest kernel boot,\nsystemd init, and the agent process starting up. For tasks that typically run\n20-120 seconds, 4 seconds of overhead is nothing.
\nEach VM also copies a 4GB rootfs image. Sparse copies make this fast (<1\nsecond), but it does use disk. On a machine with a 1TB NVMe, I'm not losing\nsleep over it.
\nThe hardware is an AMD Ryzen 5 5600GT with 30GB of RAM. Nothing exotic. About\n$400 worth of parts sitting under my desk. Each VM gets 2GB of RAM by default,\nso I can run roughly 12-13 VMs concurrently before the host runs out of memory.
\nThis was my favourite bit to figure out.
\nThe obvious way to communicate with a process inside a VM is SSH. Set up keys,\nopen a port, connect over the network. But SSH means key management, an open\nnetwork port inside the VM, and another service to configure. If the guest's\nnetwork breaks during a task, you've lost your control channel.
\nvsock\n(AF_VSOCK, address family 40) is a kernel-level host-guest communication\nchannel. It doesn't touch the network stack. No IP addresses, no ports, no keys.\nFirecracker exposes the guest's vsock as a Unix domain socket on the host side —\nyou connect to the socket, send CONNECT <port>\\n, and you're talking directly\nto a process inside the VM.
func Connect(jailID string, port int) (net.Conn, error) {\n socketPath := fmt.Sprintf("/srv/jailer/firecracker/%s/root/vsock.sock", jailID)\n conn, _ := net.Dial("unix", socketPath)\n conn.Write([]byte(fmt.Sprintf("CONNECT %d\\n", port)))\n // Read "OK <port>" response\n return conn, nil\n}\nOn the guest side, Go's standard library doesn't support AF_VSOCK — address\nfamily 40 doesn't exist in the net package. So the guest agent uses raw\nsyscalls:
fd, _ := syscall.Socket(40, syscall.SOCK_STREAM, 0) // AF_VSOCK = 40\n// Manually construct struct sockaddr_vm (16 bytes)\nsa := [16]byte{}\n*(*uint16)(unsafe.Pointer(&sa[0])) = 40 // family\n*(*uint32)(unsafe.Pointer(&sa[4])) = uint32(port) // port (9001)\n*(*uint32)(unsafe.Pointer(&sa[8])) = 0xFFFFFFFF // VMADDR_CID_ANY\nsyscall.RawSyscall(syscall.SYS_BIND, uintptr(fd), uintptr(unsafe.Pointer(&sa[0])), 16)\nsyscall.RawSyscall(syscall.SYS_LISTEN, uintptr(fd), 5, 0)\nYeah, that's unsafe.Pointer and manual struct layout. Not the prettiest Go\nyou'll ever write. But it works, it's fast, and the whole vsock layer is about\n160 lines shared between both binaries.
The wire protocol is dead simple — length-prefixed JSON frames:
\nfunc WriteFrame(w io.Writer, v interface{}) error {\n data, _ := json.Marshal(v)\n binary.Write(w, binary.BigEndian, uint32(len(data)))\n w.Write(data)\n return nil\n}\nEach operation (ping, exec, write files, read file) opens a new connection,\nsends one request, reads the response, and closes. Connection-per-request. Not\nfancy, but vsock connections are local and effectively instant, so there's no\nreason to complicate things with multiplexing.
\nThe whole system is two Go binaries — the orchestrator (runs on the host) and\nthe agent (runs inside each VM).
\ngraph TD\n subgraph "Host — orchestrator binary"\n API["REST API + WebSocket :8080"]\n MCP["MCP Server :8081"]\n VM["VM Manager"]\n NET["TAP + iptables"]\n TASK["Task Runner"]\n STREAM["Pub/Sub Hub"]\n VSOCK["vsock Client"]\n end\n\n subgraph "Guest — agent binary"\n AGENT["Guest Agent vsock:9001"]\n CLAUDE["Claude Code"]\n end\n\n API --> TASK\n MCP --> TASK\n TASK --> VM\n VM --> NET\n TASK --> VSOCK\n VSOCK --> AGENT\n AGENT --> CLAUDE\n TASK --> STREAM\n STREAM --> API\nThe orchestrator is a single 14MB binary with the React dashboard embedded via\n//go:embed. Copy it to a server, run it with sudo, done. Seven Go dependencies\ntotal — chi for routing, netlink for TAP devices, go-iptables for firewall\nrules, mcp-go for the MCP protocol, and a\nfew others.
The agent is a 2.5MB static binary compiled with CGO_ENABLED=0. It ships\ninside the VM's rootfs and starts via systemd on boot. Within about a second of\nthe VM coming up, the agent is listening on vsock port 9001 and ready to accept\ncommands.
They share exactly one file — internal/agent/protocol.go — which defines the\nwire protocol types and framing functions. Everything else is independent.
You give it a prompt. It does the rest.
\nFrom the CLI it looks like this:
\nsudo ./bin/orchestrator task run \\\n --prompt "Write a Python script that generates Fibonacci numbers" \\\n --ram 2048 \\\n --vcpus 2 \\\n --timeout 120\nOutput streams to your terminal in real time. When it's done:
\n=== Task Complete ===\nID: a3bfca80\nStatus: completed\nExit: 0\nCost: $0.0582\nFiles: [fibonacci.py]\nThe VM is gone. The rootfs is deleted. The TAP device and iptables rules are\ncleaned up. All that's left is the result files in\n/opt/firecracker/results/a3bfca80/.
Or you use the MCP server, and Claude Code on your laptop delegates the task to\na VM on the box under your desk. Claude spawning Claude. That bit is properly\ncool, and I'll get into it in Part 3.
\nQuick aside on this because people always ask.
\nGo produces static binaries. The agent needs to be a single file with zero\ndependencies that runs inside a minimal Debian guest — CGO_ENABLED=0 makes\nthis trivial. The orchestrator needs to manage concurrent VMs, and goroutines\nare a natural fit for that. Syscall support is first-class, which matters when\nyou're doing raw vsock operations. And it compiles in about 2 seconds, which is\nnice when you're iterating.
build-agent:\n\tCGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/agent -ldflags="-s -w" ./cmd/agent\nThat -ldflags="-s -w" strips debug info and DWARF tables, dropping the agent\nbinary from ~3.5MB to ~2.5MB. Every byte counts when you're baking it into a\nrootfs that gets copied for every VM.
Part 2 gets into\nthe actual build — the rootfs, the networking (including a fun bug with Ubuntu's\nUFW that had me staring at iptables rules for an embarrassing amount of time),\nthe guest agent, and the streaming pipeline that gets Claude's output from\ninside a VM to your browser.
\n","date_published":"Wed, 08 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/what-if-your-browser-built-the-ui-for-you/","url":"https://jonnonz.com/posts/what-if-your-browser-built-the-ui-for-you/","title":"What if your browser built the UI for you?","content_html":"We're at a genuinely weird inflection point in frontend development. AI can\ngenerate entire interfaces now. LLMs can reason about data and layout. And yet —\nmost SaaS products still ship hand-crafted React apps, each building its own UI,\nits own accessibility layer, its own theme system, its own responsive\nbreakpoints. Not every service, but the vast majority.
\nThat's a lot of duplicated effort for what's essentially the same job — showing\na human some data and letting them do stuff with it.
\nI've been thinking about this a lot lately, and I built a proof of concept to\ntest an idea: what if the browser itself generated the UI?
\nThe industry is circling this idea from multiple angles, but nobody's quite\nlanded on it yet.
\nServer-driven UI\nhas been around for a while — Airbnb and others pioneered it for mobile, where\napp store review cycles make shipping UI changes painful. The server sends down\na JSON tree describing what to render, and the client just follows instructions.\nIt's clever, but the server is still calling the shots. x.
\nGoogle recently shipped\nNatively Adaptive Interfaces\n— a framework that uses AI agents to make accessibility a default rather than an\nafterthought. Really cool idea, and the right instinct. But it's still operating\nwithin a single app's boundaries. Your accessibility preferences don't carry\nbetween Google's products and, say, your project management tool.
\nThen there's the\ngenerative UI\nwave — CopilotKit, Vercel's AI SDK, and others building frameworks where LLMs\ngenerate components on the fly. These are powerful developer tools, but they're\nstill developer tools. The generation happens at build time or on the server.\nThe service is still in control.
\nSee the pattern? Every approach keeps the power on the service side.
\nHere's the idea behind the\nadaptive browser: what if the\ngeneration happened on your side?
\nInstead of a service shipping you a finished frontend, it publishes a manifest —\na structured description of what it can do. Its capabilities, endpoints, data\nshapes, what actions are available. Think of it like an API spec, but semantic.\nNot just "here's a GET endpoint" but "here's a list of repositories, they're\nsortable by stars and language, you can create, delete, star, or fork them."
\nYour browser takes that manifest, calls the actual APIs, gets real data back,\nand then generates the UI based on your preferences. Your font size. Your colour\nscheme. Your preferred layout (tables vs cards vs kanban). Your accessibility\nneeds. All applied universally, across every service.
\nThe manifest for something like GitHub looks roughly like this — a service\ndescribes its capabilities and the browser figures out the rest:
\nservice:\n name: "GitHub"\n domain: "api.github.com"\n\ncapabilities:\n - id: "repositories"\n endpoints:\n - path: "/user/repos"\n semantic: "list"\n entity: "repository"\n sortable_fields: [name, updated_at, stargazers_count]\n actions: [create, delete, star, fork]\nThe browser takes that, fetches the data, and generates a bespoke interface —\nusing an LLM to reason about the best way to present it given who you are and\nwhat you're trying to do.
\nWhen I was building the app store and integrations platforms at Xero, one of the\nconstant headaches was that every third-party integration had its own UI\npatterns. Users had to learn a new interface for every app they connected. If\nthe browser was generating the UI from a shared set of preferences, that problem\njust… goes away.
\nAccessibility is the big one though. Right now, accessibility is a feature that\ngets bolted on — and often badly. When the browser generates the UI,\naccessibility isn't a feature. It's the default. Your preferences — high\ncontrast, keyboard-first navigation, screen reader optimisation, larger text —\napply everywhere. Not because every developer remembered to implement them, but\nbecause they're baked into how the UI gets generated in the first place.
\nCustomisation becomes genuinely personal too. Not "pick from three themes the\ndeveloper made" but "this is how I interact with software, full stop."
\nFrontend complexity drops dramatically, but the complexity doesn't disappear —\nit moves behind the API. And honestly, it probably increases.
\nAPI design becomes way more important. You can't just throw together some REST\nendpoints and call it a day. Your manifest needs to be semantic — describing\nwhat the data means, not just what shape it is. Data contracts between services\nmatter more. Versioning matters more.
\ngraph LR\n A[Service] -->|Publishes manifest + APIs| B[Browser Agent]\n C[User Preferences] --> B\n D[Org Guardrails] --> B\n B -->|Generates| E[Bespoke UI]\nBut here's the thing — this trade-off pushes us somewhere genuinely interesting.\nIf every service needs to describe itself semantically through APIs and\nmanifests, those APIs become the actual product surface. Not the frontend. The\nAPIs.
\nAnd once APIs are the product surface, sharing context between platforms becomes\nthe interesting problem. Your project management tool knows what you're working\non. Your email client knows who you're talking to. Your code editor knows what\nyou're building. Right now, none of these talk to each other in any meaningful\nway because they're all locked behind their own UIs. In a manifest-driven world,\nthat context flows through the APIs — and your browser can stitch it all\ntogether into something coherent.
\nI reckon we're about 3-5 years from this being mainstream. The pieces are all\nthere — LLMs that can reason about UI,\nstandardisation efforts around\nsending UI intent over APIs, and a growing expectation from users that software\nshould adapt to them, not the other way around.
\nThe services that win in this world won't be the ones with the prettiest\nhand-crafted UI. They'll be the ones with the best APIs, the richest manifests,\nand the most useful data. The frontend becomes a generated output, not a\nhand-crafted input.
\nOrganisations will set preference guardrails — "our people can use dark or light\nmode, must have destructive action confirmations, these fields are always\nvisible" — while individuals customise within those bounds. Your browser becomes\nyour agent, not just a renderer.
\nI built the adaptive browser as\na proof of concept to test this thinking — it uses Claude to generate UIs from a\nGitHub manifest and user preferences defined in YAML. It's rough, but the\ndirection feels right.
\nThe frontend isn't dying. But what we think of as "frontend development" is\nabout to change. The interesting work moves to API design, semantic data\ncontracts, and building browsers smart enough to be genuine user agents.
\n","date_published":"Sun, 05 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/stealing-nanoclaw-patterns-for-webapps-and-saas/","url":"https://jonnonz.com/posts/stealing-nanoclaw-patterns-for-webapps-and-saas/","title":"Stealing NanoClaw Patterns for Web Apps and SaaS","content_html":"In Part 1 I pulled\napart NanoClaw's codebase and found six patterns that make an 8,000-line AI\nassistant surprisingly robust. But NanoClaw is a single-user tool running on\nyour laptop. Surely these patterns fall apart once you've got real tenants, real\nmoney, and real scale?
\nNah. Four of them translate almost directly — and the ones that don't still\nteach you something useful.
\nNanoClaw's credential proxy — where containers get a placeholder API key and a\nlocalhost proxy injects the real one — sounds like a neat trick for a personal\ntool. But this exact pattern is showing up in production Kubernetes deployments\nright now.
\nThe broader version is a\nsidecar proxy that handles credential injection\nfor any service that needs API keys or tokens. Your application code never\ntouches the real secret. A sidecar container intercepts outbound requests, swaps\nin credentials, and forwards them upstream.
\nAt Vend we managed a bunch of third-party integrations — payment gateways,\nshipping providers, accounting platforms. Each one had API keys that needed to\nlive somewhere. We went through the typical evolution: environment variables,\nthen a secrets manager, then a service that distributed keys at startup. Every\nstep was an improvement, but the keys still ended up in the application's\nmemory.
\nThe sidecar approach skips that entirely. Your app sends requests with a\nplaceholder. The proxy — which is a separate process with its own security\nboundary — does the credential swap. Even if your application gets compromised,\nthe real keys aren't there to steal.
\nIf you're running any kind of multi-service architecture where services call\nexternal APIs, this pattern is worth adopting. Your API gateway might already be\ndoing a version of it — the insight is making it explicit and consistent across\nall outbound credential flows.
\nThis is the one I keep thinking about.
\nNanoClaw uses filesystem mounts to control what each container can see. No\napplication-level permission checks — the security model is the infrastructure\ntopology. If a container can't see a file, it can't access it. No bugs, no\nmissed checks, no escalation vulnerabilities.
\nIn SaaS, we spend enormous amounts of time writing authorisation logic. Role\nchecks, permission middleware, tenant-scoping queries. And it works — until\nsomeone forgets a WHERE clause.
\nAWS's own\nSaaS tenant isolation guidance\nmakes this point explicitly: authentication and authorisation are not the same\nas isolation. The fact that a user logged in doesn't mean your system has\nachieved tenant isolation. A\nsingle missed tenant filter\non a database query and you've got a cross-tenant data leak.
\nThe NanoClaw-inspired approach is to push isolation down the stack. Separate\ndatabase schemas per tenant. Separate containers. Separate cloud accounts for\nyour highest-value customers. Not instead of application-level checks — but as a\nbackstop that catches the bugs your application-level checks inevitably have.
\nAt Xero, working across the integrations and app store teams, I saw first-hand\nhow multi-tenant data isolation gets complicated fast. The teams that had the\nfewest incidents were the ones where the infrastructure itself enforced\nboundaries, not just the application code.
\nYou don't need to go full NanoClaw and give every tenant their own container.\nBut you should be asking: if my application-level authorisation has a bug,\nwhat's my second line of defence? If the answer is "nothing" — that's the\npattern to steal.
\nNanoClaw polls SQLite every 2 seconds. No WebSockets, no event bus, no pub/sub.\nJust a loop that checks for new stuff.
\nThe instinct for most teams is to treat polling as a temporary hack you'll\nreplace with "proper" event-driven architecture later. Yan Cui wrote a\nsolid breakdown of push vs poll in event-driven systems\nand the takeaway isn't that one is always better — it's that the right choice\ndepends on your throughput, ordering, and failure-handling requirements.
\nFor a lot of internal systems, polling is the correct permanent answer.
\nAdmin dashboards. Background job status. Internal reporting. Webhook retry\nqueues. Deployment pipelines. These systems don't need sub-second latency. They\nneed reliability and simplicity. A polling loop against your database gives you\nboth, with zero infrastructure overhead.
\nAt Xero we shipped multiple times per day, and some of the internal tooling that\nsupported continuous deployment was surprisingly simple under the hood. Cron\njobs. Polling loops. SQL queries on a timer. Not because anyone was cutting\ncorners — because the requirements genuinely didn't need anything more\nsophisticated.
\nThe trap is reaching for Kafka or RabbitMQ because you think you'll need it\neventually.\n70% of startups fail due to premature scaling.\nThe infrastructure you don't deploy is the infrastructure that never breaks.
\nNanoClaw uses JSON files on the filesystem for inter-process communication.\nAtomic rename, directory-based identity, simple polling to pick up new messages.\nNo Redis. No message broker.
\nThat specific approach won't scale to a multi-tenant SaaS — but the instinct\nbehind it absolutely does. The instinct is: use the infrastructure you already\nhave.
\nFor most web apps, that means Postgres. The\nPostgres-as-queue movement\nhas been gaining serious traction, and tools like\nPGMQ make it practical. You get ACID guarantees,\nyou don't need to manage another service, and your queue is backed by the same\ndatabase you're already monitoring and backing up.
\nNanoClaw's\natomic write pattern\n— write to a temp file, rename into place — maps to INSERT INTO queue_table\nfollowed by a SELECT ... FOR UPDATE SKIP LOCKED consumer. Same principle: the\nmessage either exists completely or doesn't exist at all. No partial state.
The "just add Redis" reflex is strong in our industry. Sometimes it's the right\ncall. But I've seen plenty of teams introduce a message broker for a workload\nthat Postgres could've handled without breaking a sweat — and then spend the\nnext six months debugging consumer lag and dead letter queues.
\nThe specific techniques matter less than the discipline behind them.
\nNanoClaw's developer looked at a 500,000-line framework and asked: what are my\nactual constraints? Single user. Local machine. One AI provider. And then\nbuilt exactly the architecture those constraints required — nothing more.
\nMost teams don't do this. They build for imaginary scale, imaginary\nmulti-tenancy requirements, imaginary traffic spikes. They reach for Kubernetes\nbefore they've outgrown a single server. They deploy event buses before they've\noutgrown a polling loop. They write complex authorisation middleware before\nthey've considered whether infrastructure isolation would eliminate the problem\nentirely.
\nThe pattern worth stealing isn't the credential proxy or the polling loop or\nPostgres-as-queue. It's the habit of understanding your constraints first and\nletting them delete complexity from your architecture.
\nHardest pattern to adopt, though. Because it means admitting you're smaller than\nyou think.
\n","date_published":"Sun, 05 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/what-the-data-actually-shows/","url":"https://jonnonz.com/posts/what-the-data-actually-shows/","title":"What the Data Actually Shows","content_html":"You can't just shove a tensor into a neural network and hope for the best.
\nI mean, you can. People do it all the time. But you'll have no idea whether\nyour model is learning something real or just memorising noise. Before we get\nanywhere near ConvLSTM or ST-ResNet, we need to properly understand what\npatterns actually exist in this data, and whether they're strong enough for a\nmodel to learn.
\nThis is the part that most ML blog posts skip. It's also the part that saves you\nweeks of debugging later.
\nThe monthly pattern across Auckland is surprisingly consistent year to year.\nCrime peaks in late spring and early summer (October through January) and dips\nin late summer through winter. February is reliably the quietest month at around\n7,000–8,000 victimisations, while November and December regularly push past\n9,000.
\nThis tracks with what\ncriminology research has found globally:\nwarmer months mean more people out and about, more opportunities for property\ncrime, and more interpersonal conflict. It's a well-documented pattern called\nseasonal variation in crime, and it shows up clearly in the NZ data.
\nThe seasonal signal isn't uniform across crime types though. Theft drives most\nof the swing. It surges in summer and drops in winter, accounting for nearly all\nthe monthly variance. Assault has its own rhythm. It peaks around the holiday\nperiod (December–January) and shows a secondary bump in winter weekends,\nprobably pub-related. Burglary is flatter, with a slight winter uptick when\nhouses are dark earlier.
\n2023 was the peak year across the board, with a noticeable decline through 2024\nand into early 2025. Whether that's a real trend or a reporting artefact, I\ngenuinely don't know. But it means the model's training data includes both an\nupswing and a downswing, which is useful. It can't just learn "crime always goes\nup."
\nCrime in Auckland is not randomly distributed. That's obvious to anyone who\nlives here, but it's worth quantifying.
\nRunning a\nMoran's I test\non our 500m grid confirms strong positive spatial autocorrelation. Cells with\nhigh crime counts are surrounded by other high-crime cells. The Moran's I\nstatistic comes out at 0.43 (p < 0.001), which means the clustering is highly\nsignificant. Crime begets more crime in adjacent cells.
\nThe hotspots are exactly where you'd expect. The CBD dominates: Queen Street,\nKarangahape Road, and the surrounding blocks consistently light up across all\ncrime types. South Auckland corridors (Manukau, Ōtāhuhu, Papatoetoe) form a\nsecond cluster, particularly for assault and robbery. Henderson in the west\nshows up for burglary.
\nWhat's less obvious is how stable these hotspots are over time. The top 5% of\ncells (about 227 cells) account for over 60% of all recorded crime across the\nentire four-year period. These aren't random spikes. They're persistent. A cell\nthat's hot in 2022 is almost certainly still hot in 2025. That temporal\npersistence is exactly what makes this data amenable to prediction. If hotspots\nmoved randomly month to month, no model could learn them.
\nThe six channels in our tensor don't behave independently. Theft and burglary\nshow moderate positive correlation (r ≈ 0.52). Cells with lots of theft tend to\nhave more burglary too, which makes sense given similar opportunity structures\n(commercial areas, transport hubs).
\nAssault correlates weakly with everything else (r ≈ 0.15–0.25). It has its own\nspatial logic (nightlife areas, specific residential pockets) that doesn't align\nneatly with property crime.
\nRobbery, sexual offences, and harm are so sparse at the 500m monthly resolution\nthat correlation analysis is basically meaningless. Most cells have zero counts\nfor these types in any given month. That sparsity is going to be a real headache\nfor the models.
\nWe flagged this in Part 3: 91.7% of the tensor is zeros. But the EDA makes the\nproblem even clearer.
\nThe distribution of non-zero cell values is heavily right-skewed. The median\nnon-zero value is 1. One crime, in one cell, in one month. The mean is about\n2.3. A handful of cells (the CBD, Manukau) hit 30–50+ in peak months for theft.\nThe model needs to learn the difference between "always zero" cells,\n"occasionally one" cells, and "consistently busy" cells.
\nIf you plot the crime count distribution across non-zero cells, it follows\nsomething close to a power law. A tiny number of cells carry an outsized share\nof the signal. This is textbook\nspatial concentration of crime,\ndocumented in basically every city ever studied.
\nFor modelling, this means two things. First, aggregate metrics like RMSE will be\ndominated by how well the model predicts the high-count cells. Second,\npredicting "zero" for a sparse cell is almost always correct but completely\nuninformative. We'll need to think carefully about what "accuracy" actually\nmeans when we get to evaluation.
\nThe EDA tells us a few things that should directly shape how we build and\nevaluate the models:
\nThe seasonal signal is strong and consistent. A model that can't capture monthly\nseasonality is worse than useless. It's worse than a calendar.
\nSpatial structure is real and persistent. Hotspots don't move much. A model that\nlearns static spatial patterns will get a lot of the way there, even without\nunderstanding temporal dynamics.
\nWe already know the CBD will have lots of theft next month. That's not what\nwe're trying to predict. The real value is in the margins: the cells that go\nfrom quiet to active, or the months where a normally stable area spikes. That's\nwhere deep learning might actually add something over simple baselines.
\nSpeaking of which, we need baselines. Otherwise we won't know if ConvLSTM is\nactually clever or just expensive. That's next.
\n","date_published":"Thu, 02 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/built-an-sms-gateway-with-a-20-dollar-android-phone/","url":"https://jonnonz.com/posts/built-an-sms-gateway-with-a-20-dollar-android-phone/","title":"How I Built an SMS Gateway with a $20 Android Phone","content_html":"Twilio charges around $0.05–0.06 per SMS round-trip. Doesn't sound like much\nuntil you're building an MVP that sends reminders, confirmations, and\nnotifications — suddenly you're looking at $50/month for a thousand messages.\nFor an app that's not making money yet, that's a dumb tax.
\nHere's what I did instead: grabbed a cheap Android phone, installed an\nopen-source app called\nSMS Gateway for Android, and\nturned it into a full SMS gateway with a REST API. My SMS costs dropped to\nwhatever my mobile plan charges — which on plenty of prepaid plans is zero.\nUnlimited texts.
\nThis post walks through exactly how to wire it into a Next.js app, from first\ninstall to receiving webhooks. The whole thing took an afternoon.
\nBy the end of this you'll have:
\nInstall SMS Gateway for Android from the\nGoogle Play Store\nor grab the APK from\nGitHub Releases
\nOpen the app and grant SMS permissions when prompted
\nYou'll see the main screen with toggles for Local Server and Cloud Server:
\n![\"SMS](\"https://jonnonz.com/img/posts/sms-gateway/screenshot.png\")
The app supports two modes — local and cloud. Both work well, and I'll cover\neach.
\nLocal mode runs an HTTP server directly on the phone. Your backend talks to it\nover your local network. No cloud dependency, no third-party servers — the\nsimplest setup.
\n![\"Local](\"https://jonnonz.com/img/posts/sms-gateway/local-server.png\")
Local server\nconfiguration
8080)192.168.1.50)Your phone is now running an HTTP server. Verify it:
\n# Health check\ncurl http://192.168.1.50:8080/health\n\n# Swagger docs\nopen http://192.168.1.50:8080/docs\ncurl -X POST http://192.168.1.50:8080/message \\\n -u "admin:yourpassword" \\\n -H "Content-Type: application/json" \\\n -d '{\n "textMessage": { "text": "Hello from my SMS gateway!" },\n "phoneNumbers": ["+15551234567"]\n }'\nThat's it. The phone sends the SMS from its own number, using your mobile plan's\nrates.
\nTo receive SMS messages as webhooks:
\ncurl -X POST http://192.168.1.50:8080/webhooks \\\n -u "admin:yourpassword" \\\n -H "Content-Type: application/json" \\\n -d '{\n "id": "my-webhook",\n "url": "http://192.168.1.100:4000/api/sms/webhook",\n "event": "sms:received"\n }'\nReplace 192.168.1.100 with your dev machine's local IP. Both devices need to\nbe on the same WiFi network.
Cloud mode is easier to set up and works from anywhere — no local network\nrequired. The phone connects to SMS Gateway's cloud relay (api.sms-gate.app),\nand your backend talks to the same cloud API.
![\"Cloud](\"https://jonnonz.com/img/posts/sms-gateway/cloud-server.png\")
Cloud server\nconfiguration
The cloud uses a hybrid push architecture: Firebase Cloud Messaging as the\nprimary channel, Server-Sent Events as fallback, and 15-minute polling as a last\nresort. It's well thought through.
\ncurl -X POST https://api.sms-gate.app/3rdparty/v1/messages \\\n -u "YOUR_USERNAME:YOUR_PASSWORD" \\\n -H "Content-Type: application/json" \\\n -d '{\n "textMessage": { "text": "Hello from the cloud!" },\n "phoneNumbers": ["+15551234567"]\n }'\nYour webhook URL must be HTTPS in cloud mode. For local development, use\nngrok:
\n# Start ngrok tunnel to your dev server\nngrok http 4000\n# Output: https://abc123.ngrok.app\n\n# Register the webhook\ncurl -X POST https://api.sms-gate.app/3rdparty/v1/webhooks \\\n -u "YOUR_USERNAME:YOUR_PASSWORD" \\\n -H "Content-Type: application/json" \\\n -d '{\n "url": "https://abc123.ngrok.app/api/sms/webhook",\n "event": "sms:received"\n }'\n# List webhooks\ncurl -u "YOUR_USERNAME:YOUR_PASSWORD" \\\n https://api.sms-gate.app/3rdparty/v1/webhooks\n\n# Delete a webhook\ncurl -X DELETE -u "YOUR_USERNAME:YOUR_PASSWORD" \\\n https://api.sms-gate.app/3rdparty/v1/webhooks/WEBHOOK_ID\nHere's how I integrated SMS Gateway into a Next.js app with a clean provider\nabstraction. The idea is simple — swap providers without touching business\nlogic.
\n// src/lib/sms/provider.ts\n\nexport interface InboundSms {\n from: string;\n body: string;\n receivedAt?: Date;\n}\n\nexport interface SmsProvider {\n send(to: string, body: string): Promise<string>;\n parseWebhook(req: Request): Promise<InboundSms | null>;\n webhookResponse(replyText?: string): Response;\n}\n\nexport async function getSmsProvider(): Promise<SmsProvider> {\n const provider = process.env.SMS_PROVIDER || "sms-gate";\n\n switch (provider) {\n case "sms-gate": {\n const { SmsGateProvider } = await import("./sms-gate");\n return new SmsGateProvider();\n }\n case "console": {\n const { ConsoleProvider } = await import("./console");\n return new ConsoleProvider();\n }\n default:\n throw new Error(`Unknown SMS provider: ${provider}`);\n }\n}\nThe provider handles both local and cloud API differences:
\n// src/lib/sms/sms-gate.ts\n\nimport type { InboundSms, SmsProvider } from "./provider";\n\nconst SMSGATE_URL = process.env.SMSGATE_URL || "http://localhost:8080";\nconst SMSGATE_USER = process.env.SMSGATE_USER || "";\nconst SMSGATE_PASSWORD = process.env.SMSGATE_PASSWORD || "";\n\nexport class SmsGateProvider implements SmsProvider {\n private headers(): Record<string, string> {\n const auth = Buffer.from(\n `${SMSGATE_USER}:${SMSGATE_PASSWORD}`,\n ).toString("base64");\n return {\n "Content-Type": "application/json",\n Authorization: `Basic ${auth}`,\n };\n }\n\n async send(to: string, body: string): Promise<string> {\n const isCloud = SMSGATE_URL.includes("api.sms-gate.app");\n const endpoint = isCloud\n ? `${SMSGATE_URL}/3rdparty/v1/messages`\n : `${SMSGATE_URL}/api/3rdparty/v1/message`;\n const payload = isCloud\n ? { textMessage: { text: body }, phoneNumbers: [to] }\n : { phoneNumbers: [to], message: body };\n\n const res = await fetch(endpoint, {\n method: "POST",\n headers: this.headers(),\n body: JSON.stringify(payload),\n });\n\n if (!res.ok) {\n const err = await res.text();\n throw new Error(`SMS Gate send failed: ${res.status} ${err}`);\n }\n\n const data = await res.json();\n return data.id || "sent";\n }\n\n async parseWebhook(req: Request): Promise<InboundSms | null> {\n try {\n const body = await req.json();\n\n if (body.event !== "sms:received" || !body.payload) {\n return null;\n }\n\n const { phoneNumber, message, receivedAt } = body.payload;\n if (!phoneNumber || !message) return null;\n\n return {\n from: phoneNumber,\n body: message,\n receivedAt: receivedAt ? new Date(receivedAt) : new Date(),\n };\n } catch {\n return null;\n }\n }\n\n webhookResponse(): Response {\n return new Response(JSON.stringify({ ok: true }), {\n headers: { "Content-Type": "application/json" },\n });\n }\n}\nA basic webhook handler that receives inbound SMS and replies:
\n// src/app/api/sms/webhook/route.ts\n\nimport { NextRequest } from "next/server";\nimport { getSmsProvider } from "@/lib/sms/provider";\n\nexport async function POST(req: NextRequest) {\n const provider = await getSmsProvider();\n const sms = await provider.parseWebhook(req);\n\n if (!sms) {\n return new Response("Bad request", { status: 400 });\n }\n\n const { from, body } = sms;\n\n // Look up the sender — replace with your own user lookup\n const user = await findUserByPhone(from);\n\n if (!user) {\n await provider.send(from, "Hey! Text us back once you've signed up.");\n return provider.webhookResponse();\n }\n\n // Known user — do whatever your app needs\n console.log(`[SMS from ${from}]: ${body}`);\n await provider.send(from, "Got it — we're on it!");\n return provider.webhookResponse();\n}\nFor local development without a phone:
\n// src/lib/sms/console.ts\n\nimport type { InboundSms, SmsProvider } from "./provider";\n\nexport class ConsoleProvider implements SmsProvider {\n async send(to: string, body: string): Promise<string> {\n console.log(`[SMS -> ${to}] ${body}`);\n return `console-${Date.now()}`;\n }\n\n async parseWebhook(req: Request): Promise<InboundSms | null> {\n const data = await req.json();\n return {\n from: data.from || "+15550000000",\n body: data.body || "",\n receivedAt: new Date(),\n };\n }\n\n webhookResponse(): Response {\n return new Response(JSON.stringify({ ok: true }), {\n headers: { "Content-Type": "application/json" },\n });\n }\n}\n# .env\n\n# Provider: "sms-gate" | "console"\nSMS_PROVIDER=sms-gate\n\n# Local mode\nSMSGATE_URL=http://192.168.1.50:8080\nSMSGATE_USER=admin\nSMSGATE_PASSWORD=yourpassword\n\n# Cloud mode\n# SMSGATE_URL=https://api.sms-gate.app\n# SMSGATE_USER=auto-generated-username\n# SMSGATE_PASSWORD=auto-generated-password\nWhen someone texts your Android phone, SMS Gateway sends a POST to your webhook\nURL:
\n{\n "id": "Ey6ECgOkVVFjz3CL48B8C",\n "webhookId": "LreFUt-Z3sSq0JufY9uWB",\n "deviceId": "your-device-id",\n "event": "sms:received",\n "payload": {\n "messageId": "abc123",\n "message": "Hello!",\n "sender": "+15551234567",\n "recipient": "+15559876543",\n "simNumber": 1,\n "receivedAt": "2026-04-01T12:41:59.000+00:00"\n }\n}\n| Event | \nDescription | \n
|---|---|
sms:received | \nInbound SMS received | \n
sms:sent | \nOutbound SMS sent | \n
sms:delivered | \nOutbound SMS confirmed delivered | \n
sms:failed | \nOutbound SMS failed | \n
system:ping | \nHeartbeat — device still alive | \n
SMS Gateway signs webhook payloads with HMAC-SHA256. Two headers are included:
\nX-Signature — hex-encoded HMAC-SHA256 signatureX-Timestamp — Unix timestamp used in signingimport crypto from "crypto";\n\nfunction verifyWebhook(\n signingKey: string,\n payload: string,\n timestamp: string,\n signature: string,\n): boolean {\n const expected = crypto\n .createHmac("sha256", signingKey)\n .update(payload + timestamp)\n .digest("hex");\n return crypto.timingSafeEqual(\n Buffer.from(expected, "hex"),\n Buffer.from(signature, "hex"),\n );\n}\nIf your server doesn't respond 2xx within 30 seconds, SMS Gateway retries with\nexponential backoff — starting at 10 seconds, doubling each time, up to 14\nattempts (~2 days). Solid default behaviour, you don't need to configure\nanything.
\nnpm run dev\n# Next.js running at http://localhost:4000\nngrok http 4000\n# https://abc123.ngrok.app -> http://localhost:4000\ncurl -X POST https://api.sms-gate.app/3rdparty/v1/webhooks \\\n -u "USERNAME:PASSWORD" \\\n -H "Content-Type: application/json" \\\n -d '{\n "url": "https://abc123.ngrok.app/api/sms/webhook",\n "event": "sms:received"\n }'\nText your Android phone from another phone. You should see:
\nThat moment when the reply lands on your phone — genuinely satisfying.
\n# Simulate an inbound SMS with the console provider\nSMS_PROVIDER=console npm run dev\n\ncurl -X POST http://localhost:4000/api/sms/webhook \\\n -H "Content-Type: application/json" \\\n -d '{"from": "+15551234567", "body": "Hello"}'\nsystem:ping webhook to alert if the device goes\noffline.| \n | Local | \nCloud | \n
|---|---|---|
| Latency | \nLower (direct) | \nSlightly higher (relay) | \n
| Network | \nSame network required | \nWorks from anywhere | \n
| Privacy | \nMessages never leave your network | \nMessages transit through SMS Gateway's servers | \n
| Reliability | \nDepends on your network | \nAdds FCM/SSE redundancy | \n
| Cost | \nFree | \nFree (community tier) | \n
I use cloud mode in production because my server's hosted on Railway and\ncan't reach the phone's local network. For development on the same WiFi, local\nmode is simpler and faster.
\n| Provider | \nSMS Cost | \nMonthly (1,000 msgs) | \n
|---|---|---|
| Twilio | \n~$0.05/msg | \n~$50 | \n
| SMS Gateway + Prepaid SIM | \n$0/msg (unlimited plan) | \n~$8 (plan cost) | \n
That's an 80%+ saving, and it scales linearly — 10,000 messages a month is\nstill just your plan cost.
\nIt's worth knowing this is a whole category now. httpSMS\nand textbee do similar things. I went with\nSMS Gateway for Android\nbecause the local mode is properly useful for development, the\ndocumentation is solid, and it's actively\nmaintained — v1.56.0 dropped in March 2026.
\nFor an MVP, the maths is obvious. A $20 phone and an $8/month plan gets you a\nprogrammable SMS gateway that you fully control. No per-message fees, no carrier\ncontracts, no vendor lock-in. If you outgrow it, swap the provider interface to\nTwilio and you're done — that's why the abstraction exists.
\nLinks:
\n\n","date_published":"Thu, 02 Apr 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/wrangling-a-million-crime-records/","url":"https://jonnonz.com/posts/wrangling-a-million-crime-records/","title":"Wrangling a Million Crime Records","content_html":"The very first thing NZ Police's crime dataset teaches you is that government\ndata is never straightforward.
\nYou download the CSV from\npolicedata.nz,\nexpecting to do a quick pd.read_csv() and start exploring. Instead you get a\n503MB file encoded in UTF-16 Little Endian with tab delimiters. Not a regular\nCSV. Not even close. This is a legacy format from old Excel exports and most\ntools just silently corrupt it if you try to read it as UTF-8.
df = pd.read_csv("data.csv", encoding="utf-16-le", sep="\\t")\nThat one line took longer to figure out than I'd like to admit.
\nOnce you get past the encoding, there's a lot to work with. 1,154,102 rows\ncovering every reported victimisation in New Zealand from February 2022 through\nJanuary 2026. Each row tells you the crime type (ANZSOC Division), where it\nhappened (down to meshblock level), when it happened (month, day of week, hour\nof day), and sometimes what weapon was involved.
\nThere are 20 columns, but five of them are useless: three are duplicates of\n"Year Month" and two are constants that add zero information. Every area name\nand territorial authority has a trailing period stuck on the end: "Auckland.",\n"Woodglen.", "Christchurch City.". A quirk of the export that'll break any\ngeographic join if you don't strip them.
\nAnd meshblock IDs? Some are 6 digits, some are 7. Stats NZ boundary files use\n7-digit codes consistently, so shorter ones need zero-padding. The kind of thing\nthat's invisible until your join silently drops 19% of your records and you\nspend an afternoon figuring out why.
\nThat bit actually made me stop and think. 32.2% of records have the hour of day\nrecorded as 99 (unknown). Another 23.2% have the day of week as "UNKNOWN".
\nAt first this looks like a data quality problem. But it's not. It's telling you\nsomething about the nature of the crime. If someone breaks into your house while\nyou're at work, you come home to find your stuff gone. Was it 9am or 2pm? You've\ngot no idea, and neither do the police.
\nProperty crimes (theft, burglary) make up the bulk of these unknowns. Assault,\nby contrast, almost always has a precise time because there's a victim present\nwhen it happens. The absence of data is itself a signal about what kind of crime\nyou're looking at.
\n78.6% of location type values are "." (effectively missing). That column is\nsparsely populated but still useful for the roughly one in five records that\nhave it.
\nWe built a modular pipeline where each cleaning step is its own function.\nNothing fancy, just practical:
\ndef ingest() -> pd.DataFrame:\n df = load_raw_csv(RAW_CSV) # UTF-16 LE, tab-delimited\n df = drop_redundant_columns(df) # Remove 5 useless columns\n df = rename_columns(df) # snake_case everything\n df = parse_dates(df) # "July 2022" → datetime\n df = clean_strings(df) # Strip trailing periods\n df = clean_meshblocks(df) # Zero-pad to 7 digits\n df = encode_unknowns(df) # 99 → NaN, "UNKNOWN" → NaN\n df = map_crime_types(df) # ANZSOC Division → short enum\n return df\nEach function does one thing. If something breaks, you know exactly where. If\nsomeone wants to understand the pipeline, they can read it top to bottom in\nabout thirty seconds. I've been bitten enough times by monolithic data scripts\nthat I'm allergic to them now.
\nThe crime type mapping turns six ANZSOC Division values into short enums:
\n| Crime Type | \nCount | \nShare | \n
|---|---|---|
| Theft | \n761,977 | \n66.0% | \n
| Burglary | \n247,034 | \n21.4% | \n
| Assault | \n115,383 | \n10.0% | \n
| Robbery | \n14,860 | \n1.3% | \n
| Sexual | \n13,943 | \n1.2% | \n
| Harm | \n905 | \n0.1% | \n
That 66% theft number is going to haunt us when we get to model training. Any\nloss function you throw at this data will overwhelmingly optimise for predicting\ntheft, because that's two-thirds of everything. The class imbalance is real and\nit matters.
\nThe cleaned output goes to\nApache Parquet with snappy\ncompression. The result?
\nThat's not a typo. Parquet's columnar storage is dramatically more efficient\nthan row-oriented CSV, especially when you've got columns full of repeated\nvalues like crime types and territorial authorities. The file loads in under a\nsecond compared to 3+ seconds for the CSV. When you're iterating on analysis and\nloading this data hundreds of times, that adds up fast.
\nThe 21 output columns include the original 16 we kept plus five derived ones: a\nproper datetime, year, month, day-of-week as an integer, and the short crime\ntype enum.
\nBefore calling the data clean, we verify everything that matters:
\nYou want these checks automated and running every time you regenerate the data.\nFuture you will thank past you when something upstream changes and a check\ncatches it.
\nWe've got clean, compressed crime data, but the records only have meshblock IDs\nand area unit names. No coordinates. No shapes on a map. In the next post, we'll\ndownload Stats NZ geographic boundary files and join them to our crime records,\ngiving every victimisation a place in physical space.
\n","date_published":"Thu, 26 Mar 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/crime-as-video/","url":"https://jonnonz.com/posts/crime-as-video/","title":"Crime as Video","content_html":"This is where the project gets properly fun.
\nWe've got 1.15 million clean crime records. Every one of them has coordinates:\neither precise meshblock centroids or area unit fallbacks from Part 2. But a bag\nof lat/lon points isn't what a neural network wants. ConvLSTM and ST-ResNet are\nfundamentally image-processing architectures. They expect regular 2D grids, rows\nand columns, like pixels in a photograph.
\nSo our job now is to convert the messy reality of crime locations into clean,\nregular "crime images" that a convolutional network can actually consume. And\nonce you see it framed that way, crime prediction becomes video prediction. Each\nmonth is a frame. Each grid cell is a pixel. The brightness is the crime count.
\nThis is the single most consequential decision in the entire data pipeline. Get\nthe grid resolution wrong and everything downstream suffers.
\nToo fine (say 100m cells) and the vast majority of cells are empty in any given\nmonth. The model sees an ocean of zeros with occasional spikes, which is\nincredibly hard to learn from. Too coarse (say 2km) and you've blurred away the\nspatial patterns you're trying to detect. "Auckland CBD" and "Ponsonby" become\nthe same cell, which is useless.
\nWe computed Auckland's urban crime extent from the meshblock centroids (5th to\n95th percentile to exclude outliers like Great Barrier Island):
\n| Metric | \nValue | \n
|---|---|
| Urban extent | \n27.7 km × 36.9 km | \n
| Grid resolution | \n500m × 500m | \n
| Grid dimensions | \n77 rows × 59 columns | \n
| Total cells | \n4,543 | \n
At 500m, each cell covers roughly a few city blocks. That's fine enough to\ndistinguish a commercial strip from a residential street, but coarse enough that\nmost cells accumulate at least some crime over the 48-month period. It's a sweet\nspot, and it's consistent with what\nrecent crime forecasting research uses for\nsimilar models in US cities.
\nWorking in NZTM2000 (the coordinate system we set up in Part 2, where units are\nmetres) makes the next bit easy. Assigning a crime to a grid cell is just floor\ndivision:
\ngrid_j = floor((x - xmin) / 500) # column index\ngrid_i = floor((y - ymin) / 500) # row index\nNo spatial joins, no polygon intersection, no geopandas overhead. Just\narithmetic. It processes all 400k Auckland records in under a second.
\nFor the ~22% of Auckland records that didn't get meshblock coordinates in Part\n2, we fall back to area unit centroids converted to NZTM2000. Those records land\nat the centre of their suburb rather than their exact location. Less precise,\nbut dropping them entirely would be worse.
\nThe result: 354,387 of 412,669 Auckland records (86.2%) fall within the grid.\nThe remaining 14% are in Auckland's outer fringes (Great Barrier Island, rural\nRodney, the edges of the Waitakere Ranges) beyond our urban bounding box. That's\nfine. We're modelling urban crime patterns, not rural ones.
\nWith every crime assigned to a cell, we aggregate by grid position, month, and\ncrime type:
\n(grid_i, grid_j, month, crime_type) → sum(victimisations)\nThis gives us a 4D tensor:
\n| Dimension | \nSize | \nMeaning | \n
|---|---|---|
| T (time) | \n48 | \nMonths: Feb 2022 – Jan 2026 | \n
| H (height) | \n77 | \nGrid rows (south → north) | \n
| W (width) | \n59 | \nGrid columns (west → east) | \n
| C (channels) | \n6 | \nCrime types: theft, burglary, assault, robbery, sexual, harm | \n
Think of it as a 48-frame video with 6 colour channels. A regular video has 3\nchannels: red, green, blue. Ours has 6: theft, burglary, assault, robbery,\nsexual offences, harm. Each pixel's brightness in a given channel tells you how\nmany of that crime type happened in that 500m cell during that month.
\nI genuinely love this framing. It takes a complicated spatial-temporal\nprediction problem and maps it onto something that decades of computer vision\nresearch already knows how to handle.
\nThe tensor is overwhelmingly empty. 91.7% of all cells are zero.
\nThis makes complete sense if you think about it. Most 500m squares in Auckland\ndon't have a single reported crime in any given month. Crime clusters:\ncommercial corridors, transport hubs, specific residential pockets. The non-zero\n8.3% is where all the signal lives.
\nThe sparsity does create a training challenge though. If the model just\npredicted zero everywhere, it'd be right 91.7% of the time. Useless, but\ntechnically accurate. That's why we'll use log1p normalisation during\ntraining. It compresses the range from [0, 50+] to [0, ~4], giving the model a\nmore balanced gradient to learn from. And it's why the loss function needs to\ncare more about the non-zero cells than the empty ones.
The upside of all those zeros is storage. The\ncompressed numpy format\nhandles sparse data beautifully. The full 4D tensor saves to just 0.2 MB.\nCompare that to the 21.9 MB Parquet from Part 2.
\nWe split the 48 months temporally. No shuffling, no random sampling:
\n| Set | \nMonths | \nRange | \n
|---|---|---|
| Train | \n36 | \nFeb 2022 – Jan 2025 | \n
| Validation | \n6 | \nFeb 2025 – Jul 2025 | \n
| Test | \n6 | \nAug 2025 – Jan 2026 | \n
The model trains on three years, tunes on six months, and gets evaluated on the\nmost recent six months it's never seen. There's no spatial leakage either. We\ndon't hold out specific grid cells. The model has to predict all locations for\nfuture months simultaneously.
\nThis is the only honest way to evaluate a time-series model. If you randomly\nshuffle months into train and test, the model can memorise seasonal patterns and\nlook brilliant without actually learning anything useful about temporal\ndynamics.
\nEven at this aggregate level, clear patterns jump out.
\nFebruary tends to be the quietest month (~7–8k victimisations across Auckland),\nwhile October through January (spring and early summer) consistently peaks at\n8.5–9.5k. 2023 was the peak year across the board, with a gradual decline\nthrough 2024 and into 2025.
\nTheft accounts for 72% of the tensor values (283k victimisations), burglary 17%\n(68k), and assault 9% (34k). That theft dominance from Part 1, the 66% figure,\ngets even more pronounced when you focus on Auckland, because theft clusters\nharder in urban areas than other crime types do.
\nThe tensor is built. The model input is ready. But before throwing deep learning\nat anything, we need to properly understand what patterns actually exist in this\ndata: when does crime peak, where does it cluster, and how do different crime\ntypes behave differently. Next post: exploratory data analysis.
\n","date_published":"Thu, 26 Mar 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/giving-crime-a-place-on-the-map/","url":"https://jonnonz.com/posts/giving-crime-a-place-on-the-map/","title":"Giving Crime a Place on the Map","content_html":"A crime record that says "Woodglen, meshblock 0284305" is useless for spatial\nmodelling. It's a name and a number. You can't plot it, you can't measure\ndistances from it, and you definitely can't feed it to a neural network that\nthinks in grid cells.
\nTo do anything spatial, every record needs actual coordinates: latitude,\nlongitude, or ideally metres on a proper projection. That means downloading\nStats NZ's geographic boundary files and joining them to our crime data.
\nNew Zealand has a neat nested system of geographic units maintained by\nStats NZ:
\ngraph TD\n A["Region (16)"] --> B["Territorial Authority (67)"]\n B --> C["Area Unit / SA2 (~2,000)"]\n C --> D["Meshblock (~53,000)"]\nRegions are the big ones: Auckland, Canterbury, Wellington. Territorial\nauthorities are your cities and districts. Area units are roughly suburb-sized.\nAnd meshblocks are the smallest unit, about 100 people each, roughly a city\nblock. Our crime data uses area units and meshblocks, so those are the layers we\nneed.
\nThere's a gotcha here. Stats NZ replaced "Area Units" with "Statistical Area 2"\n(SA2) in 2018 as part of a geographic classification overhaul. But the NZ Police\ncrime data still uses the old area unit names. So we need the 2017 vintage\nboundary files, not the current ones. Use the wrong vintage and your join\nsilently fails on hundreds of area units. Ask me how I know.
\nWe downloaded three layers from\nStats NZ DataFinder via their WFS API:
\n| Layer | \nFeatures | \nSize | \n
|---|---|---|
| Area Unit 2017 (generalised) | \n2,004 | \n88 MB | \n
| Meshblock 2018 (generalised) | \n53,589 | \n213 MB | \n
| Territorial Authority 2023 | \n68 | \n34 MB | \n
All three come in EPSG:2193, which is\nNZTM2000,\nNew Zealand's official projected coordinate system. The units are metres, not\ndegrees. This matters a lot later when we need to build a "500m grid". You want\nthat to be 500 actual metres, not some approximation based on latitude.
\nWe use generalised (simplified) versions rather than high-definition. The\nfull-resolution meshblock layer is over a gigabyte. For centroid calculations\nand spatial joins, the generalised versions are more than accurate enough.
\nJoining crime records to area unit boundaries by name was almost perfect.\n1,146,721 of 1,154,102 records matched, 99.4%.
\nOnly two area unit codes failed:
\n999999: the official "unspecified" catch-all (7,331 records)-29: a straight-up data entry error (50 records)That's a genuinely excellent result. The unmatched records aren't a bug in our\npipeline. They're unlocatable crimes that the police couldn't assign to a\nspecific area. Nothing we can do about those, and nothing we should try to do.
\nThe meshblock join came in lower at 81.2%, with 937,604 records matched out of\n1,154,102.
\nThis is expected and it's fine. Here's why: NZ meshblock boundaries get revised\nwith every census. We're using 2018 boundaries, but our crime data runs through\nJanuary 2026. Any crime from 2023 onwards might reference a 2023-vintage\nmeshblock code that simply doesn't exist in the 2018 file. Some meshblocks get\nsplit, some get merged, some get renumbered entirely.
\n81.2% still gives us fine-grained coordinates for the vast majority of records.\nFor the ~19% that miss, we fall back to the area unit centroid. It's less\nprecise (suburb-level instead of block-level) but better than dropping the\nrecords entirely.
\nThis is one of those things that seems like a minor detail but will bite you\nhard if you get it wrong. We use two coordinate reference systems throughout the\nproject:
\nNZTM2000 (EPSG:2193) for all spatial analysis. The units are metres, which\nmakes grid construction trivial: a 500m cell is literally 500 units on each\naxis. Distance calculations are straightforward. No need to worry about the fact\nthat a degree of longitude means different things at different latitudes.
\nWGS84 (EPSG:4326) for the frontend dashboard only. deck.gl and MapLibre\nexpect coordinates in degrees (latitude/longitude), which is the standard for\nweb mapping.
\nThe rule is simple: do everything in NZTM2000, convert to WGS84 at the very end\nwhen exporting for the dashboard. Mixing coordinate systems mid-pipeline is a\nrecipe for bugs that are incredibly annoying to track down.
\nEach crime record now has up to 8 new geographic columns: area unit centroids,\nmeshblock centroids, and areas in both coordinate systems. The enriched dataset\nsaves as crimes_with_geo.parquet at 21.9 MB with 29 columns.
Quick sanity check: Auckland's mean crime centroid lands at lat -36.90, lon\n174.78. Right in the middle of the urban area. If that number had come back as\nsomewhere in the Waikato, we'd know something went wrong.
\nEvery crime record now has a place in physical space. But individual points\naren't what the neural network needs. It needs a regular grid. In the next post,\nwe'll overlay a 500m × 500m grid on Auckland, count crimes per cell per month,\nand build the 4D tensor that turns crime prediction into a video prediction\nproblem.
\n","date_published":"Thu, 26 Mar 2026 00:00:00 GMT"},{"id":"https://jonnonz.com/posts/predicting-crime-in-aotearoa/","url":"https://jonnonz.com/posts/predicting-crime-in-aotearoa/","title":"Predicting Crime in Aotearoa","content_html":"NZ Police publish every recorded victimisation in the country, over a million\nrecords, and most people have no idea.
\nI stumbled across\npolicedata.nz\na while back and was surprised by how much is there. Every reported theft,\nassault, burglary, robbery, broken down by location, time of day, day of week,\nand month. All the way down to meshblock level, which is roughly a city block.\nUpdated monthly.\nCreative Commons licensed.\nYou can just... use it.
\nSo naturally I started wondering: what happens if you point deep learning at\nthis?
\nThe dataset I pulled covers February 2022 through January 2026. Four years,\n1,154,102 records across the whole country. The breakdown is roughly what you'd\nexpect: theft dominates at 66%, followed by burglary at 21% and assault at 10%.\nThe remaining sliver covers robbery, sexual offences, and harm/endangerment.
\nWhat makes it interesting for modelling is the spatial granularity. Each record\nmaps to one of 42,778 meshblocks (tiny geographic units defined by Stats NZ).\nThat's detailed enough to see patterns at a neighbourhood level, not just\n"Auckland has more crime than Tauranga" (which, yeah, obviously).
\nAuckland alone accounts for about 36% of all recorded crime. Then there's a long\ntail: Wellington, Christchurch, Hamilton, and then it drops off fast. NZ's urban\ngeography is weird like that. One mega-city and a bunch of mid-size towns.
\nThe core question is pretty simple. Given the crime patterns of the last few\nmonths, can we predict what the next month looks like?
\nThis isn't Minority Report. Nobody's getting arrested for crimes they haven't\ncommitted. It's pattern recognition on publicly available statistics, the same\nkind of modelling people do with weather data or traffic flows.
\nThe neat trick is how you frame it. If you overlay a grid on a city (say 500m by\n500m cells) and count crimes per cell per month, you get something that looks a\nlot like a video. Each month is a frame. Each cell is a pixel. The brightness is\nthe crime count.
\nPredicting next month's crime becomes a video prediction problem. And there are\nsome really cool deep learning architectures built exactly for that.
\nThe two models I'm building are ConvLSTM and ST-ResNet. Don't worry if those\nsound like gibberish. The short version: they're neural networks designed to\nlearn patterns that are both spatial (where things cluster) and temporal (how\nthose clusters change over time).
\nConvLSTM is the primary model. A standard LSTM network is great at learning\nsequences. It's the architecture behind a lot of language and time-series\nmodels. ConvLSTM swaps out the matrix multiplications for convolutions, which\nmeans it can process grid-structured data. Feed it the last six months of crime\ngrids and it learns both the shape of hotspots and how they evolve.\nRecent research has shown these work well\nfor crime forecasting across multiple US cities.
\nST-ResNet takes a different angle. Instead of one sequential view, it\ncaptures three temporal perspectives: what happened recently, what happened at\nthe same time last year, and what's the long-term trend. Each gets its own\nbranch of residual convolutional networks, and a learned fusion layer combines\nthem. The\noriginal paper was for\ncrowd flow prediction in Beijing, but the architecture\ntranslates well to crime data.
\nAlmost all published crime prediction research uses US data. Chicago, Los\nAngeles, New York. A\nsystematic review of spatial crime forecasting\nmakes this pretty clear. The models are well-studied, but they're trained on\nAmerican cities with American urban patterns.
\nNew Zealand doesn't look like that. Our cities are smaller, more spread out, and\nthe distribution is completely different. Auckland dominates in a way that no\nsingle US city does relative to the rest of the country. The spatial patterns\nhere are their own thing, and I couldn't find anyone who'd applied these deep\nlearning approaches to NZ data.
\nThat's what got me keen. Not because I think I'll beat the published benchmarks.\nThose researchers have GPUs and PhD students; I have a Ryzen 5 desktop with no\ngraphics card. But applying known techniques to new geography is useful work,\nand nobody else seems to have done it.
\nAll of this runs on my desktop, an AMD Ryzen 5 5600GT with 12 threads and 30GB\nof RAM. No GPU at all. That sounds limiting, but the Auckland 500m grid works\nout to about 60 by 80 cells. The ConvLSTM model ends up around 5 million\nparameters, which trains in under an hour on CPU. You don't always need a beefy\nrig.
\nIt does mean being smart about model sizing and not going crazy with\nhyperparameter searches. But for a hobby project, it's more than enough.
\nThis is the first post in a ten-part series covering the whole project end to\nend.
\nPart 1: Data Acquisition and Exploration. We start with a 503MB CSV file\nfrom NZ Police that's UTF-16 encoded (because of course it is), has trailing\nperiods on area names, and 32% of records with unknown hour-of-day. We'll\nwrangle it into a clean, typed Parquet file and get our first look at what's\nactually in there.
\nPart 2: Geographic Data Pipeline. Crime records come with meshblock IDs, but\nno coordinates. We'll join them to Stats NZ geographic boundary files using\ngeopandas, giving every record a place on the map.
\nPart 3: Spatiotemporal Grid Construction. This is where it gets fun. We\noverlay a 500m by 500m grid on Auckland, count crimes per cell per month, and\nbuild the 4D tensors that feed the neural networks. Crime prediction becomes\nvideo prediction.
\nPart 4: Exploratory Data Analysis. Before throwing deep learning at\nanything, we need to understand what patterns actually exist. When does crime\npeak? Where does it cluster? How do different crime types behave differently?
\nPart 5: Baseline Models. Simple benchmarks (historical averages, naive\npersistence) so we know whether the deep learning is actually adding value or\njust being fancy for the sake of it.
\nPart 6: ConvLSTM Architecture. Building and training the primary model.\nThree ConvLSTM layers, six-month lookback window, learning spatial hotspots and\ntemporal dynamics simultaneously.
\nPart 7: ST-ResNet Architecture. The three-branch alternative that captures\ncloseness, periodicity, and long-term trend separately, then fuses them with\nlearned weights.
\nPart 8: Model Evaluation and Comparison. Which model wins? By how much? And\nmore importantly, where do they fail?
\nPart 9: Building the Dashboard. A 3D interactive map built with deck.gl\nwhere you can watch crime patterns evolve over time. Dark theme, extruded\ncolumns, time-lapse playback.
\nPart 10: Deployment and Reflections. Shipping to Vercel, what worked, what\ndidn't, and what I'd do differently next time.
\nEvery post will include code and real results. The whole codebase will be open\nsource. And I'll be upfront about the stuff that didn't work. Trust me, there's\nplenty of it.
\nThis is a hobby project. It's not a policing tool, it's not a product, and it's\ndefinitely not claiming to solve crime. It's just me being curious about what's\nsitting in a publicly available dataset and seeing how far you can push it with\nsome Python and a bit of patience.
\n","date_published":"Tue, 24 Mar 2026 00:00:00 GMT"}]}