BlogAI Agents

Nothing was broken

An agent changed four files in production over a fault that lived in Shopify's index

Nicolás Biondi

A wall opened up in the middle with two bundles of glowing cables, one cyan and one amber, shining at different intensities.

Searching the full name of a brand we keep in the display case returned "no results". The cameras were right there, in the window and in the catalog, and the store's search engine disagreed. I reloaded the same URL and products showed up, reloaded again and the count changed once more, with nobody touching a line of code.

The same URL returned a different number every time

I saw it on my phone, testing the store the way someone who walks in to look at cameras sees it. A brand with dozens of published products came back empty, and a person who searches by brand already knows what they want: they are one click from buying or from leaving for another store.

I picked the wrong suspect right away. I blamed the theme, the code that builds the results page, because on top of it we run our own layer that spots weak searches and widens them with similar terms. The culprit looked like one of ours.

The missing parameter

The agent went through that layer and found a call to Shopify's search API without the prefix parameter, the one that says the last word is incomplete. The theory was reasonable and it matched the symptom. It edited four theme files, pushed them to the live version and tested again. The query returned products. If I had walked off to help a customer at the counter at that point, I would have sworn it was fixed.

Meme Surprised Pikachu: Tested once after the deploy and it failed again Tested once after the deploy and it failed again

Search was already intermittent before the change. With a system like that, one test after a deploy can't tell "I fixed it" apart from "it changed on its own again". So the agent ran the same URL five times in a row, touching nothing between runs. The count kept moving, so the deploy had stabilized nothing.

Two Shopify indexes disagreeing in the same second

Shopify answers through two paths: the predictive index, which feeds the suggestions while you type, and the full results page. The agent queried both at the same instant, with the same word. The suggestions returned 10 products and the search page returned 0.

There is one theme for both paths, and no misplaced parameter in our code makes two of the platform's indexes contradict each other in the same second. We had acted like the electrician who opens the wall with great care because the lights are flickering, when the voltage drop covers the whole street.

Yes

No

No

Yes

Valid search gives 0

Same URL, 5 runs

Does the count change?

Compare suggestions and page

Review theme code

Do the indexes match?

Wait for the index

The new rule: measure five times and compare the two indexes before touching the theme.

Reverting after 18 minutes

The agent reverted its four files 18 minutes after deploying them. The checksums, the fingerprint of each file, confirmed that the live theme matched the repository again, so we didn't leave a wrong fix half applied in the store.

Hours later, without a single code change, the same brand query returned 82 results. The index recovered on its own and nobody told us.

We can't look at that index from the outside, and we don't know how long a streak like that lasts. We can measure it and write down what we see while it settles, which isn't elegant but costs nothing.

The search system document now opens with a rule: if a valid search returns 0, we measure five times and compare the two indexes before touching a line of code. Reverting is now as routine as deploying. And when the search engine tells me a brand doesn't exist, I believe it less than the display case.

postmortem search data-quality metrics rollback