How accurate is Store Diagnosis? We tested it on a store whose answers we know.

The demo store

The store is a Shopify development store we built to demonstrate and test Store Diagnosis: diagnosis-d9qyllsk.myshopify.com. It sells clothing, bags and homeware under three house labels and is sized like a brand turning over about a million dollars a year. Two scripts in our repository create it, and both are deterministic.

• Twelve scenario products, each shaped by hand to produce one problem: a tote that stopped selling nine months ago, a limited scarf that never sold, an enamel mug whose margin cannot carry the discount it keeps getting, a Black Friday code at 50% off, sticker packs sold in tiny orders, and a few refunds. 111 orders.
• Seventy-two products at scale, 342 variants, and 6,493 orders over 365 days drawn from a random generator with a fixed seed (mulberry32). Each product has a profile that decides how it sells and how much stock it holds today: best-sellers with two to three weeks of cover, steady sellers, slow lines holding most of a year of stock, fading lines, winter and summer ranges, discontinued lines that stopped selling more than 200 days ago with the stock still on the shelf, and two new launches. A store-wide season, a Black Friday peak, discount codes, free shipping over $100, and 340 refunds weighted towards clothing complete it.

How the answers are known

A real store cannot tell you which of its products are really dead stock; that is the question. A generated one can. Our test harness rebuilds the store exactly as the two scripts planned it, by calling their own planning functions: the catalogue, each variant's cost and stock today, every order with its lines, discount and shipping, and every refund. No engine code runs in this step.

Because the orders are drawn from known distributions, the harness also knows the rate each variant truly sells at, which a real merchant never sees: only a noisy history. That true rate is what the labels use. The orders are dated from the moment each script ran, and those moments were read back from the synced store, where every one of the 1,724 scale orders synced so far sits at its planned time to the second.

The engines then run on the rebuilt store through the same snapshot the app builds from its database, with the configuration the app uses. We score four engines: dead stock, stock risk, profit leaks and profit by product. The discount margin engine is left out: it simulates discounts a merchant types in, so a store has nothing for it to find.

The labelling rules

Each case is labelled from what the generator meant, before any engine runs and without reading engine code. Where the generator's intent does not settle a case, it is marked not labelled and left out of the score rather than guessed. Real problems too small to be worth a merchant's time are marked immaterial and also left out. Both groups are listed in the results file, with what the engine said about them.

Two points deserve a word. First, the dead and stale thresholds are the same 90 and 180 days the engine uses; for those cases the test checks whether the engine gets the facts right from raw orders, not whether the definition is right. Second, a label needs two readings of cover to agree, because the scale script set each variant's stock from its last 90 days of generated sales: where a variant sold above its true rate by chance, the script stocked it for that luck, and neither reading alone is the whole truth.

Results by engine

Each engine is scored twice. Shown is what a merchant sees: the findings list after the engine's listing floor ($20 of capital, $50 of revenue or margin) and its cap (50 dead stock findings, 40 stock risk, 25 profit leaks). Detected is the same engine with the floor and cap lifted, to show what it recognises.

TP: real problems found. FP: flagged, but not a problem. FN: real problems missed. TN: correctly left alone. Money covered: the share of the money at stake in the real problems that the found ones account for. Left out of the score: dead stock 123 not labelled; stock risk 131 not labelled and 1 immaterial; profit leaks 1 not labelled and 3 immaterial.

The money at stake

A finding is only useful if its amount is right. On the cases each engine found:

Dead stock capital is exact wherever the store has a unit cost; the 0.3% comes from variants without one, which the engine values at an assumed 50% margin and marks as estimated. Stock risk understates the revenue at risk by 28.5% in total on the six cases it found. It measures pace from recent sales: for three of the six, recent sales ran below the true rate and the amount is understated; for the other three they ran above it and it is overstated.

Profit by product

This engine raises no findings; it produces the figures behind the others, so we checked the figures.

Per product, the median contribution error is 1.2% and 69 of 81 products are within 5%. The largest errors, up to 28%, are products with a variant that has no unit cost. The ABC classes match on all 83 products that sold. $2,773 of the $36,859 in refunds lands on the wrong product, because refunds reach the engine as one amount per order and are spread over every line in it.

What it missed, and why

Dead stock: 4 missed even with the list limit lifted

• 4 variants that never sold. The engine skips products stocked less than 30 days ago, and a product that never sold has only its creation date to prove its age. Shopify does not let a script backdate a product, so in the demo store every product is a day old or less. In a real store these would be caught; here they cannot be.

Dead stock: what the list of 50 shows

The engine ranks findings by capital and lists the top 50. Of those 50, 44 are labelled cases and 6 are cases we left unlabelled: three winter sizes that sold two pieces each in last year's season, two products the scenario script presents as healthy sellers, and one borderline slow line. The 20 labelled cases the list does not show rank below the cap or sit under the $20 floor; they are in the results file.

That list used to be mostly winter stock. On 24 September, a winter sweater that last sold in March looks dead to a rule that only counts days since the last sale, and a list of them invites a markdown weeks before the season that sells them. The engine now holds a variant back as out of season, and says how many in a note on the report, when all of these hold:

• The order history reaches back a full year, so last year's coming months are known. With less, which is what a store gets without Shopify's read_all_orders access (60 days), the rule does nothing and every variant is judged as before.
• The product sold at least twice its yearly pace, and three times its pace of the last 90 days, in the 90 days that are now ahead, one year back. The season has to stand out from the year and from the quiet months before it.
• The product still sells something, in any size or colour, in the last 90 days. With one year of data, a product that sold all last winter and then stopped for good looks exactly like a winter range; the trickle a live seasonal range keeps selling is what tells them apart, so a product that sells nothing at all is still reported.
• The size or colour itself sold at least 3 units in last year's run of the season. A size that sold two pieces then is judged like any other stock.

On this store that holds back 64 variants: 63 of the winter ranges and one bifold wallet whose sales have faded since a strong holiday quarter last year, which the rule cannot tell from a seasonal product. How much is on hand does not enter into it; a winter range bought for two seasons is overstocked, but the season will show what is left, and once it has passed the rule no longer applies and the stock is judged on its sales. Switched off, the same list would hold 38 winter and summer lines and 12 labelled cases, and shown recall would be 17.6%.

Stock risk: 3 missed, 2 false alarms

• 2 sizes measured slow. A linen shirt size selling 0.12 a day by its true rate measured 0.04, and a cap selling 0.32 measured 0.22, so their stock looked like 25 and 23 days of cover against a 21-day lead time.
• 1 size with a single sale in 90 days. A slower size is judged when it has sold at least twice in 90 days and the sales it would lose reach the $50 floor. This one sold once, and the engine's estimate of the gap is $3.92.
• 2 false alarms are the reverse of the first: two best-selling sizes had a strong last 30 days (0.30 and 0.34 a day measured, 0.12 and 0.16 true), and the engine weights the last 30 days at 60%.

These are the limits of measuring pace from a short, noisy history, and every tool that reads recent sales shares them. Two chino sizes that sold under the old minimum pace of 0.1 a day, one sold out and one down to its last unit, are now found.

Profit leaks

The enamel mug sold below cost ($28 lost over the year), its VIP40 code (6 orders at −74% margin, $35 below the floor) and the sticker-pack small orders ($1.80 in fixed fees) are real and were detected, but each is below the $50 floor, so none is listed and none is scored. At this store's size that is the right call. Shipping is not judged at all: the app does not yet ask what shipping costs the merchant.

What the first run found

The first run of this test found problems in our engines and our sync. We fixed them before publishing, and kept the first run's figures here for the record:

• The slow rule. It asked for a 90-day sell-through under 20% and more than 180 days of supply. A sell-through under 20% already means more than about 360 days of supply, so the 180-day condition never decided anything. The rule is now the one measure it was meant to be: more than 180 days of supply at the current pace.
• The pace of young products. Both engines divided 90 days of sales by 90 even for a product first sold three weeks ago, which understated its pace up to four times. They now divide by the days the product could actually sell, never fewer than 14, and never more than the history the store has.
• Slow sizes that run out. Stock risk ignored every variant selling under 0.1 units a day. A size that sells twice a month and is down to its last unit loses real sales, so it is now judged when the loss reaches the $50 floor.
• Seasons. The rule described above.
• Discounts without a code. The sync always passes a list of codes, empty when none was used, and the profit leak engine only grouped orders under "automatic discounts" when the list was missing altogether, so automatic discounts were never judged. It now groups them when the list is empty.
• Currencies without cents. The sync stored yen and won as whole units, which is correct, while every screen and the weekly email divided every amount by 100, so a store in yen read a hundred times too small; dinars, which have three decimals, lost the third, and the CSV reader reported any currency but dollars, euros and pounds in dollars. Amounts now follow each currency's own minor unit everywhere, and the engines' money thresholds, written in US cents, are scaled to the same weight of money in the store's currency.
• Our sync. Refunded revenue was taken out twice, and order lines could be written twice by two syncs of the same order at once; both were fixed before this update. The third problem, 14 sizes of one product without a stock level, is explained in the next section.

Checked against the live app

The store is also connected to a development copy of the app, which syncs it into its own database. The harness read that database (read only) to check that the synced store is the planned one and that the app's pipeline reaches the same conclusions. The scale script was still creating orders, newest first, so these figures are as of 25 September 2026, 00:14 UTC.

• Orders: 1,835 of the 1,850 synced orders match a planned order to the second, with the same lines, discounts, codes and shipping. All 111 scenario orders are synced, and scale orders back to 13 June 2026. The other 15 are not in either plan: 5 duplicates of linen shirt orders from a first seeding attempt a minute earlier, and 10 sticker-pack orders from outside both scripts.
• Catalogue: all 356 planned variants are synced with the right price and cost. 14 of the 15 sizes of one product, the first one the scale script created, have no stock level in the database although the plan gives them 1 to 3 units each. Their stock updates were handled before the product itself had been written, because the product's own notification failed once on a network error and was retried, and the sync dropped a stock level whose variant it did not know yet. It now fetches the product first and keeps the level.
• Same data, same answers: on the orders synced so far, the engines running on the database through the app's own loader and the engines running on the plan agree on 66 of 77 findings and on 337 of 351 dead stock classifications. Every difference is one of those 14 sizes, which the database holds as out of stock.
• The stored report was computed by the app at 18:06 UTC, before the fixes, when 240 orders had been synced. Its 9 open findings all agree with the harness; the fixed engines add 12 more on the same data, 3 for dead stock and 9 for stock risk.
• Rows written before the fixes. The development database still holds what the sync wrote before them: 4 refunded orders stored with the subtotal after the refund, 2 orders with a line twice, and the 14 missing stock levels. They stay until the store is synced again, and they are why the live check still reports 4 subtotal differences and the 14 sizes.

What this does not prove

• It is one synthetic store. Real stores have returns that restock, costs that change, bundles, pre-orders, several locations, and merchants who edit orders. The generator has none of the mess a real catalogue has, and a clean store flatters any engine.
• The truth shares definitions with the engine. For dead and stale stock the labels use the same 90 and 180 days the engine does. The test shows the engine applies its definitions correctly to raw orders; it does not show those definitions are the right ones for your store.
• Some numbers rest on very few cases. Profit leaks has one labelled positive and stock risk has nine. A rate over one case is not a rate, and the stock risk figures could move by 10 points with one case either way.
• Unlabelled cases are not scored. 123 dead stock cases and 131 stock risk cases, mostly seasonal lines, are excluded because the generator does not say what is right for them. What the engine said about each of them is in the results file.
• The demo store has artefacts. Its products were created in Shopify a day or less before the analysis date, which hides never-sold stock from the age check, and its refunds are still being created.
• We wrote both the engines and the test. The rules are stated on this page in full, and every number here comes from one results file that records each label, each finding and the reason for each miss. Our source repository is not public; the results file is available on request.

Reproduce it

The harness lives in apps/tools/accuracy/ in our source repository, which is not public. It needs Node.js 22 or later and the workspace dependencies, and no network or store access. From apps/:

pnpm install
node tools/accuracy/run.mjs          # rebuild the store, label it, run the engines, score them
node tools/accuracy/run.mjs --live   # also reconcile with a synced database (read only)

A run takes about half a second (about 0.6 seconds with the database check). It writes every label, every finding, every miss with its reason and the what-if runs to apps/tools/accuracy/results.json, and a summary to docs/diagnosis-accuracy.md. The labelling rules are in truth.mjs, stated in plain words in the results file, and independent of the engine code: change a threshold there and the score changes with it.