Scan your fridge with your phone: live scan is in Sugo
Sweep your phone along a shelf and Sugo names what it sees, counts each item once and lists it for you to check. Camera frames never leave your phone.
Sugo can now read your fridge through the camera while you move it. Open the pantry, tap the camera button, choose Live scan and sweep the phone slowly along a shelf. Each thing it recognises gets a bracket and a name on screen, a short buzz, and a chip on a list along the bottom. When you are done you check that list, fix anything it got wrong, and add the lot in one go. Nothing reaches your pantry before you have seen it.
It all happens on the phone. The camera frames are read by a model that lives on the device and are never uploaded. Live scan is in the iPhone and iPad app on iOS 17 or later, and in the Android app.
The rest of this post is about the part that took the longest to get right, which is counting, and about what live scan still gets wrong.
A pantry list is only useful if it is roughly true
Sugo's pantry ranks your saved recipes by how much of each one you can already cook. That works as well as the pantry list is accurate, and keeping it accurate has always been the weak point. Nobody types thirty items into a phone on a Sunday evening, and a list that was right three weeks ago is wrong in ways you cannot see.
Photo scanning helped: photograph a shelf and the items come back as a list. Live scan makes the job the length of one sweep, which is short enough that you might actually do it after the shopping is put away.
Recognising the eggs is the easy part
Naming what is in a frame is the part people expect to be hard. Counting turned out to be harder.
A phone reads about ten frames a second while you sweep. The eggs are in view for three or four seconds on the way along, so they turn up in thirty or forty frames, and again when you swing back to check something you missed. A scanner that added up what it saw would hand you a list of four hundred lines. So the question live scan spends most of its effort on is whether the thing in this frame is the same thing as the one in the last frame.
Here is one sweep along a shelf and part of the way back. The count on the left is frames read. The one on the right is rows on the list.
One sweep along the shelf, and part of the way back
- Milk
- Eggs×2
- Cheese
- Mushrooms
- Carrots
- Lime
85
Frames read
6
On the list
Three rules do most of the work.
An item has to hold. Something that shows up in one frame and not the next is usually a reflection, a shadow or the model changing its mind, so an object only makes the list once it has qualified in three of the last five frames. Until then the app draws a thin outline around it with no name, which is why names never flicker on screen.
Each object is followed from frame to frame. The scanner tracks position and appearance, so a jar half hidden behind the milk keeps its identity, and the name it ends up with is a vote across all the frames it was seen in. That vote is what stops a chip flipping between shallot and onion as the light changes.
The list counts kinds of thing. Two tomatoes on different shelves are one row, tomatoes, and the number beside it is the most it saw together in a single frame. That number cannot double-count a sweep, which is the point, but it can undercount: three peppers that were never in view at the same time come back as one. The review screen has a stepper on every countable row for exactly that.
It reads the packet as well as the shape
A lot of a fridge is packaging, and packaging is the hardest thing to recognise by shape. A carton of oat milk looks much like a carton of stock. So live scan reads the large print on anything that looks like a package, in English, French and Spanish, and trusts the words over the shape.
The words beat the shape
A carton that says "lait" is milk, whatever the box looks like. Forty-odd things, from tahini to sriracha, are only ever named from their labels.
Phrases before words
"Peanut butter" is checked before "butter", and "ice cream" before "cream", so a jar of one does not end up on the list as the other.
The picture on the tub
A salad printed on a tub of feta is a picture of food. It is ignored.
Brands decide nothing alone
A brand name or a price on its own proves nothing, so a butter pack showing only its brand is named by how it looks.
Live scan or photos
Photo scanning has not gone anywhere, and for some jobs it is still the better tool. They sit on the same sheet: live scan at the top, photos underneath.
Photos
Up to six pictures, read by a vision model in the cloud. Better for a receipt, a shopping list (printed or handwritten) or a haul spread out on the table. It can add things Sugo's ingredient list does not know, and it counts towards a monthly allowance that depends on your plan.
Live scan
The camera, read on the phone, for a fridge or a counter. Nothing is uploaded and there is no monthly limit. It adds items from Sugo's ingredient list only, with a count rather than a weight.
What leaves your phone during a live scan is the list you confirm, plus anonymous daily tallies of which kinds of item people kept or unticked, with no account, household or device attached. Those tallies are how we learn which names the model gets wrong most often.
What the list is for
Once you tap Add, the items are in your pantry, and the pantry is what the rest of Sugo cooks from. Your saved recipes are ranked by how much of each one you can make tonight, with the missing ingredients shown on the ones that nearly work.
Your pantry
The same pantry is subtracted from your shopping list, so a scan after the weekly shop also stops you buying a second jar of what is already at the back of the cupboard. If you want the cooking side of this, what to cook with what you already have is the method, and the 12 pantry staples that make anything cookable is the shelf worth scanning.
What it gets wrong
It knows a few dozen common fresh ingredients by sight: eggs, lemons, limes, tomatoes, peppers, onions, potatoes, carrots, mushrooms, salmon, bacon, spinach, lettuce, berries and so on, plus the packaged goods it reads. Plenty is missing. Ginger, shallots, kale and parsnips are not recognised by look at all. It still mixes up onions and potatoes now and then, calls a round lemon an orange, and struggles with lettuce once it is in a salad rather than a head.
It sees presence and a count, never an amount. Half a bag of spinach and a full one are both spinach. Eggs get no number unless it sees two apart, because a box of eggs is usually a whole carton and guessing one would be wrong most of the time. It only catches what is in the camera's view, so the yoghurt behind the milk stays behind the milk.
The first scan asks something of you. The model is a one-off download, which the app fetches by itself on Wi-Fi when it can, and preparing it the first time took around half a minute on an iPhone 12 Pro in our testing. After that the scanner opens in a couple of seconds. It needs iOS 17 even though the rest of Sugo runs on iOS 16.4, and it is not in the browser version at app.sugo.kitchen, where there is no model on the device to run it.
None of that is hidden from you: every row of the result lands on a review screen with a tick box, and nothing is added until you have been through it.
Before your first sweep
Let the download happen on Wi-Fi
Once you are signed in, the app usually fetches the model in the background on Wi-Fi, so the first scan is not spent waiting for it.
Turn the labels towards you
Packaged things are named from their print, and a label facing the back wall gives the scanner nothing to read.
Move slower than feels necessary
Each item has to hold for a few frames before it counts. A quick swipe across the door catches the big things and skips the rest.
Get multiples into one frame
The count is the most it sees at once, so four yoghurts in a row are four, and four yoghurts on four shelves are one.
Read the list before you add it
Untick what is wrong, fix the counts, and remember that anything it could not see is still yours to add by hand.
Sugo is free on iPhone, iPad and Mac, on Android, and in any browser at app.sugo.kitchen. If your pantry list has not matched your fridge since the week you made it, download Sugo.