Scan your fridge with your phone: live scan is in Sugo
Sweep your phone along a shelf and Sugo lists what it sees, counting each item once. Hold up a pack and it reads the barcode. Frames stay on your phone.
Sugo can now read your fridge through the camera while you move it. Open the pantry, tap the camera button, choose Live scan and sweep the phone slowly along a shelf. Each thing it recognises gets a bracket and a name on screen, a short buzz, and a chip on a list along the bottom. When you are done you check that list, fix anything it got wrong, and add the lot in one go. Nothing reaches your pantry before you have seen it.
It all happens on the phone. The camera frames are read by a model that lives on the device and are never uploaded. Live scan is in the iPhone and iPad app on iOS 17 or later, and in the Android app.
Since 30 September it also reads barcodes. Hold a pack up to the camera and its barcode names it, which has its own section below.
The rest of this post is about the part that took the longest to get right, which is counting, then barcodes, and what live scan still gets wrong.
Why scan your fridge at all
Sugo's pantry ranks your saved recipes by how much of each one you can already cook. That works as well as the pantry list is accurate, and keeping it accurate has always been the weak point. Nobody types thirty items into a phone on a Sunday evening, and a list that was right three weeks ago is wrong in ways you cannot see.
Photo scanning helped: photograph a shelf and the items come back as a list. Live scan makes the job the length of one sweep, which is short enough that you might actually do it after the shopping is put away.
How live scan counts what is in your fridge
Naming what is in a frame is the part people expect to be hard. Counting turned out to be harder.
A phone reads about ten frames a second while you sweep. The eggs are in view for three or four seconds on the way along, so they turn up in thirty or forty frames, and again when you swing back to check something you missed. A scanner that added up what it saw would hand you a list of four hundred lines. So the question live scan spends most of its effort on is whether the thing in this frame is the same thing as the one in the last frame.
Here is one sweep along a shelf and part of the way back. The count on the left is frames read. The one on the right is rows on the list.
One sweep along the shelf, and part of the way back
- Milk
- Eggs×2
- Cheese
- Mushrooms
- Carrots
- Lime
85
Frames read
6
On the list
Three rules do most of the work.
An item has to hold. Something that shows up in one frame and not the next is usually a reflection, a shadow or the model changing its mind, so an object only makes the list once it has qualified in three of the last five frames. Until then the app draws a thin outline around it with no name, which is why names never flicker on screen.
Each object is followed from frame to frame. The scanner tracks position and appearance, so a jar half hidden behind the milk keeps its identity, and the name it ends up with is a vote across all the frames it was seen in. That vote is what stops a chip flipping between shallot and onion as the light changes.
The list counts kinds of thing. Two tomatoes on different shelves are one row, tomatoes, and the number beside it is the most it saw together in a single frame. That number cannot double-count a sweep, which is the point, but it can undercount: three peppers that were never in view at the same time come back as one. The review screen has a stepper on every countable row for exactly that.
How it recognises packaged food
A lot of a fridge is packaging, and packaging is the hardest thing to recognise by shape. A carton of oat milk looks much like a carton of stock. So live scan reads the large print on anything that looks like a package, in English, French and Spanish, and trusts the words over the shape.
The words beat the shape
A carton that says "lait" is milk, whatever the box looks like. Forty-odd things, from tahini to sriracha, are only ever named from their labels.
Phrases before words
"Peanut butter" is checked before "butter", and "ice cream" before "cream", so a jar of one does not end up on the list as the other.
The picture on the tub
A salad printed on a tub of feta is a picture of food. It is ignored.
Brands decide nothing alone
A brand name or a price on its own proves nothing, so a butter pack showing only its brand is named by how it looks.
Hold a pack up and it reads the barcode
Reading the print only helps when the front says something useful. Often it is just a brand, which decides nothing, and a pot of cream and a pot of yogurt are the same white tub from most angles. In the latest version of the app, the barcode settles it. Hold the pack close to the phone and turn it until the barcode faces the lens. Live scan reads the number, looks it up and renames the chip.
One pot, held up and turned
- BarcodeNames the chip
Cream
- Label
A brand name only
- Shape
Yogurt
Holding it up is what makes it work. Before building this we tried reading barcodes during an ordinary sweep, cropping every box the scanner tracked at full resolution: 2,709 crops from one fridge, and not one readable barcode. From sweeping distance the bars are too small and too blurred. A pack held up to the phone fills a third of the frame or more, and after a moment at that size live scan reads it at full resolution. If you have held it for a second and it has found no barcode, a line above the list says "Show the barcode to name it", which usually means turning the pack round.
The barcode names the kind of thing, cream or butter, never the brand or the size, because the pantry keeps kinds of thing. It beats the print on the label, and the print beats the shape. The lookup runs in the background, so the scan never stops to wait for it, and a pack it cannot look up (with no signal, say) keeps the name it already had.
The table it looks the number up in is built from Open Food Facts, the open database of food products, and lives on our own server. The phone sends the number and nothing from the camera.
Live scan or photos
Photo scanning has not gone anywhere, and for some jobs it is still the better tool. They sit on the same sheet: live scan at the top, photos underneath.
Photos
Up to six pictures, read by a vision model in the cloud. Better for a receipt, a shopping list (printed or handwritten) or a haul spread out on the table. It can add things Sugo's ingredient list does not know, and it counts towards a monthly allowance that depends on your plan.
Live scan
The camera, read on the phone, for a fridge or a counter. Nothing is uploaded from the camera and there is no monthly limit. It adds items from Sugo's ingredient list only, with a count rather than a weight.
What leaves your phone during a live scan is the list you confirm, the number of any barcode it reads so it can be looked up, and anonymous daily tallies of which kinds of item people kept or unticked. The tallies carry no account, household or device, and they are how we learn which names the model gets wrong most often.
From a fridge scan to recipes you can cook
Once you tap Add, the items are in your pantry, and the pantry is what the rest of Sugo cooks from. Your saved recipes are ranked by how much of each one you can make tonight, with the missing ingredients shown on the ones that nearly work.
Your pantry
The same pantry is subtracted from your shopping list, so a scan after the weekly shop also stops you buying a second jar of what is already at the back of the cupboard. If you want the cooking side of this, what to cook with what you already have is the method, and the 12 pantry staples that make anything cookable is the shelf worth scanning.
What it gets wrong
It knows a few dozen common fresh ingredients by sight: eggs, lemons, limes, tomatoes, peppers, onions, potatoes, carrots, mushrooms, salmon, bacon, spinach, lettuce, berries and so on, plus the packaged goods it reads. Plenty is missing. Ginger, shallots, kale and parsnips are not recognised by look at all. It still mixes up onions and potatoes now and then, calls a round lemon an orange, and struggles with lettuce once it is in a salad rather than a head.
It sees presence and a count, never an amount. Half a bag of spinach and a full one are both spinach. Eggs get no number unless it sees two apart, because a box of eggs is usually a whole carton and guessing one would be wrong most of the time. It only catches what is in the camera's view, so the yoghurt behind the milk stays behind the milk.
The barcode table is small for now. It covers about one product in ten sold in the US, the UK, France, Spain and Italy, because we only map a product when its category or name leaves no doubt about what it is. For most packs you hold up today, the barcode is read, it is not in the table, and the chip keeps the name its label or shape gave it.
The first scan asks something of you. The model is a one-off download, which the app fetches by itself on Wi-Fi when it can, and preparing it the first time took around half a minute on an iPhone 12 Pro in our testing. After that the scanner opens in a couple of seconds. It needs iOS 17 even though the rest of Sugo runs on iOS 16.4, and it is not in the browser version at app.sugo.kitchen, where there is no model on the device to run it.
None of that is hidden from you: every row of the result lands on a review screen with a tick box, and nothing is added until you have been through it.
Questions people ask
Is there an app that scans your fridge and finds recipes?
Sugo does. Open the pantry, tap the camera button, choose Live scan and sweep the phone slowly along a shelf. You check the list it builds and add it to your pantry, and your saved recipes are then ranked by how much of each one you can cook tonight, with the missing ingredients shown on the ones that nearly work.
Does a fridge scan upload my camera?
No. The camera frames are read by a model on the phone and never uploaded. What leaves the phone is the list you confirm, the number of any barcode you hold up so it can be looked up, and anonymous daily tallies of which kinds of item people kept or unticked. Those tallies carry no account, household or device.
Can it read barcodes?
Yes. Hold a pack close to the phone and turn it until the barcode faces the lens, and the barcode renames the chip. It names the kind of thing, cream or butter, never the brand or size. The barcode table is small for now, about one product in ten, so many packs keep the name their label or shape gave them.
Which phones does live scan work on?
The iPhone and iPad app on iOS 17 or later, and the Android app. It is not in the browser version at app.sugo.kitchen, because there is no model on the device there to run it. The model is a one-off download, which the app fetches by itself on Wi-Fi when it can.
Tips for your first fridge scan
Let the download happen on Wi-Fi
Once you are signed in, the app usually fetches the model in the background on Wi-Fi, so the first scan is not spent waiting for it.
Turn the labels towards you
Packaged things are named from their print, and a label facing the back wall gives the scanner nothing to read.
Hold up anything that could be two things
A white pot or a plain carton: hold it close and turn it until the barcode faces the phone, and the barcode names it.
Move slower than feels necessary
Each item has to hold for a few frames before it counts. A quick swipe across the door catches the big things and skips the rest.
Get multiples into one frame
The count is the most it sees at once, so four yoghurts in a row are four, and four yoghurts on four shelves are one.
Read the list before you add it
Untick what is wrong, fix the counts, and remember that anything it could not see is still yours to add by hand.
Sugo is free on iPhone, iPad and Mac, on Android, and in any browser at app.sugo.kitchen. If your pantry list has not matched your fridge since the week you made it, download Sugo.