Skip to main content
Send the picture your shopper shows with their message, and Agentcard searches for the product in it. A viewer who points their phone at a jacket on screen and says “where can I get that jacket” gets product cards for that jacket, not for every jacket.

Send the picture with the words

Send the picture beside the shopper’s words. Agentcard reads the two together and names the product the picture shows, then searches with the shopper’s words and that name in place of “that”: “where can I get the Patagonia Down Sweater Jacket”. The cards, the reply and the conversation’s history all read those words.
Pass the picture as image to buy.turn:
agentcard.ts
Show products and add the pick covers how to show these cards and add the one the shopper picks.

Send the picture as data

A picture your app holds in memory goes as base64 text, the picture’s bytes written out as letters and digits, with its type. Agentcard takes image/jpeg, image/png, image/webp and image/gif, up to 3,000,000 bytes once decoded. A full-size phone photo can be larger, so scale the picture down before you send it. A picture sent by address needs an https address of at most 2,048 characters.
agentcard.ts

Send a picture with no words

A shopper who only shows a picture still gets a search. Agentcard searches as if the shopper had written “find” and the product’s name, and the answer has the same shape: a reply and the products found.
buy.turn needs the shopper’s words, so give it a short ask with the picture:
agentcard.ts
An app that calls client.buy itself can send the picture with no ask at all:
agentcard.ts

Handle an unclear picture

Agentcard searches for a product only when it is sure what the picture shows. A picture of a crowded shelf, or of several products when the words do not say which, leaves the search on the shopper’s words alone, so “where can I get that jacket” searches for jackets. A picture with no words that Agentcard cannot read is refused:
Ask the shopper what they are looking for, and send their words on the next message.

Send it only with a new request

Send the picture with the shopper’s message that asks for something, and with nothing else. A pick of a card, a yes to a total and every later message go without it, so a shopper who taps a card never uploads the picture again. Agentcard reads the picture during the call and never keeps it.
buy.turn sends the picture with the turn’s first request for products and with nothing else, so your code does nothing more.

Fix a refused picture

A picture Agentcard cannot take is refused with 400, and the error names the field:
buy.turn checks the picture’s address, its type and the size of its data before any call, so a picture those checks refuse throws a TypeError naming the field. buy.turn('that', { image: { url: 'http://frames.yourapp.com/a.jpg' } }) throws:
Send the picture from an https address instead:
agentcard.ts
The SDK’s size check takes up to 3,145,728 decoded bytes, while Agentcard takes 3,000,000. A picture between the two throws nothing in the SDK and is refused by Agentcard with image.data is at most 3 MB decoded, so handle that refusal as well as the TypeError.