Auto-populate fields with OCR

Workflow purpose

This guide demonstrates how to use Proto's platform to auto-populate structured fields from a photo or scanned document a user uploads – an ID card, receipt, or claim form – instead of asking them to type it all out. Proto's LLM action supports vision-capable models, so the same extraction pattern used for text intake works directly on an uploaded image: the user attaches a document, the model reads it, and the fields come back ready to confirm.

Who can benefit from this guide:

Any workflow that currently asks a user to retype details from a document they already have
Claims, KYC, and account-opening flows that rely on ID or form uploads
Operations teams wanting to cut data entry errors from manual re-keying
Teams building any intake flow where "just upload a photo" is faster than a form

Auto-populate fields from an uploaded document

This workflow reads an uploaded photo or scan with a vision-capable LLM action and presents the extracted fields for the user to confirm. See how to auto-populate fields with OCR.

Just starting with the platform?

For initial setup of your AI agent – including workspace setup, agent training, and channel deployment – please start here. This workflow guide assumes your AI agent is set up and ready for more advanced configurations.

1. Create a trigger

Purpose: Start the flow wherever a document upload would save the user time.

Navigate to the Actions tab in the AI agent settings.
Click “+ Add Trigger” and select the Message Received trigger type, or insert this as a step inside an existing intake flow.
Name the trigger “Document field extraction”.

2. Survey – attachment upload

Purpose: Collect the document itself before attempting to read it.

Add a Survey action titled Document upload.
Field document_image – Attachment(s) – Please upload a clear photo or scan of the document. – Required: On.

3. LLM action – field extraction

Purpose: Read the document and return only the fields you actually need.

Add an LLM action named Document field extraction.
Model: GPT-4o (or another vision-capable model).
Input: {document_image}.
Output variable: extracted_fields.
Success branch: first action is Set chat variable – extracted_fields = _.json_parse(extracted_fields) – before any field is read.
System prompt should list each expected field by name and type, and instruct the model to return null for anything it can’t read clearly rather than guessing.

4. Set chat variables – map the extracted fields

Purpose: Move the extracted values into the same variables the rest of the flow expects.

Add Set chat variable actions for each expected field, e.g. full_name = extracted_fields["full_name"].

5. Confirm or correct

Purpose: Let the user catch anything the model misread before it’s submitted anywhere.

Add a Send message action summarising the extracted fields, followed by quick replies Looks right and Something’s wrong.
Add a Set chat variable action to capture the reply as confirmation_choice – quick reply selections aren’t captured automatically, so this step reads _user_input into the variable.
Add a Branch action. Condition: _.lower(confirmation_choice) == "something's wrong" → route to a manual Survey for the affected fields. Otherwise, continue with the extracted values as they are.

6. Test the flow

Purpose: Confirm the extraction is reliable enough to trust before it feeds downstream actions.

Test with a clear, well-lit photo and confirm every field extracts correctly.
Test with a blurry or partially cropped photo and confirm the model returns null for unreadable fields instead of inventing a value.
Test the correction path to confirm a user can fix a misread field without restarting the whole flow.

550K