Multimodal Retrieval and Execution Monitoring Using Rich Recipe Representation


Reference #: 01587

The University of South Carolina is offering licensing opportunities for Multimodal Retrieval and Execution Monitoring Using Rich Recipe Representation


The content of recipes like for preparing food or assembling furniture exist as textual or image documents which makes it difficult for machines to read, reason and handle ambiguity. We consider the problem of reasoning with data in recipes in multiple representations.

Invention Description:

Our innovative representation is used in a web-based decision support system that helps users perform constrained queries using multiple modalities and monitor an agent executing (cooking) based on the recipes. This technology overcomes shortcomings with our recipe representation, which is enhanced with additional knowledge such as outcomes like allergen information, possible failures, and solutions for each atomic cooking step.

Potential Applications:


Complex instructions such as furniture assembly

Advantages and Benefits:

Ability to perform expressive and constrained queries about

– the outcome of recipe

– the process of recipe preparation

Ability to monitor progress of an executor automatically following the instruction.

Patent Information:
Software and Computing
For Information, Contact:
Technology Commercialization
University of South Carolina
Biplav Srivastava
Vishal Pallagani
Revathy Chandrasekaran Venka
Vedant Khandelwal
Kausik Lakkaraju
Multimodal representation
Recipe search and recommendation
Workflows execution
© 2024. All Rights Reserved. Powered by Inteum