Systems and methods of the present disclosure are directed to a computer-implemented method for multimodal search refinement. The method includes obtaining a visual search query from a user comprising one or more query images. The method includes providing a search interface for display to the user, the search interface comprising one or more result images responsive to the one or more query images and an interface element indicative of a request to the user to refine the visual search query. The method includes obtaining, from the user, textual data comprising a refinement to the visual search query. The method includes appending, by the computing system, the textual data to the visual search query to obtain a multimodal search query.