TL;DR
RealityCrafter is a mixed-reality (MR) 3D authoring tool that enables users to interact with and edit a 3D scene reconstructed from a single real-world image.
Abstract
We propose RealityCrafter, a mixed-reality 3D authoring tool that enables users to edit and interact with a reconstructed 3D scene from a single real-world image. Prior research has largely focused on 3D authoring tools for purely virtual spaces, insufficiently incorporating real-world context and thereby hindering user immersion during the creation process. To overcome these limitations, our approach takes a single real-world image as input, generates segmented object-level 3D meshes in a zero-shot manner, and re-constructs a 3D scene where objects can be removed or modified without occlusion through instance mask-based inpainting. We leverage LLMs to interpret user voice commands and update the style, position, scale, and orientation of 3D objects in real time, providing an interactive 3D authoring interface in mixed-reality environments. By using a single image as a baseline, this approach enables effortless generation of realistic 3D scenes and intuitive editing based on user intent, delivering a novel creative experience that seamlessly blends the real and the virtual objects.
System Implementation
The overall system consists of two main stages: (1) offline — 3D scene reconstruction from a single image; and (2) online — user-guided 3D object editing and authoring. Our system reconstructs a 3D scene from a single image in an offline stage, then moves into an online phase where users direct object edits and craft 3D content.
BibTeX
@inproceedings{kim2025realitycrafter,
title={RealityCrafter: User-guided Editable 3D Scene Generation from a Single Image in Mixed Reality},
author={Kim, Seokyoung and Kim, Dooyoung and Son, Taejun and Woo, Woontack},
booktitle={Adjunct Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology},
pages={1–3},
year={2025}
}