LEGO®-Maker: Autoregressive Image-Conditioned LEGO® Model Creation

This paper presents LEGO ® -Maker, a new learning-based generative model that can effectively consider over 100 unique brick types and rapidly generate hundreds of bricks to create LEGO ® models conditioned on images. This work has three major technical contributions that enable it to achieve surpassing capabilities beyond existing generative approaches. First, we design a compact LEGO ® tokenization scheme to serialize LEGO ® models and bricks into tokens for autoregressive learning. Second, we build LEGO ® -Maker, an autoregressive image-conditioned architecture, with a multi-token prediction strategy to encourage pre-considering multiple brick attributes and a rollback mechanism for collision-free generation. Third, we propose an effective data preparation pipeline with a procedural generator to synthesize LEGO ® models and a LEGO ® -to-real image translator distilled from a large vision language model to translate LEGO ® renderings into associated photorealistic images, leveraging rich prior to address the scarcity of image-to-LEGO ® data. Extensive evaluations and comparisons are conducted on two object categories, facade and portrait, over metrics in four aspects: geometry, color, semantics, and structural integrity, together with a user study. Experimental results demonstrate the versatility and compelling strengths of LEGO ® -Maker in producing structures and details given by the reference image. Also, the evaluation scores manifest that our method clearly surpasses the baselines, consistently for all evaluation metrics.