SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing

Jing Gu1, Nanxuan Zhao2, Wei Xiong2, Qing Liu2, Zhifei Zhang2, He Zhang2, Jianming Zhang2, HyunJoon Jung2, Yilin Wang2*, Xin Eric Wang1*,
1University of California, Santa Cruz, 2Adobe, *co-advising

ECCV 2024
Paper arXiv Code

Figure 1. SwapAnything results on various personalized image swapping tasks. SwapAnything is adept at precise, arbitrary object replacement in a source image with a personalized reference, and achieves high-fidelity swapping results without influencing any context pixels. We demonstrate its general swapping effect in single-object, multi-object, partial-object, and cross-domain swapping tasks.

Abstract

Effective editing of personal content holds a pivotal role in enabling individuals to express their creativity, weaving captivating narratives within their visual stories, and elevate the overall quality and impact of their visual content. Therefore, in this work, we introduce SwapAnything, a novel framework that can swap any objects in an image with personalized concepts given by the reference, while keeping the context unchanged. Compared with existing methods for personalized subject swapping, SwapAnything has three unique advantages: (1) precise control of arbitrary objects and parts rather than the main subject, (2) more faithful preservation of context pixels, (3) better adaptation of the personalized concept to the image. First, we propose targeted variable swapping to apply region control over latent feature maps and swap masked variables for faithful context preservation and initial semantic concept swapping. Then, we introduce appearance adaptation, to seamlessly adapt the semantic concept into the original image in terms of target location, shape, style, and content during the image generation process. Extensive results on both human and automatic evaluation demonstrate significant improvements of our approach over baseline methods on personalized swapping. Furthermore, SwapAnything shows its precise and faithful swapping abilities across single object, multiple objects, partial object, and cross-domain swapping tasks.

A single frame for different tasks.

  • Targetted swapping to keep context pixel perfectly unchaged and to transfer important features.
  • Sophisticated appearance adaptation process to adapt the concept image into the source object.
  • The framework applied to single object swapping, multi-object swapping, partial-object swapping, cross-domain swapping, text-based swapping, and tasks beyond swapping such as object insersion.

Figure 2. Overview of SwapAnything on swapping a object from a source image ($I_{src}$) into a personalized concept ($<{*}>$) to get the target image ($I_{target}$). The personalized concept is first converted into textual space to be treated as concept appearance. Meanwhile, the source image is first inverted into initial noise to obtain U-Net variables (including latent feature, attention map, and attention output). Targeted variable swapping preserves the context pixels in the source image. The appearance adaptation process then utilizes these informative variables to integrate the concept into the target image.

Single Object Swapping Result

Figure 3. Comparison on single-object swapping with baselines in their original components. SS means Object Swapping, BP means Background Preservation, and OQ means Overall Quality. SG means Object Gesture. Please zoom in for a clear visual result.

Multi-object Swapping Result

Partial-object Swapping

Cross-domain Swapping

Text-based Swapping

Object Insertion

Comparison with DALL-E in ChatGPT

Unlike SwapAnything, DALL-E in ChatGPT can only do text-based (not personalized) editing, and it can not edit real image. In other words, user can only edit image created by DALL-E itself in previous conversation. We conduct comparison by generating a source image by DALL-E itself.

Human Evaluation

We show the human preference between results generated by our method and the baseline methods. SS means Object Swapping, BP means Background Preservation, and OQ means Overall Quality. SG means Object Gesture. For the baseline methods, PS means Photoswap; MC means MasaCtrl; BP means BlipDiffusion; DE means DreamEdit; CP means CopyPaste.

BibTeX

@article{gu2024swapanythiing,
      title={SwapAnything: Enabling Arbitrary Object Swapping in Personalized Image Editing}, 
      author={Jing Gu and Nanxuan Zhao and Wei Xiong and Qing Liu and Zhifei Zhang and He Zhang and Jianming Zhang and HyunJoon Jung and Yilin Wang and Xin Eric Wang},
      year={2024},
      journal={ECCV}
}