Jun Hao Liew

I am a senior research scientist at TikTok/ByteDance Research. My research focus is controllable diffusion-based generative models. Before joining TikTok, I worked as a research fellow at NUS. Prior to that, I did my Ph.D. with A/P Sim-Heng Ong, Dr. Wei Xiong and Dr. Jiashi Feng in NUS. I also work closely with Dr. Hanshu Yan, Dr. Jianfeng Zhang and Professor Yunchao Wei.

I am actively looking for research interns and collaborators. Please feel free to drop me an email if you are interested.

Email  /  Google Scholar  /  LinkedIn  /  Twitter  /  Github

Research

We improve the appearance quality of MagicAnimate by introducing regional supervision and explicit modeling of motion blur.

We train a fast (<1s) and accurate drag-based image editing model by learning from video supervision.

We present Creativity-VLM, a vision-language assistant that can translate coarse editing hints (e.g., "spring") into precise, actionable instructions for image editing.

DiG explores the long sequence modeling capability of Gated Linear Attention Transformers (GLA) in diffusion models for long-sequence image generation tasks.

We present ClassDiffusion to mitigate the weakening of compositional ability during personalization tuning.

We present PeRFlow, a flow-based method for accelerating diffusion models. PeRFlow divides the sampling process of generative flows into several time windows and straightens the trajectories in each interval via the reflow operation.

We introduce MagicVideo-V2 that integrates the text-to-image model, video motion generator, reference image embedding module and frame interpolation module into an end-to-end video generation pipeline.

We present Symplectic Adjoint Guidance (SAG) to obtain accurate gradient guidance for training-free guided sampling in diffusion models.

We present DragDiffusion, which extends interactive point-based image editing to large-scale pretrained diffusion models.

We propose AvatarStudio, a coarse-to-fine generative model that generates explicit textured 3D meshes for animatable human avatars.

We propose MagicAnimate, a diffusion-based human image animation framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity.

XAGen is a 3D-aware generative model that enables human synthesis with high-fidelity appearance and geometry, together with disentangled controllability for body, face, and hand.

We present MixVal, a model selection method that operates solely with unlabeled target data during inference to select the best UDA model for the target domain.

We present SegRefiner, a universal segmentation refinement model that is applicable across diverse segmentation models and tasks (e.g., semantic segmentation, instance segmentation, and dichotomous image segmentation).

MagicEdit explicitly disentangles the learning of appearance and motion to achieve high-fidelity and temporally coherent video editing. It supports various editing applications, including video stylization, local editing, video-MagicMix and video outpainting.

MagicAvatar is a multi-modal framework that is capable of converting various input modalities — text, video, and audio — into motion signals that subsequently generate/ animate an avatar.

MagicProp employs the edited frame as an appearance reference and generates the remaining frames using an autoregressive rendering approach.

We developed a fast open-vocabulary semantic segmentation model that can perform comparably or better without the extra computational burden of the CLIP image encoder during inference.

We conduct an empirical study on the scaling capability of masked image modeling (MIM) methods for visual recognition.

Associating spatially-consistent grouping of self-supervised vision models with text-supervised semantic segmentation.

We propose a 3D-aware portrait video GAN, PV3D, which is capable to generate a large variety of 3D-aware portrait videos with high-quality appearance, motions, and 3D geometry. PV3D is trainable on 2D monocular videos only, without the need of any 3D or multi-view annotations.

We explored a new task called semantic mixing, aiming at mixing two different semantics to create a new concept (e.g., tiger and rabbit).

Our Slim Scissors enables quick extraction of elongated thin parts by simply brushing some coarse scribbles.

We develop a novel learning-based aggregation method that improves upon SOLO by leveraging the rich neighboring information while maintaining the architectural efficiency.

We identify the issue of indirect information propagation between deeper and shallower layers in FPN-based saliency methods and present a cross-layer communication mechanism for better salient object detection.

We present the first single-stage model for multi-person body mesh recovery. BMP introduces a new representation: each person instance is represented as a point in the spatial-depth space which is associated with a parameterized body mesh.

We revisit the use of superpixels for active learning in segmentation and demonstrate that the inappropriate choice of cost measure may cause the effectiveness of the superpixel-based approach to be under-estimated.

With our proposed attentive deformation mechanism and segment-wise matching scheme, our contour-based instance segmentation model DANCE performs comparably to existing top-performing pixel-based models.

We collect a large-scale dataset specifically for segmentation of thin elongated objects, named ThinObject-5K. In addition, we design a three-stream network called TOS-Net that integrates high-resolution boundary information with fixed resolution semantic contexts for effective segmentation of thin parts.

We investigate performance drop of Mask R-CNN on long-tail LVIS dataset, and unveil that a major cause is the inaccurate classification of object proposals. To address this, we propose a simple calibration framework to more effectively alleviate classification head bias with a bi-level class balanced sampling approach.

We present a simple Inside-Outside Guidance (IOG) for interactive segmentation. IOG only requires an inside point that is clicked near the object center and two outside points at the symmetrical corner locations (top-left and bottom-right or top-right and bottom-left) of a bounding box that encloses the target object.

We propose a deep yet light-weight saliency inference module that adopts a multi-dilated depth-wise convolution architecture for salient object detection.

We present MultiSeg, a scale-diverse interactive image segmentation network that incorporates a set of two-dimensional scale priors into the model to generate a set of scale-varying proposals that conform to the user input.

PANet introduces a prototype alignment regularization between support and query for better generalization on few-shot segmentation.

We present FSENet to tackle the class imbalance and inter-class interference problem in multi-label brain tumor segmentation.

RIS-Net expands the field-of-view of the given input clicks to capture the local regional information surrounding them for local refinement.