Frank Fundel

By leveraging intricate data generation pipelines, Segment Anything Model (SAM) excels in interactive segmentation. However, SAM has shown weaknesses in specific scenarios, primarily due to the ambiguity of single point prompts. For example, prompting SAM to segment a human clicking on his torso, the model can produce a mask of the whole human, but also of the individual parts e.g. his upper body or shirt. To mitigate this issue in an interactive segmentation scenario, we allow SAM to use information from a few example (image, mask) pairs without updating its weights i.e. few-shot prompting.

Presented at the Streiflicht 2023 at Ulm University with a custom Annotation Tool.

Diffusion models excel in image generation but lack detailed semantic control using text prompts. Additional techniques have been developed to address this limitation. However, conditioning diffusion models solely on text-based descriptions is challenging due to ambiguity and lack of structure. In contrast, scene graphs offer a more precise representation of image content, making them superior for fine-grained control and accurate synthesis in image generation models. The amount of image and scene-graph data is sparse, which makes fine-tuning large diffusion models challenging. We propose multiple approaches to tackle this problem using ControlNet and Gated Self-Attention.