A Language-Informed Loss for Visual Generation

Abstract

End-to-end visual generation with pixel-space loss has recently achieved competitive performance without the need for learned image tokenizers. In this work, we study whether language-informed supervision can provide a richer training signal than pixel loss for visual generation. We introduce LILIE (pronounced “lily”), a Language-Informed and Lossless Image Encoder. LILIE is an invertible image transform trained such that a vision-language model can recover captions from noised features. The encoder uses a normalizing flow transformer trained jointly with a vision-language model via next token prediction. We then use the encoder to extract a simple language-informed loss for text-to-image generation. We train JiT models with LILIE loss, reducing gFD by 14 points (66.84 → 53.01) against a pixel-only baseline on the GPIC dataset.

Overview

LILIE is an invertible encoder built with the TARFlow architecture. We train the encoder by passing an image through the encoder, corrupting the encoded image with additive Gaussian noise, then captioning the noised features with a LoRA-finetuned VLM. The objective is a standard cross-entropy loss against ground-truth captions from the GPIC corpus.

Once we have LILIE, we use L² distance in LILIE space to supervise JiT training. LILIE loss works on its own and improves generation quality over a pixel-only baseline (66.84 → 55.78 gFD), and combining LILIE loss with pixel loss further improves gFD to 53.01.

Samples

Click a pair to zoom in.

PixelPixel+LILIE

Pixel sample for: orange petals, brown stamens, close-up, flower, blurred background — tagorange petals, brown stamens, close-up, flower, blurred background

Pixel+LILIE sample for: orange petals, brown stamens, close-up, flower, blurred background — tagorange petals, brown stamens, close-up, flower, blurred background

Pixel sample for: ancient temple, carved reliefs, hieroglyphs, stone wall, sunlit — tagancient temple, carved reliefs, hieroglyphs, stone wall, sunlit

Pixel+LILIE sample for: ancient temple, carved reliefs, hieroglyphs, stone wall, sunlit — tagancient temple, carved reliefs, hieroglyphs, stone wall, sunlit

Pixel sample for: tabby cat, white chest, sunlight, close-up — tagtabby cat, white chest, sunlight, close-up

Pixel+LILIE sample for: tabby cat, white chest, sunlight, close-up — tagtabby cat, white chest, sunlight, close-up

Pixel sample for: red building, debris, construction, truck, cone — tagred building, debris, construction, truck, cone

Pixel+LILIE sample for: red building, debris, construction, truck, cone — tagred building, debris, construction, truck, cone

Pixel sample for: A man with gray hair and a white beard smiles while wearing glasses and a black — shortA man with gray hair and a white beard smiles while wearing glasses and a black shirt.

Pixel+LILIE sample for: A man with gray hair and a white beard smiles while wearing glasses and a black — shortA man with gray hair and a white beard smiles while wearing glasses and a black shirt.

Pixel sample for: A yellow crocheted moon and star sit on a dark blue, textured blanket. — shortA yellow crocheted moon and star sit on a dark blue, textured blanket.

Pixel+LILIE sample for: A yellow crocheted moon and star sit on a dark blue, textured blanket. — shortA yellow crocheted moon and star sit on a dark blue, textured blanket.

Pixel sample for: A mandrill monkey with bright red and blue facial markings looks directly at the — shortA mandrill monkey with bright red and blue facial markings looks directly at the camera.

Pixel+LILIE sample for: A mandrill monkey with bright red and blue facial markings looks directly at the — shortA mandrill monkey with bright red and blue facial markings looks directly at the camera.

Pixel sample for: Tall, tree-like structures with purple lights stand in a park at night. — shortTall, tree-like structures with purple lights stand in a park at night.

Pixel+LILIE sample for: Tall, tree-like structures with purple lights stand in a park at night. — shortTall, tree-like structures with purple lights stand in a park at night.

Pixel sample for: Two ducks swim side by side on calm water. The duck on the right has a green hea — mediumTwo ducks swim side by side on calm water. The duck on the right has a green head and gray body, while the one on the left is brown with mottled feathers. Their reflections ripple gently in the still surface.

Pixel+LILIE sample for: Two ducks swim side by side on calm water. The duck on the right has a green hea — mediumTwo ducks swim side by side on calm water. The duck on the right has a green head and gray body, while the one on the left is brown with mottled feathers. Their reflections ripple gently in the still surface.

Pixel sample for: Red and orange maple leaves hang against a clear blue sky. Their veins are visib — mediumRed and orange maple leaves hang against a clear blue sky. Their veins are visible, and sunlight highlights their edges as they sway slightly on thin branches.

Pixel+LILIE sample for: Red and orange maple leaves hang against a clear blue sky. Their veins are visib — mediumRed and orange maple leaves hang against a clear blue sky. Their veins are visible, and sunlight highlights their edges as they sway slightly on thin branches.

Pixel sample for: A colorful mural of a blue, cartoonish character with glowing eyes covers a conc — mediumA colorful mural of a blue, cartoonish character with glowing eyes covers a concrete wall. Snow melts into a puddle on the ground next to a black hexagonal pavement. In the background, a tree and buildings are visible under daylight.

Pixel+LILIE sample for: A colorful mural of a blue, cartoonish character with glowing eyes covers a conc — mediumA colorful mural of a blue, cartoonish character with glowing eyes covers a concrete wall. Snow melts into a puddle on the ground next to a black hexagonal pavement. In the background, a tree and buildings are visible under daylight.

Pixel sample for: A calm lake reflects the overcast sky and surrounding mountains. Green hills ris — mediumA calm lake reflects the overcast sky and surrounding mountains. Green hills rise on either side, with clouds drifting near their peaks. Sparse vegetation emerges from the water’s surface in the foreground.

Pixel+LILIE sample for: A calm lake reflects the overcast sky and surrounding mountains. Green hills ris — mediumA calm lake reflects the overcast sky and surrounding mountains. Green hills rise on either side, with clouds drifting near their peaks. Sparse vegetation emerges from the water’s surface in the foreground.

Pixel sample for: A spider with a dark, patterned body and long legs is centered on a delicate, ra — longA spider with a dark, patterned body and long legs is centered on a delicate, radiating web. The web stretches across the frame, with fine threads visible against a blurred background of green foliage and soft blue sky. Behind the spider, a faint, out-of-focus shape suggests a person wearing a light-colored garment. The background shows muted greens and blues, indicating an outdoor setting with natural lighting. The spider’s legs are spread outward, gripping the web as if it’s stationary or preparing to move. The web appears taut and intact, with no obvious damage or debris caught in it. The overall scene is shallow-focused, drawing attention to the spider’s detailed markings and the intricate geometry of its web.

Pixel+LILIE sample for: A spider with a dark, patterned body and long legs is centered on a delicate, ra — longA spider with a dark, patterned body and long legs is centered on a delicate, radiating web. The web stretches across the frame, with fine threads visible against a blurred background of green foliage and soft blue sky. Behind the spider, a faint, out-of-focus shape suggests a person wearing a light-colored garment. The background shows muted greens and blues, indicating an outdoor setting with natural lighting. The spider’s legs are spread outward, gripping the web as if it’s stationary or preparing to move. The web appears taut and intact, with no obvious damage or debris caught in it. The overall scene is shallow-focused, drawing attention to the spider’s detailed markings and the intricate geometry of its web.

Pixel sample for: A round, golden-brown pastry with visible spiral layers sits on a white plate, d — longA round, golden-brown pastry with visible spiral layers sits on a white plate, dusted lightly with powdered sugar. Beside it is a generous swirl of white whipped cream, topped with a small green leaf. To the right of the cream is a dark, glossy fruit compote with a single greenish fruit nestled inside. The plate has a smooth, slightly reflective surface and is illuminated by warm lighting. In the background, faint shadows and a blurred edge suggest the plate is placed on a table or counter. The dessert appears freshly served, with the textures of the flaky pastry and creamy topping clearly distinguishable.

Pixel+LILIE sample for: A round, golden-brown pastry with visible spiral layers sits on a white plate, d — longA round, golden-brown pastry with visible spiral layers sits on a white plate, dusted lightly with powdered sugar. Beside it is a generous swirl of white whipped cream, topped with a small green leaf. To the right of the cream is a dark, glossy fruit compote with a single greenish fruit nestled inside. The plate has a smooth, slightly reflective surface and is illuminated by warm lighting. In the background, faint shadows and a blurred edge suggest the plate is placed on a table or counter. The dessert appears freshly served, with the textures of the flaky pastry and creamy topping clearly distinguishable.

Pixel sample for: A large, rectangular swimming pool with clear blue water is centered in the scen — longA large, rectangular swimming pool with clear blue water is centered in the scene, reflecting the bright sky above. To the left stands a grand stone structure with tall columns and a triangular pediment topped by sculpted figures, while to the right is a matching colonnade with a balcony and a small statue on its steps. Two white marble statues of winged figures sit near the pool’s edge—one on the left beside a metal ladder, and another further down on the right side. The pool’s surface has dark geometric patterns etched into it, and the surrounding area includes low stone railings and green trees visible behind the structures. In the far background, distant hills rise under a clear blue sky, and the entire scene is bathed in bright sunlight. The architecture resembles ancient Greek or Roman design, with detailed carvings on the columns and pediment.

Pixel+LILIE sample for: A large, rectangular swimming pool with clear blue water is centered in the scen — longA large, rectangular swimming pool with clear blue water is centered in the scene, reflecting the bright sky above. To the left stands a grand stone structure with tall columns and a triangular pediment topped by sculpted figures, while to the right is a matching colonnade with a balcony and a small statue on its steps. Two white marble statues of winged figures sit near the pool’s edge—one on the left beside a metal ladder, and another further down on the right side. The pool’s surface has dark geometric patterns etched into it, and the surrounding area includes low stone railings and green trees visible behind the structures. In the far background, distant hills rise under a clear blue sky, and the entire scene is bathed in bright sunlight. The architecture resembles ancient Greek or Roman design, with detailed carvings on the columns and pediment.

Pixel sample for: A wide, calm lake stretches across the scene under a deep blue twilight sky with — longA wide, calm lake stretches across the scene under a deep blue twilight sky with soft white clouds. On the left side, a wooden dock extends into the water, lit by warm yellow lights that reflect on the surface. Across the lake, a dense line of bare trees borders the far shore, silhouetted against the darkening sky. The water is smooth and glassy, mirroring the sky’s hue with gentle ripples. In the distance, a small structure with a roof sits near the shoreline, partially obscured by trees. The horizon shows a faint glow from distant lights, suggesting a town or park area beyond the treeline. The overall atmosphere is quiet and still, with no visible people or moving objects.

Pixel+LILIE sample for: A wide, calm lake stretches across the scene under a deep blue twilight sky with — longA wide, calm lake stretches across the scene under a deep blue twilight sky with soft white clouds. On the left side, a wooden dock extends into the water, lit by warm yellow lights that reflect on the surface. Across the lake, a dense line of bare trees borders the far shore, silhouetted against the darkening sky. The water is smooth and glassy, mirroring the sky’s hue with gentle ripples. In the distance, a small structure with a roof sits near the shoreline, partially obscured by trees. The horizon shows a faint glow from distant lights, suggesting a town or park area beyond the treeline. The overall atmosphere is quiet and still, with no visible people or moving objects.

BibTeX

@misc{agarwal2026lilie,
  title={A Language-Informed Loss for Visual Generation},
  author={Suchir Agarwal and Keshigeyan Chandrasegaran and Ziyu Chen and Kyle Sargent and Juan Carlos Niebles and Li Fei-Fei and Jiajun Wu},
  year={2026},
  eprint={2610.23653},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.23653},
}