PreferThinker: Reasoning-Based Personalized Image Preference Assessment

Shengqi Xu, Xinpeng Zhou, Yabo Zhang, Ming Liu, Tao Liang, Tianyu Zhang, Yalong Bai, Zuxuan Wu, Wangmeng Zuo†

Code(coming soon) Paper Weight(coming soon) Data(coming soon)

Reasoning-based Personalized Assessment

Watch PreferThinker analyze and reason step-by-step

Preferred

Preferred Image 1
Processing...
Preferred Image 2
Processing...
Preferred Image 3
Processing...

Non-preferred

Non-preferred Image 1
Processing...
Non-preferred Image 2
Processing...
Non-preferred Image 3
Processing...

Candidate Images

Image A

Candidate Image A
Processing...

Image B

Candidate Image B
Processing...
Analyzing image preferences...
PreferThinker AI Reasoning
Human:

Personalized Preference Assessment

The practical yet challenging task of ​​personalized image preference assessment​​ aims to align generative models with individual user tastes. Its core challenges are the ​​limited amount of personalized data​​ per user and the ​​complex, ill-defined nature​​ of personal preferences (e.g., style, color), which make it difficult for models trained on general preferences to handle effectively.

Existing methods (CLIP-based or MLLM-based) struggle with personalization. They either lack interpretability or fail to explicitly leverage the critical prior information in a user's reference images. Merely feeding references to a model (like ViPer) is an ​​implicit​​ use that lacks interpretable reasoning steps​ to explain how the final assessment is derived from the individual's specific preferences. Therefore, a reasoning-based approach is necessary to make the assessment process transparent, grounded in the user's data, and multi-dimensional.

We propose ​​PreferThinker​​, a reasoning-based system with ​​preference profile prediction​​. Its core insight is to bridge users via a common set of visual elements (color, style, etc.).

Personalized Preference Assessment Illustration

Illustration of the motivation of personalized preference assessment.

Personalized Preference Dataset

Personalized Preference Dataset

Illustration of the proposed dataset PreferImg-CoT.

To address the difficulty in acquiring personalized data with annotated preference profiles, we constructed a large-scale dataset with ​​80K simulated users​​, each having a unique preference profile:

  • ​Preference Profile Generation​​: ​Randomly sampled five ​​visual preference elements​​ to assign preference/non-preference profiles, 20% of users (16K) assigned multiple profiles​​ to simulate real-world multi-preference
  • ​​Image Generation​​: Fed profiles and initial prompts into ​​text-to-image models​​ (e.g., Stable Diffusion) to generate ​​Reference images​​ (preferred/non-preferred pairs) and ​​Candidate images​​ (for evaluation), Selected ​​190K initial prompts​​ from ​​Lexica​​, ​​DiffusionDB​​, and ​​COCO​​ to ensure diversity
  • ​Dataset Scale​​: Finalized an annotated dataset with ​​80K user profiles​​ and ​​1.36 million images​​, laying the foundation for CoT-style dataset construction

Training Strategy and Proposed Prediction Reward

Illustration of training strategy and proposed prediction reward

Illustration of training strategy and proposed prediction reward

We employ a two-stage training strategy to elicit and incentivize the model’s structured reasoning capabilities.

Stage 1: Supervised Fine-tuning for Cold-start Initialization.

In Stage 1, the model Qwen2.5-VL-7B is initialized via supervised fine-tuning (SFT) using the PreferImg-CoT dataset. It learns to predict user preference profiles from reference images and uses them to generate interpretable scores for candidate images. Training employs autoregressive language modeling with token-level cross-entropy loss.

Stage 2: GRPO-based Reinforcement Learning for Post-training.

After SFT, Group Relative Policy Optimization (GRPO) is applied for reinforcement learning-based post-training. This method enhances generalization by generating diverse reasoning paths without a critic model. It calculates group-relative advantages from rewards of multiple responses and optimizes policies with a KL divergence constraint to balance exploration and stability.

Similarity-aware Prediction Reward Design.

To enhance the accuracy of the model’s prediction of user preference profiles, a dual-modality reward mechanism is proposed. It combines text similarity (computed via SBERT) between predicted and ground-truth preference/non-preference profiles, and image similarity (computed via DreamSim) between images generated from predicted and ground-truth profiles using a T2I model. The final reward is a weighted sum of both similarities.

Together, these two stages and the proposed reward result in a model that is not only accurate, but also interpretable and trustworthy in its reasoning process.

More Examples

Explore additional examples of PreferThinker's reasoning capabilities across diverse scenarios.

Preferred 1 Preferred 2 Preferred 3 Preferred 4 Preferred 5 Preferred 6
Non-Preferred 1 Non-Preferred 2 Non-Preferred 3 Non-Preferred 4 Non-Preferred 5 Non-Preferred 6
Candidate A A
Candidate B B
Preferred 1 Preferred 2 Preferred 3 Preferred 4 Preferred 5 Preferred 6
Non-Preferred 1 Non-Preferred 2 Non-Preferred 3 Non-Preferred 4 Non-Preferred 5 Non-Preferred 6
Candidate A A
Candidate B B
Preferred 1 Preferred 2 Preferred 3 Preferred 4 Preferred 5 Preferred 6
Non-Preferred 1 Non-Preferred 2 Non-Preferred 3 Non-Preferred 4 Non-Preferred 5 Non-Preferred 6
Candidate A A
Candidate B B
Preferred 1 Preferred 2 Preferred 3 Preferred 4 Preferred 5 Preferred 6
Non-Preferred 1 Non-Preferred 2 Non-Preferred 3 Non-Preferred 4 Non-Preferred 5 Non-Preferred 6
Candidate A A
Candidate B B

Preferred Images

Preferred 1 Preferred 2 Preferred 3 Preferred 4 Preferred 5 Preferred 6

Non-preferred Images

Non-preferred 1 Non-preferred 2 Non-preferred 3 Non-preferred 4 Non-preferred 5 Non-preferred 6

Candidate Images

Candidate A A
Candidate B B

Reasoning Process

Question: 
Given a user's reference images (both preferred and non-preferred images) and two candidate images, first predict the user's visual preference and non-preference profile based on the reference images, then assess which of the two candidate images better matches the user's personlized preference.

Answer:

<visual preference profile>
Art Style: Indian Art
Color: Fern Green
Art Medium: Ceramic
Saturation: Natural
Detail: Textured
</visual preference profile>
<visual non-preference profile>
Art Style: Naive Art
Color: Slate Blue
Art Medium: Pastel
Saturation: Vivid
Detail: Intricate
</visual non-preference profile>

1.Art Style:
Image A (Score: 4): The intricate mandala-like pattern with symmetrical design and spiritual symbolism reflects Indian Art aesthetics.
Image B (Score: 2): The simplified, childlike representation with disproportionate features aligns with non-preferred Naive Art.
2.Color:
Image A (Score: 4): Dominant earthy greens and natural tones closely match preferred Fern Green.
Image B (Score: 2): Prominent slate blue background corresponds to non-preferred Slate Blue.
3.Art Medium:
Image A (Score: 4): Solid, dimensional quality with visible textures suggests ceramic construction, aligning with preferred medium.
Image B (Score: 2): Soft blending and chalky application indicate pastel usage, matching non-preferred medium.
4.Saturation:
Image A (Score: 4): Colors appear naturally balanced without artificial enhancement, fitting natural saturation preference.
Image B (Score: 3): Moderately saturated blues lean toward vivid (non-preferred), but other elements are subdued.
5.Detail:
Image A (Score: 5): Richly textured surfaces with carved/molded techniques embody preferred textured detail.
Image B (Score: 2): Relies on line-based details rather than texture, aligning closer to non-preferred intricate detail.
Total Score:
Image A: 4 + 4 + 4 + 4 + 5 = 21
Image B: 2 + 2 + 2 + 3 + 2 = 9

Image A

Preferred Images

Preferred 1 Preferred 2 Preferred 3 Preferred 4 Preferred 5 Preferred 6

Non-preferred Images

Non-preferred 1 Non-preferred 2 Non-preferred 3 Non-preferred 4 Non-preferred 5 Non-preferred 6

Candidate Images

Candidate A A
Candidate B B

Reasoning Process

Question: 
Given a user's reference images (both preferred and non-preferred images) and two candidate images, first predict the user's visual preference and non-preference profile based on the reference images, then assess which of the two candidate images better matches the user's personlized preference.

Answer:

<visual preference profile>
Art Style: Conceptual Art
Color: Electric Lime
Art Medium: Paper Mache
Saturation: Grayish
Detail: Simplified
</visual preference profile>
<visual non-preference profile>
Art Style: Impressionism
Color: Burgundy Wine
Art Medium: Watercolor
Saturation: Vibrant
Detail: Rough
</visual non-preference profile>

1.Art Style:
Image A (Score: 2): The cartoon character under a large red flower petal leans toward illustrative design and narrative-focused storytelling, resembling Impressionism rather than Conceptual Art.
Image B (Score: 4): The black cat among oversized lotus leaves subverts proportions and challenges viewer expectations, aligning with Conceptual Art’s provocative and intellectual framework.
2.Color:
Image A (Score: 1): Dominant burgundy-red petals match the non-preference for Burgundy Wine, overshadowing hints of lime-green.
Image B (Score: 5): Prominent electric lime green lotus leaves perfectly match the preferred color, standing out against a neutral background.
3.Art Medium:
Image A (Score: 2): Fluid, transparent qualities suggest watercolor techniques, which are non-preferred.
Image B (Score: 4): Sculptural, dimensional solidity resembles molded paper mache, aligning with the preferred medium.
4.Saturation:
Image A (Score: 1): Highly vibrant, rich colors contradict the preference for grayish, subdued saturation.
Image B (Score: 4): Overall toned-down presentation with a pale background aligns with grayish, less saturated preference.
5.Detail:
Image A (Score: 3): Moderate detailing with simplified character but detailed background, falling between rough and elaborate.
Image B (Score: 4): Deliberately reduced details focus on shape and silhouette, matching the preference for simplified representations.
Total Score:
Image A: 2 + 1 + 2 + 1 + 3 = 9
Image B: 4 + 5 + 4 + 4 + 4 = 21

Image B

Preferred Images

Preferred 1 Preferred 2 Preferred 3 Preferred 4 Preferred 5 Preferred 6

Non-preferred Images

Non-preferred 1 Non-preferred 2 Non-preferred 3 Non-preferred 4 Non-preferred 5 Non-preferred 6

Candidate Images

Candidate A A
Candidate B B

Reasoning Process

Question: 
Given a user's reference images (both preferred and non-preferred images) and two candidate images, first predict the user's visual preference and non-preference profile based on the reference images, then assess which of the two candidate images better matches the user's personlized preference.

Answer:

<visual preference profile>
Art Style: Conceptual Art
Color: Mint Green
Art Medium: Macrame
Saturation: Grayish
Detail: Textured
</visual preference profile>
<visual non-preference profile>
Art Styles: Abstract Expressionism
Color: Dusty Orange
Art Medium: Drypoint
Saturation: Vibrant
Detail: Simplified
</visual non-preference profile>

1. Art style:​​
Image A (4) - The mint-colored mouse character sits atop what appears to be a stone pedestal with rope details, presenting a conceptual juxtaposition where everyday objects become symbolic representations through deliberate placement and context. This thoughtful arrangement aligns well with Conceptual Art principles focusing on ideas over pure aesthetics;
Image B (2) - The orange-hued mouse against a warm-toned background exhibits gestural brushstrokes and spontaneous markmaking qualities characteristic of Abstract Expressionism. The loose, emotive application of paint with expressive line work reflects the non-preferred abstract expressionist aesthetic rather than conceptually driven approaches.
​​2. Color:​​
Image A (5) - The dominant mint green color palette perfectly matches the preferred color attribute. From the soft teal-green background to the pale mint hues of the mouse itself, every element incorporates various shades of mint green, creating a cohesive and pleasing color scheme that strongly aligns with stated preferences;
Image B (1) - The image prominently features dusty orange tones throughout the background and parts of the mouse's body, precisely matching the non-preferred color attribute. These warm amber-orange hues dominate the composition, creating exactly the type of color environment specified as undesirable.
​​3. Art Medium:​​
Image A (4) - While primarily appearing as a digital illustration, the image incorporates actual physical textures like the woven rope around the base of the pedestal and the tactile-looking fur of the mouse. These tangible material elements evoke the dimensional quality of macrame, suggesting craft-based creation methods despite being digitally rendered;
Image B (2) - The image uses thin lines and sharp contrasts reminiscent of drypoint etching techniques, particularly evident in the defined outlines of the mouse figure. This linear approach with controlled mark-making shares similarities with the non-preferred drypoint medium, lacking the dimensional textile quality of macrame.
​​4. Saturation:​​
Image A (4) - The image employs a deliberately muted color palette with softened mint greens and subtle grays in the background, creating a subdued atmosphere. This restrained use of color intensity aligns well with the preferred grayish saturation level, allowing form and texture to take precedence over color strength;
Image B (2) - The image features highly vibrant colors with strong chromatic intensity, particularly in the bright orange background and vivid pink ear accents. This heightened color energy directly corresponds to the non-preferred vibrant saturation attribute, making the image visually stimulating but contrary to stated preferences.
​​5. Detail:​​
Image A (5) - The mouse illustration showcases rich textural elements including the intricate whiskers, fuzzy fur texture, and the dimensional rope wrapping around the pedestal. These tactile surfaces invite closer inspection, offering multiple layers of visual information that reward close viewing—perfectly embodying the preferred textured detailing;
Image B (2) - Despite having recognizable subjects, many elements appear simplified with minimal textural variation. The mouse's face and body lack nuanced surface details, instead relying on basic shapes and limited tonal shifts. This streamlined approach aligns with the non-preferred simplified detail attribute.
​​Total Score:​​
Image A: 4 + 4 + 4 + 4 + 5 = 21
Image B: 2 + 2 + 2 + 3 + 2 = 9

Image A

Preferred Images

Preferred 1 Preferred 2 Preferred 3 Preferred 4 Preferred 5 Preferred 6

Non-preferred Images

Non-preferred 1 Non-preferred 2 Non-preferred 3 Non-preferred 4 Non-preferred 5 Non-preferred 6

Candidate Images

Candidate A A
Candidate B B

Reasoning Process

Question: 
Given a user's reference images (both preferred and non-preferred images) and two candidate images, first predict the user's visual preference and non-preference profile based on the reference images, then assess which of the two candidate images better matches the user's personlized preference.

Answer:

<visual preference profile>
Art Style: Conceptual Art
Color: Electric Lime
Art Medium: Paper Mache
Saturation: Grayish
Detail: Simplified
</visual preference profile>
<visual non-preference profile>
Art Style: Impressionism
Color: Burgundy Wine
Art Medium: Watercolor
Saturation: Vibrant
Detail: Rough
</visual non-preference profile>

1.Art Style:
Image A (Score: 2): The cartoon character under a large red flower petal leans toward illustrative design and narrative-focused storytelling, resembling Impressionism rather than Conceptual Art.
Image B (Score: 4): The black cat among oversized lotus leaves subverts proportions and challenges viewer expectations, aligning with Conceptual Art’s provocative and intellectual framework.
2.Color:
Image A (Score: 1): Dominant burgundy-red petals match the non-preference for Burgundy Wine, overshadowing hints of lime-green.
Image B (Score: 5): Prominent electric lime green lotus leaves perfectly match the preferred color, standing out against a neutral background.
3.Art Medium:
Image A (Score: 2): Fluid, transparent qualities suggest watercolor techniques, which are non-preferred.
Image B (Score: 4): Sculptural, dimensional solidity resembles molded paper mache, aligning with the preferred medium.
4.Saturation:
Image A (Score: 1): Highly vibrant, rich colors contradict the preference for grayish, subdued saturation.
Image B (Score: 4): Overall toned-down presentation with a pale background aligns with grayish, less saturated preference.
5.Detail:
Image A (Score: 3): Moderate detailing with simplified character but detailed background, falling between rough and elaborate.
Image B (Score: 4): Deliberately reduced details focus on shape and silhouette, matching the preference for simplified representations.
Total Score:
Image A: 2 + 1 + 2 + 1 + 3 = 9
Image B: 4 + 5 + 4 + 4 + 4 = 21

Image B

Acknowledgments

We would like to express our gratitude to the open source community for their valuable contributions and inspiration.

This website design is inspired by and adapted from the Rex-Thinker project website, which provides an excellent template for academic paper presentations with interactive demonstrations and modern web design.

We also thank the developers and contributors of the following open-source projects and resources:

  • Font Awesome for providing excellent icons
  • GitHub Pages for free hosting
  • The open source community for inspiration and tools