fix: Allow abliterating VL models by anrp · Pull Request #108 · p-e-w/heretic

anrp · 2026-01-15T14:41:32Z

Per https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes, it indicates that "There is one class of AutoModel for each task." Since we're not using AutoModel (allowing it to pick the task automatically), expose a flag to allow for a different "task", which allows (at least) Qwen3-VL to be abliterated.

gemini-code-assist · 2026-01-15T14:41:47Z

Summary of Changes

Hello @anrp, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the system's capability to work with Visual Language (VL) models by introducing a configuration flag to identify them. By dynamically selecting the appropriate Hugging Face AutoModel class based on this flag, the system can now correctly load and process VL models, such as Qwen3-VL, which previously might not have been compatible with the default AutoModelForCausalLM class.

Highlights

New Configuration for VL Models: Introduced a new boolean setting, model_is_vl, in the Settings class to explicitly indicate if a model is a Visual Language (VL) model, defaulting to False.
Dynamic Model Class Selection: Implemented a new helper function get_model_class that dynamically returns either AutoModelForImageTextToText or AutoModelForCausalLM based on the model_is_vl setting, allowing for proper handling of different model types.
Flexible Model Loading: Modified all instances of model loading (from_pretrained) to use the new get_model_class function, ensuring that the correct AutoModel class is used for both Causal Language Models and Visual Language Models.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces support for Vision-Language (VL) models by adding a model_is_vl setting and a get_model_class helper function to select the appropriate AutoModel class. The changes are applied consistently where models are loaded. I have one comment regarding a style guide violation for a missing type hint.

A more critical concern is that the PEFT task_type appears to be hardcoded to CAUSAL_LM in src/heretic/model.py (lines 169 and 239). VL models, such as the Qwen-VL series mentioned in the context of this change, are often sequence-to-sequence models. These require task_type='SEQ_2_SEQ_LM' in the LoraConfig. Using the wrong task type will likely lead to incorrect behavior or errors when abliterating VL models. While this is outside the changed lines of this PR, it's a crucial point to address for this feature to work correctly.

src/heretic/model.py

p-e-w · 2026-01-16T07:22:44Z

Having a configuration flag for this is not the correct solution. This should be determined automatically.

anrp · 2026-01-16T11:17:32Z

Do you know if there's a reason that it wasn't just using AutoModel to begin with? I think I tried that before but got some vague error, but maybe I just did it wrong...

p-e-w · 2026-01-16T12:05:00Z

I never tried AutoModel. Does it work with VL models?

anrp · 2026-01-16T13:55:46Z

OK, per https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModel
AutoModel.from_pretrained("openai/gpt-oss-20b") returns a GptOssModel which does not have a "generate" method
AutoModel.from_pretrained("Qwen/Qwen3-VL-2B-Instruct") returns a Qwen3VLModel, same

So we need the "{model}ForConditionalGeneration" type as a return but per that page only the specific AutoModelFor{CausalLM,ImageTextToText,(others)} classes can build them. I think you have to pick the superclass?

anrp · 2026-01-16T14:02:39Z

Futher evidence that you have to choose the task https://discuss.huggingface.co/t/difference-between-automodel-and-automodelforlm/5967 https://insujang.github.io/2023-04-19/using-huggingface-transformers/

anrp · 2026-01-16T14:18:48Z

OK - figured out how to get the class from the config. No config needed now.

anrp · 2026-01-16T14:31:54Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a valuable enhancement by allowing the ablisteration of Vision-Language (VL) models. It achieves this by dynamically determining the appropriate AutoModel class based on the model's configuration, rather than hardcoding AutoModelForCausalLM. The implementation is clean and correctly applied throughout the model loading and manipulation logic. My review includes a couple of suggestions to further improve code clarity and ensure full adherence to the repository's style guide.

src/heretic/model.py

Vinay-Umrethe · 2026-01-16T19:37:20Z

I never tried AutoModel. Does it work with VL models?

That's the most generic class transformers have, as last fallback users can have after no class like AutoModelForCausalLM, AutoModelForImageTextToText, AutoModelForTextToWaveform etc don't work

Having that + AutoProcessor would truly make heretic support multi-modal

p-e-w · 2026-01-17T12:06:28Z

So is this working? Which VL models have you tested it with?

p-e-w · 2026-01-17T12:33:21Z

src/heretic/model.py



+def get_model_class(
+    settings: Settings,


We only use the model field from the settings here, so I think it's cleaner to just pass the model path directly instead of the entire settings object.

p-e-w · 2026-01-17T12:35:16Z

src/heretic/model.py

+    config_type = type(AutoConfig.from_pretrained(settings.model))
+    if config_type in MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING:
+        return AutoModelForImageTextToText
+    if config_type in MODEL_FOR_CAUSAL_LM_MAPPING:


Are those the same dicts that are used by AutoModelFor* internally? We don't want to lose the ability to load models that were previously supported.

https://github.com/huggingface/transformers/blob/57278c904c5158999d31a0db8bfcd63360c37b48/src/transformers/models/auto/modeling_auto.py#L1979
https://github.com/huggingface/transformers/blob/57278c904c5158999d31a0db8bfcd63360c37b48/src/transformers/models/auto/modeling_auto.py#L2183

Yes - exactly the same.

anrp · 2026-01-17T12:41:01Z

I've tested Qwen3-VL, I will test a few more (InternVL, Qwen2.5-VL), but given that it's the same data structure that transformers uses internally in AutoModel I don't expect trouble.

p-e-w · 2026-01-17T13:27:29Z

Let me know when you're satisfied that this works, then I'll merge.

anrp · 2026-01-17T17:05:44Z

Had to rework this as non-transformer-known-models (InternVL3, InternVL3.5) do not register for auto classes, but instead provide data in the config.json under auto_map (example: https://huggingface.co/OpenGVLab/InternVL3-1B/blob/main/config.json#L7). Also pulled param back to settings, as getting asked repeatedly about remote code was annoying.

Vinay-Umrethe · 2026-01-17T18:50:46Z

Had to rework this as non-transformer-known-models (InternVL3, InternVL3.5) do not register for auto classes, but instead provide data in the config.json under auto_map (example: https://huggingface.co/OpenGVLab/InternVL3-1B/blob/main/config.json#L7). Also pulled param back to settings, as getting asked repeatedly about remote code was annoying.

Transformers simply didn't merged those earlier versions at first but for InternVL 3.5 They already did it after sometime

"auto_map": {
"AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
"AutoModel": "modeling_internvl_chat.InternVLChatModel",
"AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
},

If it's there it would work with both the Auto Classes, just needs trust_remote_code=True which heretic already has

src/heretic/model.py

p-e-w · 2026-01-18T05:00:54Z

src/heretic/model.py

+
+    # If the model is not self registered, it may provide task class details, documented at
+    # https://huggingface.co/docs/transformers/en/custom_models#upload
+    maybe_auto_map = getattr(config, "auto_map", None)


Can't we get rid of all that complexity by checking for a vision_config key in the configuration? It seems that most or all VL models use that. Basically, we just load the config.json (which can be done without trust_remote_code), then check for the vision_config key. If it is present, we return AutoModelForImageTextToText, else we return AutoModelForCausalLM.

Per https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes, it indicates that "There is one class of AutoModel for each task." Use the presence of "vision_config" in the config.json to determine which.

p-e-w · 2026-01-20T13:27:52Z

👍 This looks great now! This has got to be the cleanest way to solve the problem.

As far as I'm concerned, this is ready to merge. Are you satisfied that it works correctly with all/most VL models?

anrp · 2026-01-20T15:10:58Z

I'll test with some models I know can come in under both superclasses (gemma3) as well as some others.

anrp · 2026-01-20T19:49:31Z

Tested working openai/gpt-oss-20b, google/gemma-3-4b-it, Qwen/Qwen3-0.6B, Qwen/Qwen3-VL-2B-Instruct, Qwen/Qwen2.5-VL-3B-Instruct, OpenGVLab/InternVL3_5-1B-hf

anrp · 2026-01-20T19:50:43Z

A couple are weird; allenai/Molmo2-8B uses "vit_config" and not "vision_config", mistralai/Ministral-3-3B-Instruct-2512 complains about chat user-assistant ordering (not related to this change)

p-e-w · 2026-01-22T13:35:24Z

Please give me a clear signal when this is ready to merge.

anrp · 2026-01-22T17:41:42Z

This is a strict improvement, and the failures are unrelated (Molmo doesn't support system prompt, ministral 2512 needs transformers 5).

p-e-w · 2026-01-23T14:05:01Z

Ok, merged. This is a great improvement!

anrp mentioned this pull request Jan 15, 2026

[Feature/Success Report with workaround] Support for Vision Models (Qwen2.5-VL) by switching to AutoModelForVision2Seq #99

Closed

gemini-code-assist bot reviewed Jan 15, 2026

View reviewed changes

src/heretic/model.py Outdated Show resolved Hide resolved

anrp force-pushed the anrp/vlsetting branch 3 times, most recently from 9259582 to 9d41693 Compare January 15, 2026 19:10

anrp force-pushed the anrp/vlsetting branch from 9d41693 to 5591c3e Compare January 16, 2026 14:18

anrp force-pushed the anrp/vlsetting branch 5 times, most recently from 1d18920 to 2cdef6b Compare January 16, 2026 14:29

gemini-code-assist bot reviewed Jan 16, 2026

View reviewed changes

src/heretic/model.py Outdated Show resolved Hide resolved

src/heretic/model.py Show resolved Hide resolved

anrp force-pushed the anrp/vlsetting branch 2 times, most recently from 40aab65 to 95a229a Compare January 16, 2026 14:39

p-e-w reviewed Jan 17, 2026

View reviewed changes

anrp force-pushed the anrp/vlsetting branch from 95a229a to b13966b Compare January 17, 2026 12:44

anrp force-pushed the anrp/vlsetting branch from b13966b to b8bf3b5 Compare January 17, 2026 17:03

p-e-w reviewed Jan 18, 2026

View reviewed changes

qfix: Allow abliterating VL models

9242e9b

Per https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes, it indicates that "There is one class of AutoModel for each task." Use the presence of "vision_config" in the config.json to determine which.

anrp force-pushed the anrp/vlsetting branch from b8bf3b5 to 9242e9b Compare January 19, 2026 13:08

p-e-w merged commit d5c834c into p-e-w:master Jan 23, 2026
4 checks passed

p-e-w mentioned this pull request Jan 23, 2026

Qwen3 32b VL support? #18

Closed

Conversation

anrp commented Jan 15, 2026

Uh oh!

gemini-code-assist bot commented Jan 15, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

p-e-w commented Jan 16, 2026

Uh oh!

anrp commented Jan 16, 2026

Uh oh!

p-e-w commented Jan 16, 2026

Uh oh!

anrp commented Jan 16, 2026

Uh oh!

anrp commented Jan 16, 2026

Uh oh!

anrp commented Jan 16, 2026

Uh oh!

anrp commented Jan 16, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Vinay-Umrethe commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

p-e-w commented Jan 17, 2026

Uh oh!

p-e-w Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

anrp Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

p-e-w Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

anrp Jan 17, 2026

Choose a reason for hiding this comment

Uh oh!

anrp commented Jan 17, 2026

Uh oh!

p-e-w commented Jan 17, 2026

Uh oh!

anrp commented Jan 17, 2026

Uh oh!

Vinay-Umrethe commented Jan 17, 2026

Uh oh!

Uh oh!

p-e-w Jan 18, 2026

Choose a reason for hiding this comment

Uh oh!

anrp Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

p-e-w commented Jan 20, 2026

Uh oh!

anrp commented Jan 20, 2026

Uh oh!

anrp commented Jan 20, 2026

Uh oh!

anrp commented Jan 20, 2026

Uh oh!

p-e-w commented Jan 22, 2026

Uh oh!

anrp commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

p-e-w commented Jan 23, 2026

Vinay-Umrethe commented Jan 16, 2026 •

edited

Loading

anrp commented Jan 22, 2026 •

edited

Loading