diff --git a/README.md b/README.md index b4ab2f5ab..1337c5b92 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ API and command-line option may change frequently.*** - [FLUX.2-dev/FLUX.2-klein](./docs/flux2.md) - [FLUX.3 Image Klein / Klein Base](./docs/flux3.md) - [MOSS-Image-Base / MOSS-Image-SFT](./docs/moss_image.md) + - [Fibo 2.0 Dev](./docs/fibo2.md) - [Lens](./docs/lens.md) - [Chroma](./docs/chroma.md) - [Chroma1-Radiance](./docs/chroma_radiance.md) @@ -79,6 +80,7 @@ API and command-line option may change frequently.*** - [Boogu Image Edit](./docs/boogu_image.md) - [Mage-Flow-Edit](./docs/mage_flow.md#image-editing) - [LLaDA-Image Edit](./docs/llada_image.md#image-editing) + - [Fibo 2.0 Dev / Edit Turbo](./docs/fibo2.md#editing-and-turbo) - Video Models - [Wan2.1/Wan2.2](./docs/wan.md) - [MiniMax-H3](./docs/minimax_h3.md) diff --git a/assets/fibo2/dev_edit.png b/assets/fibo2/dev_edit.png new file mode 100644 index 000000000..fd6e06be1 Binary files /dev/null and b/assets/fibo2/dev_edit.png differ diff --git a/assets/fibo2/t2i.png b/assets/fibo2/t2i.png new file mode 100644 index 000000000..688ffb36a Binary files /dev/null and b/assets/fibo2/t2i.png differ diff --git a/docs/adetailer.md b/docs/adetailer.md index 2bc64df2c..f42831823 100644 --- a/docs/adetailer.md +++ b/docs/adetailer.md @@ -1,9 +1,10 @@ # ADetailer `sd-cli` can run a YOLOv8 object detector on an existing or newly generated -image and perform a cropped inpaint pass for every detected object. The first -implementation supports YOLOv8 detection checkpoints. YOLOv8 segmentation and -MediaPipe models are not supported yet. +image, and `sd-server` can run it after image generation. Both perform a cropped +inpaint pass for every detected object. The implementation supports YOLOv8 +detection checkpoints. YOLOv8 segmentation and MediaPipe models are not +supported yet. ## Convert a detector @@ -68,6 +69,36 @@ ADetailer can also run automatically after normal image generation: --extra-ad-args "confidence=0.3,denoising_strength=0.4,inpaint_width=512,inpaint_height=512" ``` +The same options can be supplied when starting `sd-server`. When `--ad-model` is set, its detector runs after OpenAI-compatible and native asynchronous image generation. SDAPI requests enable ADetailer through `alwayson_scripts`, as described below. If no object is detected, the original image is returned unchanged. RGBA output keeps its original alpha channel, including when detected regions are repaired. Detector initialization or detail-pass failures are reported as request failures. + +Server image-generation requests limit base, hires, and detail-pass dimensions to 4096 pixels per side. ADetailer step overrides are limited to 1–100; `steps=0` in `extra_ad_args` keeps inheriting the main generation steps. These limits also apply to inherited dimensions and `extra_ad_args` overrides. Requests exceeding the limits return a parameter error before generation. CLI limits are unchanged. + +### SDAPI requests + +`/sdapi/v1/txt2img` and `/sdapi/v1/img2img` accept the WebUI ADetailer argument format: + +```json +{ + "prompt": "portrait photo", + "alwayson_scripts": { + "ADetailer": { + "args": [true, false, { + "ad_model": "face_yolov8n.pt", + "ad_prompt": "[PROMPT], detailed face", + "ad_use_steps": true, + "ad_steps": 20 + }] + } + } +} +``` + +The detector must be configured at server startup with `--ad-model`. `ad_model` must match that file's basename; the original `.pt` basename is also accepted for a converted detector. Requests cannot load arbitrary detector paths. Omitting the script, setting its first argument to `false`, selecting `"None"`, or setting `ad_tab_enable` to `false` disables detailing. Dictionary-only argument arrays are also accepted. Multiple enabled dictionaries run in order using the configured detector. + +Supported settings include prompts, confidence, mask filtering by area or confidence, mask count and area ratios, dilation/erosion, offsets, mask merging/inversion, blur, denoising strength, masked or full-image inpainting, padding, and optional overrides for inpaint dimensions, steps, CFG, sampler/scheduler, and CLIP skip. The corresponding `ad_use_*` flags control whether overrides apply. Otherwise dimensions, steps, CFG, sampler, scheduler, and CLIP skip inherit the main request. Denoising strength defaults to `0.4`. Pixel mask settings are limited to the larger output dimension, with a minimum limit of 32 and a maximum of 16384 pixels. + +Model/VAE switching, non-default noise multipliers, face restoration, ControlNet, and skipping the initial img2img pass are not supported; active requests for these features return an error. Unknown settings and invalid active values also return errors. During detailing, `/sdapi/v1/progress` remains active with an `adetailer` job; its percentage and preview stay fixed until detailing finishes. + An empty ADetailer prompt inherits the main prompt. `[PROMPT]` inserts the main prompt, `[SEP]` assigns different prompts to consecutive masks, and `[SKIP]` skips the corresponding mask. @@ -82,6 +113,7 @@ All settings other than the detector path and prompts are passed through | `nms` | `0.45` | NMS IoU threshold | | `max_detections` | `100` | Maximum detections retained after NMS | | `mask_k_largest` | `0` | Keep only the largest K masks; zero keeps all | +| `mask_filter_method` | `area` | Rank retained masks by `area` or `confidence` | | `mask_min_ratio` | `0` | Minimum bbox area relative to the image | | `mask_max_ratio` | `1` | Maximum bbox area relative to the image | | `dilate_erode` | `4` | Positive values dilate; negative values erode | @@ -90,6 +122,7 @@ All settings other than the detector path and prompts are passed through | `merge_masks`, `invert_mask` | `false` | Boolean alternatives to `mask_mode` | | `mask_blur` | `4` | Final composite feather radius | | `inpaint_padding` | `32` | Padding around the detected region | +| `inpaint_only_masked` | `true` | Crop to the detected region; `false` uses the full image with the detection mask | | `inpaint_width`, `inpaint_height` | mode-specific | `512x512` after generation; input/output size in `adetailer` mode | | `denoising_strength` | mode-specific | `0.4` after generation; inherits `--strength` in `adetailer` mode | | `steps` | `0` | Detail steps; zero inherits the main generation | diff --git a/docs/edit.md b/docs/edit.md index 62ecff2af..9f33862d2 100644 --- a/docs/edit.md +++ b/docs/edit.md @@ -19,6 +19,7 @@ Depending on the architecture, different models handle reference images differen | **Krea2 (Community Edit LoRAs)** | `krea2_ostris_edit` | | [**Mage-Flow-Edit**](./mage_flow.md#image-editing) | `mage_flow` | | [**LLaDA-Image**](./llada_image.md#image-editing) | `llada_image` | +| [**Fibo 2.0 Dev / Edit Turbo**](./fibo2.md#examples) | `fibo2` | | **Anima (Community Edit LoRAs)** | `cosmos_reference` | Stable-diffusion.spp also supports basic Unet-based editing models like instruct-pix2pix or CosXL-Edit. This document is not about those. @@ -50,6 +51,7 @@ The `--ref-image-args` argument accepts a comma-separated list of key-value pair | `flux_kontext` | FLUX.1 Kontext | | `longcat` | LongCat Image Edit | | `flux2` | FLUX.2 models | +| `fibo2` | Fibo 2.0 Dev / Edit Turbo | | `qwen` | Qwen Image Edit | | `qwen_layered` | Qwen Image Layered | | `z_image_omni` | Boogu, Z-Image Omni | @@ -73,7 +75,12 @@ If presets are insufficient, you can manually configure the following parameters | `ref_index_mode` | string | Behavior of the RoPE index. | `fixed`, `increase`, `decrease` | | `force_ref_timestep_zero` | bool | Forces timestep=0 for reference tokens. | `true`, `false` (Krea2 only) | | `resize_before_vae` | bool | Whether reference images are resized before VAE encoding. | `true`, `false` | -| `vae_input_max_pixels` | int | Maximum pixel area for VAE reference inputs. | Integer | +| `resize_vae_to_target` | bool | Use output dimensions for every reference, overriding area-based sizing. | `true`, `false` | +| `resize_single_ref_to_target` | bool | Use output dimensions when exactly one reference is supplied. | `true`, `false` | +| `vae_size_buckets` | bool | Choose aligned dimensions within ±10% of the requested area, minimizing aspect-ratio cropping. Ignored when using output dimensions. | `true`, `false` | +| `vae_center_crop` | bool | Scale to cover the selected dimensions with Lanczos, then center-crop. Otherwise resize directly. | `true`, `false` | +| `vae_input_max_pixels` | int | VAE area budget, also capped by output area in ordinary resizing. With size buckets, this is the target area and output area does not cap it. | Integer | +| `vlm_use_vae_input` | bool | Reuse the resized/cropped VAE input for the VLM, before the VLM's own resize. Has no effect when `resize_before_vae=false`. | `true`, `false` | | `vlm_resize_mode` | string | How to resize VLM reference inputs. | `longest_side`, `area`, `none` | | `vlm_max_size` | int | Maximum VLM input size; interpreted according to `vlm_resize_mode`. | Integer | | `vlm_min_size` | int | Minimum VLM input size; interpreted according to `vlm_resize_mode`. | Integer | @@ -88,6 +95,7 @@ For a technical overview of how each preset is configured, see the table below. | `flux_kontext` | No | `fixed` | `none` | | | `longcat` | Yes | `fixed` | `area` | | | `flux2` | No | `increase` | `none` | | +| `fibo2` | Yes | `increase` | `longest` | Shared reference crops; clean reference timesteps; see [Fibo 2.0](./fibo2.md) | | `qwen` | Yes | `increase` | `area` | | | `qwen_layered` | Yes | `decrease` | `area` | | | `mage_flow` | Yes | `increase` | `longest` | `vlm_max_size = 384`, VAE input resized to target | @@ -99,3 +107,5 @@ For a technical overview of how each preset is configured, see the table below. **Additional Default Notes:** - **VLM Input Sizes:** For most presets, `vlm_max_size` and `vlm_min_size` are set to `-1`, meaning the values are model-dependent and handled automatically. In `area` mode they represent pixel area; in `longest_side` mode they represent a side length in pixels. - **VAE Input Size:** `vae_input_max_pixels` defaults to $1024 \times 1024$ pixels (`1048576`). +- **VAE Geometry:** `resize_vae_to_target` defaults to `true` for Mage-Flow and LLaDA-Image. Fibo enables `resize_single_ref_to_target`, `vae_size_buckets`, and `vae_center_crop`. These options are otherwise `false`. Bucket dimensions use the model's alignment (normally 16, Qwen Image 32), with a minimum side of four alignment units; if no bucket fits, output dimensions are used. +- **Shared Encoder Input:** `vlm_use_vae_input` defaults to `true` for Fibo and Qwen Image 2.1, otherwise `false`. All these options can be overridden individually after selecting a preset. diff --git a/docs/fibo2.md b/docs/fibo2.md new file mode 100644 index 000000000..064d80089 --- /dev/null +++ b/docs/fibo2.md @@ -0,0 +1,53 @@ +# How to Use + +Fibo 2.0 uses a Fibo diffusion transformer, the standard FLUX.2 VAE, and Qwen3-VL 4B as the LLM text encoder. The Qwen tokenizer is built in. + +## Download weights + +- Download Fibo 2.0 Dev + - safetensors: https://huggingface.co/Comfy-Org/Fibo-2.0/tree/main/diffusion_models +- Download Fibo 2.0 Edit Turbo + - safetensors: https://huggingface.co/Comfy-Org/Fibo-2.0/tree/main/diffusion_models +- Download vae + - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-dev/tree/main +- Download Qwen3-VL 4B + - safetensors: https://huggingface.co/Comfy-Org/Fibo-2.0/tree/main/text_encoders + - gguf: https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct-GGUF/tree/main + +The original Diffusers weights from [briaai/Fibo-2.0-dev](https://huggingface.co/briaai/Fibo-2.0-dev) and [briaai/Fibo-2.0-edit-turbo](https://huggingface.co/briaai/Fibo-2.0-edit-turbo) are also supported. Pass the transformer's `diffusion_pytorch_model.safetensors.index.json` to `--diffusion-model` and keep its shards alongside it. + +INT8 convrot weights load directly; see [INT8 convrot](./int8_convrot.md) for backend requirements. + +## Examples + +### Fibo 2.0 Dev + +Dev supports both text-to-image generation and reference-image editing. + +``` +.\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\fibo2_dev_int8_convrot.safetensors --llm ..\models\text_encoders\Qwen3-VL-4B-Instruct-Q4_K_M.gguf --llm_vision ..\models\text_encoders\Qwen3-VL-4B-Instruct-mmproj-BF16.gguf --vae ..\models\vae\flux2_ae.safetensors -p "a lovely cat holding a sign says 'fibo2.cpp'" --cfg-scale 5.0 --sampling-method euler -v --offload-to-cpu +``` + + + +``` +.\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\fibo2_dev_int8_convrot.safetensors --llm ..\models\text_encoders\Qwen3-VL-4B-Instruct-Q4_K_M.gguf --llm_vision ..\models\text_encoders\Qwen3-VL-4B-Instruct-mmproj-BF16.gguf --vae ..\models\vae\flux2_ae.safetensors -r ..\assets\flux\flux1-dev-q8_0.png -p "change 'flux.cpp' to 'fibo2.cpp'" --cfg-scale 5.0 --sampling-method euler -v --offload-to-cpu +``` + + + +### Fibo 2.0 Edit Turbo + +Edit Turbo is distilled for reference-image editing in 4 steps with CFG 1. + +``` +.\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\fibo2_edit_turbo_int8_convrot.safetensors --llm ..\models\text_encoders\Qwen3-VL-4B-Instruct-Q4_K_M.gguf --llm_vision ..\models\text_encoders\Qwen3-VL-4B-Instruct-mmproj-BF16.gguf --vae ..\models\vae\flux2_ae.safetensors -r ..\assets\flux\flux1-dev-q8_0.png -p "change 'flux.cpp' to 'fibo2.cpp'" --cfg-scale 1 --steps 4 --sampling-method euler -v --offload-to-cpu +``` + +The BF16 files accept the same options. Use dimensions divisible by 16. Repeat `-r` to use multiple reference images; instructions can refer to ``, ``, etc. See [Image Editing](./edit.md) for reference-image preprocessing options. Separate GGUF text encoders need their vision projector passed with `--llm_vision` for editing. + +The automatic `fibo2` reference preset combines `resize_single_ref_to_target=true`, `vae_size_buckets=true`, `vae_center_crop=true`, and `vlm_use_vae_input=true`. A single reference is scaled to cover the output dimensions and center-cropped. Multiple references use dimensions divisible by 16, chosen within ±10% of one megapixel to minimize cropping for each image's aspect ratio; `vae_input_max_pixels` overrides that target area. The VAE and vision encoder receive the same crops, with smaller copies used by the vision encoder. These are generic reference-image options and can be overridden individually: for example, `--ref-image-args vae_center_crop=false` uses direct resizing, while `resize_before_vae=false` skips this automatic crop and resize step entirely. + +The vision encoder defaults to `vlm_resize_mode=longest_side,vlm_max_size=768`. Override these with `--ref-image-args`, for example `vlm_max_size=512`, or use `vlm_resize_mode=none` to skip size-based scaling. Vision inputs are still aligned to the encoder's patch grid. + +Plain prompts and structured JSON captions are accepted. This integration uses the caption-only text-rendering path (`render_text_images=False`); it does not generate auxiliary glyph images or expand prompts with Bria VGL. Native Fibo mask preprocessing is not connected to `--mask`. diff --git a/docs/rpc.md b/docs/rpc.md index ea9edf1c3..cb28e256f 100644 --- a/docs/rpc.md +++ b/docs/rpc.md @@ -131,7 +131,7 @@ If you want to allow connections from other machines on the network: > **Drivers & Hardware:** Ensure the Server machine has the necessary drivers installed and functional (e.g., Nvidia Drivers for CUDA, Vulkan SDK, or Metal). If no devices are found, the server will simply fallback to CPU usage. - +``` + +The options can be given in either order; no model is required. The client exits +with an error if RPC support is unavailable or a server cannot provide devices. -### Step B: Run with RPC device +### Step C: Run with RPC device If everything is working correctly, you can now run the client while offloading some or all of the work to the RPC server. @@ -217,4 +217,4 @@ The client will map these servers to sequential device IDs (e.g., RPC0 from the ## 6. Performance Considerations -RPC performance is heavily dependent on network bandwidth, as large weights and activations must be transferred back and forth over the network, especially for large models, or when using high resolutions. For best results, ensure your network connection is stable and has sufficient bandwidth (>1Gbps recommended). This shoumd not be a concern if you are running the server and client on the same machine, as the data transfer will happen over the loopback interface. \ No newline at end of file +RPC performance is heavily dependent on network bandwidth, as large weights and activations must be transferred back and forth over the network, especially for large models, or when using high resolutions. For best results, ensure your network connection is stable and has sufficient bandwidth (>1Gbps recommended). This shoumd not be a concern if you are running the server and client on the same machine, as the data transfer will happen over the loopback interface. diff --git a/docs/taesd.md b/docs/taesd.md index 954086d18..8331f441b 100644 --- a/docs/taesd.md +++ b/docs/taesd.md @@ -49,3 +49,5 @@ curl -L -o taeqi2_1.safetensors https://huggingface.co/madebyollin/taeqi2_1/reso ``` Replace `--vae PATH` with `--taesd taeqi2_1.safetensors` in the [Qwen Image 2.1 examples](qwen_image_2.1.md) to use it for encoding and decoding. For previews only, keep `--vae PATH` and add `--taesd taeqi2_1.safetensors --taesd-preview-only --preview tae`. + +`sd-server` accepts the same `--taesd` and `--taesd-preview-only` options; the preview method is selected per request instead of via `--preview`. diff --git a/examples/cli/main.cpp b/examples/cli/main.cpp index ca8ce4e79..ee1e56f22 100644 --- a/examples/cli/main.cpp +++ b/examples/cli/main.cpp @@ -47,7 +47,6 @@ struct SDCliParams { int preview_interval = 1; std::string preview_path = "preview.png"; int preview_fps = 16; - bool taesd_preview = false; bool preview_noisy = false; bool color = false; bool metadata_raw = false; @@ -114,10 +113,6 @@ struct SDCliParams { "--color", "colors the logging tags according to level", true, &color}, - {"", - "--taesd-preview-only", - std::string("prevents usage of taesd for decoding the final image. (for use with --preview ") + previews_str[PREVIEW_TAE] + ")", - true, &taesd_preview}, {"", "--preview-noisy", "enables previewing noisy inputs of the models rather than the denoised outputs", @@ -268,7 +263,6 @@ struct SDCliParams { << " preview_interval: " << preview_interval << ",\n" << " preview_path: \"" << preview_path << "\",\n" << " preview_fps: " << preview_fps << ",\n" - << " taesd_preview: " << (taesd_preview ? "true" : "false") << ",\n" << " preview_noisy: " << (preview_noisy ? "true" : "false") << ",\n" << " imatrix_out: \"" << imatrix_out << "\",\n" << " metadata_raw: " << (metadata_raw ? "true" : "false") << ",\n" @@ -301,6 +295,10 @@ void parse_args(int argc, const char** argv, SDCliParams& cli_params, SDContextP log_level = cli_params.log_level; log_color = cli_params.color; + if (ctx_params.list_devices) { + exit(ctx_params.print_devices() ? 0 : 1); + } + bool valid = cli_params.resolve_and_validate(); if (valid && cli_params.mode != METADATA) { valid = ctx_params.resolve_and_validate(cli_params.mode) && @@ -881,7 +879,7 @@ int main(int argc, const char* argv[]) { } } - sd_ctx_params_t sd_ctx_params = ctx_params.to_sd_ctx_params_t(cli_params.taesd_preview); + sd_ctx_params_t sd_ctx_params = ctx_params.to_sd_ctx_params_t(); SDImageVec results; int num_results = 0; diff --git a/examples/common/common.cpp b/examples/common/common.cpp index 9e7851fef..df76c517a 100644 --- a/examples/common/common.cpp +++ b/examples/common/common.cpp @@ -596,6 +596,11 @@ ArgOptions SDContextParams::get_options() { }; options.bool_options = { + {"", + "--list-devices", + "list available ggml backend devices, including --rpc-servers (one 'namedescription' per line), and exit; " + "the names are the device names accepted by --backend and --params-backend", + true, &list_devices}, {"", "--disable-prefetch", "disable asynchronous next-segment weight prefetch (defaults to false)", @@ -636,6 +641,10 @@ ArgOptions SDContextParams::get_options() { "--sage-attn", "use native CUDA SageAttention in the diffusion model, with flash/default attention fallback", true, &sage_attn}, + {"", + "--taesd-preview-only", + "prevents usage of taesd for decoding the final image; the VAE keeps encoding / decoding and taesd is only used for step previews", + true, &taesd_preview_only}, }; auto on_type_arg = [&](int argc, const char** argv, int index) { @@ -819,23 +828,22 @@ ArgOptions SDContextParams::get_options() { "but it usually offers faster inference speed and, in some cases, lower memory usage. " "The at_runtime mode, on the other hand, is exactly the opposite.", on_lora_apply_mode_arg}, - {"", - "--list-devices", - "list available ggml backend devices (one 'namedescription' per line) and exit; " - "the names are the device names accepted by --backend and --params-backend", - [](int /*argc*/, const char** /*argv*/, int /*index*/) { - size_t device_list_size = sd_list_devices(nullptr, 0); - std::vector devices(device_list_size + 1); - sd_list_devices(devices.data(), devices.size()); - fputs(devices.data(), stdout); - std::exit(0); - return 0; - }}, }; return options; } +bool SDContextParams::print_devices() const { + if (!sd_add_rpc_servers(rpc_servers.c_str())) { + return false; + } + size_t device_list_size = sd_list_devices(nullptr, 0); + std::vector devices(device_list_size + 1); + sd_list_devices(devices.data(), devices.size()); + fputs(devices.data(), stdout); + return true; +} + void SDContextParams::build_embedding_map() { static const std::vector valid_ext = {".gguf", ".safetensors", ".pt", ".ckpt"}; @@ -1010,6 +1018,7 @@ std::string SDContextParams::to_string() const { << " attn_scale: " << attn_scale << ",\n" << " diffusion_conv_direct: " << (diffusion_conv_direct ? "true" : "false") << ",\n" << " vae_conv_direct: " << (vae_conv_direct ? "true" : "false") << ",\n" + << " taesd_preview_only: " << (taesd_preview_only ? "true" : "false") << ",\n" << " prediction: " << sd_prediction_name(prediction) << ",\n" << " lora_apply_mode: " << sd_lora_apply_mode_name(lora_apply_mode) << ",\n" << " force_sdxl_vae_conv_scale: " << (force_sdxl_vae_conv_scale ? "true" : "false") << "\n" @@ -1017,7 +1026,7 @@ std::string SDContextParams::to_string() const { return oss.str(); } -sd_ctx_params_t SDContextParams::to_sd_ctx_params_t(bool taesd_preview) { +sd_ctx_params_t SDContextParams::to_sd_ctx_params_t() { prepare_backend_assignments(); embedding_vec.clear(); embedding_vec.reserve(embedding_map.size()); @@ -1067,7 +1076,7 @@ sd_ctx_params_t SDContextParams::to_sd_ctx_params_t(bool taesd_preview) { sd_ctx_params.sage_attn = sage_attn; sd_ctx_params.linear_scale = linear_scale; sd_ctx_params.attn_scale = attn_scale; - sd_ctx_params.tae_preview_only = taesd_preview; + sd_ctx_params.tae_preview_only = taesd_preview_only; sd_ctx_params.diffusion_conv_direct = diffusion_conv_direct; sd_ctx_params.vae_conv_direct = vae_conv_direct; sd_ctx_params.force_sdxl_vae_conv_scale = force_sdxl_vae_conv_scale; @@ -2219,7 +2228,7 @@ bool SDGenerationParams::from_json_str( if (target_custom_sigmas != nullptr && sample_json.contains("custom_sigmas") && sample_json["custom_sigmas"].is_array() && - !sample_json["custom_sigmas"].empty()) { // empty [] = "not specified", keep server/CLI default + !sample_json["custom_sigmas"].empty()) { // empty [] = "not specified", keep server/CLI default *target_custom_sigmas = sample_json["custom_sigmas"].get>(); } if (sample_json.contains("sample_method") && sample_json["sample_method"].is_string()) { diff --git a/examples/common/common.h b/examples/common/common.h index 91e75c685..7b9f691af 100644 --- a/examples/common/common.h +++ b/examples/common/common.h @@ -168,6 +168,7 @@ struct SDContextParams { std::string model_args; bool auto_fit = true; std::string rpc_servers; + bool list_devices = false; std::string effective_backend; std::string effective_params_backend; bool enable_mmap = false; @@ -179,6 +180,7 @@ struct SDContextParams { bool sage_attn = false; bool diffusion_conv_direct = false; bool vae_conv_direct = false; + bool taesd_preview_only = false; prediction_t prediction = PREDICTION_COUNT; lora_apply_mode_t lora_apply_mode = LORA_APPLY_AUTO; @@ -189,13 +191,14 @@ struct SDContextParams { float flow_shift = INFINITY; ArgOptions get_options(); + bool print_devices() const; void build_embedding_map(); void prepare_backend_assignments(); bool resolve(SDMode mode); bool validate(SDMode mode); bool resolve_and_validate(SDMode mode); std::string to_string() const; - sd_ctx_params_t to_sd_ctx_params_t(bool taesd_preview); + sd_ctx_params_t to_sd_ctx_params_t(); }; struct SDGenerationParams { diff --git a/examples/server/api.md b/examples/server/api.md index bb4252366..fc64fb029 100644 --- a/examples/server/api.md +++ b/examples/server/api.md @@ -280,7 +280,8 @@ TAE slot. | `model` | `checkpoints` | | `diffusion_model`, `high_noise_diffusion_model`, `uncond_diffusion_model` | `diffusion_models` | | `clip_l`, `clip_g`, `t5xxl`, `llm`, `audio_encoder`, `embeddings_connectors` | `text_encoders` | -| `clip_vision`, `llm_vision` | `clip_vision` | +| `clip_vision` | `clip_vision` | +| `llm_vision` | `text_encoders` or `clip_vision` | | `vae`, `audio_vae` | `vae` | | `tae` (legacy alias: `taesd`) | `vae` or `vae_approx` | | `tokenizer` | `tokenizers` | @@ -302,11 +303,15 @@ new model architectures. Optional `context` settings apply to each model load and override the matching startup setting. Supported fields are `vae_format` (`auto`, `flux`, `sd3`, -`flux2`, or `wan`) and `model_args` (a string in the same format as -`--model-args`; replaces the startup string). Other startup settings remain -unchanged. Switching to a recipe that omits these fields restores the startup -values. For example, PiD uses `{"vae_format": "flux"}` and Qwen Image Edit -2511 uses `{"model_args": "qwen_image_zero_cond_t=true"}`. +`flux2`, or `wan`), `model_args` (a string in the same format as +`--model-args`; replaces the startup string), and `backend` / `params_backend` +(strings in the same format as `--backend` / `--params-backend`; replace the +startup assignments for that load, so a recipe can place components on +specific devices, e.g. keep a large text encoder on CPU while another recipe +uses the GPU). Other startup settings remain unchanged. Switching to a +recipe that omits these fields restores the startup values. For example, PiD +uses `{"vae_format": "flux"}` and Qwen Image Edit 2511 uses +`{"model_args": "qwen_image_zero_cond_t=true"}`. Supported defaults: @@ -1032,7 +1037,11 @@ Notes: #### `POST /sdcpp/v1/jobs/{id}/cancel` -Attempts to cancel an accepted job. +Cancels a queued job immediately, or requests interruption of a generating image or +video job. A generating job returns `200 OK` with `cancel_requested: true` and +remains `generating` until execution stops at a cancellation checkpoint. Continue +polling until its status becomes `cancelled`. Cancelled jobs have no result. +Repeated cancellation requests are safe; terminal jobs are returned unchanged. Typical status codes: diff --git a/examples/server/async_jobs.cpp b/examples/server/async_jobs.cpp index 7f3c8a1bf..3e40e506f 100644 --- a/examples/server/async_jobs.cpp +++ b/examples/server/async_jobs.cpp @@ -3,13 +3,46 @@ #include "async_jobs.h" #include +#include #include #include +#include #include "common/log.h" #include "common/media_io.h" #include "common/resource_owners.hpp" +class JobCancellationMonitor { + std::mutex mutex_; + std::condition_variable cv_; + bool done_ = false; + std::thread thread_; + +public: + // The caller holds sd_ctx_mutex until this monitor has joined, keeping the + // context alive and preventing cancellation from reaching another request. + JobCancellationMonitor(sd_ctx_t* ctx, const AsyncGenerationJob& job) + : thread_([this, ctx, &job]() { + std::unique_lock lock(mutex_); + while (!done_) { + if (job.cancel_requested.load()) { + // Each generation pass resets the library flag, including ADetailer. + sd_cancel_generation(ctx, SD_CANCEL_ALL); + } + cv_.wait_for(lock, std::chrono::milliseconds(10), [this]() { return done_; }); + } + }) {} + + ~JobCancellationMonitor() { + { + std::lock_guard lock(mutex_); + done_ = true; + } + cv_.notify_one(); + thread_.join(); + } +}; + static void preview_callback(int step, int frame_count, sd_image_t* frames, bool is_noisy, void* data) { (void)is_noisy; if (frame_count <= 0 || frames == nullptr || frames[0].data == nullptr) { @@ -142,6 +175,7 @@ bool cancel_queued_job(AsyncJobManager& manager, AsyncGenerationJob& job) { } manager.queue.erase(new_end, manager.queue.end()); + job.cancel_requested.store(true); job.status = AsyncJobStatus::Cancelled; job.completed_at = unix_timestamp_now_ms(); job.result_images_b64.clear(); @@ -156,13 +190,14 @@ bool cancel_queued_job(AsyncJobManager& manager, AsyncGenerationJob& job) { json make_async_job_json(const AsyncJobManager& manager, const AsyncGenerationJob& job) { json result; - result["id"] = job.id; - result["kind"] = async_job_kind_name(job.kind); - result["status"] = async_job_status_name(job.status); - result["created"] = job.created_at; - result["started"] = job.started_at == 0 ? json(nullptr) : json(job.started_at); - result["completed"] = job.completed_at == 0 ? json(nullptr) : json(job.completed_at); - result["queue_position"] = 0; + result["id"] = job.id; + result["kind"] = async_job_kind_name(job.kind); + result["status"] = async_job_status_name(job.status); + result["cancel_requested"] = job.cancel_requested.load(); + result["created"] = job.created_at; + result["started"] = job.started_at == 0 ? json(nullptr) : json(job.started_at); + result["completed"] = job.completed_at == 0 ? json(nullptr) : json(job.completed_at); + result["queue_position"] = 0; if (job.status == AsyncJobStatus::Queued) { size_t position = 1; @@ -235,6 +270,10 @@ bool execute_img_gen_job(ServerRuntime& runtime, { std::lock_guard lock(*runtime.sd_ctx_mutex); + if (job.cancel_requested.load()) { + return false; + } + JobCancellationMonitor cancellation(runtime.sd_ctx, job); set_preview_callback(job); sd_image_t* raw_results = nullptr; int num_results = 0; @@ -244,10 +283,17 @@ bool execute_img_gen_job(ServerRuntime& runtime, } clear_preview_callback(); results.adopt(raw_results, num_results); + if (job.cancel_requested.load()) { + return false; + } + if (!results.empty() && !apply_server_adetailer(runtime, job.img_gen, params, results)) { + error_message = "ADetailer failed"; + return false; + } } const int num_results = results.count(); - if (num_results <= 0) { + if (job.cancel_requested.load() || num_results <= 0) { error_message = "generate_image returned no results"; return false; } @@ -310,6 +356,10 @@ bool execute_vid_gen_job(ServerRuntime& runtime, { std::lock_guard lock(*runtime.sd_ctx_mutex); + if (job.cancel_requested.load()) { + return false; + } + JobCancellationMonitor cancellation(runtime.sd_ctx, job); set_preview_callback(job); sd_image_t* raw_results = nullptr; if (!generate_video(runtime.sd_ctx, ¶ms, &raw_results, &num_results, &generated_audio, &output_fps)) { @@ -320,7 +370,7 @@ bool execute_vid_gen_job(ServerRuntime& runtime, } num_results = results.count(); - if (num_results <= 0) { + if (job.cancel_requested.load() || num_results <= 0) { free_sd_audio(generated_audio); error_message = "generate_video returned no results"; return false; @@ -406,7 +456,11 @@ void async_job_worker(ServerRuntime& runtime) { } job->completed_at = unix_timestamp_now_ms(); - if (ok) { + if (job->cancel_requested.load()) { + job->status = AsyncJobStatus::Cancelled; + job->error_code = "cancelled"; + job->error_message = "job cancelled by client"; + } else if (ok) { job->status = AsyncJobStatus::Completed; job->result_images_b64 = std::move(output_images); job->result_media_b64 = std::move(output_media_b64); diff --git a/examples/server/async_jobs.h b/examples/server/async_jobs.h index 049a9d73b..aa54370b1 100644 --- a/examples/server/async_jobs.h +++ b/examples/server/async_jobs.h @@ -1,5 +1,6 @@ #pragma once +#include #include #include #include @@ -32,9 +33,10 @@ struct AsyncGenerationJob { std::string id; AsyncJobKind kind = AsyncJobKind::ImgGen; AsyncJobStatus status = AsyncJobStatus::Queued; - int64_t created_at = unix_timestamp_now_ms(); - int64_t started_at = 0; - int64_t completed_at = 0; + std::atomic cancel_requested{false}; + int64_t created_at = unix_timestamp_now_ms(); + int64_t started_at = 0; + int64_t completed_at = 0; ImgGenJobRequest img_gen; VidGenJobRequest vid_gen; std::vector result_images_b64; diff --git a/examples/server/frontend b/examples/server/frontend index e36e34a2e..34e5ad6ed 160000 --- a/examples/server/frontend +++ b/examples/server/frontend @@ -1 +1 @@ -Subproject commit e36e34a2e2204e3df654b16cadc112d8ce8f3174 +Subproject commit 34e5ad6edcaf827b7fe1e9ffa55ee86cd2fc43d2 diff --git a/examples/server/main.cpp b/examples/server/main.cpp index 170877686..a10cb66f1 100644 --- a/examples/server/main.cpp +++ b/examples/server/main.cpp @@ -48,6 +48,10 @@ static void parse_args(int argc, log_level = svr_params.log_level; log_color = svr_params.color; + if (ctx_params.list_devices) { + exit(ctx_params.print_devices() ? 0 : 1); + } + const bool random_seed_requested = default_gen_params.seed < 0; if (!svr_params.resolve_and_validate() || @@ -89,7 +93,7 @@ int main(int argc, const char** argv) { LOG_DEBUG("%s", ctx_params.to_string().c_str()); LOG_DEBUG("%s", default_gen_params.to_string().c_str()); - sd_ctx_params_t sd_ctx_params = ctx_params.to_sd_ctx_params_t(false); + sd_ctx_params_t sd_ctx_params = ctx_params.to_sd_ctx_params_t(); ServerModels models; if (!ctx_params.model_path.empty() || !ctx_params.diffusion_model_path.empty()) { const auto started = std::chrono::steady_clock::now(); diff --git a/examples/server/model_catalog.cpp b/examples/server/model_catalog.cpp index a3c59206d..8efe624b8 100644 --- a/examples/server/model_catalog.cpp +++ b/examples/server/model_catalog.cpp @@ -32,7 +32,7 @@ static const ModelSlot k_model_slots[] = { {"clip_vision", "CLIP Vision", {"clip_vision"}, &SDContextParams::clip_vision_path}, {"t5xxl", "T5", {"text_encoders"}, &SDContextParams::t5xxl_path}, {"llm", "LLM Text Encoder", {"text_encoders"}, &SDContextParams::llm_path}, - {"llm_vision", "LLM Vision Encoder", {"clip_vision"}, &SDContextParams::llm_vision_path}, + {"llm_vision", "LLM Vision Encoder", {"text_encoders", "clip_vision"}, &SDContextParams::llm_vision_path}, {"tokenizer", "Tokenizer", {"tokenizers"}, &SDContextParams::tokenizer}, {"vae", "VAE", {"vae"}, &SDContextParams::vae_path}, {"audio_vae", "Audio VAE", {"vae"}, &SDContextParams::audio_vae_path}, @@ -273,7 +273,7 @@ static void apply_recipe_defaults(const json& defaults, SDGenerationParams& para } static void apply_recipe_context(const json& settings, SDContextParams& params) { - check_keys(settings, {"vae_format", "model_args"}); + check_keys(settings, {"vae_format", "model_args", "backend", "params_backend"}); if (settings.contains("vae_format")) { const auto format = settings.at("vae_format").get(); if (format != "auto" && format != "flux" && format != "sd3" && format != "flux2" && format != "wan") @@ -283,6 +283,12 @@ static void apply_recipe_context(const json& settings, SDContextParams& params) if (settings.contains("model_args")) { params.model_args = settings.at("model_args").get(); } + if (settings.contains("backend")) { + params.backend = settings.at("backend").get(); + } + if (settings.contains("params_backend")) { + params.params_backend = settings.at("params_backend").get(); + } } static const std::map k_recipe_features = { diff --git a/examples/server/recipes/fibo2-dev.json b/examples/server/recipes/fibo2-dev.json new file mode 100644 index 000000000..aa7e81460 --- /dev/null +++ b/examples/server/recipes/fibo2-dev.json @@ -0,0 +1,39 @@ +{ + "schema_version": 1, + "id": "fibo2-dev", + "name": "Fibo 2.0 Dev", + "description": "Text-to-image generation and native reference-image editing. Select a separate vision projector for editing when it is not bundled in the text encoder.", + "model_url": "https://huggingface.co/briaai/Fibo-2.0-dev", + "slots": [ + { + "id": "diffusion_model", + "required": true + }, + { + "id": "llm", + "required": true, + "label": "Qwen 3 VL 4B Text Encoder" + }, + { + "id": "llm_vision", + "required": false, + "label": "Vision projector (required for separate GGUF text encoders)" + }, + { + "id": "vae", + "required": true, + "label": "FLUX.2 VAE" + } + ], + "defaults": { + "sampling_method": "euler", + "cfg_scale": 5, + "steps": 20, + "width": 1024, + "height": 1024 + }, + "features": [ + "flow_shift", + "ref_images" + ] +} diff --git a/examples/server/recipes/fibo2-edit-turbo.json b/examples/server/recipes/fibo2-edit-turbo.json new file mode 100644 index 000000000..b84e1cce4 --- /dev/null +++ b/examples/server/recipes/fibo2-edit-turbo.json @@ -0,0 +1,39 @@ +{ + "schema_version": 1, + "id": "fibo2-edit-turbo", + "name": "Fibo 2.0 Edit Turbo", + "description": "Dedicated reference-image editing checkpoint; supply reference images. Select a separate vision projector when it is not bundled in the text encoder.", + "model_url": "https://huggingface.co/briaai/Fibo-2.0-edit-turbo", + "slots": [ + { + "id": "diffusion_model", + "required": true + }, + { + "id": "llm", + "required": true, + "label": "Qwen 3 VL 4B Text Encoder" + }, + { + "id": "llm_vision", + "required": false, + "label": "Vision projector (required for separate GGUF text encoders)" + }, + { + "id": "vae", + "required": true, + "label": "FLUX.2 VAE" + } + ], + "defaults": { + "sampling_method": "euler", + "cfg_scale": 1, + "steps": 4, + "width": 1024, + "height": 1024 + }, + "features": [ + "flow_shift", + "ref_images" + ] +} diff --git a/examples/server/routes_models.cpp b/examples/server/routes_models.cpp index ea14eb821..9e4e3963c 100644 --- a/examples/server/routes_models.cpp +++ b/examples/server/routes_models.cpp @@ -94,7 +94,7 @@ static void change_model(ServerRuntime& runtime, const httplib::Request& req, ht *runtime.ctx_params = unload ? models.base_context : std::move(context); *runtime.default_gen_params = unload ? models.base_defaults : std::move(defaults); if (!unload) { - auto params = runtime.ctx_params->to_sd_ctx_params_t(false); + auto params = runtime.ctx_params->to_sd_ctx_params_t(); models.context.reset(new_sd_ctx(¶ms)); if (!models.context) { throw std::runtime_error("Model initialization failed; check the server log for component compatibility and memory errors"); diff --git a/examples/server/routes_openai.cpp b/examples/server/routes_openai.cpp index 29dfe313a..27a7e18f9 100644 --- a/examples/server/routes_openai.cpp +++ b/examples/server/routes_openai.cpp @@ -75,7 +75,7 @@ static bool build_openai_generation_request(const httplib::Request& req, error_message = "invalid params"; return false; } - return true; + return validate_server_img_gen_request(runtime, request, error_message); } static bool build_openai_edit_request(const httplib::Request& req, @@ -158,22 +158,22 @@ static bool build_openai_edit_request(const httplib::Request& req, request.gen_params.height = height; request.gen_params.batch_count = n; - std::string sd_cpp_extra_args_str = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt); - const json extra_args = sd_cpp_extra_args_str.empty() - ? json::object() - : json::parse(sd_cpp_extra_args_str, nullptr, false); - const bool has_extra_init_image_override = extra_args.is_object() && extra_args.contains("init_image"); - const bool has_extra_ref_images_override = extra_args.is_object() && extra_args.contains("ref_images"); - const bool has_extra_strength_override = extra_args.is_object() && - extra_args.contains("strength") && - extra_args["strength"].is_number(); - const bool has_extra_mask_image = extra_args.is_object() && - extra_args.contains("mask_image") && - !extra_args["mask_image"].is_null(); + std::string sd_cpp_extra_args_str = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt); + const json extra_args = sd_cpp_extra_args_str.empty() + ? json::object() + : json::parse(sd_cpp_extra_args_str, nullptr, false); + const bool has_extra_init_image_override = extra_args.is_object() && extra_args.contains("init_image"); + const bool has_extra_ref_images_override = extra_args.is_object() && extra_args.contains("ref_images"); + const bool has_extra_strength_override = extra_args.is_object() && + extra_args.contains("strength") && + extra_args["strength"].is_number(); + const bool has_extra_mask_image = extra_args.is_object() && + extra_args.contains("mask_image") && + !extra_args["mask_image"].is_null(); const bool supports_ref_image_conditioning = sd_ctx_supports_ref_image_conditioning(runtime.sd_ctx); const bool may_need_uploaded_init_image = !has_extra_init_image_override && - (!supports_ref_image_conditioning || !mask_bytes.empty() || - has_extra_mask_image || has_extra_strength_override || has_extra_ref_images_override); + (!supports_ref_image_conditioning || !mask_bytes.empty() || + has_extra_mask_image || has_extra_strength_override || has_extra_ref_images_override); SDImageOwner uploaded_image_for_init; bool uploaded_image_was_first_reference = false; for (auto& bytes : images_bytes) { @@ -252,7 +252,7 @@ static bool build_openai_edit_request(const httplib::Request& req, return false; } - return true; + return validate_server_img_gen_request(runtime, request, error_message); } static bool execute_sync_img_gen_request(ServerRuntime& runtime, @@ -270,6 +270,10 @@ static bool execute_sync_img_gen_request(ServerRuntime& runtime, num_results = 0; } results.adopt(raw_results, num_results); + if (!results.empty() && !apply_server_adetailer(runtime, request, img_gen_params, results)) { + error_message = "ADetailer failed"; + return false; + } } if (results.empty()) { diff --git a/examples/server/routes_sdapi.cpp b/examples/server/routes_sdapi.cpp index 4b8e7a5b1..014b0b5f8 100644 --- a/examples/server/routes_sdapi.cpp +++ b/examples/server/routes_sdapi.cpp @@ -3,9 +3,12 @@ #include #include #include +#include #include +#include #include #include +#include #include "common/common.h" #include "common/media_io.h" @@ -81,6 +84,202 @@ static enum sample_method_t get_sdapi_sample_method(std::string name) { return it != hardcoded.end() ? it->second : SAMPLE_METHOD_COUNT; } +static bool parse_sdapi_adetailer(const json& body, + ServerRuntime& runtime, + bool img2img, + ImgGenJobRequest& request, + std::string& error_message) { + request.adetailer_enabled = false; + request.adetailer_passes.clear(); + if (!body.contains("alwayson_scripts")) { + return true; + } + try { + const auto& scripts = body.at("alwayson_scripts"); + if (!scripts.is_object()) { + throw std::invalid_argument("alwayson_scripts must be an object"); + } + if (!scripts.contains("ADetailer")) { + return true; + } + const auto& args = scripts.at("ADetailer").at("args"); + if (!args.is_array()) { + throw std::invalid_argument("ADetailer.args must be an array"); + } + if (args.empty() || (args[0].is_boolean() && !args[0].get())) { + return true; + } + static const std::unordered_set keys = { + "ad_model", "ad_model_classes", "ad_tab_enable", "ad_prompt", "ad_negative_prompt", + "ad_confidence", "ad_mask_filter_method", "ad_mask_k", "ad_mask_min_ratio", "ad_mask_max_ratio", + "ad_dilate_erode", "ad_x_offset", "ad_y_offset", "ad_mask_merge_invert", "ad_mask_blur", + "ad_denoising_strength", "ad_inpaint_only_masked", "ad_inpaint_only_masked_padding", + "ad_use_inpaint_width_height", "ad_inpaint_width", "ad_inpaint_height", "ad_use_steps", "ad_steps", + "ad_use_cfg_scale", "ad_cfg_scale", "ad_use_sampler", "ad_sampler", "ad_scheduler", + "ad_use_checkpoint", "ad_checkpoint", "ad_use_vae", "ad_vae", "ad_use_noise_multiplier", + "ad_noise_multiplier", "ad_use_clip_skip", "ad_clip_skip", "ad_restore_face", "ad_controlnet_model", + "ad_controlnet_module", "ad_controlnet_weight", "ad_controlnet_guidance_start", "ad_controlnet_guidance_end", "is_api"}; + for (const auto& config : args) { + if (!config.is_object()) { + continue; + } + auto boolean = [&](const char* key, bool fallback) { + if (!config.contains(key)) + return fallback; + if (!config.at(key).is_boolean()) + throw std::invalid_argument(std::string(key) + " must be boolean"); + return config.at(key).get(); + }; + auto text = [&](const char* key, const char* fallback) { + if (!config.contains(key)) + return std::string(fallback); + if (!config.at(key).is_string()) + throw std::invalid_argument(std::string(key) + " must be a string"); + return config.at(key).get(); + }; + auto number = [&](const char* key, double fallback, double minimum, double maximum) { + double value = fallback; + if (config.contains(key)) { + if (!config.at(key).is_number()) + throw std::invalid_argument(std::string(key) + " must be numeric"); + value = config.at(key).get(); + } + if (!std::isfinite(value) || value < minimum || value > maximum) + throw std::invalid_argument(std::string(key) + " is out of range"); + return value; + }; + auto integer = [&](const char* key, int fallback, int minimum, int maximum = std::numeric_limits::max()) { + if (config.contains(key) && !config.at(key).is_number_integer()) + throw std::invalid_argument(std::string(key) + " must be an integer"); + return static_cast(number(key, fallback, minimum, maximum)); + }; + const std::string model = text("ad_model", "None"); + if (model == "None" || !boolean("ad_tab_enable", true)) { + continue; + } + for (const auto& item : config.items()) { + if (keys.count(item.key()) == 0) + throw std::invalid_argument("unsupported ADetailer setting: " + item.key()); + } + const auto& configured = runtime.default_gen_params->ad_model_path; + if (configured.empty()) + throw std::invalid_argument("ADetailer requires a server-configured --ad-model"); + const auto path = fs::u8path(configured); + if (model.find_first_of("/\\") != std::string::npos || + (lower_ascii(model) != lower_ascii(path.filename().u8string()) && lower_ascii(model) != lower_ascii(path.stem().u8string() + ".pt"))) { + throw std::invalid_argument("ADetailer model must match the server-configured detector"); + } + if (!text("ad_model_classes", "").empty()) + throw std::invalid_argument("ad_model_classes is not supported"); + if (boolean("ad_use_checkpoint", false)) { + const auto checkpoint = config.contains("ad_checkpoint") && config.at("ad_checkpoint").is_null() ? "" : text("ad_checkpoint", "Use same checkpoint"); + if (!checkpoint.empty() && checkpoint != "None" && checkpoint != "Use same checkpoint") + throw std::invalid_argument("ADetailer checkpoint switching is not supported"); + } + if (boolean("ad_use_vae", false)) { + const auto vae = config.contains("ad_vae") && config.at("ad_vae").is_null() ? "" : text("ad_vae", "Use same VAE"); + if (!vae.empty() && vae != "None" && vae != "Use same VAE") + throw std::invalid_argument("ADetailer VAE switching is not supported"); + } + if (boolean("ad_use_noise_multiplier", false) && number("ad_noise_multiplier", 1, 0.5, 1.5) != 1) + throw std::invalid_argument("ADetailer noise multiplier is not supported"); + if (boolean("ad_restore_face", false)) + throw std::invalid_argument("ADetailer face restoration is not supported"); + if (text("ad_controlnet_model", "None") != "None") + throw std::invalid_argument("ADetailer ControlNet is not supported"); + + ServerADetailerPass pass; + pass.prompt = text("ad_prompt", ""); + pass.negative_prompt = text("ad_negative_prompt", ""); + pass.width = request.gen_params.get_resolved_width(); + pass.height = request.gen_params.get_resolved_height(); + pass.strength = static_cast(number("ad_denoising_strength", 0.4, 0, 1)); + auto append = [&](const char* key, const std::string& value) { + if (!pass.extra_args.empty()) + pass.extra_args += ","; + pass.extra_args += std::string(key) + "=" + value; + }; + auto append_number = [&](const char* target, const char* key, double fallback, double minimum, double maximum) { + append(target, json(number(key, fallback, minimum, maximum)).dump()); + }; + auto append_integer = [&](const char* target, const char* key, int fallback, int minimum, int maximum = std::numeric_limits::max()) { + append(target, std::to_string(integer(key, fallback, minimum, maximum))); + }; + append_number("confidence", "ad_confidence", 0.3, 0, 1); + append_integer("mask_k_largest", "ad_mask_k", 0, 0); + const auto filter = text("ad_mask_filter_method", "Area"); + if (filter != "Area" && filter != "Confidence") + throw std::invalid_argument("invalid ad_mask_filter_method"); + append("mask_filter_method", lower_ascii(filter)); + const auto min_ratio = number("ad_mask_min_ratio", 0, 0, 1); + const auto max_ratio = number("ad_mask_max_ratio", 1, 0, 1); + if (min_ratio > max_ratio) + throw std::invalid_argument("ad_mask_min_ratio exceeds ad_mask_max_ratio"); + append("mask_min_ratio", json(min_ratio).dump()); + append("mask_max_ratio", json(max_ratio).dump()); + // Bound pixel operations before converting them to the native mask parameters. + const int mask_limit = std::clamp(std::max(pass.width, pass.height), 32, 16384); + append_integer("dilate_erode", "ad_dilate_erode", 4, -mask_limit, mask_limit); + append_integer("x_offset", "ad_x_offset", 0, -mask_limit, mask_limit); + append_integer("y_offset", "ad_y_offset", 0, -mask_limit, mask_limit); + append_integer("mask_blur", "ad_mask_blur", 4, 0, mask_limit); + append_integer("inpaint_padding", "ad_inpaint_only_masked_padding", 32, 0, mask_limit); + append("inpaint_only_masked", boolean("ad_inpaint_only_masked", true) ? "true" : "false"); + const auto merge = text("ad_mask_merge_invert", "None"); + if (merge != "None" && merge != "Merge" && merge != "Merge and Invert") + throw std::invalid_argument("invalid ad_mask_merge_invert"); + append("mask_mode", merge == "None" ? "none" : merge == "Merge" ? "merge" + : "merge_invert"); + if (boolean("ad_use_inpaint_width_height", false)) { + pass.width = integer("ad_inpaint_width", 512, 1, k_max_generation_dimension); + pass.height = integer("ad_inpaint_height", 512, 1, k_max_generation_dimension); + } + if (boolean("ad_use_steps", false)) + append_integer("steps", "ad_steps", 28, 1, k_max_adetailer_steps); + if (boolean("ad_use_cfg_scale", false)) + append_number("cfg_scale", "ad_cfg_scale", 7, 0, std::numeric_limits::max()); + if (boolean("ad_use_clip_skip", false)) { + pass.clip_skip = integer("ad_clip_skip", 1, 1); + if (pass.clip_skip > 12) + throw std::invalid_argument("ad_clip_skip is out of range"); + } + if (boolean("ad_use_sampler", false)) { + std::string sampler = text("ad_sampler", "DPM++ 2M Karras"); + if (sampler != "Use same sampler") { + const std::string suffix = " karras"; + const bool legacy_karras = sampler.size() > suffix.size() && lower_ascii(sampler.substr(sampler.size() - suffix.size())) == suffix; + if (legacy_karras) + sampler.resize(sampler.size() - suffix.size()); + const auto method = get_sdapi_sample_method(sampler); + if (method == SAMPLE_METHOD_COUNT) + throw std::invalid_argument("unsupported ADetailer sampler"); + append("sample_method", sd_sample_method_name(method)); + if (legacy_karras && text("ad_scheduler", "Use same scheduler") == "Use same scheduler") + append("scheduler", sd_scheduler_name(KARRAS_SCHEDULER)); + } + const auto scheduler = text("ad_scheduler", "Use same scheduler"); + if (scheduler == "Automatic") { + pass.automatic_scheduler = true; + } else if (scheduler != "Use same scheduler") { + const auto value = str_to_scheduler(scheduler.c_str()); + if (value == SCHEDULER_COUNT) + throw std::invalid_argument("unsupported ADetailer scheduler"); + append("scheduler", sd_scheduler_name(value)); + } + } + request.adetailer_passes.push_back(std::move(pass)); + } + if (!request.adetailer_passes.empty() && img2img && args.size() > 1 && args[1].is_boolean() && args[1].get()) { + throw std::invalid_argument("ADetailer skip img2img is not supported"); + } + request.adetailer_enabled = !request.adetailer_passes.empty(); + return true; + } catch (const std::exception& error) { + error_message = std::string("invalid ADetailer args: ") + error.what(); + return false; + } +} + static bool build_sdapi_img_gen_request(const json& j, ServerRuntime& runtime, bool img2img, @@ -259,7 +458,8 @@ static bool build_sdapi_img_gen_request(const json& j, return false; } - return true; + return parse_sdapi_adetailer(j, runtime, img2img, request, error_message) && + validate_server_img_gen_request(runtime, request, error_message); } static nlohmann::json prepare_info_field(const SDContextParams& ctx_params, @@ -349,7 +549,7 @@ static void preview_callback(int step, int frame_count, sd_image_t* frames, bool } struct SDAPIPreviewGuard { - SDAPIPreviewGuard(const std::string& id_task, const sd_img_gen_params_t& params, const std::string& job_type) { + SDAPIPreviewGuard(const std::string& id_task, const sd_img_gen_params_t& params, const std::string& job_type, bool detailing = false) { std::lock_guard lock(g_preview.mutex); g_preview.id_task = id_task; g_preview.total = 0; @@ -358,11 +558,20 @@ struct SDAPIPreviewGuard { g_preview.step = 0; g_preview.active = true; // Each batch item has a base pass and, when enabled, a hires pass. - g_preview.job_count = static_cast(std::max(1, params.batch_count)) * (params.hires.enabled ? 2 : 1); + g_preview.job_count = static_cast(std::max(1, params.batch_count)) * (params.hires.enabled ? 2 : 1) + (detailing ? 1 : 0); g_preview.job_no = 0; sd_set_preview_callback(preview_callback, PREVIEW_PROJ, 1, true, false, nullptr); } + void begin_detailing() { + sd_set_preview_callback(nullptr, PREVIEW_NONE, 1, false, false, nullptr); + std::lock_guard lock(g_preview.mutex); + g_preview.job_type = "adetailer"; + g_preview.job_no = static_cast(g_preview.job_count - 1); + g_preview.step = 0; + g_preview.total = 0; + } + ~SDAPIPreviewGuard() { sd_set_preview_callback(nullptr, PREVIEW_NONE, 1, false, false, nullptr); std::lock_guard lock(g_preview.mutex); @@ -410,13 +619,21 @@ void register_sdapi_endpoints(httplib::Server& svr, ServerRuntime& rt) { { std::lock_guard lock(*runtime->sd_ctx_mutex); - SDAPIPreviewGuard preview_guard(id_task, img_gen_params, img2img ? "img2img" : "txt2img"); + const bool detailing = request.adetailer_enabled && !runtime->default_gen_params->ad_model_path.empty(); + SDAPIPreviewGuard preview_guard(id_task, img_gen_params, img2img ? "img2img" : "txt2img", detailing); sd_image_t* raw_results = nullptr; if (!generate_image(runtime->sd_ctx, &img_gen_params, &raw_results, &num_results)) { raw_results = nullptr; num_results = 0; } results.adopt(raw_results, num_results); + if (detailing && !results.empty()) + preview_guard.begin_detailing(); + if (!results.empty() && !apply_server_adetailer(*runtime, request, img_gen_params, results)) { + res.status = 500; + res.set_content(R"({"error":"ADetailer failed"})", "application/json"); + return; + } } if (results.empty()) { diff --git a/examples/server/routes_sdcpp.cpp b/examples/server/routes_sdcpp.cpp index 7e93e5edb..7839d5c53 100644 --- a/examples/server/routes_sdcpp.cpp +++ b/examples/server/routes_sdcpp.cpp @@ -206,7 +206,7 @@ static json make_img_gen_features_json() { {"hires", true}, {"cache", true}, {"cancel_queued", true}, - {"cancel_generating", false}, + {"cancel_generating", true}, {"preview", true}, }; } @@ -221,7 +221,7 @@ static json make_vid_gen_features_json() { {"vae_tiling", true}, {"cache", true}, {"cancel_queued", true}, - {"cancel_generating", false}, + {"cancel_generating", true}, {"preview", true}, }; } @@ -381,7 +381,7 @@ static json make_capabilities_json(ServerRuntime& runtime) { json top_level_output_formats = json::array(); json top_level_features = { {"cancel_queued", true}, - {"cancel_generating", false}, + {"cancel_generating", true}, }; std::string current_mode = ""; if (supports_img) { @@ -410,9 +410,9 @@ static json make_capabilities_json(ServerRuntime& runtime) { result["defaults_by_mode"] = defaults_by_mode; result["limits"] = { {"min_width", 64}, - {"max_width", 4096}, + {"max_width", k_max_generation_dimension}, {"min_height", 64}, - {"max_height", 4096}, + {"max_height", k_max_generation_dimension}, {"max_batch_count", 8}, {"max_queue_size", manager.max_pending_jobs}, {"max_upscale_width", k_max_upscale_dimension}, @@ -457,7 +457,7 @@ static bool parse_img_gen_request(const json& body, error_message = "invalid generation parameters"; return false; } - return true; + return validate_server_img_gen_request(runtime, request, error_message); } static bool parse_vid_gen_request(const json& body, @@ -840,8 +840,9 @@ void register_sdcpp_api_endpoints(httplib::Server& svr, ServerRuntime& rt) { } if (job.status == AsyncJobStatus::Generating) { - res.status = 409; - res.set_content(R"({"error":"job is currently generating and cannot be interrupted yet"})", "application/json"); + job.cancel_requested.store(true); + res.status = 200; + res.set_content(make_async_job_json(manager, job).dump(), "application/json"); return; } diff --git a/examples/server/runtime.cpp b/examples/server/runtime.cpp index 34140ea35..5b33b338a 100644 --- a/examples/server/runtime.cpp +++ b/examples/server/runtime.cpp @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -18,6 +19,157 @@ namespace fs = std::filesystem; +bool validate_server_img_gen_request(const ServerRuntime& runtime, + const ImgGenJobRequest& request, + std::string& error_message) { + auto dimensions_valid = [&](double width, double height) { + if (!std::isfinite(width) || !std::isfinite(height) || + width < 1 || height < 1 || width > k_max_generation_dimension || height > k_max_generation_dimension) { + error_message = "generation width and height must be between 1 and " + std::to_string(k_max_generation_dimension); + return false; + } + return true; + }; + const auto& params = request.gen_params; + const int width = params.get_resolved_width(); + const int height = params.get_resolved_height(); + if (!dimensions_valid(width, height)) { + return false; + } + if (params.hires_enabled) { + const double hires_width = params.hires_width > 0 ? params.hires_width : params.hires_height > 0 ? params.hires_height + : std::round(width * params.hires_scale); + const double hires_height = params.hires_height > 0 ? params.hires_height : params.hires_width > 0 ? params.hires_width + : std::round(height * params.hires_scale); + if (!dimensions_valid(hires_width, hires_height)) { + return false; + } + } + if (!request.adetailer_enabled || runtime.default_gen_params->ad_model_path.empty()) { + return true; + } + auto pass_valid = [&](const ServerADetailerPass& pass) { + if (!dimensions_valid(pass.width, pass.height)) { + return false; + } + // Match the native key normalization, but leave full option parsing to ADetailer. + auto trim = [](const std::string& text) { + const auto first = text.find_first_not_of(" \t\r\n"); + return first == std::string::npos ? std::string() : text.substr(first, text.find_last_not_of(" \t\r\n") - first + 1); + }; + std::stringstream stream(pass.extra_args); + std::string item; + while (std::getline(stream, item, ',')) { + const auto equals = item.find('='); + if (equals == std::string::npos) { + continue; + } + auto key = trim(item.substr(0, equals)); + std::transform(key.begin(), key.end(), key.begin(), [](unsigned char c) { return static_cast(std::tolower(c)); }); + if (key != "inpaint_width" && key != "inpaint_height" && key != "steps") { + continue; + } + const auto value = trim(item.substr(equals + 1)); + const int minimum = key == "steps" ? 0 : 1; + const int maximum = key == "steps" ? k_max_adetailer_steps : k_max_generation_dimension; + try { + size_t used = 0; + const int parsed = std::stoi(value, &used); + if (used == value.size() && parsed >= minimum && parsed <= maximum) { + continue; + } + } catch (const std::exception&) { + } + error_message = "ADetailer " + key + " must be between " + std::to_string(minimum) + " and " + std::to_string(maximum); + return false; + } + return true; + }; + if (request.adetailer_passes.empty()) { + ServerADetailerPass pass; + pass.extra_args = params.extra_ad_args; + return pass_valid(pass); + } + return std::all_of(request.adetailer_passes.begin(), request.adetailer_passes.end(), pass_valid); +} + +bool apply_server_adetailer(ServerRuntime& runtime, + const ImgGenJobRequest& request, + const sd_img_gen_params_t& params, + SDImageVec& results) { + const auto& gen_params = request.gen_params; + const auto& ad_model_path = runtime.default_gen_params->ad_model_path; + if (!request.adetailer_enabled || ad_model_path.empty()) { + return true; + } + std::string error_message; + if (!validate_server_img_gen_request(runtime, request, error_message)) { + LOG_ERROR("%s", error_message.c_str()); + return false; + } + + std::vector passes = request.adetailer_passes; + if (passes.empty()) { + ServerADetailerPass pass; + pass.prompt = gen_params.ad_prompt; + pass.negative_prompt = gen_params.ad_negative_prompt; + pass.extra_args = gen_params.extra_ad_args; + passes.push_back(std::move(pass)); + } + + ADetailerCtxPtr ad_ctx(new_adetailer_ctx(ad_model_path.c_str(), + runtime.ctx_params->n_threads, + runtime.ctx_params->effective_backend.c_str(), + runtime.ctx_params->effective_params_backend.c_str())); + if (ad_ctx == nullptr) { + LOG_ERROR("new_adetailer_ctx failed"); + return false; + } + + for (const auto& pass : passes) { + sd_adetailer_params_t ad_params{}; + ad_params.prompt = pass.prompt.empty() ? nullptr : pass.prompt.c_str(); + ad_params.negative_prompt = pass.negative_prompt.empty() ? nullptr : pass.negative_prompt.c_str(); + ad_params.extra_ad_args = pass.extra_args.c_str(); + for (int i = 0; i < results.count(); ++i) { + if (results[i].data == nullptr) { + continue; + } + sd_img_gen_params_t ad_generation_params = params; + ad_generation_params.seed = params.seed + i; + ad_generation_params.width = pass.width; + ad_generation_params.height = pass.height; + ad_generation_params.strength = pass.strength; + if (pass.clip_skip > 0) { + ad_generation_params.clip_skip = pass.clip_skip; + } + if (pass.automatic_scheduler) { + ad_generation_params.sample_params.scheduler = SCHEDULER_COUNT; + } + + sd_image_t* detailed_images = nullptr; + int detailed_count = 0; + if (!adetail_image(ad_ctx.get(), + runtime.sd_ctx, + results[i], + &ad_params, + &ad_generation_params, + &detailed_images, + &detailed_count) || + detailed_count <= 0 || detailed_images == nullptr || detailed_images[0].data == nullptr) { + free_sd_images(detailed_images, detailed_count); + LOG_ERROR("ADetailer failed for image %d", i + 1); + return false; + } + free(results[i].data); + results[i] = detailed_images[0]; + detailed_images[0] = {0, 0, 0, nullptr}; + free_sd_images(detailed_images, detailed_count); + } + } + return true; +} + static std::string lower_ascii(std::string value) { std::transform(value.begin(), value.end(), value.begin(), [](unsigned char c) { return static_cast(std::tolower(c)); diff --git a/examples/server/runtime.h b/examples/server/runtime.h index d9272f6c2..8986f3827 100644 --- a/examples/server/runtime.h +++ b/examples/server/runtime.h @@ -67,18 +67,38 @@ struct ServerRuntime { ServerModels* models; }; +struct ServerADetailerPass { + std::string prompt; + std::string negative_prompt; + std::string extra_args; + int width = 512; + int height = 512; + float strength = 0.4f; + int clip_skip = -1; + bool automatic_scheduler = false; +}; + struct ImgGenJobRequest { SDGenerationParams gen_params; std::string output_format = "png"; int output_compression = 100; std::string preview_mode = "none"; int preview_interval = 1; + bool adetailer_enabled = true; + std::vector adetailer_passes; sd_img_gen_params_t to_sd_img_gen_params_t() { return gen_params.to_sd_img_gen_params_t(); } }; +constexpr int k_max_generation_dimension = 4096; +constexpr int k_max_adetailer_steps = 100; + +bool validate_server_img_gen_request(const ServerRuntime& runtime, + const ImgGenJobRequest& request, + std::string& error_message); + struct VidGenJobRequest { SDGenerationParams gen_params; std::string output_format = "webm"; @@ -111,3 +131,7 @@ void refresh_lora_cache(ServerRuntime& rt); std::string get_lora_full_path(ServerRuntime& rt, const std::string& path); void refresh_upscaler_cache(ServerRuntime& rt); int64_t unix_timestamp_now_ms(); +bool apply_server_adetailer(ServerRuntime& runtime, + const ImgGenJobRequest& request, + const sd_img_gen_params_t& params, + SDImageVec& results); diff --git a/ggml b/ggml index dc998771e..c1f80e507 160000 --- a/ggml +++ b/ggml @@ -1 +1 @@ -Subproject commit dc998771e700d49bbdf9a39cede22bb5b3311767 +Subproject commit c1f80e507db1d0ce49f3bf7f81bc79246d7a2611 diff --git a/include/stable-diffusion.h b/include/stable-diffusion.h index 404d8cbdd..85fe51d46 100644 --- a/include/stable-diffusion.h +++ b/include/stable-diffusion.h @@ -640,6 +640,11 @@ SD_API const char* sd_version(void); // terminator. Passing nullptr or buffer_size 0 only queries the required size. SD_API size_t sd_list_devices(char* buffer, size_t buffer_size); +// Register comma-separated RPC endpoints (host:port) before listing devices. +// Repeated endpoints are safe; nullptr or an empty string is a no-op. +// Returns false if RPC is unavailable or an endpoint has no usable devices. +SD_API bool sd_add_rpc_servers(const char* servers); + // for C API, caller needs to call free_sd_images to free the memory after use // This helps avoid CRT problems on Windows when memory is allocated in the library but freed in the caller, which may use a different CRT. SD_API void free_sd_images(sd_image_t* result_images, int num_images); diff --git a/src/conditioning/conditioner.hpp b/src/conditioning/conditioner.hpp index 7bbc38348..a146716d9 100644 --- a/src/conditioning/conditioner.hpp +++ b/src/conditioning/conditioner.hpp @@ -2008,7 +2008,8 @@ struct LLMEmbedder : public Conditioner { arch = LLM::LLMArch::GPT_OSS_20B; } else if (sd_version_is_pid(version)) { arch = LLM::LLMArch::GEMMA2_2B; - } else if (version == VERSION_QWEN_IMAGE_2_1 || + } else if (version == VERSION_FIBO2 || + version == VERSION_QWEN_IMAGE_2_1 || sd_version_is_flux3(version) || sd_version_is_lingbot_video(version) || sd_version_is_ideogram4(version) || @@ -2147,10 +2148,11 @@ struct LLMEmbedder : public Conditioner { std::tuple, std::vector, std::vector> tokenize(std::string text, const std::pair& attn_range, - size_t min_length = 0, - size_t max_length = 100000000, - bool spell_quotes = false, - const std::string& suffix = "") { + size_t min_length = 0, + size_t max_length = 100000000, + bool spell_quotes = false, + const std::string& suffix = "", + const std::vector>>& image_embeds = {}) { std::vector> parsed_attention; if (attn_range.first >= 0 && attn_range.second > 0) { if (attn_range.first > 0) { @@ -2195,6 +2197,19 @@ struct LLMEmbedder : public Conditioner { weights.insert(weights.end(), curr_tokens.size(), curr_weight); } + for (const auto& image_embed : image_embeds) { + const int index = image_embed.first; + const int64_t count = image_embed.second.shape()[1]; + if (index < 0 || static_cast(index) >= tokens.size() || count < 1) { + LOG_ERROR("Invalid reference image token range"); + return {}; + } + const int token = tokens[index]; + const float weight = weights[index]; + tokens.insert(tokens.begin() + index, count - 1, token); + weights.insert(weights.begin() + index, count - 1, weight); + } + if (!suffix.empty()) { std::vector suffix_tokens; if (!tokenizer->encode(suffix, suffix_tokens) || suffix_tokens.size() >= max_length) { @@ -2215,6 +2230,17 @@ struct LLMEmbedder : public Conditioner { std::vector mask; tokenizer->pad_tokens(tokens, &weights, &mask, min_length, max_length); + // Keep masks and weights in expanded coordinates, but only look up one token per image. + for (auto it = image_embeds.rbegin(); it != image_embeds.rend(); ++it) { + const int64_t begin = it->first; + const int64_t end = begin + it->second.shape()[1]; + if (begin < 0 || end > static_cast(tokens.size())) { + LOG_ERROR("Reference image exceeds the tokenized context"); + return {}; + } + tokens.erase(tokens.begin() + begin + 1, tokens.begin() + end); + } + // for (int i = 0; i < tokens.size(); i++) { // std::cout << tokens[i] << ":" << weights[i] << ", " << i << std::endl; // } @@ -2237,7 +2263,7 @@ struct LLMEmbedder : public Conditioner { const std::vector& image_grids = {}, const std::string& prompt_suffix = "", sd::Tensor* output_mask = nullptr) { - auto tokens_weights_mask = tokenize(prompt, prompt_attn_range, min_length, max_length, spell_quotes, prompt_suffix); + auto tokens_weights_mask = tokenize(prompt, prompt_attn_range, min_length, max_length, spell_quotes, prompt_suffix, image_embeds); auto& tokens = std::get<0>(tokens_weights_mask); auto& weights = std::get<1>(tokens_weights_mask); auto& mask = std::get<2>(tokens_weights_mask); @@ -2267,10 +2293,10 @@ struct LLMEmbedder : public Conditioner { sd::Tensor position_ids; if (sd_version_is_flux3(version)) { - position_ids = sd::Tensor::zeros({static_cast(tokens.size() * 4)}); + position_ids = sd::Tensor::zeros({static_cast(weights.size() * 4)}); for (size_t axis = 0; axis < 4; ++axis) { - for (size_t i = 0; i < tokens.size(); ++i) { - position_ids[axis * tokens.size() + i] = mask[i] == 0.f ? 1 : static_cast(i); + for (size_t i = 0; i < weights.size(); ++i) { + position_ids[axis * weights.size() + i] = mask[i] == 0.f ? 1 : static_cast(i); } } } @@ -2367,6 +2393,50 @@ struct LLMEmbedder : public Conditioner { } } + bool expand_vision_marker(const std::string& prompt, + size_t& marker_start, + std::vector> outputs, + int grid_h, + int grid_w, + std::vector>>& image_embeds, + LLM::DeepStackImageEmbeds& deepstack_image_embeds, + std::vector& image_grids, + int max_length = std::numeric_limits::max()) { + if (outputs.empty()) { + return false; + } + const std::string placeholder = "<|image_pad|>"; + size_t position = prompt.find(placeholder, marker_start); + if (position == std::string::npos) { + LOG_ERROR("Reference image marker is missing"); + return false; + } + std::vector prefix_tokens; + if (!tokenizer->encode(prompt.substr(0, position), prefix_tokens, nullptr)) { + return false; + } + int index = static_cast(prefix_tokens.size()); + for (const auto& image_embed : image_embeds) { + index += static_cast(image_embed.second.shape()[1]) - 1; + } + int count = static_cast(outputs[0].shape()[1]); + if (index > max_length || count > max_length - index) { + LOG_ERROR("Reference images exceed the %d-token context", max_length); + return false; + } + if (deepstack_image_embeds.empty()) { + deepstack_image_embeds.resize(outputs.size() - 1); + } + GGML_ASSERT(deepstack_image_embeds.size() == outputs.size() - 1); + image_embeds.emplace_back(index, std::move(outputs[0])); + for (size_t layer = 1; layer < outputs.size(); ++layer) { + deepstack_image_embeds[layer - 1].emplace_back(index, std::move(outputs[layer])); + } + image_grids.push_back({index, count, grid_h, grid_w}); + marker_start = position + placeholder.size(); + return true; + } + SDCondition get_learned_condition(int n_threads, const ConditionerParams& conditioner_params) override { std::string prompt; @@ -2394,9 +2464,8 @@ struct LLMEmbedder : public Conditioner { prompt_attn_range = {0, 0}; if (llm->enable_vision) { - const std::string placeholder = "<|image_pad|>"; - const int patch_size = llm->config.vision.patch_size; - const int factor = patch_size * llm->config.vision.spatial_merge_size; + const int patch_size = llm->config.vision.patch_size; + const int factor = patch_size * llm->config.vision.spatial_merge_size; auto resize_for_vision = [&](const sd::Tensor& image) { int height = static_cast(image.shape()[1]); @@ -2424,29 +2493,11 @@ struct LLMEmbedder : public Conditioner { int grid_h, int grid_w) { GGML_ASSERT(image_outputs.size() == 4); - auto image_embed = std::move(image_outputs[0]); prompt += "<|vision_start|>"; - std::vector prefix_tokens; - if (!tokenizer->encode(prompt, prefix_tokens, nullptr)) { - return false; - } - int image_embed_idx = static_cast(prefix_tokens.size()); - image_embeds.emplace_back(image_embed_idx, image_embed); - if (deepstack_image_embeds.empty()) { - deepstack_image_embeds.resize(image_outputs.size() - 1); - } - for (size_t layer = 0; layer < deepstack_image_embeds.size(); ++layer) { - deepstack_image_embeds[layer].emplace_back(image_embed_idx, std::move(image_outputs[layer + 1])); - } - image_grids.push_back({image_embed_idx, - static_cast(image_embed.shape()[1]), - grid_h, - grid_w}); - for (int64_t i = 0; i < image_embed.shape()[1]; ++i) { - prompt += placeholder; - } - prompt += "<|vision_end|>"; - return true; + size_t marker_start = prompt.size(); + prompt += "<|image_pad|><|vision_end|>"; + return expand_vision_marker(prompt, marker_start, std::move(image_outputs), grid_h, grid_w, + image_embeds, deepstack_image_embeds, image_grids); }; const auto* references = conditioner_params.minimax_h3_references; @@ -2609,14 +2660,13 @@ struct LLMEmbedder : public Conditioner { return {}; } int image_embed_idx = static_cast(prefix_tokens.size()); + for (const auto& previous_image : image_embeds) { + image_embed_idx += static_cast(previous_image.second.shape()[1]) - 1; + } image_embeds.emplace_back(image_embed_idx, image_embed); img_prompt += "<|vision_start|>"; - int64_t num_image_tokens = image_embed.shape()[1]; - img_prompt.reserve(img_prompt.size() + static_cast(num_image_tokens) * placeholder.size() + 32); - for (int j = 0; j < num_image_tokens; j++) { - img_prompt += placeholder; - } + img_prompt += placeholder; img_prompt += "<|vision_end|>"; } prompt += img_prompt; @@ -2660,26 +2710,14 @@ struct LLMEmbedder : public Conditioner { return {}; } prompt += (i == 0 ? "" : " ") + std::string("<|vision_start|>"; - std::vector prefix_tokens; - if (!tokenizer->encode(prompt, prefix_tokens, nullptr)) { + size_t marker_start = prompt.size(); + prompt += "<|image_pad|><|vision_end|>"; + if (!expand_vision_marker(prompt, marker_start, std::move(outputs), + static_cast(height) / llm->config.vision.patch_size, + static_cast(width) / llm->config.vision.patch_size, + image_embeds, deepstack_image_embeds, image_grids)) { return {}; } - int index = static_cast(prefix_tokens.size()); - int count = static_cast(outputs[0].shape()[1]); - image_embeds.emplace_back(index, std::move(outputs[0])); - if (deepstack_image_embeds.empty()) { - deepstack_image_embeds.resize(outputs.size() - 1); - } - for (size_t layer = 1; layer < outputs.size(); ++layer) { - deepstack_image_embeds[layer - 1].emplace_back(index, std::move(outputs[layer])); - } - image_grids.push_back({index, count, - static_cast(height) / llm->config.vision.patch_size, - static_cast(width) / llm->config.vision.patch_size}); - for (int j = 0; j < count; ++j) { - prompt += "<|image_pad|>"; - } - prompt += "<|vision_end|>"; } } prompt_attn_range.first = static_cast(prompt.size()); @@ -2690,7 +2728,6 @@ struct LLMEmbedder : public Conditioner { if (llm->enable_vision && conditioner_params.ref_images != nullptr && !conditioner_params.ref_images->empty()) { LOG_INFO("%s", sd_version_is_mage_flow(version) ? "MageFlowEditPipeline" : "QwenImageEditPlusPipeline"); prompt_template_encode_start_idx = 64; - int image_embed_idx = 64 + 6; int min_pixels = conditioner_params.ref_image_params.vlm_min_size; if (min_pixels <= 0) { @@ -2707,6 +2744,7 @@ struct LLMEmbedder : public Conditioner { } } + prompt = "<|im_start|>system\nDescribe the key features of the input image (color, shape, size, texture, objects, background), then explain how the user's text instruction should alter or modify the image. Generate a new image that meets the user's requirements while maintaining consistency with the original input where appropriate.<|im_end|>\n<|im_start|>user\n"; std::string placeholder = "<|image_pad|>"; std::string img_prompt; @@ -2726,19 +2764,22 @@ struct LLMEmbedder : public Conditioner { auto image_embed = llm->encode_image(n_threads, resized_image, false); GGML_ASSERT(!image_embed.empty()); + std::string image_prefix = prompt + img_prompt + (sd_version_is_mage_flow(version) ? "Image " : "Picture ") + std::to_string(i + 1) + ": <|vision_start|>"; + std::vector prefix_tokens; + if (!tokenizer->encode(image_prefix, prefix_tokens, nullptr)) { + return {}; + } + int image_embed_idx = static_cast(prefix_tokens.size()); + for (const auto& previous_image : image_embeds) { + image_embed_idx += static_cast(previous_image.second.shape()[1]) - 1; + } image_embeds.emplace_back(image_embed_idx, image_embed); - image_embed_idx += 1 + static_cast(image_embed.shape()[1]) + 6; img_prompt += (sd_version_is_mage_flow(version) ? "Image " : "Picture ") + std::to_string(i + 1) + ": <|vision_start|>"; - int64_t num_image_tokens = image_embed.shape()[1]; - img_prompt.reserve(num_image_tokens * placeholder.size()); - for (int j = 0; j < num_image_tokens; j++) { - img_prompt += placeholder; - } + img_prompt += placeholder; img_prompt += "<|vision_end|>"; } - prompt = "<|im_start|>system\nDescribe the key features of the input image (color, shape, size, texture, objects, background), then explain how the user's text instruction should alter or modify the image. Generate a new image that meets the user's requirements while maintaining consistency with the original input where appropriate.<|im_end|>\n<|im_start|>user\n"; prompt += img_prompt; prompt_attn_range.first = static_cast(prompt.size()); @@ -2816,14 +2857,13 @@ struct LLMEmbedder : public Conditioner { return {}; } int image_embed_idx = static_cast(prefix_tokens.size()); + for (const auto& previous_image : image_embeds) { + image_embed_idx += static_cast(previous_image.second.shape()[1]) - 1; + } image_embeds.emplace_back(image_embed_idx, image_embed); img_prompt += "<|vision_start|>"; - int64_t num_image_tokens = image_embed.shape()[1]; - img_prompt.reserve(img_prompt.size() + static_cast(num_image_tokens) * placeholder.size() + 32); - for (int j = 0; j < num_image_tokens; j++) { - img_prompt += placeholder; - } + img_prompt += placeholder; img_prompt += "<|vision_end|>"; } @@ -2888,14 +2928,13 @@ struct LLMEmbedder : public Conditioner { return {}; } int image_embed_idx = static_cast(prefix_tokens.size()); + for (const auto& previous_image : image_embeds) { + image_embed_idx += static_cast(previous_image.second.shape()[1]) - 1; + } image_embeds.emplace_back(image_embed_idx, image_embed); img_prompt += "Picture " + std::to_string(i + 1) + ": <|vision_start|>"; - int64_t num_image_tokens = image_embed.shape()[1]; - img_prompt.reserve(img_prompt.size() + static_cast(num_image_tokens) * placeholder.size() + 32); - for (int j = 0; j < num_image_tokens; j++) { - img_prompt += placeholder; - } + img_prompt += placeholder; img_prompt += "<|vision_end|>"; } prompt += img_prompt; @@ -2913,7 +2952,6 @@ struct LLMEmbedder : public Conditioner { LOG_INFO("LongCatEditPipeline"); prompt_template_encode_start_idx = 67; min_length = 512 + prompt_template_encode_start_idx; - int image_embed_idx = 36 + 6; int min_pixels = conditioner_params.ref_image_params.vlm_min_size; if (min_pixels <= 0) { @@ -2930,6 +2968,7 @@ struct LLMEmbedder : public Conditioner { } } + prompt = "<|im_start|>system\nAs an image editing expert, first analyze the content and attributes of the input image(s). Then, based on the user's editing instructions, clearly and precisely determine how to modify the given image(s), ensuring that only the specified parts are altered and all other aspects remain consistent with the original(s).<|im_end|>\n<|im_start|>user\n"; std::string placeholder = "<|image_pad|>"; std::string img_prompt; @@ -2948,19 +2987,22 @@ struct LLMEmbedder : public Conditioner { auto resized_image = clip_preprocess(image, w_bar, h_bar); auto image_embed = llm->encode_image(n_threads, resized_image, false); GGML_ASSERT(!image_embed.empty()); + std::string image_prefix = prompt + img_prompt + "<|vision_start|>"; + std::vector prefix_tokens; + if (!tokenizer->encode(image_prefix, prefix_tokens, nullptr)) { + return {}; + } + int image_embed_idx = static_cast(prefix_tokens.size()); + for (const auto& previous_image : image_embeds) { + image_embed_idx += static_cast(previous_image.second.shape()[1]) - 1; + } image_embeds.emplace_back(image_embed_idx, image_embed); - image_embed_idx += 1 + static_cast(image_embed.shape()[1]) + 6; img_prompt += "<|vision_start|>"; - int64_t num_image_tokens = image_embed.shape()[1]; - img_prompt.reserve(num_image_tokens * placeholder.size()); - for (int j = 0; j < num_image_tokens; j++) { - img_prompt += placeholder; - } + img_prompt += placeholder; img_prompt += "<|vision_end|>"; } - prompt = "<|im_start|>system\nAs an image editing expert, first analyze the content and attributes of the input image(s). Then, based on the user's editing instructions, clearly and precisely determine how to modify the given image(s), ensuring that only the specified parts are altered and all other aspects remain consistent with the original(s).<|im_end|>\n<|im_start|>user\n"; prompt += img_prompt; } else { prompt_template_encode_start_idx = 36; @@ -3028,7 +3070,10 @@ struct LLMEmbedder : public Conditioner { } else if (sd_version_is_z_image(version) || sd_version_is_z_image_l2p(version)) { prompt_template_encode_start_idx = 0; out_layers = {35}; // -2 - + if (conditioner_params.ref_images != nullptr && !conditioner_params.ref_images->empty()) { + LOG_ERROR("Z-Image currently does not support reference-image conditioning"); + return {}; + } if (conditioner_params.ref_images != nullptr && !conditioner_params.ref_images->empty()) { LOG_INFO("ZImageOmniPipeline"); prompt = "<|im_start|>user\n<|vision_start|>"; @@ -3563,12 +3608,10 @@ struct LLaDAImageEmbedder : public Conditioner { auto inputs_embeds = llm->compute_input_embeds(n_threads, text_ids); auto query_embeds = query_former->compute(n_threads, inputs_embeds); - // splice_image_embeds() replaces tokens in place, so the query slots have to exist in - // input_ids; their ids are irrelevant because the embeddings are overwritten. - std::vector padded(tokens.begin(), tokens.end()); - padded.resize(static_cast(n_text + num_queries), tokenizer->PAD_TOKEN_ID); - int64_t n_total = static_cast(padded.size()); - sd::Tensor input_ids({n_total}, padded); + std::vector input_tokens(tokens.begin(), tokens.end()); + input_tokens.push_back(tokenizer->PAD_TOKEN_ID); + sd::Tensor input_ids({n_text + 1}, std::move(input_tokens)); + int64_t n_total = n_text + num_queries; // Bidirectional everywhere except that the text tokens must not see the appended // queries, matching backbone_attention_mask[:, :, :text_length, text_length:] = min. diff --git a/src/conditioning/fibo2.hpp b/src/conditioning/fibo2.hpp new file mode 100644 index 000000000..43b2ee084 --- /dev/null +++ b/src/conditioning/fibo2.hpp @@ -0,0 +1,182 @@ +#ifndef __SD_CONDITIONING_FIBO2_HPP__ +#define __SD_CONDITIONING_FIBO2_HPP__ + +#include "conditioning/conditioner.hpp" + +struct Fibo2Conditioner : public LLMEmbedder { + using JSON = nlohmann::ordered_json; + + Fibo2Conditioner(ggml_backend_t backend, const String2TensorStorage& weights, std::shared_ptr weight_manager, const TokenizerConfig& tokenizers) + : LLMEmbedder(backend, weights, VERSION_FIBO2, "", true, weight_manager, tokenizers) { + if (llm->config.hidden_size != 2560 || llm->config.num_layers != 36) { + throw std::runtime_error("Fibo 2.0 requires its Qwen3-VL-4B text encoder"); + } + } + + static bool empty_value(const JSON& value) { + return value.is_null() || + (value.is_string() && value.get_ref().empty()) || + ((value.is_array() || value.is_object()) && value.empty()); + } + + static JSON clean_caption(const JSON& value) { + if (value.is_object()) { + auto result = JSON::object(); + for (auto it = value.begin(); it != value.end(); ++it) { + auto cleaned = clean_caption(it.value()); + if (!empty_value(cleaned)) { + result[it.key()] = std::move(cleaned); + } + } + return result; + } + if (value.is_array()) { + auto result = JSON::array(); + for (const auto& item : value) { + auto cleaned = clean_caption(item); + if (!empty_value(cleaned)) { + result.push_back(std::move(cleaned)); + } + } + return result; + } + return value; + } + + static std::pair parse_prompt(const std::string& prompt) { + auto caption = JSON::parse(prompt, nullptr, false); + if (!caption.is_object()) { + return {prompt, nullptr}; + } + JSON instruction; + if (caption.contains("edit_instruction")) { + instruction = caption["edit_instruction"]; + caption.erase("edit_instruction"); + } + if (caption.contains("structured_caption")) { + caption = JSON(caption["structured_caption"]); + if (!caption.is_object()) { + return {caption, instruction}; + } + } + bool short_form = caption.size() == 2 && caption.contains("short_description") && caption.contains("text_render"); + caption = clean_caption(caption); + for (const char* score : {"aesthetic_score", "preference_score"}) { + caption.erase(score); + for (auto& value : caption) { + if (value.is_object()) { + value.erase(score); + } + } + } + caption = clean_caption(caption); + if (short_form) { + caption = JSON{{"short_description", caption.value("short_description", JSON(""))}, + {"text_render", caption.value("text_render", JSON::array())}}; + } + return {caption, instruction}; + } + + static std::string format_prompt(const std::string& text, int num_images) { + auto [caption, instruction] = parse_prompt(text); + if (num_images > 0) { + if (instruction.is_null()) { + caption = nullptr; + instruction = text; + } + auto edit = JSON::object(); + for (int i = 0; i < num_images; ++i) { + edit[num_images == 1 ? "image" : "image_" + std::to_string(i + 1)] = "<|vision_start|><|image_pad|><|vision_end|>"; + } + if (!empty_value(caption)) { + edit["structured_caption"] = caption; + } + edit["edit_instruction"] = instruction; + return edit.dump(); + } + if (text.empty()) { + return R"({"short_description":"","objects":[],"background_setting":"","lighting":{},"aesthetics":{},"photographic_characteristics":{},"style_medium":"","context":"","artistic_style":""})"; + } + if (caption.is_string()) { + return caption.get(); + } + return empty_value(caption) ? "{}" : caption.dump(); + } + + static sd::Tensor make_bundles(sd::Tensor states, const std::set& layers) { + constexpr int width = 2560; + constexpr int bundles[6][3] = {{9, 20, 31}, {5, 16, 27}, {7, 18, 29}, {9, 20, 31}, {11, 22, 33}, {13, 24, 35}}; + for (int64_t token = 0; token < states.shape()[1]; ++token) { + for (size_t layer = 0; layer < layers.size(); ++layer) { + float sum = 0.f; + int64_t offset = token * states.shape()[0] + layer * width; + for (int i = 0; i < width; ++i) { + sum += states[offset + i] * states[offset + i]; + } + float scale = 1.f / std::sqrt(sum / width + 1e-6f); + for (int i = 0; i < width; ++i) { + states[offset + i] *= scale; + } + } + } + sd::Tensor result({6 * 3 * width, states.shape()[1]}); + for (int b = 0; b < 6; ++b) { + for (int j = 0; j < 3; ++j) { + int index = static_cast(std::distance(layers.begin(), layers.find(bundles[b][j]))); + for (int64_t token = 0; token < states.shape()[1]; ++token) { + std::copy_n(states.data() + token * states.shape()[0] + index * width, width, + result.data() + token * result.shape()[0] + (b * 3 + j) * width); + } + } + } + return result; + } + + SDCondition get_learned_condition(int n_threads, const ConditionerParams& p) override { + int image_count = p.ref_images ? static_cast(p.ref_images->size()) : 0; + if (image_count && !llm->enable_vision) { + LOG_ERROR("Fibo 2.0 editing requires vision weights"); + return {}; + } + std::string prompt = "<|im_start|>user\n" + format_prompt(p.text, image_count) + "<|im_end|>\n<|im_start|>assistant\n"; + std::vector>> images; + LLM::DeepStackImageEmbeds deepstack; + std::vector grids; + size_t marker_start = 0; + for (int i = 0; i < image_count; ++i) { + const auto& image = (*p.ref_images)[i]; + int factor = llm->config.vision.patch_size * llm->config.vision.spatial_merge_size; + int height = static_cast(image.shape()[1]), width = static_cast(image.shape()[0]); + int h = std::max(factor, static_cast(std::round(static_cast(height) / factor)) * factor); + int w = std::max(factor, static_cast(std::round(static_cast(width) / factor)) * factor); + resize_image_dims(height, width, h, w, factor, + p.ref_image_params.vlm_min_size, + p.ref_image_params.vlm_max_size, + p.ref_image_params.vlm_resize_mode); + auto pixels = sd::ops::interpolate(image, {w, h, image.shape()[2], image.shape()[3]}, sd::ops::InterpolateMode::Bicubic, false, true); + auto rgb = sd::Tensor({w, h, 3, 1}); + for (int64_t pixel = 0; pixel < int64_t(w) * h; ++pixel) { + float alpha = pixels.shape()[2] == 4 ? pixels[pixel + 3 * w * h] : 1.f; + for (int channel = 0; channel < 3; ++channel) { + rgb[pixel + channel * w * h] = 2.f * std::clamp(pixels[pixel + channel * w * h] * alpha + 1.f - alpha, 0.f, 1.f) - 1.f; + } + } + auto outputs = llm->encode_image_outputs(n_threads, rgb, false); + if (!expand_vision_marker(prompt, marker_start, std::move(outputs), + h / llm->config.vision.patch_size, w / llm->config.vision.patch_size, + images, deepstack, grids, 4096)) { + return {}; + } + } + const std::set layers = {5, 7, 9, 11, 13, 16, 18, 20, 22, 24, 27, 29, 31, 33, 35}; + auto states = encode_prompt(n_threads, prompt, {0, 0}, 0, 0, images, layers, 0, false, 4096, deepstack, grids); + if (states.empty()) { + return {}; + } + SDCondition result; + result.c_crossattn = make_bundles(std::move(states), layers); + return result; + } +}; + +#endif // __SD_CONDITIONING_FIBO2_HPP__ diff --git a/src/core/ggml_extend_backend.cpp b/src/core/ggml_extend_backend.cpp index 9eafe6f1d..f1ff3795a 100644 --- a/src/core/ggml_extend_backend.cpp +++ b/src/core/ggml_extend_backend.cpp @@ -296,6 +296,10 @@ bool add_rpc_devices(const std::string& servers) { return false; } ggml_backend_reg_t rpc_reg = ggml_backend_reg_by_name("RPC"); + if (!rpc_reg) { + ggml_backend_load_all(); + rpc_reg = ggml_backend_reg_by_name("RPC"); + } if (!rpc_reg) { LOG_ERROR("RPC backend not found, cannot add RPC servers"); return false; @@ -309,12 +313,19 @@ bool add_rpc_devices(const std::string& servers) { for (const auto& server : rpc_servers) { LOG_INFO("Adding RPC server: %s", server.c_str()); auto reg = ggml_backend_rpc_add_server_fn(server.c_str()); - // no return value to check for success but should print errors from the RPC backend if it fails to add the server + if (!reg) { + LOG_ERROR("failed to add RPC server: %s", server.c_str()); + return false; + } ggml_backend_register(reg); } return true; } +bool sd_add_rpc_servers(const char* servers) { + return add_rpc_devices(servers ? servers : ""); +} + static void ggml_backend_load_all_once() { // If the registry already has devices and the CPU backend is present, // assume either static registration or explicit host-side preloading has diff --git a/src/core/util.cpp b/src/core/util.cpp index c1d463e4d..a691b423c 100644 --- a/src/core/util.cpp +++ b/src/core/util.cpp @@ -343,7 +343,7 @@ int32_t sd_get_num_physical_cores() { // enumerate the set of thread siblings, num entries is num cores std::unordered_set siblings; for (uint32_t cpu = 0; cpu < UINT32_MAX; ++cpu) { - std::ifstream thread_siblings("/sys/devices/system/cpu" + std::to_string(cpu) + "/topology/thread_siblings"); + std::ifstream thread_siblings("/sys/devices/system/cpu/cpu" + std::to_string(cpu) + "/topology/thread_siblings"); if (!thread_siblings.is_open()) { break; // no more cpus } @@ -568,50 +568,59 @@ bool parse_strict_bool(const std::string& text, bool& value) { } static std::string build_progress_bar(int step, int steps, char progress_char = '=', bool show_head = true) { - std::string progress = " |"; - int max_progress = 50; - int32_t current = 0; + std::string progress = " |"; + constexpr int max_progress = 50; + int32_t current = 0; if (steps > 0) { - current = (int32_t)(step * 1.f * max_progress / steps); + current = (int32_t)(std::clamp(step, 0, steps) * 1.f * max_progress / steps); } - for (int i = 0; i < 50; i++) { - if (i > current) { - progress += " "; - } else if (show_head && i == current && i != max_progress - 1) { - progress += ">"; - } else { - progress += progress_char; - } - } - progress += "|"; + int head_length = (current == max_progress) ? 0 : 1; + progress += std::string(current, progress_char) + std::string(head_length, show_head ? '>' : progress_char) + std::string(max_progress - current - head_length, ' ') + "|"; return progress; } -static void print_progress_line(int step, int steps, const std::string& speed_text, char progress_char = '=', bool show_head = true) { +static void print_progress_line(int step, int steps, const std::string& speed_text, char progress_char = '=', bool show_head = true, const std::string& eta_text = "") { if (step == 0) { return; } std::string progress = build_progress_bar(step, steps, progress_char, show_head); + int step_width = static_cast(std::to_string(steps).size()); const char* lf = (step == steps ? "\n" : ""); - printf("\r%s %i/%i - %s\033[K%s", progress.c_str(), step, steps, speed_text.c_str(), lf); + printf("\r%s %*i/%i - %10s%s%s\033[K%s", progress.c_str(), step_width, step, steps, speed_text.c_str(), + eta_text.empty() ? "" : " - ETA ", eta_text.c_str(), lf); fflush(stdout); // for linux } +static int show_eta = -1; // -1 - estimate duration, 0 - hide, 1 - show +static float passed_time = 0.f; + void pretty_progress(int step, int steps, float time) { if (sd_progress_cb) { sd_progress_cb(step, steps, time, sd_progress_cb_data); return; } if (step == 0) { + show_eta = -1; // reinitialize for batch generations + passed_time = 0.f; return; } + std::string remainder_text = ""; + if (show_eta && step < steps) { // if timer is not disabled and bar is not filled, + if (show_eta == -1 && step) // estimate the total time once, then + show_eta = (std::abs(time * steps / step) < 15.f) ? 0 : 1; // hide the timer if total is less than 15 seconds + if (show_eta) { + unsigned remainder = (passed_time += time) / step * (steps - step); // average speed; time*(steps-step) for current + remainder_text = remainder < 3600 ? sd_format("%02u:%02u", remainder / 60, remainder % 60) + : sd_format("%u:%02u:%02u", remainder / 3600, (remainder / 60) % 60, remainder % 60); + } + } const char* unit = "s/it"; float speed = time; if (speed < 1.0f && speed > 0.f) { speed = 1.0f / speed; unit = "it/s"; } - print_progress_line(step, steps, sd_format("%.2f%s", speed, unit)); + print_progress_line(step, steps, sd_format("%.2f%s", speed, unit), '=', true, remainder_text); } void pretty_bytes_progress(int step, int steps, uint64_t bytes_processed, float elapsed_seconds) { diff --git a/src/detailer.cpp b/src/detailer.cpp index 72d01dc6f..ec4d6065a 100644 --- a/src/detailer.cpp +++ b/src/detailer.cpp @@ -291,7 +291,9 @@ static void filter_and_sort_detections(std::vector* detectio std::partial_sort(detections->begin(), detections->begin() + params.mask_k_largest, detections->end(), - [](const auto& a, const auto& b) { return detection_area(a) > detection_area(b); }); + [&](const auto& a, const auto& b) { + return params.mask_filter_by_confidence ? a.confidence > b.confidence : detection_area(a) > detection_area(b); + }); detections->resize(params.mask_k_largest); } @@ -545,7 +547,16 @@ static sd_image_t resize_crop_mask(const Mask& source, const CropRegion& crop, i return output; } -static sd_image_t copy_as_rgb(const sd_image_t& source) { +static sd_image_t copy_for_detailing(const sd_image_t& source) { + if (source.channel == 4) { + sd_image_t output = source; + size_t bytes = static_cast(source.width) * source.height * source.channel; + output.data = static_cast(malloc(bytes)); + if (output.data != nullptr) { + memcpy(output.data, source.data, bytes); + } + return output; + } CropRegion full{0, 0, static_cast(source.width), static_cast(source.height)}; return resize_crop_image(source, full, source.width, source.height); } @@ -566,7 +577,7 @@ static void composite_crop(sd_image_t* destination, continue; } float source_x = (x - crop.x1 + 0.5f) * scale_x - 0.5f; - size_t destination_index = (static_cast(y) * destination->width + x) * 3; + size_t destination_index = (static_cast(y) * destination->width + x) * destination->channel; for (int c = 0; c < 3; ++c) { float generated_value = bilinear_channel(generated, source_x, source_y, c) * 255.f; float original_value = destination->data[destination_index + c]; @@ -787,6 +798,10 @@ static bool parse_adetailer_params(const sd_adetailer_params_t& public_params, ok = parse_int(value, ¶ms->max_detections); } else if (key == "mask_k_largest") { ok = parse_int(value, ¶ms->mask_k_largest); + } else if (key == "mask_filter_method") { + const std::string method = lower_copy(value); + ok = method == "area" || method == "confidence"; + params->mask_filter_by_confidence = method == "confidence"; } else if (key == "mask_min_ratio") { ok = parse_float(value, ¶ms->mask_min_ratio); } else if (key == "mask_max_ratio") { @@ -805,6 +820,8 @@ static bool parse_adetailer_params(const sd_adetailer_params_t& public_params, ok = parse_int(value, ¶ms->mask_blur); } else if (key == "inpaint_padding") { ok = parse_int(value, ¶ms->inpaint_padding); + } else if (key == "inpaint_only_masked") { + ok = parse_bool(value, ¶ms->inpaint_only_masked); } else if (key == "inpaint_width") { ok = parse_int(value, ¶ms->inpaint_width); } else if (key == "inpaint_height") { @@ -920,7 +937,7 @@ bool adetail_image(adetailer_ctx_t* context, filter_and_sort_detections(&detections, input_image.width, input_image.height, params); auto masks = make_masks(detections, input_image.width, input_image.height, params); - sd_image_t current = copy_as_rgb(input_image); + sd_image_t current = copy_for_detailing(input_image); if (current.data == nullptr) { return false; } @@ -939,12 +956,13 @@ bool adetail_image(adetailer_ctx_t* context, if (!mask_bbox(masks[i], &crop)) { continue; } - crop = expand_crop(crop, - current.width, - current.height, - params.inpaint_padding, - params.inpaint_width, - params.inpaint_height); + crop = params.inpaint_only_masked ? expand_crop(crop, + current.width, + current.height, + params.inpaint_padding, + params.inpaint_width, + params.inpaint_height) + : CropRegion{0, 0, static_cast(current.width), static_cast(current.height)}; sd_image_t local_image = resize_crop_image(current, crop, params.inpaint_width, params.inpaint_height); sd_image_t local_mask = resize_crop_mask(masks[i], crop, params.inpaint_width, params.inpaint_height); if (local_image.data == nullptr || local_mask.data == nullptr) { diff --git a/src/detailer.h b/src/detailer.h index 17a81a1dd..0e3ae50f5 100644 --- a/src/detailer.h +++ b/src/detailer.h @@ -27,30 +27,32 @@ enum ADetailerSort { }; struct ADetailerParams { - const char* prompt = nullptr; - const char* negative_prompt = nullptr; - int input_size = 640; - float confidence = 0.3f; - float nms_threshold = 0.45f; - int max_detections = 100; - int mask_k_largest = 0; - float mask_min_ratio = 0.f; - float mask_max_ratio = 1.f; - int dilate_erode = 4; - int x_offset = 0; - int y_offset = 0; - bool merge_masks = false; - bool invert_mask = false; - int mask_blur = 4; - int inpaint_padding = 32; - int inpaint_width = 512; - int inpaint_height = 512; - float denoising_strength = 0.4f; - int steps = 0; - float cfg_scale = -1.f; - sample_method_t sample_method = SAMPLE_METHOD_COUNT; - scheduler_t scheduler = SCHEDULER_COUNT; - ADetailerSort sort_by = ADETAILER_SORT_NONE; + const char* prompt = nullptr; + const char* negative_prompt = nullptr; + int input_size = 640; + float confidence = 0.3f; + float nms_threshold = 0.45f; + int max_detections = 100; + int mask_k_largest = 0; + bool mask_filter_by_confidence = false; + float mask_min_ratio = 0.f; + float mask_max_ratio = 1.f; + int dilate_erode = 4; + int x_offset = 0; + int y_offset = 0; + bool merge_masks = false; + bool invert_mask = false; + int mask_blur = 4; + int inpaint_padding = 32; + bool inpaint_only_masked = true; + int inpaint_width = 512; + int inpaint_height = 512; + float denoising_strength = 0.4f; + int steps = 0; + float cfg_scale = -1.f; + sample_method_t sample_method = SAMPLE_METHOD_COUNT; + scheduler_t scheduler = SCHEDULER_COUNT; + ADetailerSort sort_by = ADETAILER_SORT_NONE; }; struct ADetailerGGML { diff --git a/src/model.h b/src/model.h index 0c1a37a66..73a369301 100644 --- a/src/model.h +++ b/src/model.h @@ -68,6 +68,7 @@ enum SDVersion { VERSION_IRIS, VERSION_FLUX3, VERSION_MOSS_IMAGE, + VERSION_FIBO2, VERSION_COUNT, }; @@ -293,7 +294,8 @@ static inline bool sd_version_uses_flux2_vae(SDVersion version) { sd_version_is_ideogram4(version) || sd_version_is_sefi_image(version) || sd_version_is_llada_image(version) || - version == VERSION_MOSS_IMAGE) { + version == VERSION_MOSS_IMAGE || + version == VERSION_FIBO2) { return true; } return false; @@ -351,7 +353,8 @@ static inline bool sd_version_is_dit(SDVersion version) { sd_version_is_sensenova_u1(version) || sd_version_is_pixart(version) || version == VERSION_IRIS || - version == VERSION_MOSS_IMAGE) { + version == VERSION_MOSS_IMAGE || + version == VERSION_FIBO2) { return true; } return false; @@ -385,7 +388,8 @@ static inline bool sd_version_supports_ref_image_conditioning(SDVersion version) sd_version_is_mage_flow(version) || sd_version_is_anima(version) || sd_version_is_sefi_image(version) || - sd_version_is_pid(version); + sd_version_is_pid(version) || + version == VERSION_FIBO2; } static bool sd_version_is_inpaint_or_unet_edit(SDVersion version) { diff --git a/src/model/diffusion/fibo2.hpp b/src/model/diffusion/fibo2.hpp new file mode 100644 index 000000000..f53b6ecd1 --- /dev/null +++ b/src/model/diffusion/fibo2.hpp @@ -0,0 +1,395 @@ +#ifndef __SD_MODEL_DIFFUSION_FIBO2_HPP__ +#define __SD_MODEL_DIFFUSION_FIBO2_HPP__ + +#include "model/diffusion/z_image.hpp" + +// https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_bria_fibo2.py +namespace Fibo2 { + struct Fibo2Config { + int64_t dim = 3584; + int64_t channels = 128; + int64_t text_dim = 7680; + int64_t head_dim = 128; + int64_t depth = 44; + int64_t refiner_depth = 2; + int64_t perceiver_depth = 2; + int64_t max_gist = 512; + int64_t ffn_dim = 9536; + bool fused_qkv = false; + bool fused_kv = false; + bool fused_ffn = false; + std::vector injection_layers = {6, 14, 22, 30, 38}; + + static Fibo2Config detect_from_weights(const String2TensorStorage& weights, const std::string& prefix) { + Fibo2Config config; + int64_t depth = 0, refiner_depth = 0, perceiver_depth = 0; + std::set injection_layers; + for (const auto& [name, tensor] : weights) { + if (!starts_with(name, prefix + ".")) { + continue; + } + auto key = name.substr(prefix.size() + 1); + if (key == "x_embedder.weight") { + config.channels = tensor.ne[0]; + config.dim = tensor.ne[1]; + } else if (key == "perceiver.latent_queries") { + config.max_gist = tensor.ne[1]; + } else if (key == "perceiver.layers.0.cross_attention.to_k.weight" || key == "perceiver.layers.0.cross_attention.to_kv.weight") { + config.text_dim = tensor.ne[0]; + config.fused_kv = key == "perceiver.layers.0.cross_attention.to_kv.weight"; + } else if (key == "layers.0.attention.norm_q.weight") { + config.head_dim = tensor.ne[0]; + } else if (key == "layers.0.attention.qkv.weight") { + config.fused_qkv = true; + } else if (key == "layers.0.feed_forward.w1.weight") { + config.ffn_dim = tensor.ne[1]; + } else if (key == "layers.0.feed_forward.w13.weight") { + config.ffn_dim = tensor.ne[1] / 2; + config.fused_ffn = true; + } + if (starts_with(key, "layers.")) { + int index = std::stoi(key.substr(7)); + depth = std::max(depth, index + 1); + if (ends_with(key, ".cross_attn_gate")) { + injection_layers.insert(index); + } + } else if (starts_with(key, "noise_refiner.")) { + refiner_depth = std::max(refiner_depth, std::stoi(key.substr(14)) + 1); + } else if (starts_with(key, "perceiver.layers.")) { + perceiver_depth = std::max(perceiver_depth, std::stoi(key.substr(17)) + 1); + } + } + if (depth) { + config.depth = depth; + config.refiner_depth = refiner_depth; + config.perceiver_depth = perceiver_depth; + config.injection_layers.assign(injection_layers.begin(), injection_layers.end()); + LOG_VERBOSE("fibo2: dim = %" PRId64 ", depth = %" PRId64 ", head_dim = %" PRId64 ", text_dim = %" PRId64, + config.dim, config.depth, config.head_dim, config.text_dim); + } + return config; + } + + int gist_count(int text_length) const { + double ratio = std::clamp((text_length - 256.0) / (2048.0 - 256.0), 0.0, 1.0); + return std::min(static_cast(max_gist), static_cast(std::ceil((256 + ratio * (max_gist - 256)) / 64)) * 64); + } + }; + + struct Attention : public GGMLBlock { + int64_t head_dim; + int64_t heads; + bool fused_qkv; + bool fused_kv; + + Attention(const Fibo2Config& config, int64_t kv_dim, bool joint = false) + : head_dim(config.head_dim), heads(config.dim / config.head_dim), fused_qkv(joint && config.fused_qkv), fused_kv(!joint && config.fused_kv) { + if (fused_qkv) { + blocks["qkv"] = std::make_shared(config.dim, 3 * config.dim, false); + } else { + blocks["to_q"] = std::make_shared(config.dim, config.dim, false); + if (fused_kv) { + blocks["to_kv"] = std::make_shared(kv_dim, 2 * config.dim, false); + } else { + blocks["to_k"] = std::make_shared(kv_dim, config.dim, false); + blocks["to_v"] = std::make_shared(kv_dim, config.dim, false); + } + } + blocks["to_out.0"] = std::make_shared(config.dim, config.dim, false); + blocks["norm_q"] = std::make_shared(head_dim, 1e-5f); + blocks["norm_k"] = std::make_shared(head_dim, 1e-5f); + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* kv, ggml_tensor* pe = nullptr) { + auto c = ctx->ggml_ctx; + auto project = [&](const char* name, ggml_tensor* h) { + return std::dynamic_pointer_cast(blocks[name])->forward(ctx, h); + }; + ggml_tensor *q, *k, *v; + if (fused_qkv) { + auto parts = ggml_ext_chunk(c, project("qkv", x), 3, 0); + q = parts[0]; + k = parts[1]; + v = parts[2]; + } else { + q = project("to_q", x); + if (fused_kv) { + auto parts = ggml_ext_chunk(c, project("to_kv", kv), 2, 0); + k = parts[0]; + v = parts[1]; + } else { + k = project("to_k", kv); + v = project("to_v", kv); + } + } + auto reshape = [&](ggml_tensor* h) { + return ggml_reshape_4d(c, h, head_dim, heads, h->ne[1], h->ne[2]); + }; + q = std::dynamic_pointer_cast(blocks["norm_q"])->forward(ctx, reshape(q)); + k = std::dynamic_pointer_cast(blocks["norm_k"])->forward(ctx, reshape(k)); + v = reshape(v); + auto prepare = [&](ggml_tensor* h) { + if (pe) { + return Rope::apply_rope(ctx, h, pe); + } + h = ggml_cont(c, ggml_permute(c, h, 0, 2, 1, 3)); + return ggml_reshape_3d(c, h, head_dim, h->ne[1], heads * h->ne[3]); + }; + auto y = ggml_ext_attention_ext(ctx, prepare(q), prepare(k), v, heads, nullptr, true, ctx->flash_attn_enabled); + return std::dynamic_pointer_cast(blocks["to_out.0"])->forward(ctx, y); + } + }; + + struct FeedForward : public ZImage::FeedForward { + bool fused; + + FeedForward(const Fibo2Config& config) + : ZImage::FeedForward(config.dim, config.ffn_dim, 1), fused(config.fused_ffn) { + if (fused) { + blocks.erase("w1"); + blocks.erase("w3"); + blocks["w13"] = std::make_shared(config.dim, 2 * config.ffn_dim, false); + } + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x) { + if (!fused) { + return ZImage::FeedForward::forward(ctx, x); + } + auto w2 = std::dynamic_pointer_cast(blocks["w2"]); + if (sd_backend_is(ctx->backend, "Vulkan") || sd_backend_is(ctx->backend, "ROCm")) { + w2->set_force_prec_f32(true); + } + x = std::dynamic_pointer_cast(blocks["w13"])->forward(ctx, x); + return w2->forward(ctx, ggml_swiglu(ctx->ggml_ctx, x)); + } + }; + + struct PerceiverLayer : public GGMLBlock { + PerceiverLayer(const Fibo2Config& config) { + for (const char* name : {"cross_attn_norm", "self_attn_norm", "ffn_norm"}) { + blocks[name] = std::make_shared(config.dim, 1e-5f); + } + blocks["cross_attention"] = std::make_shared(config, config.text_dim); + blocks["self_attention"] = std::make_shared(config, config.dim); + blocks["ffn"] = std::make_shared(config); + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* text) { + auto norm = [&](const char* name, ggml_tensor* h) { + return std::dynamic_pointer_cast(blocks[name])->forward(ctx, h); + }; + auto h = std::dynamic_pointer_cast(blocks["cross_attention"])->forward(ctx, norm("cross_attn_norm", x), text); + x = ggml_add(ctx->ggml_ctx, x, h); + h = norm("self_attn_norm", x); + h = std::dynamic_pointer_cast(blocks["self_attention"])->forward(ctx, h, h); + x = ggml_add(ctx->ggml_ctx, x, h); + h = std::dynamic_pointer_cast(blocks["ffn"])->forward(ctx, norm("ffn_norm", x)); + return ggml_add(ctx->ggml_ctx, x, h); + } + }; + + struct Perceiver : public GGMLBlock { + Fibo2Config config; + + Perceiver(const Fibo2Config& config) + : config(config) { + for (int i = 0; i < config.perceiver_depth; ++i) { + blocks["layers." + std::to_string(i)] = std::make_shared(config); + } + blocks["norm_out"] = std::make_shared(config.dim, 1e-5f); + } + + void init_params(ggml_context* ctx, const String2TensorStorage& weights = {}, const std::string prefix = "") override { + params["latent_queries"] = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, config.dim, config.max_gist); + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* text, int gist_count) { + auto x = ggml_ext_slice(ctx->ggml_ctx, params["latent_queries"], 1, 0, gist_count); + for (int i = 0; i < config.perceiver_depth; ++i) { + x = std::dynamic_pointer_cast(blocks["layers." + std::to_string(i)])->forward(ctx, x, text); + } + return std::dynamic_pointer_cast(blocks["norm_out"])->forward(ctx, x); + } + }; + + struct TransformerBlock : public GGMLBlock { + bool injection; + + TransformerBlock(const Fibo2Config& config, bool injection) + : injection(injection) { + blocks["attention"] = std::make_shared(config, config.dim, true); + blocks["feed_forward"] = std::make_shared(config); + for (const char* name : {"attention_norm1", "attention_norm2", "ffn_norm1", "ffn_norm2"}) { + blocks[name] = std::make_shared(config.dim, 1e-5f); + } + blocks["adaLN_modulation.0"] = std::make_shared(std::min(config.dim, 256), 4 * config.dim); + if (injection) { + blocks["cross_attention"] = std::make_shared(config, config.text_dim); + blocks["cross_attn_norm1"] = std::make_shared(config.dim, 1e-5f); + blocks["cross_attn_norm2"] = std::make_shared(config.dim, 1e-5f); + } + } + + void init_params(ggml_context* ctx, const String2TensorStorage& weights = {}, const std::string prefix = "") override { + if (injection) { + params["cross_attn_gate"] = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1); + } + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* pe, ggml_tensor* noisy, ggml_tensor* clean, int image_start, int image_count, ggml_tensor* text = nullptr) { + auto c = ctx->ggml_ctx; + auto norm = [&](const char* name, ggml_tensor* h) { + return std::dynamic_pointer_cast(blocks[name])->forward(ctx, h); + }; + auto modulation = std::dynamic_pointer_cast(blocks["adaLN_modulation.0"]); + auto mn = ggml_ext_chunk(c, modulation->forward(ctx, noisy), 4, 0); + auto mc = ggml_ext_chunk(c, modulation->forward(ctx, clean), 4, 0); + auto select = [&](int index, bool gate) { + auto n = gate ? ggml_tanh(c, mn[index]) : mn[index]; + auto k = gate ? ggml_tanh(c, mc[index]) : mc[index]; + auto repeat = [&](ggml_tensor* h, int64_t count) { + return ggml_repeat(c, h, ggml_new_tensor_2d(c, GGML_TYPE_F32, h->ne[0], count)); + }; + auto m = repeat(n, image_count); + if (image_start) { + m = ggml_concat(c, repeat(k, image_start), m, 1); + } + int64_t tail = x->ne[1] - image_start - image_count; + if (tail) { + m = ggml_concat(c, m, repeat(k, tail), 1); + } + return m; + }; + auto h = ZImage::modulate(c, norm("attention_norm1", x), select(0, false)); + h = std::dynamic_pointer_cast(blocks["attention"])->forward(ctx, h, h, pe); + x = ggml_add(c, x, ggml_mul(c, norm("attention_norm2", h), select(1, true))); + if (injection) { + h = std::dynamic_pointer_cast(blocks["cross_attention"])->forward(ctx, norm("cross_attn_norm1", x), text); + h = ggml_mul(c, norm("cross_attn_norm2", h), ggml_tanh(c, params["cross_attn_gate"])); + x = ggml_add(c, x, h); + } + h = ZImage::modulate(c, norm("ffn_norm1", x), select(2, false)); + h = std::dynamic_pointer_cast(blocks["feed_forward"])->forward(ctx, h); + return ggml_add(c, x, ggml_mul(c, norm("ffn_norm2", h), select(3, true))); + } + }; + + struct Fibo2Model : public GGMLBlock { + Fibo2Config config; + + Fibo2Model(const Fibo2Config& config) + : config(config) { + blocks["x_embedder"] = std::make_shared(config.channels, config.dim); + blocks["t_embedder"] = std::make_shared(1024, 256, std::min(config.dim, 256)); + blocks["perceiver"] = std::make_shared(config); + for (int i = 0; i < config.refiner_depth; ++i) { + blocks["noise_refiner." + std::to_string(i)] = std::make_shared(config, false); + } + for (int i = 0; i < config.depth; ++i) { + bool injection = std::find(config.injection_layers.begin(), config.injection_layers.end(), i) != config.injection_layers.end(); + blocks["layers." + std::to_string(i)] = std::make_shared(config, injection); + } + blocks["final_layer"] = std::make_shared(config.dim, 1, config.channels); + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* t, ggml_tensor* text, ggml_tensor* pe, int gist_count, const std::vector& refs) { + auto c = ctx->ggml_ctx; + int64_t width = x->ne[0], height = x->ne[1]; + int image_count = static_cast(width * height); + // The shared FLUX.2 VAE already packs (channel, dy, dx) into 128 channels. + auto tokens = [&](ggml_tensor* h) { + h = ggml_reshape_3d(c, h, h->ne[0] * h->ne[1], config.channels, 1); + return ggml_cont(c, ggml_permute(c, h, 1, 0, 2, 3)); + }; + x = tokens(x); + for (auto ref : refs) { + x = ggml_concat(c, x, tokens(ref), 1); + } + x = std::dynamic_pointer_cast(blocks["x_embedder"])->forward(ctx, x); + auto embedder = std::dynamic_pointer_cast(blocks["t_embedder"]); + auto noisy = embedder->forward(ctx, t); + auto clean = embedder->forward(ctx, ggml_scale(c, t, 0.f)); + auto image_pe = ggml_ext_slice(c, pe, 3, gist_count, pe->ne[3]); + for (int i = 0; i < config.refiner_depth; ++i) { + x = std::dynamic_pointer_cast(blocks["noise_refiner." + std::to_string(i)])->forward(ctx, x, image_pe, noisy, clean, 0, image_count); + sd::ggml_graph_cut::mark_graph_cut(x, "fibo2.noise_refiner." + std::to_string(i), "x"); + } + auto bundle = [&](int i) { return ggml_ext_slice(c, text, 0, i * config.text_dim, (i + 1) * config.text_dim); }; + auto gist = std::dynamic_pointer_cast(blocks["perceiver"])->forward(ctx, bundle(0), gist_count); + x = ggml_concat(c, gist, x, 1); + int bundle_index = 1; + for (int i = 0; i < config.depth; ++i) { + auto block = std::dynamic_pointer_cast(blocks["layers." + std::to_string(i)]); + auto context = block->injection ? bundle(bundle_index++) : nullptr; + x = block->forward(ctx, x, pe, noisy, clean, gist_count, image_count, context); + sd::ggml_graph_cut::mark_graph_cut(x, "fibo2.layers." + std::to_string(i), "x"); + } + x = ggml_ext_slice(c, x, 1, gist_count, gist_count + image_count); + x = std::dynamic_pointer_cast(blocks["final_layer"])->forward(ctx, x, noisy); + x = ggml_cont(c, ggml_permute(c, x, 1, 0, 2, 3)); + return ggml_reshape_4d(c, x, width, height, config.channels, 1); + } + }; + + struct Fibo2Runner : public DiffusionModelRunner { + Fibo2Config config; + Fibo2Model model; + std::vector pe_values; + + Fibo2Runner(ggml_backend_t backend, const String2TensorStorage& weights, const std::string& prefix, std::shared_ptr weight_manager = nullptr) + : DiffusionModelRunner(backend, prefix, weight_manager), + config(Fibo2Config::detect_from_weights(weights, prefix)), + model(config) { + model.init(params_ctx, weights, prefix); + } + + std::string get_desc() override { return "fibo2"; } + + void get_param_tensors(std::map& tensors, const std::string& prefix) override { + model.get_param_tensors(tensors, prefix); + } + + sd::Tensor compute(int n_threads, const DiffusionParams& p) override { + if (!p.x || !p.timesteps || !p.context || p.context->shape()[0] != config.text_dim * 6) { + LOG_ERROR("Fibo 2.0 requires six text feature bundles"); + return {}; + } + auto build = [&]() { + auto graph = new_graph_custom(65536); + auto x = make_input(*p.x); + GGML_ASSERT(x->ne[3] == 1 && x->ne[2] == config.channels); + int count = config.gist_count(static_cast(p.context->shape()[1])); + Rope::Embedding embedding; + for (int i = 0; i < count; ++i) { + embedding.ids.push_back({static_cast(i + 1), 0.f, 0.f}); + } + embedding.positions.append_tokens(count); + auto append_image = [&](ggml_tensor* image, int plane) { + auto ids = Rope::gen_flux_img_ids(static_cast(image->ne[1]), static_cast(image->ne[0]), + 1, 1, 3, plane, 0, 0, false, &embedding.positions); + embedding.ids.insert(embedding.ids.end(), ids.begin(), ids.end()); + }; + append_image(x, static_cast(config.max_gist + 1)); + std::vector refs; + if (p.ref_latents && p.ref_image_params.pass_to_dit) { + for (const auto& ref : *p.ref_latents) { + refs.push_back(make_input(ref)); + append_image(refs.back(), static_cast(config.max_gist + 1 + refs.size())); + } + } + embedding.values = Rope::embed_nd(embedding.ids, 1, 256.f, {32, 48, 48}, embedding.layout, &embedding.frequencies); + pe_values = finish_rope_pe(std::move(embedding)); + auto pe = ggml_new_tensor_4d(compute_ctx, GGML_TYPE_F32, 2, 2, 64, static_cast(pe_values.size() / 256)); + set_backend_tensor_data(pe, pe_values.data()); + auto ctx = get_context(); + auto out = model.forward(&ctx, x, make_input(*p.timesteps), make_input(*p.context), pe, count, refs); + ggml_build_forward_expand(graph, out); + return graph; + }; + return restore_trailing_singleton_dims(GGMLRunner::compute(build, n_threads, false), p.x->dim()); + } + }; +} + +#endif // __SD_MODEL_DIFFUSION_FIBO2_HPP__ diff --git a/src/model/diffusion/hidream_o1.hpp b/src/model/diffusion/hidream_o1.hpp index 2befb723a..ca1559bcb 100644 --- a/src/model/diffusion/hidream_o1.hpp +++ b/src/model/diffusion/hidream_o1.hpp @@ -56,15 +56,6 @@ namespace HiDreamO1 { } }; - static inline std::string repeat_special_token(const std::string& token, int64_t count) { - std::string out; - out.reserve(static_cast(count) * token.size()); - for (int64_t i = 0; i < count; ++i) { - out += token; - } - return out; - } - static inline std::pair calculate_dimensions(int max_size, double ratio) { int width = static_cast(std::sqrt(max_size * max_size * ratio)); int height = static_cast(width / ratio); @@ -405,7 +396,7 @@ namespace HiDreamO1 { txt = LLM::splice_image_embeds(&runner_ctx, txt, image_embeds); auto t_emb = t_embedder1->forward(&runner_ctx, timestep); - int64_t txt_seq_len = input_ids->ne[0]; + int64_t txt_seq_len = txt->ne[1]; if (txt_seq_len > 1) { auto prefix = ggml_ext_slice(compute_ctx, txt, 1, 0, txt_seq_len - 1); txt = ggml_concat(compute_ctx, prefix, ggml_reshape_3d(compute_ctx, t_emb, t_emb->ne[0], 1, 1), 1); @@ -616,8 +607,7 @@ namespace HiDreamO1 { sd::Tensor vlm_image = sd::ops::interpolate( resized_ref, {dims.first, dims.second, resized_ref.shape()[2], resized_ref.shape()[3]}); - vlm_image = vlm_image * 2.0f - 1.0f; - int64_t image_tokens = static_cast(dims.first / PATCH_SIZE) * static_cast(dims.second / PATCH_SIZE); + vlm_image = vlm_image * 2.0f - 1.0f; auto patch_img = resized_ref * 2.0f - 1.0f; result.c_ref_images.push_back(std::move(patch_img)); @@ -626,8 +616,11 @@ namespace HiDreamO1 { return {}; } int64_t prompt_start = static_cast(prefix_tokens.size()); + for (const auto& grid : image_grids) { + prompt_start += static_cast(grid[1]) * grid[2] - 1; + } prompt += "<|vision_start|>"; - prompt += repeat_special_token("<|image_pad|>", image_tokens); + prompt += "<|image_pad|>"; prompt += "<|vision_end|>"; vlm_images.emplace_back(static_cast(prompt_start), std::move(vlm_image)); image_grids.push_back({1, dims.second / PATCH_SIZE, dims.first / PATCH_SIZE}); @@ -642,6 +635,13 @@ namespace HiDreamO1 { } std::vector input_ids_pad = input_ids; + for (size_t i = 0; i < vlm_images.size(); ++i) { + const auto& grid = image_grids[i]; + const int64_t count = static_cast(grid[1]) * grid[2]; + input_ids_pad.insert(input_ids_pad.begin() + vlm_images[i].first, count - 1, IMAGE_TOKEN_ID); + } + + int txt_seq_len = static_cast(input_ids_pad.size()); input_ids_pad.push_back(VISION_START_TOKEN_ID); input_ids_pad.insert(input_ids_pad.end(), target_image_len - 1, IMAGE_TOKEN_ID); image_grids.push_back({1, static_cast(height / PATCH_SIZE), static_cast(width / PATCH_SIZE)}); @@ -656,8 +656,7 @@ namespace HiDreamO1 { } std::vector token_types(input_ids_pad.size(), 0); - int txt_seq_len = static_cast(input_ids.size()); - int bgn = txt_seq_len - TIMESTEP_TOKEN_NUM; + int bgn = txt_seq_len - TIMESTEP_TOKEN_NUM; for (int i = bgn; i < static_cast(token_types.size()); ++i) { token_types[i] = 1; } diff --git a/src/model/diffusion/model.hpp b/src/model/diffusion/model.hpp index 955c5277a..7c14d9d1f 100644 --- a/src/model/diffusion/model.hpp +++ b/src/model/diffusion/model.hpp @@ -27,9 +27,25 @@ struct RefImageParams { int vlm_min_size = -1; int vlm_max_size = -1; bool resize_vae_to_target = false; + bool resize_single_ref_to_target = false; + bool vae_size_buckets = false; + bool vae_center_crop = false; + bool vlm_use_vae_input = false; }; const std::unordered_map REF_IMAGE_PRESETS = { + {"fibo2", [] { + RefImageParams params; + params.pass_to_vlm = true; + params.ref_index_mode = Rope::RefIndexMode::INCREASE; + params.vlm_resize_mode = RefImageResizeMode::LONGEST_SIDE; + params.vlm_max_size = 768; + params.resize_single_ref_to_target = true; + params.vae_size_buckets = true; + params.vae_center_crop = true; + params.vlm_use_vae_input = true; + return params; + }()}, {"flux_kontext", {false, true, Rope::RefIndexMode::FIXED, false, true, -1, RefImageResizeMode::NONE, -1, -1}}, {"longcat", {true, true, Rope::RefIndexMode::FIXED, false, true, -1, RefImageResizeMode::AREA, -1, -1}}, {"flux2", {false, true, Rope::RefIndexMode::INCREASE, false, true, -1, RefImageResizeMode::NONE, -1, -1}}, diff --git a/src/model/te/llm.hpp b/src/model/te/llm.hpp index 8ea918025..85f751169 100644 --- a/src/model/te/llm.hpp +++ b/src/model/te/llm.hpp @@ -924,6 +924,8 @@ namespace LLM { int64_t txt_token_start = 0; int64_t txt_token_end = 0; ggml_tensor* input_embed = nullptr; + // Image and DeepStack indices remain in the expanded sequence coordinates. + int64_t removed_tokens = 0; for (int i = 0; i < image_embeds.size(); i++) { if (i == 0) { @@ -931,7 +933,10 @@ namespace LLM { } else { txt_token_start = image_embeds[i - 1].first + image_embeds[i - 1].second->ne[1]; } - txt_token_end = image_embeds[i].first; + txt_token_start -= removed_tokens; + txt_token_end = image_embeds[i].first - removed_tokens; + removed_tokens += image_embeds[i].second->ne[1] - 1; + GGML_ASSERT(txt_token_start >= 0 && txt_token_end >= txt_token_start && txt_token_end < raw_x->ne[1]); // An embed can sit flush against the previous one or at the very start/end of the // sequence, leaving no text tokens to splice around it. @@ -951,14 +956,14 @@ namespace LLM { } } - txt_token_start = image_embeds[image_embeds.size() - 1].first + image_embeds[image_embeds.size() - 1].second->ne[1]; + txt_token_start = image_embeds[image_embeds.size() - 1].first + image_embeds[image_embeds.size() - 1].second->ne[1] - removed_tokens; txt_token_end = raw_x->ne[1]; if (txt_token_end > txt_token_start) { auto final_txt_embed = ggml_ext_slice(ctx->ggml_ctx, raw_x, 1, txt_token_start, txt_token_end); input_embed = ggml_concat(ctx->ggml_ctx, input_embed, final_txt_embed, 1); } - GGML_ASSERT(raw_x->ne[1] == input_embed->ne[1]); + GGML_ASSERT(raw_x->ne[1] + removed_tokens == input_embed->ne[1]); return input_embed; } @@ -2506,6 +2511,9 @@ namespace LLM { } int64_t n_tokens = input_ids->ne[0]; + for (const auto& image_embed : image_embeds) { + n_tokens += image_embed.second->ne[1] - 1; + } if (config.arch == LLMArch::MISTRAL_SMALL_3_2 || config.arch == LLMArch::MINISTRAL_3_3B || config.arch == LLMArch::QWEN3 || @@ -3014,20 +3022,17 @@ namespace LLM { LOG_VERBOSE("llm encode_image test done in %lldms", t1 - t0); } - std::string placeholder = "<|image_pad|>"; - std::string img_prompt = "Picture 1: <|vision_start|>"; // [24669, 220, 16, 25, 220, 151652] - int64_t num_image_tokens = image_embed.shape()[1]; - img_prompt.reserve(num_image_tokens * placeholder.size()); - for (int i = 0; i < num_image_tokens; i++) { - img_prompt += placeholder; - } - img_prompt += "<|vision_end|>"; - - std::vector>> image_embeds; - image_embeds.emplace_back(64, image_embed); + const std::string image_prefix = "Picture 1: <|vision_start|>"; + const std::string img_prompt = image_prefix + "<|image_pad|><|vision_end|>"; std::pair prompt_attn_range; std::string text = "<|im_start|>system\nDescribe the key features of the input image (color, shape, size, texture, objects, background), then explain how the user's text instruction should alter or modify the image. Generate a new image that meets the user's requirements while maintaining consistency with the original input where appropriate.<|im_end|>\n<|im_start|>user\n"; + std::vector prefix_tokens; + if (!tokenizer->encode(text + image_prefix, prefix_tokens, nullptr)) { + return; + } + std::vector>> image_embeds; + image_embeds.emplace_back(static_cast(prefix_tokens.size()), image_embed); text += img_prompt; prompt_attn_range.first = static_cast(text.size()); text += "change 'flux.cpp' to 'edit.cpp'"; diff --git a/src/model_loader.cpp b/src/model_loader.cpp index 90ab65eed..03bcd74a9 100644 --- a/src/model_loader.cpp +++ b/src/model_loader.cpp @@ -439,6 +439,10 @@ bool ModelLoader::init_from_diffusers_file(const std::string& file_path, const s SDVersion ModelLoader::get_sd_version() const { TensorStorage token_embedding_weight, input_block_weight, context_ebedding_weight; + if (tensor_storage_map.count("model.diffusion_model.perceiver.latent_queries")) { + return VERSION_FIBO2; + } + if (tensor_storage_map.find("model.diffusion_model.emb_in.image.weight") != tensor_storage_map.end() && tensor_storage_map.find("model.diffusion_model.txt_mode_blocks.0.q_proj.weight") != tensor_storage_map.end()) { return VERSION_FLUX3; diff --git a/src/model_manager.cpp b/src/model_manager.cpp index 05ee55f67..4a3103aa1 100644 --- a/src/model_manager.cpp +++ b/src/model_manager.cpp @@ -2004,12 +2004,12 @@ bool ModelManager::ensure_compute_backend_capacity( }; size_t last_eviction_count = 0; auto release_eviction_states = [&]() { - if (eviction_states.size() == last_eviction_count) { - return false; + if (eviction_states.size() != last_eviction_count) { + last_eviction_count = eviction_states.size(); + release_compute_staging_blocks(false, &eviction_states); + release_params_storage_blocks(false, &eviction_states); } - last_eviction_count = eviction_states.size(); - release_compute_staging_blocks(false, &eviction_states); - release_params_storage_blocks(false, &eviction_states); + // Device memory reports can lag behind releases even when no new candidates are added. return fits(); }; diff --git a/src/name_conversion.cpp b/src/name_conversion.cpp index 6a519ba21..f37c4ddbe 100644 --- a/src/name_conversion.cpp +++ b/src/name_conversion.cpp @@ -197,7 +197,11 @@ std::string convert_cond_stage_model_name(std::string name, std::string prefix, replace_with_name_map(name, llm_vision_name_map); } else { replace_with_name_map(name, llm_name_map); - name = convert_llada2_moe_te_name(name); + name = convert_llada2_moe_te_name(name); + const std::string language_prefix = "llm.model.language_model."; + if (starts_with(name, language_prefix)) { + name = "llm.model." + name.substr(language_prefix.size()); + } } } else { name = convert_open_clip_to_hf_clip_name(name); @@ -1034,6 +1038,18 @@ static std::string convert_ming_image_dit_name(std::string name) { return name; } +static std::string convert_fibo2_dit_name(std::string name) { + static const std::vector> name_map = { + {".attention.q_norm.", ".attention.norm_q."}, + {".attention.k_norm.", ".attention.norm_k."}, + {".attention.out.", ".attention.to_out.0."}, + {".cross_attention.to_out.weight", ".cross_attention.to_out.0.weight"}, + {".self_attention.to_out.weight", ".self_attention.to_out.0.weight"}, + }; + replace_with_name_map(name, name_map); + return name; +} + std::string convert_diffusers_dit_to_original_minimax_h3(std::string name) { static const std::vector> prefix_map = { {"transformer_blocks.", "blocks."}, @@ -1067,6 +1083,8 @@ std::string convert_diffusion_model_name(std::string name, std::string prefix, S name = convert_moss_image_dit_name(name); } else if (version == VERSION_MING_IMAGE) { name = convert_ming_image_dit_name(name); + } else if (version == VERSION_FIBO2) { + name = convert_fibo2_dit_name(name); } else if (sd_version_is_z_image(version) || sd_version_is_z_image_l2p(version)) { name = convert_diffusers_dit_to_original_lumina2(name); } else if (sd_version_is_llada_image(version)) { @@ -1715,7 +1733,7 @@ std::string convert_tensor_name(std::string name, SDVersion version) { } } - if (version == VERSION_QWEN_IMAGE_2_1 || sd_version_is_boogu_image(version) || sd_version_is_krea2(version) || sd_version_is_mage_flow(version) || sd_version_is_minimax_h3(version)) { + if (version == VERSION_FIBO2 || version == VERSION_QWEN_IMAGE_2_1 || sd_version_is_boogu_image(version) || sd_version_is_krea2(version) || sd_version_is_mage_flow(version) || sd_version_is_minimax_h3(version)) { const std::string hf_vision_prefix = "text_encoders.llm.model.visual."; if (starts_with(name, hf_vision_prefix)) { name = "text_encoders.llm.visual." + name.substr(hf_vision_prefix.size()); diff --git a/src/pipeline/diffusion_engine.cpp b/src/pipeline/diffusion_engine.cpp index 1297ca020..75a234542 100644 --- a/src/pipeline/diffusion_engine.cpp +++ b/src/pipeline/diffusion_engine.cpp @@ -114,6 +114,7 @@ const char* model_version_to_str[] = { "Iris-3B", "Flux.3 Image Klein", "MOSS-Image", + "Fibo-2.0", }; static_assert(VERSION_COUNT == sizeof(model_version_to_str) / sizeof(model_version_to_str[0]), @@ -1520,7 +1521,8 @@ bool StableDiffusionGGML::build_denoiser() { sd_version_is_pid(version) || sd_version_is_ideogram4(version) || version == VERSION_IRIS || - version == VERSION_MOSS_IMAGE) { + version == VERSION_MOSS_IMAGE || + version == VERSION_FIBO2) { pred_type = FLOW_PRED; if (sd_version_is_wan(version) || version == VERSION_MOSS_IMAGE) { default_flow_shift = 5.f; @@ -3283,7 +3285,9 @@ bool StableDiffusionGGML::is_flow_denoiser() { } std::string StableDiffusionGGML::get_default_ref_image_preset(SDVersion version) const { - if (sd_version_is_longcat(version)) { + if (version == VERSION_FIBO2) { + return "fibo2"; + } else if (sd_version_is_longcat(version)) { return "longcat"; } else if (sd_version_is_flux(version)) { return "flux_kontext"; @@ -3297,7 +3301,7 @@ std::string StableDiffusionGGML::get_default_ref_image_preset(SDVersion version) return "qwen"; } else if (sd_version_is_mage_flow(version)) { return "mage_flow"; - } else if (sd_version_is_z_image(version) || sd_version_is_boogu_image(version)) { + } else if (sd_version_is_boogu_image(version)) { return "z_image_omni"; } else if (sd_version_is_llada_image(version)) { return "llada_image"; @@ -3335,6 +3339,9 @@ RefImageParams StableDiffusionGGML::resolve_ref_image_params(const char* ref_ima LOG_INFO("Using '%s' preset for reference images", preset_name.c_str()); params = REF_IMAGE_PRESETS.at(preset_name); } + if (version == VERSION_QWEN_IMAGE_2_1) { + params.vlm_use_vae_input = true; + } for (const auto& [key, value] : parse_key_value_args(ref_image_args, "reference image args")) { if (key == "pass_to_vlm") { @@ -3363,6 +3370,26 @@ RefImageParams StableDiffusionGGML::resolve_ref_image_params(const char* ref_ima if (!parse_strict_bool(value, params.resize_before_vae)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); } + } else if (key == "resize_vae_to_target") { + if (!parse_strict_bool(value, params.resize_vae_to_target)) { + LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + } + } else if (key == "resize_single_ref_to_target") { + if (!parse_strict_bool(value, params.resize_single_ref_to_target)) { + LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + } + } else if (key == "vae_size_buckets") { + if (!parse_strict_bool(value, params.vae_size_buckets)) { + LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + } + } else if (key == "vae_center_crop") { + if (!parse_strict_bool(value, params.vae_center_crop)) { + LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + } + } else if (key == "vlm_use_vae_input") { + if (!parse_strict_bool(value, params.vlm_use_vae_input)) { + LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); + } } else if (key == "vae_input_max_pixels") { if (!parse_strict_int(value, params.vae_input_max_pixels)) { LOG_WARN("ignoring invalid reference image arg '%s=%s'", key.c_str(), value.c_str()); diff --git a/src/pipeline/image.cpp b/src/pipeline/image.cpp index 7de85b359..d20c02e6b 100644 --- a/src/pipeline/image.cpp +++ b/src/pipeline/image.cpp @@ -286,26 +286,34 @@ namespace sd::pipeline { LOG_VERBOSE("auto resize ref images"); double vae_width; double vae_height; - if (ref_image_params.resize_vae_to_target) { + int factor = sd_version_is_qwen_image(sd->version) ? 32 : 16; + if (ref_image_params.resize_vae_to_target || (ref_image_params.resize_single_ref_to_target && ref_images.size() == 1)) { vae_width = request->width; vae_height = request->height; } else { int default_pixels = sd->version == VERSION_QWEN_IMAGE_2_1 ? request->width * request->height : 1024 * 1024; int target_pixels = ref_image_params.vae_input_max_pixels > 0 ? ref_image_params.vae_input_max_pixels : default_pixels; - int vae_image_size = std::min(target_pixels, request->width * request->height); - vae_width = sqrt(vae_image_size * ref_images[i].shape()[0] / ref_images[i].shape()[1]); - vae_height = vae_width * ref_images[i].shape()[1] / ref_images[i].shape()[0]; + if (ref_image_params.vae_size_buckets) { + auto size = sd::select_image_size_bucket(static_cast(ref_images[i].shape()[0]), static_cast(ref_images[i].shape()[1]), target_pixels, factor, request->width, request->height); + vae_width = size.first; + vae_height = size.second; + } else { + int vae_image_size = std::min(target_pixels, request->width * request->height); + vae_width = sqrt(vae_image_size * ref_images[i].shape()[0] / ref_images[i].shape()[1]); + vae_height = vae_width * ref_images[i].shape()[1] / ref_images[i].shape()[0]; + } } - int factor = sd_version_is_qwen_image(sd->version) ? 32 : 16; vae_height = round(vae_height / factor) * factor; vae_width = round(vae_width / factor) * factor; - auto resized_ref_img = sd::ops::interpolate(ref_images[i], - {static_cast(vae_width), - static_cast(vae_height), - ref_images[i].shape()[2], - ref_images[i].shape()[3]}); + auto resized_ref_img = ref_image_params.vae_center_crop + ? sd::resize_and_center_crop(ref_images[i], static_cast(vae_width), static_cast(vae_height)) + : sd::ops::interpolate(ref_images[i], + {static_cast(vae_width), + static_cast(vae_height), + ref_images[i].shape()[2], + ref_images[i].shape()[3]}); LOG_VERBOSE("resize vae ref image %d from %" PRId64 "x%" PRId64 " to %" PRId64 "x%" PRId64, static_cast(i), @@ -315,7 +323,7 @@ namespace sd::pipeline { resized_ref_img.shape()[0]); ref_latent = sd->encode_first_stage(resized_ref_img); - if (sd->version == VERSION_QWEN_IMAGE_2_1) { + if (ref_image_params.vlm_use_vae_input) { ref_images[i] = std::move(resized_ref_img); } } else { @@ -484,8 +492,8 @@ namespace sd::pipeline { } condition_params.text = request->negative_prompt; condition_params.zero_out_masked = zero_out_masked; - if (sd_version_is_llada_image(sd->version)) { - // LLaDA-Image CFG keeps the source latent but drops its SigVQ features. + if (sd_version_is_llada_image(sd->version) || sd->version == VERSION_FIBO2) { + // Keep source latents in CFG, but omit source images from the text encoder. condition_params.ref_images = nullptr; } uncond = sd->get_learned_condition(condition_params); diff --git a/src/pipeline/model_builders.cpp b/src/pipeline/model_builders.cpp index 4b435170f..ca4d1998b 100644 --- a/src/pipeline/model_builders.cpp +++ b/src/pipeline/model_builders.cpp @@ -5,6 +5,7 @@ #include #include "conditioning/conditioner.hpp" +#include "conditioning/fibo2.hpp" #include "core/ggml_extend_backend.h" #include "core/util.h" #include "extensions/generation_extension.h" @@ -14,6 +15,7 @@ #include "model/diffusion/boogu.hpp" #include "model/diffusion/control.hpp" #include "model/diffusion/ernie_image.hpp" +#include "model/diffusion/fibo2.hpp" #include "model/diffusion/flux.hpp" #include "model/diffusion/flux3.hpp" #include "model/diffusion/hidream_o1.hpp" @@ -94,7 +96,12 @@ namespace sd::model_builders { return false; } - if (sd_version_is_sd3(version)) { + if (version == VERSION_FIBO2) { + result.conditioner = std::make_shared(ctx.backends.runtime_backend(SDBackendModule::TE), + tensor_storage_map, weight_manager, tokenizers); + result.diffusion = std::make_shared(ctx.backends.runtime_backend(SDBackendModule::DIFFUSION), + tensor_storage_map, "model.diffusion_model", weight_manager); + } else if (sd_version_is_sd3(version)) { result.conditioner = std::make_shared(ctx.backends.runtime_backend(SDBackendModule::TE), tensor_storage_map, weight_manager, diff --git a/src/pipeline/request.cpp b/src/pipeline/request.cpp index 4256fee4e..830a5fb99 100644 --- a/src/pipeline/request.cpp +++ b/src/pipeline/request.cpp @@ -65,7 +65,7 @@ namespace sd::pipeline { return LCM_SCHEDULER; } else if (sample_method == DDIM_TRAILING_SAMPLE_METHOD) { return SIMPLE_SCHEDULER; - } else if (sd != nullptr && (sd_version_is_flux(sd->version) || sd->version == VERSION_QWEN_IMAGE_2_1)) { + } else if (sd != nullptr && (sd_version_is_flux(sd->version) || sd->version == VERSION_QWEN_IMAGE_2_1 || sd->version == VERSION_FIBO2)) { return FLUX_SCHEDULER; } else if (sd != nullptr && sd_version_is_flux2(sd->version)) { return FLUX2_SCHEDULER; @@ -329,6 +329,11 @@ namespace sd::pipeline { } void GenerationRequest::resolve(StableDiffusionGGML* sd) { + if (sd_version_is_minit2i(sd->version) && (width != 512 || height != 512)) { + width = 512; + height = 512; + LOG_WARN("align to fixed size 512x512 for MiniT2I"); + } align_generation_request_size(); resolve_hires(); seed = resolve_seed(seed); diff --git a/src/runtime/image_preprocess.cpp b/src/runtime/image_preprocess.cpp index a8df7c3b5..ec649390b 100644 --- a/src/runtime/image_preprocess.cpp +++ b/src/runtime/image_preprocess.cpp @@ -6,6 +6,42 @@ namespace sd { + std::pair select_image_size_bucket(int source_width, int source_height, int area, int factor, int fallback_width, int fallback_height) { + int width = fallback_width; + int height = fallback_height; + double ratio = static_cast(source_width) / source_height; + double best_crop = INFINITY; + int64_t best_area_error = INT64_MAX; + int min_size = 4 * factor; + for (int w = min_size; w < 2LL * area / min_size; w += factor) { + for (int h = min_size; h < 2LL * area / w; h += factor) { + int64_t pixels = int64_t(w) * h; + if (pixels < 0.9 * area || pixels > 1.1 * area) { + continue; + } + double candidate_ratio = static_cast(w) / h; + double crop = std::round((1.0 - std::min(candidate_ratio / ratio, ratio / candidate_ratio)) * 1e9); + int64_t area_error = std::abs(pixels - area); + if (crop < best_crop || (crop == best_crop && (area_error < best_area_error || (area_error == best_area_error && w > width)))) { + width = w; + height = h; + best_crop = crop; + best_area_error = area_error; + } + } + } + return {width, height}; + } + + Tensor resize_and_center_crop(const Tensor& image, int width, int height, ops::InterpolateMode filter) { + double scale = std::max(static_cast(width) / image.shape()[0], static_cast(height) / image.shape()[1]); + int w = static_cast(std::ceil(image.shape()[0] * scale)); + int h = static_cast(std::ceil(image.shape()[1] * scale)); + auto resized = ops::interpolate(image, {w, h, image.shape()[2], image.shape()[3]}, filter); + resized = ops::slice(resized, 0, (w - width) / 2, (w - width) / 2 + width); + return ops::slice(resized, 1, (h - height) / 2, (h - height) / 2 + height); + } + static constexpr std::pair image_targets[] = { {"init", ImageTarget::Init}, {"end", ImageTarget::End}, diff --git a/src/runtime/image_preprocess.h b/src/runtime/image_preprocess.h index b75c4e0e5..7dfc45899 100644 --- a/src/runtime/image_preprocess.h +++ b/src/runtime/image_preprocess.h @@ -3,6 +3,7 @@ #include #include +#include #include #include "core/tensor.hpp" @@ -10,6 +11,9 @@ namespace sd { + std::pair select_image_size_bucket(int source_width, int source_height, int area, int factor, int fallback_width, int fallback_height); + Tensor resize_and_center_crop(const Tensor& image, int width, int height, ops::InterpolateMode filter = ops::InterpolateMode::Lanczos); + enum class ImageTarget { Init, End,