fix: open model files with UTF-8 paths on Windows

[therealshulgin]

Companion of #2145 (the same fix in the CLI and server, for outputs and input images). The two PRs are independent and each builds on its own.

Problem

Paths reach the library as UTF-8: the CLI converts its arguments with
argv_to_utf8() and the server reads JSON. On Windows, the library then
opens them with narrow APIs that Windows decodes in the ANSI code page:
GetFileAttributesA/CreateFileA (file_exists, is_directory,
MmapWrapper::create) and std::ifstream/std::ofstream/std::fstream
constructed from a std::string. Any non-ASCII character in a path then
names a different file, so a model stored under e.g. C:\Users\Hélène\ or
D:\modèles\ cannot be loaded.

read_file_stamp() in model_loader_files.cpp already uses
std::filesystem::u8path; this applies the same idiom to the other opens.

Changes

• src/core/util.cpp: file_exists, is_directory and
  MmapWrapper::create use GetFileAttributesW / CreateFileW on
  std::filesystem::u8path(path).
• Model readers and writers (gguf, safetensors, torch legacy, the
  safetensors index and shard paths, the loader's non-mmap path),
  --convert output, imatrix load/save, the HF tokenizer and tensor
  dumps open their files through std::filesystem::u8path(...).
• The GGUF writer uses the existing ggml_fopen, which already converts
  UTF-8 to UTF-16 on Windows.

No public API change. On Linux/macOS u8path constructs the same path as
before.

Verification

Windows 11, MSVC 14.44, -DSD_CUDA=OFF, master 8cd731e, FLUX.2 klein 4B
Q4_0 (256x256, 4 steps, seed 1), the diffusion model hard-linked into a
directory named modèles:

	before	after
--diffusion-model ...\modèles\klein.gguf	loading diffusion model from '...' failed, get sd version from file failed	image written
same model through an ASCII path	image written	identical image (pixel for pixel)

Known limitation

Zip-based torch checkpoints (.ckpt / .pt) are opened by miniz, which
converts UTF-8 to UTF-16 only under _MSC_VER; a MinGW build still opens
them with narrow fopen. That is in thirdparty/, so it is left out here.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Improved compatibility with non-ASCII file paths when loading and saving models, tensors, tokenizer data, and other files, particularly on Windows.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: a7593ac6-ff75-454b-8358-c0638bb38b54

📥 Commits

Reviewing files that changed from the base of the PR and between 8cd731e and 9320d46.

📒 Files selected for processing: 11

• src/convert.cpp
• src/core/ggml_tensor_utils.cpp
• src/core/tensor_ggml.hpp
• src/core/util.cpp
• src/model_io/gguf_io.cpp
• src/model_io/gguf_reader_ext.h
• src/model_io/safetensors_io.cpp
• src/model_io/torch_legacy_io.cpp
• src/model_loader.cpp
• src/runtime/imatrix.cpp
• src/tokenizers/hf_tokenizer.cpp

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Walkthrough

File access paths across conversion, model loading, tensor utilities, tokenizers, and imatrix now use UTF-8 filesystem path conversion. Windows file checks and memory mapping use wide-character APIs. GGUF streaming output now uses ggml_fopen.

Changes: UTF-8 filesystem path handling

Layer / File(s)	Summary
UTF-8 paths for model and tensor inputs src/model_io/safetensors_io.cpp, src/model_io/gguf_io.cpp, src/model_io/gguf_reader_ext.h, src/model_io/torch_legacy_io.cpp, src/model_loader.cpp, src/core/ggml_tensor_utils.cpp, src/core/tensor_ggml.hpp, src/tokenizers/hf_tokenizer.cpp	Input streams now construct paths with std::filesystem::u8path. Safetensors shard paths are resolved and normalized as filesystem paths.
UTF-8 paths for output and runtime files src/convert.cpp, src/model_io/gguf_io.cpp, src/model_io/safetensors_io.cpp, src/runtime/imatrix.cpp	Conversion, Safetensors, and imatrix file operations use UTF-8 filesystem paths. GGUF streaming output uses ggml_fopen.
Windows file checks and mapping src/core/util.cpp	Windows file and directory checks use GetFileAttributesW. Memory mapping opens paths with CreateFileW after UTF-8 path conversion.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: leejet, stduhpf

Merge Risk: ⚪ Minimal · up to 9320d

The described model and runtime paths use UTF-8-aware filesystem handling, and the inspected Safetensors shard path preserves UTF-8. No actionable merge-blocking regression is established.

Security architecture risk: 🔵 Low · up to 9320d

The change makes Windows filename handling consistently UTF-8-aware while preserving existing access modes and validation. No introduced security concern was identified in the inspected paths. Application-level path authorization and Windows failure/recovery behavior remain outside the demonstrated coverage.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The affected authority is filesystem access available to the hosting process: reading supplied model and tokenizer paths, following index-selected shard paths, and writing configured outputs. Correct Unicode filename reachability expands which intended filenames work on Windows, but the inspected changes introduce no new credential, elevation mechanism, or filesystem access mode.

Trust Boundaries and Controls

• observed — Safetensors shard resolution already accepted absolute paths and normalized relative paths against the index directory. It is not a directory-containment control, and this PR does not remove one. Whether callers may supply untrusted indexes or arbitrary filesystem paths requires application-level authorization context not supplied here.

Resilience and Maintainability Implications

• observed — Streaming output now uses consistent UTF-8 interpretation for metadata creation, preallocation, and worker writes. Existing sequencing, offset-based writes, worker joins, and failure propagation remain. Output is still modified in place without transactional replacement or deletion of partial files on failure; interruption, conflicting destination writers, and retry behavior are pre-existing lifecycle limitations, not introduced concerns.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 4.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 25 functions across 11 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly and concisely describes the main change: enabling UTF-8 model file paths on Windows.
Description check		The description is detailed and relevant. It explains the problem, lists the affected areas, documents verification results, and states a known limitation. It does not use the template headings or inc…

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[laurentvv]

Verified on the official master-970 release binaries (2f016e8) — this PR and companion #2145. Thank you.

Datapoint (Windows 11, ANSI code page 1252, RX 6950 XT Vulkan, official sd-master-*-bin-win-vulkan-x64.zip builds, SDXL 512×512 20 steps cfg 7 seed 42, same-session A/B):

Before (8cd731e): loading an SDXL checkpoint through a path with accented characters fails hard. Rig: a junction named with é/à pointing at the model directory, so the argument is -m "C:\modèles\juggernautXL….safetensors":

[W] file C:\modèles\juggernautXL….safetensors not found
[E] init model loader from file failed: 'C:\modèles\juggernautXL….safetensors'

exit 1 — while the identical file loads fine through its non-accented path. Exactly the narrow-API (GetFileAttributesA/CreateFileA/ifstream) re-decode described in this PR.

After (2f016e8): the same accented path loads and renders normally, and the output at the same seed is pixel-identical to the ASCII-path run (0 differing pixels out of 262 144; the file hash differs only by the PNG metadata, which embeds the model path). The fix does not touch generation, as expected.

Companion #2145 validated in the same run — its "before" behavior is nastier than a clean error, see the comment there.