• Keep the proven ofxStableDiffusion structure first; stable-diffusion.cpp

• Keep wrapper APIs, examples, and docs close to ofxStableDiffusion while the

• Treat ofxGgmlDiffusion, GGUF GAN experiments, and unrelated model workflows

• Use ofxGgmlCore as the default ggml provider for ecosystem builds. A

Requirements

• openFrameworks: 0.11.0 or later (tested with 0.12.0)
• C++ Standard: C++17 or later
• Platform Support: Windows (x64), Linux (x64), macOS (experimental Metal support)

• typed request/config/result objects
• background-thread generation
• owned image/video outputs
• explicit native-library staging
• wrapper-level callback seams for optional companion-addon scoring
• lightweight unit tests for the new video helper layer

Highlights

• Text-to-image and image-to-image generation
• Native image modes: TextToImage, ImageToImage, and Inpainting
• Best-of-N image reranking through a callback seam that can be driven by an

• Image-to-video generation with Standard, Loop, PingPong, and Boomerang presentation modes
• ofxGgmlStableDiffusionRealtimeVideoSession for live creative-loop previews using prompt coalescing, low-step frames, and optional previous-frame img2img feedback
• ESRGAN upscaling support
• Progress callbacks for diffusion steps
• getCapabilities() plus capability/model-family helpers for UI gating and backend-aware workflows
• parameter-tuning helpers for image/video defaults, ranges, and clamping by model family
• Optional ofxGgmlStableDiffusionHoloscanBridge scaffold for live frame -> conditioning -> diffusion -> preview pipelines, with a native Holoscan runtime path on Linux and a clean fallback path when Holoscan is not installed or the platform is not supported yet
• ofxGgmlStableDiffusionVideoWorkflowHelpers.h for reusable video-generation presets, request validation, and richer render-manifest export on top of the existing request/result layer
• Legacy entry points still available for wrapper-level migration
• Core-backed ggml runtime management by default, with bundled ggml fallback staging

Repo Layout

• src/

• src/core/

• src/video/

• FastPreview, LowVram, Balanced, Quality, and BatchStoryboard presets
• preflight request validation before a longer render starts
• richer JSON render manifests that capture prompt, dimensions, fps, seed, LoRAs, and clip summary data
• libs/stable-diffusion/

• scripts/

• tests/

• docs/

Feature Readiness

Features

• stable-diffusion.cpp wrapper lane
• typed image generation requests
• image-to-video helper workflows
• ofxGgmlCore ggml provider by default
• bundled ggml fallback runtime staging
• local wrapper validation entrypoint

Threading Contract

• ofxGgmlStableDiffusion is internally synchronized unless a method explicitly documents borrowed-pointer or blocking behavior.
• generate(), generateVideo(), configureContext(), and setUpscalerSettings() may be called from any thread, but only one long-running task can run at a time.
• isGenerating(), isBusy(), requestCancellation(), isCancellationRequested(), and copied-result accessors (getLastResult(), getImages(), getVideoClip()) are safe to query from any thread.
• getImagePixels(), getVideoFramePixels(), and returnImages() expose borrowed buffers; use them only with care and never cache the returned pointers across generations.
• setProgressCallback() and setImageRankCallback() run on the worker thread. Keep those callbacks lightweight and defer UI / texture updates to your main update() / draw() flow.
• ofxGgmlStableDiffusionRealtimeSession and ofxGgmlStableDiffusionRealtimeVideoSession dispatch their callbacks from whichever thread calls update().

API Shape

Modern typed wrapper API

#include "ofxGgmlStableDiffusion.h"

ofxGgmlStableDiffusion sd;

ofxGgmlStableDiffusionContextSettings context;
context.modelPath = "data/models/sd/sd_turbo.safetensors";
context.nThreads = 8;
context.weightType = SD_TYPE_COUNT;
// Optional: let the current native runtime fit the model to available GPUs.
// A negative maxVram value keeps that many GiB free per selected device.
context.maxVram = "-1";
context.autoFit = true;
context.streamLayers = true;
sd.configureContext(context);

ofxGgmlStableDiffusionImageRequest request;
request.prompt = "cinematic portrait, rim lighting";
request.width = 512;
request.height = 512;
request.sampleSteps = 20;
sd.generate(request);

// Optional: apply a stack of LoRA/LoCon adapters per request
ofxGgmlStableDiffusionLora loraA;
loraA.path = "data/loras/edge.safetensors";
loraA.strength = 0.7f;
request.loras = {loraA};
sd.generate(request);

// Update the active LoRA stack globally
sd.setLoras({loraA});

// Hot-reload textual-inversion embeddings (reloads the context)
sd.reloadEmbeddings("data/embeddings");

// List currently discoverable embeddings (name, absolute path)
const auto embeddings = sd.listEmbeddings();

// Inspect what the currently configured model/runtime can actually do
const auto capabilities = sd.getCapabilities();

// Save generated video frames plus a JSON sidecar with per-frame prompts/seeds
sd.saveVideoFramesWithMetadata("output/storyboard", "shot");

const auto imageProfile =
    ofxGgmlStableDiffusionParameterTuningHelpers::resolveImageProfile(
        context,
        ofxGgmlStableDiffusionImageMode::ImageToImage);

request.cfgScale = imageProfile.defaultCfgScale;
request.sampleSteps = imageProfile.defaultSampleSteps;
request.strength = imageProfile.defaultStrength;

Error Handling

ofxGgmlStableDiffusionImageRequest request;
request.width = 513;  // Invalid: not a multiple of 64
request.height = 512;
request.batchCount = 20;  // Invalid: exceeds maximum of 16

sd.generate(request);

// Check for errors programmatically
if (sd.getLastErrorCode() != ofxGgmlStableDiffusionErrorCode::None) {
    // Get detailed error information
    const auto& errorInfo = sd.getLastErrorInfo();
    ofLogError() << "Error: " << errorInfo.message;
    ofLogError() << "Suggestion: " << errorInfo.suggestion;
    ofLogError() << "Error code: " << ofxGgmlStableDiffusionErrorCodeLabel(errorInfo.code);
}

// Or use the simple string-based error (backward compatible)
if (!sd.getLastError().empty()) {
    ofLogError() << "Error: " << sd.getLastError();
}

// Access error history for debugging
const auto& history = sd.getErrorHistory();
for (const auto& error : history) {
    ofLogNotice() << "[" << error.timestampMicros << "] "
                  << error.message << " -> " << error.suggestion;
}

// Clear error history
sd.clearErrorHistory();

• None - No error
• ModelNotFound - Model file not found
• ModelCorrupted - Model file corrupted
• ModelLoadFailed - Model loading failed
• OutOfMemory - Insufficient memory
• InvalidDimensions - Invalid width/height
• InvalidBatchCount - Invalid batch count
• InvalidFrameCount - Invalid video frame count
• MissingInputImage - Input image required but not provided
• GenerationFailed - Generation process failed
• ThreadBusy - Another task is running
• UpscaleFailed - Upscaling failed
• Unknown - Unknown error

Legacy compatibility API

Realtime Creative Loop

#include "ofxGgmlStableDiffusion.h"

ofxGgmlStableDiffusionRealtimeVideoSettings liveSettings;
liveSettings.previewSteps = 4;
liveSettings.refineSteps = 16;
liveSettings.previewStrength = 0.68f;
liveSettings.refineStrength = 0.35f;
liveSettings.usePreviousFrameFeedback = true;

ofxGgmlStableDiffusionRealtimeVideoSession liveVideo;
liveVideo.start(liveSettings, sd);

ofxGgmlStableDiffusionRealtimeVideoRequest liveRequest;
liveRequest.prompt = "a fashion portrait reacting to colored stage lights";
liveVideo.submit(liveRequest);

// In update():
liveVideo.update();

Video Behavior

• ofxGgmlStableDiffusionVideoClip::saveMetadataJson(...)
• ofxGgmlStableDiffusionVideoClip::saveFrameSequenceWithMetadata(...)
• ofxGgmlStableDiffusionVideoClip::saveWebm(...)
• ofxGgmlStableDiffusion::saveVideoMetadata(...)
• ofxGgmlStableDiffusion::saveVideoFramesWithMetadata(...)

• ofxGgmlStableDiffusionLongVideoManifest + ofxGgmlStableDiffusionLongVideoChunk

• ofxGgmlStableDiffusion::renderLongVideo(...) renders all chunks sequentially,

• Standard

• Loop

• PingPong

• Boomerang

Video Performance Recommendations

• Use fast models for previews: Prefer LCM/Turbo-style checkpoints with 4-8 steps for quick iteration; only re-run hero frames at higher quality.
• Quantize when possible: F16 or Q8/Q5 levels often cut VRAM use by 50-75% and speed up inference, enabling higher resolutions or longer clips without swapping.
• Keep the model warm and resident: Reuse the same loaded context via the model manager; avoid swapping checkpoints mid-run and consider a throwaway warmup frame to eliminate first-run latency.
• Minimize unique frames: Generate the smallest necessary source frame count, then stretch duration with PingPong, Boomerang, or Loop playback instead of regenerating.
• Right-size resolution, fps, and steps: Lower resolution and fps where acceptable; clamp sampleSteps to ~15-25 for finals (lower for previews). Per-frame time scales directly with unique frames.
• Trim preview I/O: Skip metadata/JSON exports on preview passes; save sidecar files only on final renders to avoid extra disk churn.

Coding Conventions (openFrameworks-aligned)