ofxAddons
← Back

ofxStableDiffusion

github.com/Jonathhhan/ofxStableDiffusion · @Jonathhhan
OF 0.12 Artificial Intelligence Machine Learning

Open Frameworks addon that wraps this stable diffusion library: https://github.com/leejet/stable-diffusion.cpp

16 2 Last commit 5mo ago Created 2y ago

Branches ahead of main

These branches have commits not yet on main — could be unmerged fixes or work in progress worth a look.

claude/fix-image-variation-crash 2 commits ahead · last commit 5mo ago
claude/improve-interaction-ofxggml 1 commit ahead · last commit 6mo ago
claude/explain-repository-structure 7 commits ahead · last commit 6mo ago

README

ofxStableDiffusion



ofxStableDiffusion is an openFrameworks addon that wraps stable-diffusion.cpp for text-to-image, image-to-image, image-to-video, and upscaling workflows.

Current addon version: 1.0.0

Requirements



The addon is now structured more like a production addon:

Highlights



Repo Layout



Addon wrapper, typed request/result model, and background-thread integration Enums and owned result types Video clip behavior, pure helper utilities, and reusable workflow helpers for: Bundled header/libs and vendoring location for upstream native source Native rebuild scripts Lightweight CMake-based unit tests Architecture and native-build notes

Feature Readiness



SurfaceStatusNotes
Typed image generation (generate)StablePrimary production-ready wrapper API
Typed video generation (generateVideo)StableIncludes owned frame results and metadata export
Legacy compatibility entry pointsSupportedKept for migration; new work should prefer typed requests
ofxStableDiffusionRealtimeSessionStableCall update() every frame; callbacks fire from update()
ofxStableDiffusionRealtimeVideoSessionExperimentalUseful preview workflow, but still evolving
ofxStableDiffusionCreativeWorkflowExperimentalUnifies realtime preview with queued higher-quality image/video renders and session snapshots
ofxStableDiffusionHoloscanBridgeExperimentalLinux-first runtime path with fallback behavior elsewhere
ofxStableDiffusionBatchProcessorExperimentalRuns generator-backed grids, sweeps, A/B comparisons, and batch artifact export

Threading Contract



API Shape



The addon now supports two layers of use:

Modern typed wrapper API



#include "ofxStableDiffusion.h"

ofxStableDiffusion sd;

ofxStableDiffusionContextSettings context;
context.modelPath = "data/models/sd/sd_turbo.safetensors";
context.nThreads = 8;
context.weightType = SD_TYPE_F16;
sd.configureContext(context);

ofxStableDiffusionImageRequest request;
request.prompt = "cinematic portrait, rim lighting";
request.width = 512;
request.height = 512;
request.sampleSteps = 20;
sd.generate(request);

// Optional: apply a stack of LoRA/LoCon adapters per request
ofxStableDiffusionLora loraA;
loraA.path = "data/loras/edge.safetensors";
loraA.strength = 0.7f;
request.loras = {loraA};
sd.generate(request);

// Update the active LoRA stack globally
sd.setLoras({loraA});

// Hot-reload textual-inversion embeddings (reloads the context)
sd.reloadEmbeddings("data/embeddings");

// List currently discoverable embeddings (name, absolute path)
const auto embeddings = sd.listEmbeddings();

// Inspect what the currently configured model/runtime can actually do
const auto capabilities = sd.getCapabilities();

// Save generated video frames plus a JSON sidecar with per-frame prompts/seeds
sd.saveVideoFramesWithMetadata("output/storyboard", "shot");


The parameter-tuning helpers can also be used outside the wrapper instance when you want model-family-aware defaults before building a request:

const auto imageProfile =
    ofxStableDiffusionParameterTuningHelpers::resolveImageProfile(
        context,
        ofxStableDiffusionImageMode::ImageToImage);

request.cfgScale = imageProfile.defaultCfgScale;
request.sampleSteps = imageProfile.defaultSampleSteps;
request.strength = imageProfile.defaultStrength;


Error Handling



The addon provides advanced error handling with error codes, messages, suggestions, and history tracking:

ofxStableDiffusionImageRequest request;
request.width = 513;  // Invalid: not a multiple of 64
request.height = 512;
request.batchCount = 20;  // Invalid: exceeds maximum of 16

sd.generate(request);

// Check for errors programmatically
if (sd.getLastErrorCode() != ofxStableDiffusionErrorCode::None) {
    // Get detailed error information
    const auto& errorInfo = sd.getLastErrorInfo();
    ofLogError() << "Error: " << errorInfo.message;
    ofLogError() << "Suggestion: " << errorInfo.suggestion;
    ofLogError() << "Error code: " << ofxStableDiffusionErrorCodeLabel(errorInfo.code);
}

// Or use the simple string-based error (backward compatible)
if (!sd.getLastError().empty()) {
    ofLogError() << "Error: " << sd.getLastError();
}

// Access error history for debugging
const auto& history = sd.getErrorHistory();
for (const auto& error : history) {
    ofLogNotice() << "[" << error.timestampMicros << "] "
                  << error.message << " -> " << error.suggestion;
}

// Clear error history
sd.clearErrorHistory();


Available Error Codes: Each error code automatically provides an actionable suggestion to help resolve the issue.

Legacy compatibility API



The older newSdCtx, txt2img, img2img, and img2vid entry points are still available so existing call sites, including addon-to-addon bridges, do not have to migrate immediately.

Realtime Creative Loop



ofxStableDiffusionRealtimeVideoSession adapts the interaction pattern of realtime video tools to the local stable-diffusion.cpp backend. It keeps only the latest prompt while a frame is generating, uses low-step preview requests for responsiveness, can refine an idle prompt with a higher step budget, and can feed the previous output frame back through img2img for visual continuity.

#include "ofxStableDiffusion.h"

ofxStableDiffusionRealtimeVideoSettings liveSettings;
liveSettings.previewSteps = 4;
liveSettings.refineSteps = 16;
liveSettings.previewStrength = 0.68f;
liveSettings.refineStrength = 0.35f;
liveSettings.usePreviousFrameFeedback = true;

ofxStableDiffusionRealtimeVideoSession liveVideo;
liveVideo.start(liveSettings, sd);

ofxStableDiffusionRealtimeVideoRequest liveRequest;
liveRequest.prompt = "a fashion portrait reacting to colored stage lights";
liveVideo.submit(liveRequest);

// In update():
liveVideo.update();


This is a near-realtime preview workflow, not a proprietary HD world model. For best results use fast models, modest preview dimensions, and low preview step counts.

Video Behavior



Video generation returns owned frames through ofxStableDiffusionVideoClip.

Animated requests can also drive per-frame prompt interpolation, parameter animation, and seed sequencing through ofxStableDiffusionVideoRequest::animationSettings. Each returned frame now carries the prompt / negative prompt / CFG / strength / seed values that were actually used, and clips can export both PNG sequences and JSON metadata.

Useful video-export helpers:

Long video generation (chunked rendering):

describe a long-form timeline as multiple short clips. optionally handing off the previous chunk's last frame as the next init image, and saves each chunk's frame sequence + metadata.json.

Supported playback/presentation modes:

Return generated frames as-is Repeat the first frame at the end for easier closed-loop playback Play forward, then back through the interior frames Play forward, then fully reverse including the endpoints

This is especially useful when a UI layer or another addon wants a more natural preview clip without re-asking the native runtime for more frames.

Video Performance Recommendations



Coding Conventions (openFrameworks-aligned)



Image Modes



The typed image request layer exposes the native image modes:

CLIP-Rerank Integration



ofxStableDiffusion now exposes a wrapper-level image ranking callback for Best-of-N workflows.

That is the intended integration point for ofxGgml CLIP scoring:

This keeps diffusion and CLIP loosely coupled while still enabling a strong cross-addon workflow.

ofxGgml Integration Guidance



This addon can be integrated with ofxGgml in two ways:

1. Addon-Level Integration (Recommended)



Why:

There is already an addon-level bridge seam on the ofxGgml side through ofxGgmlStableDiffusionAdapters.h. This addon also exposes a small bridge helper header:

2. System GGML Integration (Optional)



For advanced use cases where you want a single GGML binary shared across addons, you can build ofxStableDiffusion to consume GGML from ofxGgml:

# Linux/macOS
./scripts/build-stable-diffusion.sh --use-system-ggml --ofxggml-path ../ofxGgml


Windows

.\scripts\build-stable-diffusion.ps1 -UseSystemGgml -OfxGgmlPath ..\ofxGgml


This enables stable-diffusion.cpp's built-in system GGML support via -DSD_USE_SYSTEM_GGML=ON. The build scripts will: Important considerations: More detail: docs/ARCHITECTURE.md, docs/OFXGGML_BRIDGE.md

Native Runtime



The addon stages native artifacts into addon-local paths:

Rebuild helpers:

More detail: docs/NATIVE_BUILD.md

Backend flags now follow the same style as ofxGgml, but the build selects one backend per build:

Force CPU-only native builds (default) Enable CUDA explicitly Enable Vulkan explicitly Enable Metal explicitly where supported Build every available backend variant and leave the canonical runtime on the best one in this priority order: cuda, then vulkan, then cpu-only

Each backend build is also snapshotted under libs/variants/<backend>, so you can switch the canonical addon runtime later without rebuilding everything.

Variant selector helpers:

For Windows, scripts/setup_windows.bat and scripts/setup_addon.ps1 now always refresh the vendored source from the latest upstream release-tag source snapshot, then build the native runtime locally.

Override the upstream release tag used for the vendored source snapshot

When an upstream Windows zip does not include stable-diffusion.lib, the addon setup script synthesizes the Visual Studio import library from the downloaded DLL exports automatically.

Current Native Source Status



The repo now includes a vendored upstream stable-diffusion.cpp source snapshot under libs/stable-diffusion/source, pinned to:

The optional Windows prebuilt-runtime flow is currently pinned to the upstream GitHub release tag master-585-44cca3d, which was the latest upstream release published on 2026-04-19. Override it with --release-tag if you want a different upstream runtime. Source: stable-diffusion.cpp releases

The addon now includes the upstream header directly through libs/stable-diffusion/include/stable-diffusion.h, without re-exporting the older enum aliases. Addon code should use the current upstream names such as EULER_A_SAMPLE_METHOD, DPMPP2Mv2_SAMPLE_METHOD, and SCHEDULER_COUNT.

Testing



The test suite focuses on wrapper-level logic that should stay stable even when the native runtime changes.

Run:

cmake -S tests -B tests/build
cmake --build tests/build --config Release
ctest --test-dir tests/build -C Release --output-on-failure


You can also use:

Test notes: tests/README.md

Example



The example project lives in ofxStableDiffusionExample/ and now exposes:

Troubleshooting



Model Loading Issues



Problem: Model fails to load or crashes during loading

Get this addon

HTTPS
SSH
View on GitHub · Open with Github Desktop · Download ZIP