ofxIC

Inference Connector for openFrameworks. ofxIC connects an openFrameworks application to external model endpoints. Its compact surface covers chat with one allowlisted document tool, explicit OpenAI and whisper.cpp transcription protocols, explicit SAM bridge point segmentation, OpenAI-compatible image generation, and native asynchronous image/video jobs exposed by stable-diffusion.cpp. A separate Hugging Face media path routes text-to-image and text-to-video through fal-ai. Dedicated music clients connect either to a local ACE-Step server or to hosted Stability Audio 3 jobs. Inference always remains in a separate local or hosted process.

The addon does not embed ggml, llama.cpp, CUDA, SAM, or another native model runtime. Run llama-server separately and update it independently.

Install

Clone the repository into the openFrameworks addons directory:

cd path/to/openFrameworks/addons
git clone https://github.com/Jonathhhan/ofxIC.git

Open example-chat with the openFrameworks Project Generator for the minimal OF lifecycle example. It uses ofxIC and ofxImGui. Add ofxIC to an existing project in the usual addons.make form. No model runtime or model file is installed with the addon.

Open example-documents for the complete grounded-document path: explicitly load a text file, expose only search_documents, and require cited source identifiers without blocking the openFrameworks frame thread.

The complete ofxICExample endpoint workbench additionally uses a pinned ofxImGui revision:

cd path/to/openFrameworks/addons
git clone https://github.com/jvcleave/ofxImGui.git
git -C ofxImGui checkout 4b5d04a3e73bbae34d564c9b78143e021f67377a

Optional Windows CUDA server

The addon remains an HTTP client, but these scripts can install pinned, external CUDA runtimes into %LOCALAPPDATA%\ofxIC\servers. Executables, CUDA runtime files, logs, and models remain outside the repository. Pinned prebuilt archives are accepted only when their recorded digest matches; source-based environments pin their upstream release or commit.

.\scripts\install-llama-server.ps1
.\scripts\start-llama-server.ps1 -Model C:\models\model.gguf -FlashAttention
.\scripts\install-stable-diffusion-server.ps1
.\scripts\start-stable-diffusion-server.ps1 -DiffusionModel C:\models\image-model.gguf -FlashAttention
.\scripts\install-whisper-server.ps1
.\scripts\install-acestep-server.ps1
.\scripts\install-sam-server.ps1 -ExistingModel G:\Models\sam_vit_b_01ec64.pth

Use -Plan to inspect downloads or launch arguments without installing or starting anything. The pinned package is llama.cpp b10516, Windows x64, CUDA 13.3. Other backends may be added later; models remain a separate choice. The official stable-diffusion.cpp Windows package currently targets CUDA 12; its bundled runtime works with a compatible newer NVIDIA driver. Its server defaults to port 8081, so it can run beside llama-server on port 8080. The ACE-Step installer builds a pinned native acestep.cpp CUDA server and uses an existing complete GGUF set such as G:\Models; it does not copy model weights into the addon. The Windows example discovers versioned installations below that directory at runtime, lets a manually selected executable take precedence, and starts the external process from the GUI. Detection is version-independent and belongs to the example application; no runtime is linked into the addon. Overview > Runtime setup (Windows) offers Copy plan and Copy install for all five runtimes. Paste the command into PowerShell, review the plan, install explicitly, then use Rescan installed runtimes. The GUI does not run installers or query upstream releases. Rescan refreshes detection only: it does not overwrite selected executable/model paths, reapply environment settings, or restart servers. Use Detect installed in a task tab to explicitly select the detected executable. Expected script destinations count as detected only when the file exists; missing preferred paths fall back to version-independent discovery. A configured dashboard row means paths are set, not that the runtime is reachable or inference has passed. Whisper installs a verified base-q5_1 model by default; ACE-Step uses a separately selected complete GGUF model directory; the SAM installer can reuse a selected Meta sam_vit_b, vit_l, or vit_h checkpoint and creates an isolated PyTorch CUDA 13 environment.

Runtime version pins are owned by the installer scripts, not duplicated in a GUI version registry. A detected package folder identifies an installation; it does not prove API compatibility or successful inference. A newer upstream release is unvalidated, not automatically an available addon update. Changing a pin requires reviewing start flags and protocol changes, updating affected adapters/tests when necessary, and running both deterministic tests and the relevant marker-gated live smoke. An unchanged protocol may need only a script update; incompatible protocol changes require an addon update too.

The addon does not override the C++ language standard selected by the generated openFrameworks project. Its standalone deterministic CMake tests require C++20; the canonical example is built with the standard supplied by the current openFrameworks platform template. This keeps the addon aligned with openFrameworks instead of imposing a separate compiler mode.

Professional Windows build and validation

Use the repository scripts instead of relying on Visual Studio's ambiguous incremental-build message:

.\scripts\build-example.ps1
.\scripts\validate-windows.ps1

build-example.ps1 verifies that every addon/example .cpp belongs to the generated project and performs a fast incremental Release/x64 build by default, then reports whether the exact EXE was rebuilt or already current. Use -Rebuild for an unconditional full rebuild; validate-windows.ps1 does this automatically for release validation. -Incremental remains accepted for compatibility. Use -UpdateProject only when source files were added and the local openFrameworks Project Generator needs to refresh the solution. The validation script runs deterministic CMake tests, rebuilds both focused examples and the complete workbench, and then executes the model-free GUI lifecycle smoke plus a no-start plan of all five local runtime configurations. Current development builds expose the version string 0.2.1-dev through ofxIC::versionString.

The default target is the full ofxICExample. The same verified path builds the minimal example after generating its ignored platform project once:

.\scripts\build-example.ps1 -Example example-chat -UpdateProject
.\scripts\build-example.ps1 -Example example-documents -UpdateProject

Current API

#include "ofxIC.h"

ofxIC::Endpoint endpoint("http://127.0.0.1:8080");
ofxIC::ChatSession chat(endpoint);

chat.setSystemPrompt("Be concise.");
const auto result = chat.send("Hello");

if (result) {
	ofLogNotice() << result.text;
} else {
	ofLogError() << result.error;
}

Long-running endpoint inspection, chat, tool-loop, and transcription requests accept an operation-scoped RequestControl. Run the blocking call on a worker thread and set the flag from the openFrameworks thread; the example exposes this as Cancel request:

std::atomic<bool> cancelRequested{ false };
ofxIC::RequestControl control;
control.timeoutSeconds = 30; // zero keeps the operation's bounded default
control.shouldCancel = [&]() {
	return cancelRequested.load();
};
const auto result = loop.run("Summarize the loaded sources", 4, control);

Cancellation is forwarded to the HTTP transport, rolls the incomplete turn back out of chat history, and is reported separately from timeout, transport, provider, and invalid-response failures through RequestFailure. The previous standalone cancellation-predicate overloads remain available during 0.2.x.

The example's Stream direct chat option renders text chunks while a plain chat request is still running. It deliberately bypasses document tools because streaming tool calls are not part of the supported protocol yet; leave it off for grounded search_documents answers.

The complete document workflow remains small:

ofxIC::DocumentIndex documents;
documents.addFile("notes/architecture.md");

ofxIC::ToolRegistry tools;
tools.addDocumentSearch(documents);

ofxIC::ToolLoop loop(chat, tools);
const auto answer = loop.run("Why is llama-server a separate process?");

ToolLoop::run can also report typed progress when it requests the model or executes an allowlisted tool. The example uses this to expose multi-request grounding work without leaking worker-thread state into the GUI thread.

These objects deliberately use non-owning references: keep Endpoint alive while its ChatSession is used, keep both ChatSession and ToolRegistry alive while using ToolLoop, and keep DocumentIndex alive after registering document search.

search_documents receives only a query. It cannot choose a file path or run an arbitrary function; it searches only text the application loaded first. The index accepts at most 128 documents, 16,384 chunks, and 8 MiB per document; rejected additions do not partially change the index. Tool results label source text as untrusted evidence. This supports safer prompting but does not turn arbitrary web content into trusted instructions.

Controlled web snapshots

In the Windows workbench, open Documents → Import webpage, enter a public HTTP(S) URL and choose Fetch preview. Review the extracted text and its source URL, title and retrieval date, then choose Add to Documents. Fetching alone does not expose the page to chat. Cancel import stops the fetch; Discard preview removes the draft. Imports are session-only.

The example launches the existing Python utility as a separate process, automatically finding Python on PATH or in the installed SAM environment. It needs Python 3.10+ and access to scripts/web_snapshot.py. The GUI enforces a 60-second overall deadline and a bounded preview. This is webpage import, not search-engine integration. The standalone command remains available:

Web ingestion stays outside the addon and outside the model tool loop. The bundled utility fetches one URL selected by the user and writes a provenance- bearing .txt or .md snapshot that can be loaded through the regular example:

python scripts\web_snapshot.py https://example.com/ 
  --output "$env:TEMP\ofxic-article.txt"



Then choose Load .md / .txt in ofxICExample, or set OFXIC_DOCUMENT_PATH to that output file before launch. The snapshot records the final URL, retrieval time, title when present, and a SHA-256 hash of the cleaned text.

This utility is intentionally not a crawler: it does not follow page links, execute JavaScript, use authenticated browser state, or run as a model-selected tool. It revalidates redirect destinations, rejects hosts resolving to private or otherwise non-public addresses, accepts only HTML and plain text, limits the download to 4 MiB, and defaults to a 15-second timeout. Dynamic sites that need a browser renderer should be exported by a separate user-controlled process and loaded as a local .txt or .md file.

Inspect the endpoint and advertised model identity before use:

const auto status = endpoint.inspect();
if (status && !status.models.empty()) {
	ofxIC::ChatOptions options;
	options.model = status.models.front();
	chat.setOptions(options);
}

Inspection proves endpoint reachability and parses the advertised model list. It does not by itself prove that a token is valid, that provider credit is available, or that a selected model can complete the intended request. The example reports Inference completed only after a chat request has actually returned successfully.

Image generation uses the same endpoint and token configuration:

ofxIC::MediaClient media(endpoint);
ofxIC::ImageRequest image;
image.prompt = "A paper sculpture in soft studio light";
image.width = 1024;
image.height = 1024;

const auto result = media.generateImage(image);
// result.imagesBase64 or result.urls contains the generated output.

The native stable-diffusion.cpp video path is asynchronous:

ofxIC::MediaJobRequest video;
video.kind = ofxIC::MediaKind::Video;
video.prompt = "The paper sculpture slowly turns";
video.videoFrames = 33;
video.fps = 16;

auto job = media.submit(video);
// Poll later; do not block the openFrameworks draw thread.
job = media.poll(job);

Before submitting local video, MediaClient reads /sdcpp/v1/capabilities. Submission stops with an actionable error when the model currently loaded by sd-server does not advertise vid_gen; common image models such as SD-Turbo cannot generate video. Start the server with a video-capable model such as a compatible WAN, LTX-Video, or AnimateDiff model. When no output format is requested explicitly, the client selects a format advertised by that server, preferring AVI for broad Windows playback support.

Hugging Face media deliberately uses its task protocol instead of pretending that the chat router implements OpenAI image/video routes. The first compact adapter supports fal-ai for both image and video:

ofxIC::Endpoint hf("https://router.huggingface.co");
hf.setBearerToken(environmentToken);
ofxIC::MediaClient hostedMedia(hf);

ofxIC::MediaJobRequest video;
video.kind = ofxIC::MediaKind::Video;
video.model = "Wan-AI/Wan2.2-TI2V-5B";
video.prompt = "A paper sculpture slowly turns";

auto job = hostedMedia.submitHuggingFaceFal(video);
// fal-ai video is queued; call poll(job) later until it is terminal.

The client resolves the HF model's current fal-ai mapping, keeps the HF token on router requests, and downloads the returned media bytes without forwarding that token to the output CDN.

Music generation is intentionally provider-specific rather than another value in MediaKind. The model runtime stays in a separate process:

ofxIC::Endpoint aceStep("http://127.0.0.1:8085");
ofxIC::AceStepMusicClient music(aceStep);

ofxIC::AceStepMusicRequest request;
request.caption = "Warm modular synthesizer, instrumental, no vocals";
request.durationSeconds = 30;
request.outputFormat = "wav";
request.protocol = ofxIC::AceStepMusicProtocol::NativeCpp;

auto job = music.submit(request);
// Direct servers may complete immediately; async servers use /job polling.
if (!job.terminal()) job = music.poll(job);

The client supports both the official ACE-Step 1.5 /release_task, /query_result, /v1/audio flow and native acestep.cpp /lm, /synth, /job jobs. It bounds responses, validates task IDs, extracts native multipart audio, and verifies the returned WAV or MP3 container. It never sends a configured bearer token to either local route.

Stability Audio 3 has its own asynchronous API and credential:

ofxIC::Endpoint stability("https://api.stability.ai");
stability.setBearerToken(stabilityApiKey);
ofxIC::StabilityAudioClient music(stability);

ofxIC::StabilityAudioRequest request;
request.prompt = "Warm modular synthesizer, instrumental, no vocals";
request.durationSeconds = 30;
request.outputFormat = "mp3";

auto job = music.submit(request);
// Poll later. A completed job carries validated MP3 or WAV bytes.
job = music.poll(job);

The client accepts only stable-audio-3, validates the provider's documented duration and generation controls, bounds downloads, and checks the requested audio container before reporting completion. This is an external paid provider operation; deterministic tests do not spend credits.

Audio transcription is also an external endpoint operation. Select the OpenAI protocol for /v1/audio/transcriptions, or point the custom endpoint at a separately running whisper.cpp server and select its /inference protocol:

ofxIC::TranscriptionClient transcription(endpoint);
ofxIC::TranscriptionRequest request;
request.audioBytes = wavBytes;
request.filename = "recording.wav";
request.model = "whisper-1"; // OpenAI only; whisper.cpp selects at server start.

const auto transcript = transcription.transcribeOpenAI(request);

The two methods are deliberately separate because these endpoints do not share one universal transcription contract.

Prompted image segmentation uses the explicitly named ofxIC SAM bridge v1, not a claimed provider standard. The client uploads a PPM image plus normalized positive or negative points to /v1/segmentations and receives one PGM mask. The regular example can queue multiple positive and negative prompts; an empty queue preserves the single positive point workflow. Left-clicking its input preview adds a positive prompt, right-clicking adds a negative prompt, and the rendered markers can be undone or cleared:

ofxIC::SegmentationClient segmentation(endpoint);
ofxIC::SegmentationRequest request;
request.imageBytes = ppmBytes;
request.points.push_back({ 0.5f, 0.5f, true });
const auto mask = segmentation.segmentSamBridge(request);

scripts\sam-bridge-server.py implements the localhost process boundary. Its real mode invokes an independently supplied sam-runner with the existing --model, --image, --output, and repeated point-flag contract. For runners that expose it, --backend cpu|cuda is forwarded explicitly instead of silently falling back to a slow CPU default. Its fixture mode exists only for deterministic GUI evidence and performs no inference. The client and bridge require X-ofxIC-SAM-Bridge-Version: 1, accept at most 64 points, bound request and mask sizes, validate complete PPM/PGM payloads, and report stable text error codes. The bridge rejects concurrent model runs and supports --runner-timeout <seconds> rather than allowing an orphaned request to wait indefinitely. scripts\install-sam-server.ps1 provides one concrete external consumer of this contract: an isolated Meta SAM Python runner with a CUDA 13 PyTorch environment. Its bounded GET /health response reports bridge version, fixture/runner mode, and selected backend; the regular example exposes this as Check bridge before an image needs to be submitted.

Scope

Implemented:

• /v1/models endpoint inspection
• /v1/chat/completions
• /v1/images/generations
• OpenAI /v1/audio/transcriptions and whisper.cpp /inference
• explicit /v1/segmentations SAM bridge v1 point segmentation
• native stable-diffusion.cpp image/video submission and job polling
• Hugging Face text-to-image and queued text-to-video through fal-ai routing
• local ACE-Step 1.5 task-API text-to-music generation
• Stability Audio 3 text-to-music submission, polling, and MP3/WAV download
• conversational history
• optional streaming transport when openFrameworks exposes curl
• injected HTTP transport for deterministic tests
• local document index
• controlled one-URL web-to-text snapshot ingestion outside the addon
• one allowlisted search_documents tool
• bounded tool loop with source citations

Proven end to end:



• deterministic protocol tests on Linux and Windows
• compilation and GUI launch against the current openFrameworks Linux nightly
• automated keyboard input through the real example GUI
• local whisper.cpp transcription through the regular Windows GUI using

ggml-tiny.en and the jfk.wav speech sample

• local SAM2.1 point segmentation through the regular Windows GUI, explicit

localhost bridge, and independently built ofxGgmlSam runner

• marker-gated local ACE-Step music generation through the ofxIC client,

including asynchronous jobs and multipart WAV extraction

• a marker-gated Hugging Face run with two model requests, local

search_documents execution, and a cited final answer

The latest recorded live proof used openai/gpt-oss-120b and returned architecture.md#chunk-1`. This