ofxGgmlVision

ofxGgmlVision is the companion addon for CLIP, image embeddings, captions, VLM-style image understanding, and visual search workflows on top of ofxGgmlCore.

ofxGgmlCore stays the dependency. This addon owns vision-specific workflow code so core can stay small and boring.

Family map: https://jonathhhan.github.io/ofxGgmlCore/

Current addon API version: 1.0.1.

Features

• image embedding workflow boundary
• captioning workflow boundary
• image understanding request lane
• CLIP model ownership
• local GGUF and multimodal-projector selection with a CUDA llama.cpp launch path
• OpenAI-compatible vision bridge for caption and image-understanding requests

First Milestone

• define small request/result types
• define typed task/result helpers for embeddings, captions, image

understanding, and visual search

• keep one root-level smoke example
• keep generated models, media, builds, and IDE files out of git
• validate the addon with local headless tests

Example

ofxGgmlVisionImageExample defaults to local llama.cpp inference. Choose the vision-model GGUF and its matching mmproj GGUF with file dialogs, start the CUDA-backed local model, then analyze the selected image. Inference runs on a dedicated ofThread worker so the openFrameworks render thread remains responsive. An external OpenAI-compatible server remains an explicit alternative. Generate the example with the openFrameworks projectGenerator using addons ofxGgmlVision, ofxGgmlCore, and ofxImGui.

Agent Workflows

Use docs/VISION_WORKFLOWS.md when planning Codex, Copilot, or Hermes changes for image understanding, CLIP-style embeddings, captions, VLM-style workflows, and visual search.

For a fresh local checkout, use docs/FIRST_RUN.md.

Dependencies

• openFrameworks
• ofxGgmlCore
• ofxImGui for examples

Validate

scripts\doctor-vision.bat
scripts\run-vision-runtime-smoke.bat -Json -SummaryOnly
scripts\run-vision-server-smoke.bat -DryRun -Model local-vision -Image C:\path\to\image.png
scripts\validate-local.bat

On macOS/Linux:

./scripts/doctor-vision.sh
./scripts/run-vision-runtime-smoke.sh -Json -SummaryOnly
./scripts/run-vision-server-smoke.sh -DryRun -Model local-vision -Image /path/to/image.png
./scripts/validate-local.sh

scripts\run-vision-runtime-smoke. is the lane-owned runtime-smoke entrypoint for ecosystem planning and CI rollouts. It currently proves the deterministic request/helper boundary, doctor readiness, and metadata-only GGUF bridge without claiming model-backed CLIP, captioning, VLM, or visual-search inference. Add model-backed checks here only after the local backend or compatible server, model identifiers, media inputs, and generated-artifact cleanup contract are explicit.

Use scripts\run-vision-server-smoke. when an OpenAI-compatible local VLM server is already running. Pass -Model plus either -Image for a local file that should be sent as a data URL, or -ImageUrl for a server-readable image URL. Normal validation exercises the script in dry-run mode only.

Boundary

Keep vision-specific preprocessing, postprocessing, model launch, media handling, and examples here. Move code down into ofxGgmlCore only when it becomes a stable, domain-neutral primitive with focused tests.