Skip to content
This repository has been archived by the owner on Dec 18, 2024. It is now read-only.

Commit

Permalink
Add new ONNX models and update brew bundle command
Browse files Browse the repository at this point in the history
  • Loading branch information
royshil committed Mar 27, 2024
1 parent 50fec23 commit 0dd49ac
Show file tree
Hide file tree
Showing 16 changed files with 1,007 additions and 628 deletions.
2 changes: 1 addition & 1 deletion .github/scripts/utils.zsh/check_macos
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,6 @@ if (( ! ${+commands[brew]} )) {
return 2
}

brew bundle --file ${SCRIPT_HOME}/.Brewfile
brew bundle --no-upgrade --file ${SCRIPT_HOME}/.Brewfile
rehash
log_group
163 changes: 17 additions & 146 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,159 +1,30 @@
# OBS Plugin Template
# OBS Detect - Object Detection and Masking Filter

## Introduction
<div align="center">

The plugin template is meant to be used as a starting point for OBS Studio plugin development. It includes:
[![GitHub](https://img.shields.io/github/license/occ-ai/obs-detect)](https://github.com/occ-ai/obs-detect/blob/main/LICENSE)
[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/occ-ai/obs-detect/push.yaml)](https://github.com/occ-ai/obs-detect/actions/workflows/push.yaml)
[![Total downloads](https://img.shields.io/github/downloads/occ-ai/obs-detect/total)](https://github.com/occ-ai/obs-detect/releases)
![Flathub](https://img.shields.io/flathub/downloads/com.obsproject.Studio.Plugin.BackgroundRemoval?label=Flathub%20Installs)
[![GitHub release (latest by date)](https://img.shields.io/github/v/release/occ-ai/obs-detect)](https://github.com/occ-ai/obs-detect/releases)
[![Discord](https://img.shields.io/discord/1200229425141252116)](https://discord.gg/KbjGU2vvUz)

* Boilerplate plugin source code
* A CMake project file
* GitHub Actions workflows and repository actions
</div>

## Set Up
A plugin for [OBS Studio](https://obsproject.com/) that allows you to detect many types of objects in any source, track them and apply masking.

The plugin project is set up using the included `buildspec.json` file. The following fields should be customized for an actual plugin:
If you like this work, which is given to you completely free of charge, please consider supporting it by sponsoring us on GitHub:

* `name`: The plugin name
* `version`: The plugin version
* `author`: Actual name or nickname of the plugin's author
* `website`: URL of a website associated with the plugin
* `email`: Contact email address associated with the plugin
* `uuids`
* `macosPackage`: Unique (**!**) identifier for the macOS plugin package
* `macosInstaller`: Unique (**!**) identifier for the macOS plugin installer
* `windowsApp`: Unique (**!**) identifier for the Windows plugin installer
- https://github.com/sponsors/royshil
- https://github.com/sponsors/umireon

These values are read and processed automatically by the CMake build scripts, so no further adjustments in other files are needed.
## Usage

### Platform Configuration
<div align="center">
<video src="https://github.com/occ-ai/obs-backgroundremoval/assets/1067855/5ba5aae2-7ea2-4c90-ad45-fba5ccde1a4e" width="320"></video>
</div>

Platform-specific settings are set up in the `platformConfig` section of the buildspec file:

* `bundleId`: macOS bundle identifier for the plugin. Should be unique and follow reverse domain name notation.

### Set Up Build Dependencies

Just like OBS Studio itself, plugins need to be built using dependencies available either via the `obs-deps` repository (Windows and macOS) or via a distribution's package system (Linux).

#### Choose An OBS Studio Version

By default the plugin template specifies the most current official OBS Studio version in the `buildspec.json` file, which makes most sense for plugins at the start of development. As far as updating the targeted OBS Studio version is concerned, a few things need to be considered:

* Plugins targeting _older_ versions of OBS Studio should _generally_ also work in newer versions, with the exception of breaking changes to specific APIs which would also be explicitly called out in release notes
* Plugins targeting the _latest_ version of OBS Studio might not work in older versions because the internal data structures used by `libobs` might not be compatible
* Users are encouraged to always update to the most recent version of OBS Studio available within a reasonable time after release - plugin authors have to choose for themselves if they'd rather keep up with OBS Studio releases or stay with an older version as their baseline (which might of course preclude the plugin from using functionality introduced in a newer version)

On Linux, the version used for development might be decided by the specific version available via a distribution's package management system, so OBS Studio compatibility for plugins might be determined by those versions instead.

#### Windows and macOS

Windows and macOS dependency downloads are configured in the `buildspec.json` file:

* `dependencies`:
* `obs-studio`: Version of OBS Studio to build plugin with (needed for `libobs` and `obs-frontend-api`)
* `prebuilt`: Prebuilt OBS Studio dependencies
* `qt6`: Prebuilt version of Qt6 as used by OBS Studio
* `tools`: Contains additional build tools used by CI

The values should be kept in sync with OBS Studio releases and the `buildspec.json` file in use by the main project to ensure that the plugin is developed and built in sync with its target environment.

To update a dependency, change the `version` and associated `hashes` entries to match the new version. The used hash algorithm is `sha256`.

#### Linux

Linux dependencies need to be resolved using the package management tools appropriate for the local distribution. As an example, building on Ubuntu requires the following packages to be installed:

* Build System Dependencies:
* `cmake`
* `ninja-build`
* `pkg-config`
* Build Dependencies:
* `build-essential`
* `libobs-dev`
* Qt6 Dependencies:
* `qt6-base-dev`
* `libqt6svg6-dev`
* `qt6-base-private-dev`

## Build System Configuration

To create a build configuration, `cmake` needs to be installed on the system. The plugin template supports CMake presets using the `CMakePresets.json` file and ships with default presets:

* `macos`
* Universal architecture (supports Intel-based CPUs as Apple Silicon)
* Defaults to Qt version `6`
* Defaults to macOS deployment target `11.0`
* `macos-ci`
* Inherits from `macos`
* Enables compile warnings as error
* `windows-x64`
* Windows 64-bit architecture
* Defaults to Qt version `6`
* Defaults to Visual Studio 17 2022
* Defaults to Windows SDK version `10.0.18363.657`
* `windows-ci-x64`
* Inherits from `windows-x64`
* Enables compile warnings as error
* `linux-x86_64`
* Linux x86_64 architecture
* Defaults to Qt version `6`
* Defaults to Ninja as build tool
* Defaults to `RelWithDebInfo` build configuration
* `linux-ci-x86_64`
* Inherits from `linux-x86_64`
* Enables compile warnings as error
* `linux-aarch64`
* Provided as an experimental preview feature
* Linux aarch64 (ARM64) architecture
* Defaults to Qt version `6`
* Defaults to Ninja as build tool
* Defaults to `RelWithDebInfo` build configuration
* `linux-ci-aarch64`
* Inherits from `linux-aarch64`
* Enables compile warnings as error

Presets can be either specified on the command line (`cmake --preset <PRESET>`) or via the associated select field in the CMake Windows GUI. Only presets appropriate for the current build host are available for selection.

Additional build system options are available to developers:

* `ENABLE_CCACHE`: Enables support for compilation speed-ups via ccache (enabled by default on macOS and Linux)
* `ENABLE_FRONTEND_API`: Adds OBS Frontend API support for interactions with OBS Studio frontend functionality (disabled by default)
* `ENABLE_QT`: Adds Qt6 support for custom user interface elements (disabled by default)
* `CODESIGN_IDENTITY`: Name of the Apple Developer certificate that should be used for code signing
* `CODESIGN_TEAM`: Apple Developer team ID that should be used for code signing

## GitHub Actions & CI

Default GitHub Actions workflows are available for the following repository actions:

* `push`: Run for commits or tags pushed to `master` or `main` branches.
* `pr-pull`: Run when a Pull Request has been pushed or synchronized.
* `dispatch`: Run when triggered by the workflow dispatch in GitHub's user interface.
* `build-project`: Builds the actual project and is triggered by other workflows.
* `check-format`: Checks CMake and plugin source code formatting and is triggered by other workflows.

The workflows make use of GitHub repository actions (contained in `.github/actions`) and build scripts (contained in `.github/scripts`) which are not needed for local development, but might need to be adjusted if additional/different steps are required to build the plugin.

### Retrieving build artifacts

Successful builds on GitHub Actions will produce build artifacts that can be downloaded for testing. These artifacts are commonly simple archives and will not contain package installers or installation programs.

### Building a Release

To create a release, an appropriately named tag needs to be pushed to the `main`/`master` branch using semantic versioning (e.g., `12.3.4`, `23.4.5-beta2`). A draft release will be created on the associated repository with generated installer packages or installation programs attached as release artifacts.

## Signing and Notarizing on macOS

Plugins released for macOS should be codesigned and notarized with a valid Apple Developer ID for best user experience. To set this up, the private and personal key of a **paid Apple Developer ID** need to be downloaded from the Apple Developer portal:

* On your Apple Developer dashboard, go to "Certificates, IDs & Profiles" and create two signing certificates:
* One of the "Developer ID Application" type. It will be used to sign the plugin's binaries
* One of the "Developer ID Installer" type. It will be used to sign the plugin's installer

The developer certificate will usually carry a name similar in form to

`Developer ID Application: <FIRSTNAME> <LASTNAME> (<LETTERS_AND_NUMBERS>)`

This entire string should be specified as `CODESIGN_IDENTITY`, the `LETTERS_AND_NUMBERS` part as `CODESIGN_TEAM` to CMake to set up codesigning properly.

### GitHub Actions Set Up

Expand Down
95 changes: 95 additions & 0 deletions data/effects/kawase_blur.effect
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
uniform float4x4 ViewProj;
uniform texture2d image;
uniform texture2d focalmask;

uniform float xOffset;
uniform float yOffset;

sampler_state textureSampler {
Filter = Linear;
AddressU = Clamp;
AddressV = Clamp;
};

struct VertDataIn {
float4 pos : POSITION;
float2 uv : TEXCOORD0;
};

struct VertDataOut {
float4 pos : POSITION;
float2 uv : TEXCOORD0;
};

VertDataOut VSDefault(VertDataOut v_in)
{
VertDataOut vert_out;
vert_out.pos = mul(float4(v_in.pos.xyz, 1.0), ViewProj);
vert_out.uv = v_in.uv;
return vert_out;
}

/**
* Standard Kawase blur
*/
float4 PSKawaseBlur(VertDataOut v_in) : TARGET
{
// Calculate the blur value from neighboring pixels
float4 sum = float4(0.0, 0.0, 0.0, 0.0);
sum += image.Sample(textureSampler, v_in.uv + float2( xOffset, yOffset));
sum += image.Sample(textureSampler, v_in.uv + float2(-xOffset, yOffset));
sum += image.Sample(textureSampler, v_in.uv + float2( xOffset, -yOffset));
sum += image.Sample(textureSampler, v_in.uv + float2(-xOffset, -yOffset));
sum *= 0.25;
return sum;
}

/**
* Mask aware Kawase blur
* Only uses pixels which are in the masked area for blur. This prevents the "Halo Effect" on
* the border pixels of the mask.
*/
float4 PSKawaseBlurMaskAware(VertDataOut v_in) : TARGET
{
if (focalmask.Sample(textureSampler, v_in.uv).r == 0) {
// No mask - return the original image value without any blur
return image.Sample(textureSampler, v_in.uv);
}

// Calculate the blur value from neighboring pixels

float alphaValue1 = focalmask.Sample(textureSampler, v_in.uv + float2( xOffset, yOffset)).r;
float4 sum = image.Sample(textureSampler, v_in.uv + float2( xOffset, yOffset)) * alphaValue1;

float alphaValue2 = focalmask.Sample(textureSampler, v_in.uv + float2(-xOffset, yOffset)).r;
sum += image.Sample(textureSampler, v_in.uv + float2(-xOffset, yOffset)) * alphaValue2;

float alphaValue3 = focalmask.Sample(textureSampler, v_in.uv + float2( xOffset, -yOffset)).r;
sum += image.Sample(textureSampler, v_in.uv + float2( xOffset, -yOffset)) * alphaValue3;

float alphaValue4 = focalmask.Sample(textureSampler, v_in.uv + float2(-xOffset, -yOffset)).r;
sum += image.Sample(textureSampler, v_in.uv + float2(-xOffset, -yOffset)) * alphaValue4;

float pixelCounter = alphaValue1 + alphaValue2 + alphaValue3 + alphaValue4;

// Complement the blur pixels with a relative fraction of the center pixel
return (sum + image.Sample(textureSampler, v_in.uv) * (4.0 - pixelCounter)) * 0.25;
}

technique Draw
{
pass
{
vertex_shader = VSDefault(v_in);
pixel_shader = PSKawaseBlur(v_in);
}
}

technique DrawMaskAware
{
pass
{
vertex_shader = VSDefault(v_in);
pixel_shader = PSKawaseBlurMaskAware(v_in);
}
}
78 changes: 78 additions & 0 deletions data/effects/masking.effect
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
uniform float4x4 ViewProj;
uniform texture2d image;
uniform texture2d focalmask;
uniform float4 color = {1.0, 1.0, 1.0, 1.0};

sampler_state textureSampler {
Filter = Linear;
AddressU = Clamp;
AddressV = Clamp;
};

struct VertDataIn {
float4 pos : POSITION;
float2 uv : TEXCOORD0;
};

struct VertDataOut {
float4 pos : POSITION;
float2 uv : TEXCOORD0;
};

VertDataOut VSDefault(VertDataOut v_in)
{
VertDataOut vert_out;
vert_out.pos = mul(float4(v_in.pos.xyz, 1.0), ViewProj);
vert_out.uv = v_in.uv;
return vert_out;
}

/* where the mask is 1 draw a solid color */
float4 PSSolid(VertDataOut v_in) : TARGET
{
if (focalmask.Sample(textureSampler, v_in.uv).r != 0) {
return color.bgra;
}
return image.Sample(textureSampler, v_in.uv);
}

/* draw just the mask */
float4 PSMask(VertDataOut v_in) : TARGET
{
if (focalmask.Sample(textureSampler, v_in.uv).r != 0) {
return float4(1,1,1,1);
}
return float4(0,0,0,1);
}

float4 PSDefault(VertDataOut v_in) : TARGET
{
return image.Sample(textureSampler, v_in.uv);
}

technique DrawSolidColor
{
pass
{
vertex_shader = VSDefault(v_in);
pixel_shader = PSSolid(v_in);
}
}

technique DrawMask
{
pass
{
vertex_shader = VSDefault(v_in);
pixel_shader = PSMask(v_in);
}
}

technique Draw
{
pass
{
vertex_shader = VSDefault(v_in);
pixel_shader = PSDefault(v_in);
}
}
Binary file added data/models/edgeyolo_tiny_lrelu_coco_480x800.onnx
Binary file not shown.
Binary file not shown.
14 changes: 14 additions & 0 deletions src/FilterData.h
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,28 @@
struct filter_data {
std::string useGPU;
uint32_t numThreads;
float conf_threshold;
std::string modelSize;

int objectCategory;
bool maskingEnabled;
std::string maskingType;
int maskingColor;
int maskingBlurRadius;
bool trackingEnabled;
float zoomFactor;
std::string zoomObject;

obs_source_t *source;
gs_texrender_t *texrender;
gs_stagesurf_t *stagesurface;
gs_effect_t *effect;
gs_effect_t *kawaseBlurEffect;
gs_effect_t *maskingEffect;

cv::Mat inputBGRA;
cv::Mat outputPreviewBGRA;
cv::Mat outputMask;

bool isDisabled;
bool preview;
Expand Down
3 changes: 1 addition & 2 deletions src/consts.h
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,8 @@ const char *const USEGPU_CUDA = "cuda";
const char *const USEGPU_TENSORRT = "tensorrt";
const char *const USEGPU_COREML = "coreml";

const char *const EFFECT_PATH = "effects/mask_alpha_filter.effect";
const char *const KAWASE_BLUR_EFFECT_PATH = "effects/kawase_blur.effect";
const char *const BLEND_EFFECT_PATH = "effects/blend_images.effect";
const char *const MASKING_EFFECT_PATH = "effects/masking.effect";

const char *const PLUGIN_INFO_TEMPLATE =
"<a href=\"https://github.com/occ-ai/obs-detect/\">Detect Plugin</a> (%1) by "
Expand Down
Loading

0 comments on commit 0dd49ac

Please sign in to comment.