Class VirtualBackgroundProcessor

The VirtualBackgroundProcessor, when added to a VideoTrack, replaces the background in each video frame with a given image, and leaves the foreground (person(s)) untouched. Each instance of VirtualBackgroundProcessor should be added to only one VideoTrack at a time to prevent overlapping of image data from multiple VideoTracks.

Example

import { createLocalVideoTrack } from 'twilio-video';
import { Pipeline, VirtualBackgroundProcessor } from '@twilio/video-processors';

let virtualBackground;
const img = new Image();

img.onload = () => {
  virtualBackground = new VirtualBackgroundProcessor({
    assetsPath: 'https://my-server-path/assets',
    backgroundImage: img,
    pipeline: Pipeline.WebGL2,

    // Desktop Safari and iOS browsers do not support SIMD.
    // Set debounce to true to achieve an acceptable performance.
    debounce: isSafari(),
  });

  virtualBackground.loadModel().then(() => {
    createLocalVideoTrack({
      // Increasing the capture resolution decreases the output FPS
      // especially on browsers that do not support SIMD
      // such as desktop Safari and iOS browsers, or on Chrome
      // with capture resolutions above 640x480 for webgl2.
      width: 640,
      height: 480,
      // Any frame rate above 24 fps on desktop browsers increase CPU
      // usage without noticeable increase in quality.
      frameRate: 24
    }).then(track => {
      track.addProcessor(virtualBackground, {
        inputFrameBufferType: 'video',
        outputFrameBufferContextType: 'webgl2',
      });
    });
  });
};
img.src = '/background.jpg';
Copy