Stereo rendering of two intertwine toroids.
I have been working with on a 3D scanning/printing project using a time-of-flight (TOF) camera. One of the features is to manipulate and visualize the scene in 3D. To accomplish this project, I used OpenFrameworks (OF), which has OpenGL under the hood to support rendering and manipulation of 3D scene. To support 3D viewing, I followed a colleague’s advice and went with the old-fashion color-glasses approach, known as anaglyph. This article describes the process of creating anaglyph under OF.
OF offers a cross-platform run-time framework for rapid application development. It integrates powerful libraries, including OpenGL, OpenCV, UI, audio, video, file and serial I/O, math, and keyboard and mouse. There are also many third-party addons that will further facilitate application development. OF provides the ofCamera class to view a 3D scene, and the ofEasyCam, a derived class of ofCamera, to manipulate the viewing angle with the mouse.
To render the scene for stereo viewing, the scene needs to be rendered twice: once from the left-eye perspective with a color mask that passes only red, and one from the right that passes only blue. The two perspectives are separated by the interpupilary distance, for which I used a ofxUI slider to adjust. I used ofEasyCam for virtual camera, because it facilitates simple mouse interactions with the 3D world, but since the mouse movements manipulates the virtual camera and not the world, when the scene is rotated, the red-blue separation is no longer along the eye-line, which distorts stereo perception. To address this shortcoming, I created a new camera class called ofStereoCam by modifying the ofEasyCam. I kept the ofEasyCam mouse controls to record the movements, but removed the code that alters the virtual camera. Instead, the detected movements are used to rotate and move the scene inside the testApp::draw() method.
The ofStereoCam class is defined as follow:
[code language=”cpp”]
#pragma once
#include "ofCamera.h"
#include "ofEvents.h"
class ofStereoCam : public ofCamera {
public:
ofStereoCam();
~ofStereoCam();
// TODO: this should be ofGetViewRect() eventually
virtual void begin(ofRectangle viewport = ofGetCurrentViewport());
void reset();
//—————————————-
// advanced functions
void setTarget(const ofVec3f& target);
void setTarget(ofNode& target);
ofNode& getTarget();
void setDistance(float distance);
float getDistance() const;
// drag is how quickly the camera picks up and slows down
// it is a normalized value between 0-1
void setDrag(float drag);
float getDrag() const;
// the translation key is the key used to switch between rotation and translation.
// translation happens only when the key is pressed.
void setTranslationKey(char key);
char getTranslationKey();
// enable or disable mouse input to navigate
void enableMouseInput();
void disableMouseInput();
bool getMouseInputEnabled();
void enableMouseMiddleButton();
void disableMouseMiddleButton();
bool getMouseMiddleButtonEnabled();
void setAutoDistance(bool bAutoDistance);
public:
float mRotX;
float mRotY;
float mRotZ;
float mMoveX;
float mMoveY;
float mMoveZ;
private:
void setDistance(float distance, bool save);
ofNode target;
bool bEnableMouseMiddleButton;
bool bApplyInertia;
bool bDoTranslate;
bool bDoRotate;
bool bValidClick;
bool bInsideArcball;
bool bMouseInputEnabled;
bool bDistanceSet;
bool bAutoDistance;
float lastDistance;
float drag;
float xRot;
float yRot;
float zRot;
float moveX;
float moveY;
float moveZ;
float sensitivityXY;
float sensitivityZ;
float sensitivityRot;
float rotationFactor;
ofVec2f mouse;
ofVec2f lastMouse;
ofVec2f mouseVel;
void updateRotation();
void updateTranslation();
void update(ofEventArgs & args);
void updateMouse();
char doTranslationKey;
unsigned long lastTap;
ofQuaternion curRot;
ofRectangle viewport;
};
[/code]
The main changes are the addition of mRotX, mRotY, mRotZ, and mMoveX, mMoveY, mMoveZ. These are newly added public properties accessible by testApp::draw() to manipulate the scene. The ofStereCam implementation is exactly the same as that of the ofEasyCam, except the calls to set positions and rotations are removed. Instead, the variables, mRotX, mRotY, mRotZ, and mMoveX, mMoveY, mMoveZ, record the intended movements to be used by testApp::draw().
[code language=”cpp”]
#include "ofStereoCam.h"
#include "ofMath.h"
#include "ofUtils.h"
// when an ofStereoCam is moving due to momentum, this keeps it
// from moving forever by assuming small values are zero.
float minDifference = 0.1e-5;
// this is the default on windows os
unsigned long doubleclickTime = 200;
//—————————————-
ofStereoCam::ofStereoCam(){
lastTap = 0;
lastDistance = 0;
drag = 0.9f;
//when 1 moving the mouse from one side to the other of the
//arcball (min(viewport.width, viewport.height)) will rotate
//180degrees. when .5, 90 degrees
sensitivityRot = 1.0f;
sensitivityXY = .5;
sensitivityZ= .7;
bDistanceSet = false;
bMouseInputEnabled = false;
bDoRotate = false;
bApplyInertia =false;
bDoTranslate = false;
bInsideArcball = true;
bValidClick = false;
bEnableMouseMiddleButton = true;
bAutoDistance = true;
doTranslationKey = ‘m’;
reset();
enableMouseInput();
}
//—————————————-
ofStereoCam::~ofStereoCam(){
disableMouseInput();
}
//—————————————-
void ofStereoCam::update(ofEventArgs & args){
if(!bDistanceSet && bAutoDistance){
setDistance(getImagePlaneDistance(viewport), true);
}
rotationFactor = sensitivityRot * 180 / min(viewport.width, viewport.height);
if (bMouseInputEnabled) {
updateMouse();
}
if (bDoRotate) {
updateRotation();
}else if (bDoTranslate) {
updateTranslation();
}
}
//—————————————-
void ofStereoCam::begin(ofRectangle viewport){
this->viewport = viewport;
ofCamera::begin(viewport);
}
//—————————————-
void ofStereoCam::reset(){
target.resetTransform();
target.setPosition(0,0, 0);
lookAt(target);
resetTransform();
setPosition(0, 0, lastDistance);
xRot = 0;
yRot = 0;
zRot = 0;
moveX = 0;
moveY = 0;
moveZ = 0;
mRotX = 0;
mRotY = 0;
mRotZ = 0;
mMoveX = 0;
mMoveY = 0;
mMoveZ = 0;
}
//—————————————-
void ofStereoCam::setTarget(const ofVec3f& targetPoint){
target.setPosition(targetPoint);
lookAt(target);
}
//—————————————-
void ofStereoCam::setTarget(ofNode& targetNode){
target = targetNode;
lookAt(target);
}
//—————————————-
ofNode& ofStereoCam::getTarget(){
return target;
}
//—————————————-
void ofStereoCam::setDistance(float distance){
setDistance(distance, true);
}
//—————————————-
void ofStereoCam::setDistance(float distance, bool save){
if (distance > 0.0f){
if(save){
this->lastDistance = distance;
}
setPosition(target.getPosition() + (distance * getZAxis()));
bDistanceSet = true;
}
}
//—————————————-
float ofStereoCam::getDistance() const {
return target.getPosition().distance(getPosition());
}
//—————————————-
void ofStereoCam::setAutoDistance(bool bAutoDistance){
this->bAutoDistance = bAutoDistance;
if (bAutoDistance) {
bDistanceSet = false;
}
}
//—————————————-
void ofStereoCam::setDrag(float drag){
this->drag = drag;
}
//—————————————-
float ofStereoCam::getDrag() const {
return drag;
}
//—————————————-
void ofStereoCam::setTranslationKey(char key){
doTranslationKey = key;
}
//—————————————-
char ofStereoCam::getTranslationKey(){
return doTranslationKey;
}
//—————————————-
void ofStereoCam::enableMouseInput(){
if(!bMouseInputEnabled){
bMouseInputEnabled = true;
// ofRegisterMouseEvents(this);
ofAddListener(ofEvents().update , this, &ofStereoCam::update);
}
}
//—————————————-
void ofStereoCam::disableMouseInput(){
if(bMouseInputEnabled){
bMouseInputEnabled = false;
//ofUnregisterMouseEvents(this);
ofRemoveListener(ofEvents().update, this, &ofStereoCam::update);
}
}
//—————————————-
bool ofStereoCam::getMouseInputEnabled(){
return bMouseInputEnabled;
}
//—————————————-
void ofStereoCam::enableMouseMiddleButton(){
bEnableMouseMiddleButton = true;
}
//—————————————-
void ofStereoCam::disableMouseMiddleButton(){
bEnableMouseMiddleButton = false;
}
//—————————————-
bool ofStereoCam::getMouseMiddleButtonEnabled(){
return bEnableMouseMiddleButton;
}
//—————————————-
void ofStereoCam::updateTranslation(){
if (bApplyInertia) {
moveX *= drag;
moveY *= drag;
moveZ *= drag;
if (ABS(moveX) <= minDifference && ABS(moveY) <= minDifference && ABS(moveZ) <= minDifference) {
bApplyInertia = false;
bDoTranslate = false;
}
}
mMoveX += moveX;
mMoveY -= moveY;
mMoveZ += moveZ;
}
//—————————————-
void ofStereoCam::updateRotation(){
if (bApplyInertia) {
xRot *=drag;
yRot *=drag;
zRot *=drag;
if (ABS(xRot) <= minDifference && ABS(yRot) <= minDifference && ABS(zRot) <= minDifference) {
bApplyInertia = false;
bDoRotate = false;
}
}
mRotX += xRot;
mRotY -= yRot;
mRotZ += zRot;
}
//—————————————-
void ofStereoCam::updateMouse(){
mouse = ofVec2f(ofGetMouseX(), ofGetMouseY());
if(viewport.inside(mouse.x, mouse.y) && !bValidClick && ofGetMousePressed()){
unsigned long curTap = ofGetElapsedTimeMillis();
if(lastTap != 0 && curTap – lastTap < doubleclickTime){
reset();
}
if ((bEnableMouseMiddleButton && ofGetMousePressed(OF_MOUSE_BUTTON_MIDDLE)) || ofGetKeyPressed(doTranslationKey) || ofGetMousePressed(OF_MOUSE_BUTTON_RIGHT)){
bDoTranslate = true;
bDoRotate = false;
bApplyInertia = false;
}else if (ofGetMousePressed(OF_MOUSE_BUTTON_LEFT)) {
bDoTranslate = false;
bDoRotate = true;
bApplyInertia = false;
if(ofVec2f(mouse.x – viewport.x – (viewport.width/2), mouse.y – viewport.y – (viewport.height/2)).length() < min(viewport.width/2, viewport.height/2)){
bInsideArcball = true;
}else {
bInsideArcball = false;
}
}
lastTap = curTap;
lastMouse = mouse;
bValidClick = true;
bApplyInertia = false;
}
if (bValidClick) {
if (!ofGetMousePressed()) {
bApplyInertia = true;
bValidClick = false;
}else {
int vFlip;
if(isVFlipped()){
vFlip = -1;
}else{
vFlip = 1;
}
mouseVel = mouse – lastMouse;
if (bDoTranslate) {
moveX = 0;
moveY = 0;
moveZ = 0;
if (ofGetMousePressed(OF_MOUSE_BUTTON_RIGHT)) {
moveZ = mouseVel.y * sensitivityZ * (getDistance() + FLT_EPSILON)/ viewport.height;
}else {
moveX = -mouseVel.x * sensitivityXY * (getDistance() + FLT_EPSILON)/viewport.width;
moveY = vFlip * mouseVel.y * sensitivityXY * (getDistance() + FLT_EPSILON)/viewport.height;
}
}else {
xRot = 0;
yRot = 0;
zRot = 0;
if (bInsideArcball) {
xRot = vFlip * -mouseVel.y * rotationFactor;
yRot = -mouseVel.x * rotationFactor;
}else {
ofVec2f center(viewport.width/2, viewport.height/2);
zRot = – vFlip * ofVec2f(mouse.x – viewport.x – center.x, mouse.y – viewport.y – center.y).angle(lastMouse – ofVec2f(viewport.x, viewport.y) – center);
}
}
lastMouse = mouse;
}
}
}
[/code]
One can diff the above code with that of ofEasyCam and find that ofStereoCam is derived from ofEasyCam but with a few small but important changes.
The testApp::draw() renders the scene twice if stereo viewing is enabled. The GL_COLOR_BUFFER_BIT and GL_DEPTH_BUFFER_BIT are cleared to allow both color-masked scenes to overlap without completely covering up one another, thus weakens the stereo effect. The call, renderScene(), is where the 3D scene is actually generated. An offset value is passed into it to let it know which view, left or right, is to be rendered. The offset is one-half of the interpupilary distance, a signed value.
[code language=”cpp”]
void testApp::renderScene(float offset)
{
mCam.begin();
mCam.setFov(mFOV);
// Left eye
if (offset > 0) {
ofRotateY(180-mVergence);
glColorMask(true, false, false, false);
}
// Right eye
else if (offset < 0) {
ofRotateY(180+mVergence);
glColorMask(false, false, true, false);
}
// Mono
else {
ofRotateY(180);
glColorMask(true, true, true, true);
}
ofTranslate(ofPoint(offset, 0, 300));
ofPushMatrix();
ofTranslate(mCam.getXAxis() * mCam.mMoveX
+ mCam.getYAxis() * mCam.mMoveY
+ mCam.getZAxis() * mCam.mMoveZ);
ofRotateX(mCam.mRotX);
ofRotateY(mCam.mRotY);
ofRotateZ(mCam.mRotZ);
for (int i=0; i < 3; i++)
mPointLight[i].enable();
mDirLight.enable();
if (!mDepthCam->getColorEn())
glDisable(GL_COLOR_MATERIAL);
mMaterial.begin();
if (mbStereoTest) {
renderTestScene();
}
// Draw axis
if (mbAxisEn) drawAxis();
for (int i=0; i < 3; i++)
mPointLight[i].disable();
mDirLight.disable();
mMaterial.end();
ofDisableLighting();
if (mbShowPointLight1) {
ofSetColor(mPointLight[0].getDiffuseColor());
mPointLight[0].draw();
}
if (mbShowPointLight2) {
ofSetColor(mPointLight[1].getDiffuseColor());
mPointLight[1].draw();
}
if (mbShowPointLight3) {
ofSetColor(mPointLight[2].getDiffuseColor());
mPointLight[2].draw();
}
if (mbShowDirLight) {
ofSetColor(mDirLight.getDiffuseColor());
mDirLight.draw();
}
ofPopMatrix();
glColorMask(true, true, true, true);
mCam.end();
}
[/code]
The ofPushMatrix() and ofPopMatrix() are a pair of critical functions. Any rotations and translations between the pair are relative to the current pose. Note that the rotations and translations collected by ofStereoCam are used to move the scene after the ofPushMatrix() call. Why is this important? It is because this allows the line of color separation to stay consistent with the eye-line of the viewer and independent of the scene’s pose.
Anaglyph is a low-cost way of creating stereo effects that, together with a pair of color glasses, enhances one’s understanding of a 3D scene. The method described herein can be applied to view artificially generated scenes, or live scenes from captured by 3D Time-of-Flight cameras, such as the Microsoft Kinect and the Creative Gesture Camera. Chipsets that allows one to create embedded 3D-TOF modules are also becoming readily available, enabling an entirely new generation of smart applications.
Further Reading:
(The above article is solely the expressed opinion of the author and does not necessarily reflect the position of his current and past employers)