Processing Data to QC via OSC

idlefon's picture

Hi! I was trying to use the blob detection in processing (with the OpenCV and OscP5 library) and then send the blob points (blobs[i].points) and the blobs' rectangular vertex data (blobs[i].rectangle) to QC.

Since I've just started processing, My knowledge is very small about it's work flow. Thought maybe one of you guys can help me with this:

1- does QC accept structures via OSC, for example can I send the whole "blobs" object to QC and then use Key member Patch to take the data I want in QC. if I can do this please let me know what should I write in Processing.

2- if not, What is the most efficient way to send the data to QC ( I can make multiple Arrays and copy the "blobs" data to them one by one but I think it slows down everything)

Cheers for any suggestions!

dust's picture
Re: Processing Data to QC via OSC

set up oscp5 address like normal and send your message like this to qc.

 OscMessage myMessage = new OscMessage("/blob/data");
 
myMessage.add(new float[] {blob.x,blob.y,blob.w,blob.h})

use the name space "/blob/data" as type floats in qc osc receiver as it is the only type that will parse a structure. or you can pack all your data into a string and then do string explode in qc as well. i like floats. make sure to cast your data to float, or int qc will parse ints as a float as well.

a single osc message can look very different but the message you want to send for qc structure of floats would be a string like this.

"/name/space float float float float...."

idlefon's picture
Re: Processing Data to QC via OSC

Cheers dust for the quick reply. It worked and it's all thanks to you!

gtoledo3's picture
Re: Processing Data to QC via OSC

Could you give a rundown of what the blob detector in OpenCV/OscP5 library should be putting out when it's achieving what you wish ...I know you have your problem solved, but this might be helpful for getting blob tracking going in QC itself at some point.

Is there any pre-built little processing app/sketch that puts out OSC info for blobs... or rather, could you point to what you're using as your starting point? I used to use processing quite a bit, but never looked at openCV blob tracking for it.

idlefon's picture
Re: Processing Data to QC via OSC

Sorry George for the late reply,

Blob detection in Processing is available via the excellent OpenCV Library. The library accesses the blobs through the opencv.blobs object and the Blob[] class. so one can activate the blobs with a command like this:

Blob[] blobs = opencv.blobs( 100, w*h/3, 20, true );

The library gives access to some important properties of the blobs. for instance if the blob is inside another blob or the area of the blob etc.

What I do here is simply use the OscP5 Library to send 2 things to QC:

1- The rectangular dimensions of the blob via a command like this:

myMessage.add(new float[] {blobs[i].rectangle.x,blobs[i].rectangle.y,blobs[i].rectangle.width,blobs[i].rectangle.height});

2- The Points with which you make draw the blob itself in QC via the "Point" class and the "points[j].x" and "points[j].y" objects.

Since you can only send one-dimension float arrays (as far as I know) via OSC, you can separate the blobs' data by using a separator variable like 666666 for instance, so when you reach points.x =666666 and points.y=666666 in the sent array you'll know that you have moved to the next blob.

I can post the whole processing code and the QC patch if you'd like.

jersmi's picture
Re: Processing Data to QC via OSC

i for one would love to see your setup, idlefon!

monobrau's picture
Re: Processing Data to QC via OSC

count me in as well!

idlefon's picture
Re: Processing Data to QC via OSC

OK everyone,

I attached a .qtz comp wich draws the blob and the rect

the processing code is just like the example in the OpenCV library, just remember to adjust the ip address in the code to your system's ip.

I couldn't attach the .pde file (it's not allowed apparently), so I'm gonna copy the code in the next post. by the way I adjusted the code so that it only sends the biggest blob (the biggest area) to QC. you can configure it easily to send all the blobs (use the method I suggested in my previous post)

Have fun!!

PreviewAttachmentSize
Simple Blob.qtz11.17 KB

idlefon's picture
Re: Processing Data to QC via OSC

import hypermedia.video.*;
import java.awt.*;


import oscP5.*;
import netP5.*;

OscP5 oscP5;
NetAddress myRemoteLocation;


OpenCV opencv;

int w = 320;
int h = 240;
int threshold = 80;
float k=0;
final int COLOR_SPACE = OpenCV.RGB;

boolean find=true;

PFont font;

void setup() {

  size( w*2+30, h*2+30 );

  oscP5 = new OscP5(this,12000);

  opencv = new OpenCV( this );
  opencv.capture(w,h);

  font = loadFont( "AndaleMono.vlw" );
  textFont( font );

  println( "Drag mouse inside sketch window to change threshold" );
  println( "Press space bar to record background image" );

  myRemoteLocation = new NetAddress("192.168.2.10",12000);
}



void draw() {

  background(0);
  opencv.read();
  //opencv.flip( OpenCV.FLIP_HORIZONTAL );
  OscMessage myMessage = new OscMessage("/blob/data/Rect");
  OscMessage myMessagePointsX = new OscMessage("/blob/data/PointsX");
  OscMessage myMessagePointsY = new OscMessage("/blob/data/PointsY");
  if ( COLOR_SPACE!=OpenCV.RGB ) opencv.convert( COLOR_SPACE );
  opencv.restore( COLOR_SPACE );
  opencv.blur( OpenCV.GAUSSIAN, 30 );
  image( opencv.image(), 10, 10 );
  image( opencv.image(), 10, 10 );               // RGB image
  image( opencv.image(OpenCV.GRAY), 20+w, 10 );   // GRAY image
  image( opencv.image(OpenCV.MEMORY), 10, 20+h ); // image in memory

  opencv.absDiff();
  opencv.threshold(threshold);
  image( opencv.image(OpenCV.GRAY), 20+w, 20+h ); // absolute difference image


  // working with blobs
  Blob[] blobs = opencv.blobs( 100, w*h/3, 20, true );

  noFill();

  pushMatrix();
  translate(20+w,20+h);
  k=0;
  for( int i=0; ilength; i++ ) {

    Rectangle bounding_rect   = blobs[i].rectangle;
    float area = blobs[i].area;
    float circumference = blobs[i].length;
    Point centroid = blobs[i].centroid;
    Point[] points = blobs[i].points;

    if (blobs[i].isHole==false)
    {
      myMessage.add(new float[] {
        blobs[i].rectangle.x,blobs[i].rectangle.y,blobs[i].rectangle.width,blobs[i].rectangle.height
      }
      );
    }
    // rectangle
    noFill();
    stroke( blobs[i].isHole ? 128 : 64 );
    rect( bounding_rect.x, bounding_rect.y, bounding_rect.width, bounding_rect.height );


    // centroid
    stroke(0,0,255);
    line( centroid.x-5, centroid.y, centroid.x+5, centroid.y );
    line( centroid.x, centroid.y-5, centroid.x, centroid.y+5 );
    noStroke();
    fill(0,0,255);
    text(bounding_rect.x,centroid.x+5, centroid.y+5 );


    fill(255,0,255,64);
    stroke(255,0,255);
    if ( points.length>0 ) {
      beginShape();
      for( int j=0; jlength; j++ ) {
        vertex( points[j].x, points[j].y );
        if (blobs[i].isHole==false && kif (j==0)
          {
            myMessagePointsX=new OscMessage("/blob/data/PointsX");
            ;
            myMessagePointsY=new OscMessage("/blob/data/PointsY");
            ;
          }
          myMessagePointsX.add(points[j].x );
          myMessagePointsY.add(points[j].y );
        }
      }
      endShape(CLOSE);
    }
    if (knoStroke();
    fill(255,0,255);
    text( circumference, centroid.x+5, centroid.y+15 );
  }
  popMatrix();
  oscP5.send(myMessage, myRemoteLocation);
  oscP5.send(myMessagePointsX, myRemoteLocation);
  oscP5.send(myMessagePointsY, myRemoteLocation);
}

void keyPressed() {
  if ( key==' ' ) opencv.remember();
}

void mouseDragged() {
  threshold = int( map(mouseX,0,width,0,255) );
}

public void stop() {
  opencv.stop();
  super.stop();
}

gtoledo3's picture
Re: Processing Data to QC via OSC

Hey, that's cool! (haven't looked yet)...

@smokris, et al.: is there a way to use processing.js to have sketches embed in the site? If it's really trivial (I suspect it may be), it might be a cool feature... but I'm not sure how much people would use it (one use case so far!).

@idlefon: You could attach the pde (or any other non-supported file) by zipping it. (not a prob to just post the source like you did either, thanks!)

BTW, one "blob" tracking, as you mention this attachment being, is doable using QC, with CI, GLSL or CL. Multi-blob tracking is also doable...but no one has released anything. I guess I'll have to check out with the code to send multiples; I'm interested to see that part working robustly, so that I can tune my QC multi-blob tracking to be at least equivalent.

gtoledo3's picture
Re: Processing Data to QC via OSC

It seems like this could be exposed in the CVTools library in way similar to the HAAR object; image goes to blobber filter-> to tracker that has similar settings to the HAAR tracker.

I went ahead and attached it as a PDE... still haven't played w/ it :)

PreviewAttachmentSize
blob.pde_.zip1.38 KB

gtoledo3's picture
Re: Processing Data to QC via OSC

What version of Processing did you use, and do I need to add any other libraries from the standard dl? I'm getting an unexpected token result right where the "myMessagePoints.X..." clause starts.

I haven't used Processing for a bit, but my old comps are working, so I think my install is ok. I'm probably using the first or second official release (as opposed to the prerelease stuff that was around forever and a day).

idlefon's picture
Re: Processing Data to QC via OSC

I used Processing 1.2.1 Goerge and yes you'll need two extra libraries:

The OpenCV Library: http://ubaa.net/shared/processing/opencv/

The OSCP5 Library: http://www.sojamo.de/libraries/oscP5/

idlefon's picture
Re: Processing Data to QC via OSC

Sorry, You're right, I should have zipped it (Done!)

I really didn't know one could do blob tracking (with a fast FPS) with CI, GLSL or OpenCL in QC . I for one would kill to know how. I thought it's a CV thing and since it was not in the Kineme OpenCV, i assumed it was not doable in QC.

PreviewAttachmentSize
blobs_send_to_QC.pde_.zip1.43 KB

gtoledo3's picture
Re: Processing Data to QC via OSC

Thanks, I appreciate it. Hopefully I'll get a chance to give it a run tonight, and report back.

gtoledo3's picture
Re: Processing Data to QC via OSC

Checkout smokris and jstrecker's composition Centroid Calc (I think that's the name), in the composition repository. On strecker's machine, it's sorta slow (I can't remember the fps, but I think he's using the nVidia 8600), but on my machine, I get an easy 60fps, using the nVidia 9400 or 9600 (mbpro laptop). I don't know what the dif with that qtz is performance wise across a wide variety of machines.

That composition returns 1 centroid/blob.

Blob tracking is basically just a generic image routine that doesn't necessarily have to be linked to any library... just a library that supports the steps needed to derive the routine.

What's less intuitive to do in QC is to derive the region of interest, and then have the blob bounds track to just that area. It's super easy to derive one blob.

I've been been deriving multiple with a loop, and have taken the CI image routine, and converted it step by step to CL. I'm happy with fps, and it seems ok, I've just been looking for a decent example of "blob tracking" happening in some other environment, so that I can compare, and make sure I'm not unleashing anything dumb.

I've also been thinking that maybe I can be using local memory and add some index pixel offsets in a certain way (OpenCL), that would make stuff be even better... also, also, I've been noting certain kernels run more effectively when converting an RGB structure to a single lane structure, then converting back at the end. I want to play around w/ all that before I'm satisfied with a multi-blob QC routine.

However, as you say, this is sorta written into the CV library, so my hope would be that it would just get piped into that at some point, and be reasonably performant, rather than rewrite the whole routine in a C derivative (OpenCL or GLSL) and run it in QC...groan. (I may end up just rewriting it for speed/performance/"fun" though...).

I still think it hasn't fully occurred to people the breadth of things that OpenCL can do, because of sort of lacking QC examples.

smokris's picture
Re: Processing Data to QC via OSC

@gtoledo3, we're discussing adding a processing.js filter to allow inline compositions on here. Meanwhile I've added pde as an allowed attachment file type.

jersmi's picture
Re: Processing Data to QC via OSC

Thank you for posting this. (And thank you, GT, for the nice overview on blob tracking re: OpenCL, et al. Good luck with your pursuits in OpenCL. Exciting stuff!)

gtoledo3's picture
Re: Processing Data to QC via OSC

Cool... this is obviously a QC-centric forum, but I've always really loved the way that processing.js can embed in a webpage. I would love to see more people of other disciplines enter the mix.

dust's picture
Re: Processing Data to QC via OSC

I got a basic blob tracker going with open cl. I'm using it as a depth tracker with kinect. Right now I'm getting decent results. I have added a variable resolution so you can get a faster track if you want, it seems the higher the grid count the better more accurate it is for reporting blob size. a really small grid like screen size / 40 tracks and runs much faster. When I get online I will post.

monobrau's picture
Re: Processing Data to QC via OSC

Wow, sound cool! can't wait!

smokris's picture
Re: Processing Data to QC via OSC

You can now embed Processing.js sketches in kineme.net forum posts, using markup like

[processing] ... [/processing]

...and you'll get a source and rendered view:

 
void setup() {
   size(200, 200);
   background(100);
   stroke(255);
   ellipse(50, 50, 25, 25);
}
 

Thanks, @mradcliffe!

idlefon's picture
Re: Processing Data to QC via OSC

Marvelous!

Thanks Smokris and mradcliffe!

mradcliffe's picture
Re: Processing Data to QC via OSC

Javascript is loading on init now instead of trying to do it in the filter itself. A bit disappointing.

Edit: found another bug when posting this :|

 
void setup() {
  size(100, 100);
  background(0);
}
 

gtoledo3's picture
Re: Processing Data to QC via OSC

It's kind awesome to see that pop up on here though...

gtoledo3's picture
Re: Processing Data to QC via OSC

 
final int W=100;
final int H=100;
final int D=1;
final int Skip = 4;
final int N=10000;
 
final float dt = 0.02;
final float dx = 0.02;
final float dy = 0.02;
final float rho = 0.2;
final float nu = 0.001;
 
float[] px = new float[N];
float[] py = new float[N];
float[][][] u = new float[W][H][2];
float[][][] v = new float[W][H][2];
 
void setup(){
  size(W*D,H*D,P2D);
 
   frameRate(30);
 
  for(int i=0; i<W; i++){
    for(int j=0; j<H; j++){
      for(int h=0; h<2; h++){
        if(random(100)<5){
          u[i][j][h] = random(0.004)-0.002;
          v[i][j][h] =  random(0.004)-0.002;
        }else{
          u[i][j][h] = 0;
          v[i][j][h] =  0;
        }
      }
    }
  }
 
  for(int i=0; i<N; i++){
    px[i] =  random(W-1)+1;
    py[i] = random(H-1)+1;
  }
}
 
void draw(){
  background(0,0,0);
 
  //drawFlow();
  drawParticles();
 
  for(int i=0; i<Skip; i++){
    calc();
  }
 
  fill(255);
  text("fps: " + int(frameRate), 20, 20); 
}
 
void drawFlow(){
  stroke(255);
 
  for(int i=0; i<W; i++){
    for(int j=0; j<H; j++){
      point(i, u[i][j][0]);
    }
  }
}
 
void drawParticles(){
  stroke(255);
 
  int count = 0;
 
  for(int i=0; i<N; i++){
    if(px[i]>2 && px[i]<W-2 && py[i]>2 && py[i]<H-2){
      px[i] += u[int(px[i])][int(py[i])][0]*100;
 
      if(px[i]>2 && px[i]<W-2 && py[i]>2 && py[i]<H-2){
        py[i] += v[int(px[i])][int(py[i])][0]*100;
      }
    }
    else{
      if(count<N*0.05){
        px[i] = random(W);
        py[i] = random(H);
      }
      count++;
    }
    noStroke();
    fill(200,30);
    rect(px[i]*D,py[i]*D,4,4);
  }
}
 
void calc(){
  for(int i=0; i<W; i++){
    for(int j=0; j<H; j++){
      if(i==0 || i==W-1){
        u[i][j][1] = u[i][j][0]/2;
        v[i][j][1] = v[i][j][0]/2;
      }
      else if(j==0 || j==H-1){
        u[i][j][1] = u[i][j][0]/2;
        v[i][j][1] = v[i][j][0]/2;
      }
      else{
        u[i][j][1] = u[i][j][0];
        u[i][j][1] += u[i][j][0]*-dt/(2*dx)*(u[i+1][j][0] - u[i-1][j][0]);
        u[i][j][1] += v[i][j][0]*-dt/(2*dy)*(u[i][j+1][0] - u[i][j-1][0]);
        u[i][j][1] += nu*dt/(dx*dx)*(u[i+1][j][0] - 2*u[i][j][0] + u[i-1][j][0]);
        u[i][j][1] += nu*dt/(dy*dy)*(u[i][j+1][0] - 2*u[i][j][0] + u[i][j-1][0]);
 
        v[i][j][1] = v[i][j][0];
        v[i][j][1] += u[i][j][0]*-dt/(2*dx)*(v[i+1][j][0] - v[i-1][j][0]);
        v[i][j][1] += v[i][j][0]*-dt/(2*dy)*(v[i][j+1][0] - v[i][j-1][0]);
        v[i][j][1] += nu*dt/(dx*dx)*(v[i+1][j][0] - 2*v[i][j][0] + v[i-1][j][0]);
        v[i][j][1] += nu*dt/(dy*dy)*(v[i][j+1][0] - 2*v[i][j][0] + v[i][j-1][0]);
      }
    }
  }
 
  for(int i=0; i<W; i++){
    for(int j=0; j<H; j++){
      if(abs(u[i][j][1])>dx/dt) u[i][j][1]=(u[i-1][j][1]+u[i+1][j][1] + u[i][j-1][1]+u[i][j+1][1])/4;
 
      u[i][j][0] = u[i][j][1];
 
      if(abs(v[i][j][1])>dy/dt) v[i][j][1]=(v[i-1][j][1]+v[i+1][j][1] + v[i][j-1][1]+v[i][j+1][1])/4;
 
      v[i][j][0] = v[i][j][1];
    }
  }
}
 
void mouseDragged(){
  int x = constrain(mouseX/D,0,W-1);
  int y = constrain(mouseY/D,0,H-1);
  int px = constrain(pmouseX/D,0,W-1);
  int py = constrain(pmouseY/D,0,H-1);
 
  if(keyPressed){
    u[x][y][0] =- (x - px)*0.05;
    v[x][y][0] =- (y - py)*0.05;
  } 
  else {
    u[x][y][0] =+ (x - px)*0.05;
    v[x][y][0] =+ (y - py)*0.05;
  }
}
 
 

This is a tweak on a sketch "navie" from openprocessing.org, from the recent posts.

mradcliffe's picture
Re: Processing Data to QC via OSC

What sort of fps do you get on that sketch?

I guess it's a good example of how nice WebKit is. It renders at 6fps (Safari) v 1fps (Firefox 3.6). And that's cross-platform Firefox too.

gtoledo3's picture
Re: Processing Data to QC via OSC

It's a choker for sure... that's sort of why I posted it; I wanted to see how stable everything would be with a harder one to deal with. I get 8fps in Safari.

dust's picture
Re: Processing Data to QC via OSC

wow this is cool you can embed processing into kineme. i want to try.

 
/**
 * Esfera is located in the processing openGL examples. Esfera means sphere - globe or heaven or
 * heavenly orb. Esfera is by David Pena. Sudden Motion Sensor by Daniel Shiffman.
 * English translation by Dustin O'Connor.  
 * 
 * 
 */
//import the required libs
//import sms.*;
import processing.opengl.*;
 
 
//init hairball count to iterate.
int hairCount = 8000;
//declare hair as arrayList;
hair[] hairList ;
//init some variables for randomizing the hair lines placement and length;
float[] z = new float[hairCount]; 
float[] phi = new float[hairCount]; 
float[] hairLength = new float[hairCount]; 
 
float rad = 200;
float rx = 0;
float ry =0;
float x;
float y;
 
 
// interpolation lower limit and higher limits. accelerometer is in the range of -255 <-> 255
//float limLow;
//float limHi;
 
void setup() {
  //init x and y with the raw x and y values. this can be any number
 // x=Unimotion.getSMSX();
 // y=Unimotion.getSMSY();
 
  x = width/2;
  y = height/2;
 
 
//  limLow=-255;
 // limHi=255;
 
 
  size(1024, 768, OPENGL);
 
 
  rad = height/PI;
 
 
  hairList = new hair[hairCount];
  for (int i=0; i<hairCount; i++){
    hairList[i] = new hair();
 
  }
 
 
  noiseDetail(3);
 
}
 
void draw() {
 
 
  background(0);
  translate(width/2,height/2);
 
  //int[] vals = Unimotion.getSMSArray();
 
  // x = map((float)Unimotion.getSMSX(),limLow,limHi,0,width);
  // y = map((float)Unimotion.getSMSY(),limLow,limHi,0,height);
 
  float x = ((mouseX-(width/2))*0.005);
  float y = ((mouseY-(height/2))*0.005);
 
  rx = (rx*0.9)+(x*0.1);
  ry = (ry*0.9)+(y*0.1);
  rotateY(rx);
  rotateX(ry*-1.0);
 
 
   float rxp = ((x-(width/2))*0.005);
   float ryp = ((y-(height/2))*0.005);
 
 
  fill(0);
  noStroke();
  sphere(rad);
 
 
  for (int i=0;i<hairCount;i++){
    hairList[i].sketchHairBall();
  }
}
 
 
 
class hair
{
 
 // we are randomizing the z depth here so that our hair looks more natural.
  float z = random(-rad,rad);
  float phi = random(TWO_PI);
  float hairLength = random(1.15,1.2);
  float theta = asin(z/rad);
 
    void sketchHairBall(){
 
 
    float off = (noise(millis() * 0.0005,sin(phi))-0.5) * 0.3;
    float offb = (noise(millis() * 0.0007,sin(z) * 0.01)-0.5) * 0.3;
 
    float thetaff = theta+off;
    float phff = phi+offb;
 
 
    float x = rad * cos(theta) * cos(phi);
    float y = rad * cos(theta) * sin(phi);
    float z = rad * sin(theta);
 
 
 
    float msx= screenX(x,y,z);
    float msy= screenY(x,y,z);
 
 
    float xo = rad * cos(thetaff) * cos(phff);
    float yo = rad * cos(thetaff) * sin(phff);
    float zo = rad * sin(thetaff);
 
 
    float xb = xo * hairLength;
    float yb = yo * hairLength;
    float zb = zo * hairLength;
 
 
 
    beginShape(LINES);
    //this is our start vertex point that we can not see becasue the stroke is 0. 
    stroke(0,0,0);
   vertex(x,y,z);
 
 
 
    stroke(random(0,255),random(100,150), random(150,200));
    vertex(xb,yb,zb);
 
 
 
    //fin 
    endShape();
  }
 
}
 
 

(edit) damn this doesn't render must be the lib ?