From GPWiki

Introduction

There are a number of ways object selection can be performed. Using OpenGL you can use the special Selection Buffer, which allows you to select objects in the scene having unique ID's preassigned to them. A tutorial demonstrating this method can be found here: OpenGL:Tutorials:Picking

But there is another method. This method can not only be applied to OpenGL apps, but to DirectX apps as well. For the sake of this tutorial, the code given below will be using OpenGL.

How It Works

Essentially the way this method works, is that each object in our world is assigned a unique color. Since we're using a 24bit color scheme, this means we can have A LOT of objects with unique colors. If we wish to find out which object the user has clicked, we simply do the following:

• Render the scene with textures, lighting, fog turned off
• Render each object using its unique color
• Read back data from the color buffer at the current mouse position
• Search through the list of objects looking for a color ID match
• If we find one, we have a selection
  
  

This is extremely simple to implement and is not API dependent.

So the first thing we must do is declare a base class for every object in our world. This base class must take care of initializing every object's colorID. It does this by declaring a static variable in the class, gColorID[3], and setting it to the color black, (0, 0, 0). From there, each time an object is created, gColorID is incremented. The first component, or red color channel, is incremented first, and when it reaches a value of 255, it is reset back to 0 and the second component, or green channel, is incremented. The same goes for the third, or blue channel.

class BaseObject
 {
 private:
      unsigned char m_colorID[3];
      static unsigned char gColorID[3];
  
 public:
      BaseObject()
      {
           m_colorID[0] = gColorID[0];
           m_colorID[1] = gColorID[1];
           m_colorID[2] = gColorID[2];
  
           gColorID[0]++;
           if(gColorID[0] > 255)
           {
                gColorID[0] = 0;
                gColorID[1]++;
                if(gColorID[1] > 255)
                {
                     gColorID[1] = 0;
                     gColorID[2]++;
                }
           }
      }
  
      ~BaseObject() {};
 };
 
 unsigned char BaseObject::gColorID[3] = {0, 0, 0};

Now every class we declare in our game or app should be derived from this base class. So suppose we wish to declare a Scene Object class. This class should be derived from the Base Object class and, depending on our implementation, you can simply insert a function to render the object only using its colorID. Hence in the example code below the only function of the function Picking() is to render the scene object with a solid color, specifically the object's colorID color.

class SceneObject : public BaseObject
 {
 private:
      float m_position[3];
      string m_name;
  
 public:
      SceneObject();
      ~SceneObject();
  
      void Render();
      void Picking()
      {
           // set mesh position
           glPushMatrix();
           glTranslatef(m_position[0], m_position[1], m_position[2]);
  
           glColor3f(m_colorID[0]/255.0f, m_colorID[1]/255.0f, m_colorID[2]/255.0f);
  
           // render object's vertices here
  
           glPopMatrix();
      }
 };

Now as stated before we need to turn off texturing, lighting and fog before doing any of this. We also probably only want to perform object selection when a mouse button is pressed. So in the event of a mouse button being pressed, we render every object in the scene to the color buffer, read back the color information and search through our object list.

void MouseDownEvent(int x, int y)
 {
      // turn off texturing, lighting and fog
      glDisable(GL_TEXTURE_2D);
      glDisable(GL_FOG);
      glDisable(GL_LIGHTING);
  
      // render every object in our scene
      // suppose every object is stored in a list container called SceneObjects
      list<SceneObject *>::iterator itr = SceneObjects.begin();
      while(itr != SceneObjects.end())
      {
           (*itr)->Picking();
           itr++;
      }
  
      // get color information from frame buffer
      unsigned char pixel[3];
       
      GLint viewport[4];
      glGetIntegerv(GL_VIEWPORT, viewport);
       
      glReadPixels(x, viewport[3] - y, 1, 1, GL_RGB, GL_UNSIGNED_BYTE, pixel);
 
      // now our picked screen pixel color is stored in pixel[3]
      // so we search through our object list looking for the object that was selected
      itr = SceneObjects.begin();
      while(itr != SceneObjects.end())
      {
           if((*itr)->m_colorID[0] == pixel[0] && (*itr)->m_colorID[1] == pixel[1] && (*itr)->m_colorID[2] == pixel[2])
           {
                // flag object as selected
                SetSelected((*itr);
                break;
           }
           itr++;
      }
 }

And that's it! Its extremely fast, depending on how many objects you have in the scene. It is API independent and is fairly straightward to implement. If for some crazy reason you feel that only using 24bits is not enough for how many objects you have in your world, then simply add an alpha channel to each object's colorID. Then when calling glReadPixels, you would change GL_RGB to GL_RGBA.

unsigned char pixels[4];
 glReadPixels(x, viewport[3] - y, 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, pixel);

Also, some of you may be wondering why for the y - coordinate of glReadPixels, we passed in

viewport[3] - y

This is because of the way OpenGL sets up the viewport, specially the lower left is the origin. Since in Windows the upper left is the origin, we must subtract the y mouse coordinate from the height of the viewport to get the correct OpenGL y coordinate.

To see this technique in action you can run the terrain editing software, Freeworld3D. This software uses this technique, not only for object selection, but also for Axis selection when translating, rotating and scaling objects. This is a very powerful and flexible way of handling object selection.