Touch Gesture Components

Touch gesture support in Windows Embedded CE 6.0 R3 consists of three new OS components shown below:

The first new component is the “Single-Touch Gesture Recognition” component.  For OEMs, nothing new is required here as far as the touch panel hardware or the touch driver is concerned.  The gesture engine sits logically between the touch driver and the application.  It obtains touch events and the stream of touch points from the touch driver, and passes an array of touch points to every registered gesture recognizer.  If a recognizer determines that the touch points in the array represent a gesture, a gesture event message is placed into the touch events message queue. 

The second new component, the physics engine API, called “Gesture Animation Support” in the component catalog, allows an application to create a physics engine object that generates a set of animation points as a function of time.   The application supplies an initial position and velocity vector for the animation, along with a bounding rectangle and boundary rules, and the physics engine will give the position and velocity for the animation for a given timestamp.

The third new component, Gesture Support for Win32 Controls (also known as window auto gesture support), provides an API that allows you to easily add pan and flick support to any Win32 window with the WS_HSCROLL or WS_VSCROLL styles, by simply adding code that calls the SetWindowAutoGesture function to enable or disable automatic gesture support for your window.  The auto gesture support component uses the physics engine to calculate animation points for the gesture, so if you add the Gesture Support for Win32 component to your OS configuration, the Gesture Animation Support component will be added automatically.

Predefined Gestures

Currently Windows Embedded CE 6.0 R3 supports only single-touch gestures, in contrast to multi-touch support recently introduced in Windows 7 and Mac laptops.  This limitation prevents the use of multi-touch gestures such as zoom, rotate and two-finger taps.

The touch gesture engine component recognizes five distinct standard gestures:

The Single-Touch Gesture API

Applications that want to utilize the touch gesture API directly must first enable the gestures they want to support.  This is done using the EnableGestures function.  The function takes three parameters: a window handle, a combination of flags indicating which gestures to enable, and a scope parameter that determines whether gestures will be enabled for just the window specified (TGF_SCOPE_WINDOW) or for the application process (TGF_SCOPE_PROCESS).  If TGF_SCOPE_PROCESS is specified for the third parameter, the window handle parameter is ignored.

In addition to enabling the gestures it wants to support, an application must handle the WM_GESTURE message, which is sent to the application’s window procedure whenever a gesture is recognized.  The wParam of the WM_GESTURE message contains the gesture identifier of the gesture command.  The following predefined gesture commands are defined by Windows Embedded CE 6.0 R3:

When an application receives a WM_GESTURE message, it can use the gesture identifier to decide if it wants to process the command further.  If so, it can proceed by calling the GetGestureInfo function which populates a GESTUREINFO structure with detailed information about the gesture.  When calling GetGestureInfo, the application will need to provide a handle for the GESTUREINFO structure.  This handle is provided by the lParam of the WM_GESTURE message.

The GESTUREINFO handle passed to WM_GESTURE must be closed in order to release the memory resource used by the GESTUREINFO structure.  To prevent memory leaks, an application that does not pass a WM_GESTURE message on to the default window procedure must call CloseGestureInfoHandle as part of its WM_GESTURE processing.

GestureSample1 is a sample application that shows when various gesture messages are received and the information contained with each gesture.  The application creates a multiline edit control for easy text output and enables all gestures by calling EnableGestures using the TGF_GID_ALL constant.  For each WM_GESTURE message received, it calls DisplayGestureInfo to display information about the gesture.  The contents of the edit control are reset whenever the GID_BEGIN gesture command is received so that the commands for only one gesture are displayed at a time.

  Excerpt from GestureSample1.cpp

//----------------------------------------------------------------
//  WndProc(HWND, unsigned, WORD, LONG)
//
//  Processes messages for the main window.
//----------------------------------------------------------------
LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
    switch (message) 
    {
    case WM_CREATE:
      g_hEditWnd = CreateWindowW(L"EDIT", L"", WS_VISIBLE|ES_LEFT|ES_MULTILINE|ES_READONLY|ES_AUTOVSCROLL, 
                    0, 0, 0, 0, hWnd, NULL, g_hInst, 0);
      if (g_hEditWnd == NULL)
        return -1;
      EnableGestures(hWnd, TGF_GID_ALL,TGF_SCOPE_PROCESS);
      break;

    case WM_SIZE:
      if (g_hEditWnd != NULL)
      {
        RECT cr;
        GetClientRect(hWnd, &cr);
        MoveWindow(g_hEditWnd, 0,0,cr.right,cr.bottom,TRUE);
      }
      break;

    case WM_GESTURE:
      if (wParam == GID_BEGIN)
        Edit_SetText(g_hEditWnd, L"");
      if (wParam <= GID_LAST)
      {
        SetCapture(hWnd);
        DisplayGestureInfo((HGESTUREINFO)lParam);
        ReleaseCapture();
      }
      CloseGestureInfoHandle((HGESTUREINFO)lParam);
      return 1;

    case WM_DESTROY:
            PostQuitMessage(0);
            break;
        default:
            return DefWindowProc(hWnd, message, wParam, lParam);
   }
   return 0;
}

//----------------------------------------------------------------
//  DisplayGestureInfo(HGESTUREINFO hGestureInfo)
//
//  Displays the relavent fields of the GESTUREINFO structure
//    from the handle provided.
//----------------------------------------------------------------
void DisplayGestureInfo(HGESTUREINFO hGestureInfo)
{
  GESTUREINFO gi;
  wchar_t buf[180];

  gi.cbSize = sizeof(GESTUREINFO);
  GetGestureInfo(hGestureInfo, &gi);
  StringCchPrintf(buf, sizeof(buf)/sizeof(wchar_t), 
    L"%s - dwFlags:0x%8.8X, ptsLocation:(%d,%d), dwSequenceID:0x%08X\r\n",
    g_GIDName[gi.dwID],
    gi.dwFlags,
    gi.ptsLocation.x, gi.ptsLocation.y,
    gi.dwSequenceID );
  Edit_AppendText(g_hEditWnd, buf);
  if (gi.dwID == GID_SCROLL)
  {
    StringCchPrintf(buf, sizeof(buf)/sizeof(wchar_t), 
      L"\tuulArguments => Direction:%s, Velocity:%d pixels/sec., Angle:%3.4g degrees\r\n",
      (GID_SCROLL_DIRECTION(gi.ullArguments) <= ARG_SCROLL_RIGHT ? g_ArgScrollName[GID_SCROLL_DIRECTION(gi.ullArguments)] : L"Undefined"),
      GID_SCROLL_VELOCITY(gi.ullArguments),
      (180 * GID_ROTATE_ANGLE_FROM_ARGUMENT(GID_SCROLL_ANGLE(gi.ullArguments))/3.1415926535) );
    Edit_AppendText(g_hEditWnd, buf);
  }
}

Let’s have a look at the output of this example for each type of gesture:

Figure 1 - Tap

Figure 2 - Double Tap

Figure 3 - Hold

Figure 4 - Pan

Figure 5 - Flick

You’ll see that every gesture starts with a GID_BEGIN command that has a valid ptsLocation value indicating where the gesture began and a dwSequenceID of zero.  In addition, every gesture ends with a GID_END command.  The GID_END command does not specify a ptsLocation value – it’s always (0,0) - but it does specify a valid timestamp in dwSequenceID.

You can also see that some gestures will start out looking like other gestures.  A flick gesture typically starts out looking like a pan gesture, so a GID_SCROLL command can be preceded by one or more GID_PAN commands.  Likewise, a pan gesture may start out as a hold gesture before panning begins.  It’s useful to keep these points in mind as you develop the code to process gestures in your application.

At this point you should have at least a basic understanding of how to enable gestures in your application and receive WM_GESTURE messages.  We’ve talked about EnableGestures, GetGestureInfo, and CloseGestureInfoHandle.  We’ll take a quick look at the other functions in the Touch Gesture API before moving on to discuss the “Gesture Animation Support” component.

DisableGestures – Disables the specified touch gestures for a given window or application process.

Gesture –Posts an asynchronous gesture event to a message queue.

QueryGestures – Provides a bit masked value indicating which gestures are enabled for a window or process.

RegisterGesture – Used to register a custom gesture with the gesture recognizer engine or to query the ID of a previously registered gesture.

GetGestureExtraArguments – For gestures that provide more information than can fit into the GESTUREINFO structure, this function retrieves the extra information.  The number of bytes of extra information is given by the cbExtraArguments member of the GESTUREINFO structure.  Note that none of the current predefined gestures provide any extra gesture information.

Getting and Setting Gesture Metrics

Your code can retrieve and set the constraints used by the gesture recognizer to detect touch gestures.  These constraints provide the tolerances on time and movement for the various gestures, such as the minimum period of contact for a hold gesture, or the maximum time period between two consecutive taps recognized as a double select gesture.

Two new uiAction constants have been added to the SystemParametersInfo function for this purpose – SPI_GETGESTUREMETRICS and SPI_SETGESTUREMETRICS.  The pvParam parameter must be set to point to a GESTUREMETRICS structure that provides or receives the recognizer constraints values.

The Gesture Animation Physics Engine

The Gesture Animation Support API provides consistent gesture animation physics across all applications on a system instead of forcing every application developer implement his or her own unique gesture physics.

The Gesture Animation physics engine is pretty simple, and it’s limited to the generation of 1- or 2-dimensional scrolling animations.  An application first calls the CreatePhysicsEngine function to create a physics engine object, providing a pointer to a PHYSICSENGINEINIT structure that contains an initial position, speed, and direction for the animation, a bounding rectangle, and the desired animation behavior when a boundary is reached.

The dwFlags member of the PHYSICSENGINEINIT allows the application to control the time source for the physics engine object.  If the value PHYSICSENGINE_FLAG_USERTIME is set, the physics engine will use a user provided time, set by calling the SetPhysicsEngineUserTime function.  Otherwise, the default behavior is for the physics engine to use the system clock as the time source.

The sample application PESample1 demonstrates how to use the physics engine object to animate a PacMan ghost bitmap around the screen.  Menus allow you to set the initial animation parameters and boundary rules, and to start the animation.  The position and velocity of the animation for each frame is shown in the autoscrolling text box above the animation area.

Figure 6 - PESample1 Screen

 

Gesture Support for Win32 Controls (or WindowAutoGesture)

Microsoft has provided a simpler alternative to processing WM_GESTURE messages directly if you have existing applications that already implement scrolling using the standard WS_HSCROLL and WS_VSCROLL window styles.  Using the “Gesture Support for Win32 Controls” component, you can quickly add automatic pan and flick scrolling gesture support to these applications.

With the automatic gesture support component in your OS image, your applications can call the SetWindowAutoGesture function to enable and disable automatic scrolling gestures.  This function takes two parameters: the window handle for the window to enable or disable scrolling gestures for, and a pointer to a WAGINFO structure that contains the auto gesture settings to set for the window.  The dwFlags member of the WAGINFO structure is a bit mask for controlling various aspects of automatic scrolling gestures.  The flags that can be set in dwFlags are:

 The nOwnerAnimateMessage member of the WAGINFO structure lets an application define a message ID that the window auto gesture system sends to the application whenever the application needs to redraw its window content as the result of a touch interaction.  However, by setting nOwnerAnimateMessage to zero, the application can tell the auto gesture system to send WM_HSCROLL and/or WM_VSCROLL messages when touch interactions occur.

If an application wants to receive animation messages instead of scroll messages, it can do so by specifying a valid message ID number for nOwnerAnimateMessage.  Valid IDs must be greater than WM_USER.  Applications that want to handle their own window gesture animation will use the GetAnimateMessageInfo function to populate an ANIMATEMESSAGEINFO structure with details about the animation message.

The sample application GestureSample2 demonstrates the use of the “Gesture Support for Win32 Controls” API using WM_HSCROLL and WM_VSCROLL messages.  In this example, a large bitmap with a street map of downtown Cleveland, OH is displayed in a scrollable 320x320 pixel viewport using both the WS_HSCROLL and WS_VSCROLL window styles.  By inserting a call to SetWindowAutoGesture in the WM_CREATE message handler for the viewport window, I’ve added the ability to use flick and pan gestures to the applications UI.

 Excerpt from GestureSample2.cpp

//-------------------------------------------------------
//  FUNCTION: WndProc(HWND, unsigned, WORD, LONG)
//
//  PURPOSE:  Processes messages for the main window.
//-------------------------------------------------------
LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
  PAINTSTRUCT ps;
  POINT p;
  RECT r;
  SCROLLINFO si;
  BITMAP bmp;
  HDC hdc, hdcComp;
  BOOL b;
  DWORD dwErr;
  int nPos;
  static HBITMAP hbMap;

    switch (message) 
    {
    case WM_CREATE:
      WAGINFO wi;
      wi.cbSize = sizeof(WAGINFO);
      wi.dwFlags = WAGIF_VSCROLLABLE | WAGIF_HSCROLLABLE | WAGIF_OWNERANIMATE;
      wi.nItemHeight = 0;
      wi.nItemWidth = 0;
      wi.nOwnerAnimateMessage = 0;
      wi.nAnimateStatusMessage = 0;
      wi.bHorizontalExtent = wi.bVerticalExtent = 0;
      hbMap = SHLoadDIBitmap(L"\\windows\\cleveland.bmp");
      if (hbMap == NULL)
        return -1;

      if (!SetWindowAutoGesture(hWnd, &wi))
        return -1;
      break;

    case WM_SIZE:
      GetClientRect(hWnd, &r);
      GetObject(hbMap, sizeof(BITMAP), &bmp);
      si.cbSize = sizeof(SCROLLINFO);
      si.nMin = 0;
      si.nMax = max(bmp.bmWidth - r.right, 0);
      si.nPage = si.nMax / 8;
      si.nPos = si.nMax / 2;
      si.fMask = SIF_RANGE | SIF_PAGE | SIF_POS;
      SetScrollInfo(hWnd, SB_HORZ, &si, TRUE);

      si.cbSize = sizeof(SCROLLINFO);
      si.nMin = 0;
      si.nMax = max(bmp.bmHeight - r.bottom, 0);
      si.nPage = si.nMax / 8;
      si.nPos = si.nMax / 2;
      si.fMask = SIF_RANGE | SIF_PAGE | SIF_POS;
      SetScrollInfo(hWnd, SB_VERT, &si, TRUE);
      break;

    case WM_VSCROLL:
    case WM_HSCROLL:
      si.cbSize = sizeof(SCROLLINFO);
      si.fMask = SIF_ALL;
      GetScrollInfo(hWnd, (message == WM_VSCROLL) ? SB_VERT : SB_HORZ, &si);
      nPos = si.nPos;
      switch (LOWORD(wParam))
      {
      case SB_TOP:
        si.nPos = si.nMin;
        break;

      case SB_BOTTOM:        // or SB_RIGHT
        si.nPos = si.nMax;
        break;

      case SB_LINEUP:        // or SB_LINELEFT
        -- si.nPos;
        break;

      case SB_LINEDOWN:      // or SB_LINERIGHT
        ++ si.nPos;
        break;

      case SB_PAGEUP:        // or SB_PAGELEFT
        si.nPos -= si.nPage;
        break;

      case SB_PAGEDOWN:      // or SB_PAGERIGHT
        si.nPos += si.nPage;
        break;

      case SB_THUMBTRACK:
        si.nPos = si.nTrackPos;
        break;

      default:
        break;
      }
      si.fMask = SIF_POS;
      SetScrollInfo(hWnd, (message == WM_VSCROLL) ? SB_VERT : SB_HORZ, &si, TRUE);
      GetScrollInfo(hWnd, (message == WM_VSCROLL) ? SB_VERT : SB_HORZ, &si);

      if (si.nPos != nPos)
      {
        GetClientRect(hWnd, &r);
        if (message == WM_VSCROLL)
        {
          ScrollWindowEx(hWnd, 0, (nPos - si.nPos),NULL, &r,NULL, NULL, SW_INVALIDATE);
        }
        else
        {
          ScrollWindowEx(hWnd, (nPos - si.nPos), 0,NULL, &r,NULL, NULL, SW_INVALIDATE);
        }
      }
      break;

        case WM_PAINT:
      if (GetUpdateRect(hWnd, &r, FALSE))
      {
        hdc = BeginPaint(hWnd, &ps);
        hdcComp = CreateCompatibleDC(hdc);
        SelectObject(hdcComp, hbMap);

        // Get offsets from scroll bars
        si.cbSize = sizeof(SCROLLINFO);
        si.fMask = SIF_POS;
        GetScrollInfo(hWnd, SB_HORZ, &si);
        p.x = si.nPos;
        si.cbSize = sizeof(SCROLLINFO);
        si.fMask = SIF_POS;
        GetScrollInfo(hWnd, SB_VERT, &si);
        p.y = si.nPos;

        // Draw at new position
        BitBlt(hdc, r.left, r.top, r.right-r.left, r.bottom-r.top, hdcComp, r.left+p.x, r.top+p.y, SRCCOPY);
        DeleteDC(hdcComp);
        EndPaint(hWnd, &ps);
      }
            break;

        case WM_DESTROY:
            PostQuitMessage(0);
            break;
        default:
            return DefWindowProc(hWnd, message, wParam, lParam);
   }
   return 0;
}

Figure 7 - GestureSample2 Screen

Conclusion

The Touch Gesture support Microsoft released with Windows Embedded CE 6.0 R3 will allow OEMs and application developers to implement richer touch-based user interfaces for their products quicker and with more uniform results.  There are limitations in this first release of CE gesture support, such as the absence of multi-touch, and a lack of documentation on how to create your own gesture recognizers (a capability that is alluded to several times in the existing documentation).  Still, the addition of built-in gesture support and animation physics will likely improve the types of user interfaces we’ll see on Windows Embedded CE 6.0 R3 based devices in the very near future.