OTAPI

Beneath the hood of RAD Studio

Multi-Touch Playground - Moving Controls

13th October 2014
by Simon J Stuart
1 Comment

Working with Multi-Touch Input in Delphi XE7 (Part 2)

Previously…

In the first part of this series of articles, we took a look at how the OnTouch event introduced in XE7 works, and discovered some potential stumbling blocks when it comes to taking advantage of this new feature.

In this part of the series, we’re going to look at potential “workarounds” for these problems; alternative (albeit slightly more complicated) methods of harnessing multi-touch input information for practical purposes in Delphi applications.

Another “quirk” in the way XE7’s Multi-Touch Input Handler Works

Since posting the first article in this series, I have conducted many more tests, and discovered that it continues to track Touch Move and Touch Up events even after one or more Touch Points leave the constraints of your Form.

Now, this shouldn’t necessarily be considered a “bug” or even a problem, as it means we can (in theory anyway) use multi-touch input to perform “drag and drop” operations on Windows (possibly also Mac) beyond the constraints of your Form(s).

It is, of course, fairly trivial to determine whether or not a Touch Point is inside the constraints of your Form.

I just wanted to share this observation for the benefit of others, as it may just effect the way you work with multi-touch input in your applications.

Detecting when a Multi-Touch Input Event Begins and Ends

We previously discovered that the Touch Down action seems to occur once for each additional Touch Point (finger) introduced to the touch screen (effectively making that action “re-entrant”), yet the Touch Up action doesn’t necessarily occur separately for each Touch Point removed from the touch screen.

So, since the OnTouch event behaves in a somewhat-inconsistent way, we need an inventive solution to decide when a multi-touch input event begins and ends.

Since we know that a multi-touch input event will begin with a Touch Down action, we can define a Boolean member to act as a “flag”. Also, we can say with relative certainty that a multi-touch input event will end either with a Touch Up action, or a Touch Cancel action, and I’ve determined that, where it ends with a Touch Up action, that action will contain either one or zero items in the Touch Point array (Touches parameter).

This means we have a logical principle on which to base our multi-touch input event detection.

unit Unit2;

interface

uses
  System.SysUtils, System.Types, System.UITypes, System.Classes, System.Variants,
  FMX.Types, FMX.Controls, FMX.Forms, FMX.Graphics, FMX.Dialogs, FMX.StdCtrls;

type
  TForm2 = class(TForm)
    Label1: TLabel;
    procedure FormCreate(Sender: TObject);
    procedure FormTouch(Sender: TObject; const Touches: TTouches; const Action: TTouchAction);
  private
    FTouching: Boolean;
    FTouchHistory: Array of TTouches;
    procedure TouchNone(Sender: TObject; const Touches: TTouches);
    procedure TouchUp(Sender: TObject; const Touches: TTouches);
    procedure TouchDown(Sender: TObject; const Touches: TTouches);
    procedure TouchMove(Sender: TObject; const Touches: TTouches);
    procedure TouchCancel(Sender: TObject; const Touches: TTouches);
  public
    { Public declarations }
  end;

var
  Form2: TForm2;

implementation

{$R *.fmx}

procedure TForm2.FormCreate(Sender: TObject);
begin
  FTouching := False;
end;

procedure TForm2.FormTouch(Sender: TObject; const Touches: TTouches; const Action: TTouchAction);
begin
  case Action of
    TTouchAction.None: TouchNone(Sender, Touches);
    TTouchAction.Up: TouchUp(Sender, Touches);
    TTouchAction.Down: TouchDown(Sender, Touches);
    TTouchAction.Move: TouchMove(Sender, Touches);
    TTouchAction.Cancel: TouchCancel(Sender, Touches);
  end;
end;

procedure TForm2.TouchCancel(Sender: TObject; const Touches: TTouches);
begin
  FTouching := False;
  Label1.Text := 'Touch Cancelled!';
end;

procedure TForm2.TouchDown(Sender: TObject; const Touches: TTouches);
begin
  if (not FTouching) then
  begin
    FTouching := True;
    Label1.Text := Format('Touch Points: %d', [Length(Touches)]);
  end;
end;

procedure TForm2.TouchMove(Sender: TObject; const Touches: TTouches);
begin
  Label1.Text := Format('Touch Points: %d', [Length(Touches)]);
end;

procedure TForm2.TouchNone(Sender: TObject; const Touches: TTouches);
begin
  // I genuinely don't know what this is for!
end;

procedure TForm2.TouchUp(Sender: TObject; const Touches: TTouches);
begin
  if (FTouching) then
  begin
    if Length(Touches) <= 1 then
    begin
      FTouching := False;
      Label1.Text := 'Not Touching';
    end else
      Label1.Text := Format('Touch Points: %d', [Length(Touches)]);
  end;
end;

end.

This code snip (above) works on Win32, Win64, Mac, Android and iOS. It illustrates a way of determining when an individual multi-touch input event begins and ends (in this case, by displaying the current “Touch State” on a TLabel control on the Form).

Pay special attention to the TouchUp procedure, which uses a simple if statement to determine whether or not we should consider the multi-touch input event as completed.

Put simply, while we won’t necessarily always get a Touch Up action when each separate Touch Point is lifted from the screen, we always get at least one (assuming that we don’t get a Touch Cancel event instead, I mean), and this final Touch Up action will only ever contain either one or zero Touch Points in the Touches array.

This gives us a very simple workaround for that inconsistent behaviour, one which (according to my extensive testing, at least) appears to work 100% of the time.

The FTouching boolean member plays a fairly important role, in that it prevents TouchDown from acting in a re-entrant manner. It would be problematic (to say the least) if we allowed the TouchDown procedure to perform its function multiple times during an existing Touch Event… particularly when you consider a “drag and drop” operation as an example.

I also just want to point out that I have been completely unable to get TouchNone to occur, on any platform, under any set of circumstances. At this point, I have genuinely no idea what that particular Touch Action Type is for, or under what set of bizarre circumstances it will occur.

You can download the working demo (tidied up a lot) of this code snip here.

Something Fun: Moving Controls At Runtime (using multi-touch input)

Okay, now that we’ve solved the problem of adequately detecting when a multi-touch event starts and ends, let’s put this solution to use and make something entertaining.

What we’re going to do now is make a little “multi-touch playground” project, allowing us to drag multiple controls around the screen at the same time with individual fingers on a touch screen. At face value, this little “toy project” might not seem all that useful, however this same principal can be applied to a simultaneous drag-and-drop solution that could be useful.

 

Multi-Touch Playground - Moving Controls

Multi-Touch Playground – Moving Controls

Basically, with this demo we can simply drop a bunch of controls onto the Form, and use multi-touch input to move one or more of those controls around.

Here’s the code for this demo:

unit uMainForm;

interface

uses
  System.SysUtils, System.Types, System.UITypes, System.Classes, System.Variants,
  FMX.Types, FMX.Controls, FMX.Forms, FMX.Graphics, FMX.Dialogs, FMX.Objects, FMX.StdCtrls;

type
  TControlTouch = record
    Control: TControl;
    Origin: TPointF;
  end;

  TfrmMain = class(TForm)
    Rectangle1: TRectangle;
    RoundRect1: TRoundRect;
    Ellipse1: TEllipse;
    Pie1: TPie;
    Label1: TLabel;
    Label2: TLabel;
    procedure FormTouch(Sender: TObject; const Touches: TTouches; const Action: TTouchAction);
  private
    FTouching: Boolean;
    FControls: Array of TControlTouch;
    function GetControlAtPoint(const APoint: TPointF): TControl;
    procedure GetControlsAtEachTouchPoint(const Touches: TTouches);
    procedure TouchEnd;
    procedure TouchNone(Sender: TObject; const Touches: TTouches);
    procedure TouchUp(Sender: TObject; const Touches: TTouches);
    procedure TouchDown(Sender: TObject; const Touches: TTouches);
    procedure TouchMove(Sender: TObject; const Touches: TTouches);
    procedure TouchCancel(Sender: TObject; const Touches: TTouches);
  public
    { Public declarations }
  end;

var
  frmMain: TfrmMain;

implementation

{$R *.fmx}

procedure TfrmMain.FormTouch(Sender: TObject; const Touches: TTouches; const Action: TTouchAction);
begin
  case Action of
    TTouchAction.None: TouchNone(Sender, Touches);
    TTouchAction.Up: TouchUp(Sender, Touches);
    TTouchAction.Down: TouchDown(Sender, Touches);
    TTouchAction.Move: TouchMove(Sender, Touches);
    TTouchAction.Cancel: TouchCancel(Sender, Touches);
  end;
end;

function TfrmMain.GetControlAtPoint(const APoint: TPointF): TControl;
var
  I: Integer;
  LObject: TControl;
  LRect: TRectF;
begin
  Result := nil;
  for I := ChildrenCount - 1 downto 0 do
  begin
    if (Children[I] is TControl) then
      if TControl(Children[I]).HitTest then
      begin
        LObject := TControl(Children[I]);
        LRect := RectF(LObject.Position.X,
                       LObject.Position.Y,
                       LObject.Position.X + LObject.Width,
                       LObject.Position.Y + LObject.Height);
        if LRect.Contains(APoint) then
        begin
          Result := LObject;
          Break;
        end;
      end;
  end;
end;

procedure TfrmMain.GetControlsAtEachTouchPoint(const Touches: TTouches);
var
  I: Integer;
begin
  // Hold a reference to whatever controls are under each respective touch point
  SetLength(FControls, Length(Touches));
  for I := Low(Touches) to High(Touches) do
  begin
    FControls[I].Control := GetControlAtPoint(Touches[I].Location);
    FControls[I].Origin := Touches[I].Location;
  end;
end;

procedure TfrmMain.TouchCancel(Sender: TObject; const Touches: TTouches);
begin
  TouchEnd;
end;

procedure TfrmMain.TouchDown(Sender: TObject; const Touches: TTouches);
begin
  if (not FTouching) then
  begin
    FTouching := True;
    // Release any existing history
    SetLength(FControls, 0);
  end;
  GetControlsAtEachTouchPoint(Touches);
end;

procedure TfrmMain.TouchEnd;
begin
  FTouching := False;
end;

procedure TfrmMain.TouchMove(Sender: TObject; const Touches: TTouches);
var
  I: Integer;
  LDifference: TPointF;
begin
  if Length(Touches) = Length(FControls) then
  begin
    // Move the controls
    for I := Low(Touches) to High(Touches) do
      if FControls[I].Control <> nil then
      begin
        LDifference := PointF(Touches[I].Location.X - FControls[I].Origin.X,
                              Touches[I].Location.Y - FControls[I].Origin.Y);
        FControls[I].Control.Position.X := FControls[I].Control.Position.X + LDifference.X;
        FControls[I].Control.Position.Y := FControls[I].Control.Position.Y + LDifference.Y;
        FControls[I].Origin := Touches[I].Location;
      end;
  end else
    GetControlsAtEachTouchPoint(Touches);
end;

procedure TfrmMain.TouchNone(Sender: TObject; const Touches: TTouches);
begin

end;

procedure TfrmMain.TouchUp(Sender: TObject; const Touches: TTouches);
begin
  if (FTouching) and (Length(Touches) <= 1) then
    TouchEnd
  else
    GetControlsAtEachTouchPoint(Touches);
end;

end.

This code snip (above) obtains a handle on whatever control is underneath each Touch Point, and basically moves them by their respective offsets on each Touch Move event.

Please note that this demo source observes the value of each control’s HitTest property, so if you set HitTest to False, you won’t be able to drag it around (such as the two TLabel controls included in the downloadable sample)

When the number of touch points changes, it re-establishes the Array of relevant controls for each Touch Point.

As with all the other demos in this series of articles, this will work on Windows, Mac, iOS and Android.

You can download the working demo of this code snip here.

Coming up next…

In the next part of this article, we’ll take a look at solving the “Order of Contact” problem (discussed in the first part of this series) and take a look at some other fun (and useful) uses for multi-touch input.

FireMonkey Form's OnTouch Event

7th October 2014
by Simon J Stuart
2 Comments

Working with Multi-Touch Input in Delphi XE7 (Part 1)

Introduction

Before the release of XE7, handling multi-touch inputs in Delphi (and C++ Builder) applications was quite a complicated affair. While Delphi 2010 introduced “Gesture” support, this solution was less than ideal for real-time multi-touch input handling, as an action bound to a Gesture only executed after the Gesture had been completed, with no simple method of directly intercepting individual touch points to integrate custom behaviour.

Fortunately, XE7 solves this problem by giving us direct access to real-time multi-touch input data. With this data, we can implement custom touch handling for individual points, including our own Gesture detection solutions.

Best of all, it would appear that this new multi-touch feature works identically across all platforms presently supported by Delphi (Win32, Win64, Mac, Android and iOS).

The “OnTouch” Property

FireMonkey Form's OnTouch Event

FireMonkey Form’s OnTouch Event

By hooking the OnTouch Event of a FireMonkey Form, we can begin to implement our own multi-touch input handler.

procedure TForm2.FormTouch(Sender: TObject; const Touches: TTouches; const Action: TTouchAction);
begin

end;

As you can see from this code snip (above), the OnTouch event provides us with several key parameters:

  • Sender is of course a reference to the Control from whence the OnTouch event was executed.
  • Touches provides us with an Array of Point Information representing each individual touch point.
  • Action tells us whether our touch points have ended (Up), begun (Down), are in motion (Move), have been “cancelled” (Cancel), or something unknown and crazy (None).

You may notice that the multi-touch input handling is handled very differently from keyboard and mouse inputs, where we’re provided with OnKeyDown, OnKeyUp, OnMouseDown, OnMouseMove and OnMouseUp events respectively. I am unsure as to why Embarcadero chose to provide a single “catch-all” event property for all Touch input actions, but we can quite easily separate the singular OnTouch event into its discrete actions of influence.

type
  TForm2 = class(TForm)
    procedure FormTouch(Sender: TObject; const Touches: TTouches; const Action: TTouchAction);
  private
    procedure TouchNone(Sender: TObject; const Touches: TTouches);
    procedure TouchUp(Sender: TObject; const Touches: TTouches);
    procedure TouchDown(Sender: TObject; const Touches: TTouches);
    procedure TouchMove(Sender: TObject; const Touches: TTouches);
    procedure TouchCancel(Sender: TObject; const Touches: TTouches);
  public
    { Public declarations }
  end;

var
  Form2: TForm2;

implementation

{$R *.fmx}

procedure TForm2.FormTouch(Sender: TObject; const Touches: TTouches; const Action: TTouchAction);
begin
  case Action of
    TTouchAction.None: TouchNone(Sender, Touches);
    TTouchAction.Up: TouchUp(Sender, Touches);
    TTouchAction.Down: TouchDown(Sender, Touches);
    TTouchAction.Move: TouchMove(Sender, Touches);
    TTouchAction.Cancel: TouchCancel(Sender, Touches);
  end;
end;

procedure TForm2.TouchCancel(Sender: TObject; const Touches: TTouches);
begin

end;

procedure TForm2.TouchDown(Sender: TObject; const Touches: TTouches);
begin

end;

procedure TForm2.TouchMove(Sender: TObject; const Touches: TTouches);
begin

end;

procedure TForm2.TouchNone(Sender: TObject; const Touches: TTouches);
begin

end;

procedure TForm2.TouchUp(Sender: TObject; const Touches: TTouches);
begin

end;

So we now have (in the code snip above) a separate procedure defined for each multi-touch input action, and a simple “case-of” statement passes the appropriate parameter data along to the appropriate handling procedure. All very basic stuff, so let’s move on…

How Delphi handles Touch Actions

At this point, it is critical to understand when the OnTouch event is called, and what Touch Action is taking place.

Since you can introduce one or more Touch Points subsequent to the start of a one or more previous Touch Points, and likewise you can remove one or more Touch Points at any time while other Touch Points remain in play, it’s important to understand how the OnTouch event responds to changes in the number of Touch Points taking place from the moment a Touch Action begins, and that Touch Action ends.

A Touch Action begins when one Touch Point is placed on the screen. So, if you place one finger on the screen, you have begun a Touch Action. When that finger leaves the screen (either by going outside of the touch area, or by being lifted from the screen entirely) that Touch Action ends.

When it comes to Multi-Touch, however, the rules change.

Problems determining the Start and End of a Multi-Touch Event

For one thing, you cannot assume that a new Touch Event is beginning each time a Touch Down Action occurs, or that a Touch Event is ending each time a Touch Up or Touch Cancel Action occurs. This is because the introduction of a new Touch Point, or removal of an existing Touch Point can occur at any time.

To illustrate this point, I added a second form to the example code snip provided above, dropped a single TMemo control on it, and modified the main unit as follows:

type
  TForm2 = class(TForm)
    procedure FormShow(Sender: TObject);
    procedure FormTouch(Sender: TObject; const Touches: TTouches; const Action: TTouchAction);
  private
    procedure TouchNone(Sender: TObject; const Touches: TTouches);
    procedure TouchUp(Sender: TObject; const Touches: TTouches);
    procedure TouchDown(Sender: TObject; const Touches: TTouches);
    procedure TouchMove(Sender: TObject; const Touches: TTouches);
    procedure TouchCancel(Sender: TObject; const Touches: TTouches);
  public
    { Public declarations }
  end;

var
  Form2: TForm2;

implementation

uses
  Unit3;

{$R *.fmx}

procedure TForm2.FormShow(Sender: TObject);
begin
  Form3.Show;
end;

procedure TForm2.FormTouch(Sender: TObject; const Touches: TTouches; const Action: TTouchAction);
begin
  case Action of
    TTouchAction.None: TouchNone(Sender, Touches);
    TTouchAction.Up: TouchUp(Sender, Touches);
    TTouchAction.Down: TouchDown(Sender, Touches);
    TTouchAction.Move: TouchMove(Sender, Touches);
    TTouchAction.Cancel: TouchCancel(Sender, Touches);
  end;
end;

procedure TForm2.TouchCancel(Sender: TObject; const Touches: TTouches);
var
  I: Integer;
begin
  Form3.Memo1.Lines.Add('Touch Cancel:');
  for I := Low(Touches) to High(Touches) do
    Form3.Memo1.Lines.Add(Format(#9 + #9 + 'Point %d (%n, %n)', [I, Touches[I].Location.X, Touches[I].Location.Y]));
end;

procedure TForm2.TouchDown(Sender: TObject; const Touches: TTouches);
var
  I: Integer;
begin
  Form3.Memo1.Lines.Add('Touch Down:');
  for I := Low(Touches) to High(Touches) do
    Form3.Memo1.Lines.Add(Format(#9 + #9 + 'Point %d (%n, %n)', [I, Touches[I].Location.X, Touches[I].Location.Y]));
end;

procedure TForm2.TouchMove(Sender: TObject; const Touches: TTouches);
var
  I: Integer;
begin
  Form3.Memo1.Lines.Add('Touch Move:');
  for I := Low(Touches) to High(Touches) do
    Form3.Memo1.Lines.Add(Format(#9 + #9 + 'Point %d (%n, %n)', [I, Touches[I].Location.X, Touches[I].Location.Y]));
end;

procedure TForm2.TouchNone(Sender: TObject; const Touches: TTouches);
var
  I: Integer;
begin
  Form3.Memo1.Lines.Add('Touch None:');
  for I := Low(Touches) to High(Touches) do
    Form3.Memo1.Lines.Add(Format(#9 + #9 + 'Point %d (%n, %n)', [I, Touches[I].Location.X, Touches[I].Location.Y]));
end;

procedure TForm2.TouchUp(Sender: TObject; const Touches: TTouches);
var
  I: Integer;
begin
  Form3.Memo1.Lines.Add('Touch Up:');
  for I := Low(Touches) to High(Touches) do
    Form3.Memo1.Lines.Add(Format(#9 + #9 + 'Point %d (%n, %n)', [I, Touches[I].Location.X, Touches[I].Location.Y]));
end;

What this program does is output a dump of the Touch Actions and their respective Touch Point Data into that TMemo control on the other Form so that we can see exactly what Touch Actions are taking place each time OnTouch is called.

This is actually a much more reliable way of debugging multi-touch event data than using the debugger, as the debugger would cause fragmentation of the input data each time the IDE broke out at a breakpoint for introspection.

I ran this program and momentarily touched three fingers onto my multi-touch Windows tablet device for less than one second. From my perspective, all three fingers made contact with the screen at exactly the same moment, and broke contact with the screen again at exactly the same moment, however the data log tells a very different story:

Touch Down:
        Point 0 (196.00, 275.00)
Touch Down:
        Point 0 (196.00, 275.00)
        Point 1 (285.00, 142.00)
Touch Down:
        Point 0 (196.00, 275.00)
        Point 1 (285.00, 142.00)
        Point 2 (416.00, 123.00)
Touch Move:
        Point 0 (196.00, 275.00)
        Point 1 (285.00, 142.00)
        Point 2 (416.00, 125.00)
Touch Move:
        Point 0 (196.00, 275.00)
        Point 1 (285.00, 142.00)
        Point 2 (416.00, 125.00)
Touch Move:
        Point 0 (196.00, 275.00)
        Point 1 (285.00, 142.00)
        Point 2 (416.00, 125.00)
Touch Move:
        Point 0 (196.00, 275.00)
        Point 1 (285.00, 142.00)
        Point 2 (416.00, 125.00)
Touch Move:
        Point 0 (197.00, 276.00)
        Point 1 (285.00, 142.00)
        Point 2 (415.00, 127.00)
Touch Move:
        Point 0 (197.00, 276.00)
        Point 1 (285.00, 142.00)
        Point 2 (415.00, 127.00)
Touch Move:
        Point 0 (285.00, 142.00)
        Point 1 (418.00, 135.00)
Touch Up:
        Point 0 (285.00, 142.00)
        Point 1 (418.00, 135.00)
Touch Move:
        Point 0 (288.00, 141.00)
Touch Up:
        Point 0 (288.00, 141.00)

What we can see in the input dump above is that the Touch Down action was executed three separate times, once for each additional input, suggesting that – despite appearing from my perspective that all three fingers contacted the screen at the same exact moment – there was an imperceptible delay between each of my fingers making contact with the screen.

This means that, if we were to try and determine when a Touch Action begins by toggling its state using Touch Down, we would have problems as this Touch Action is essentially re-entrant (it can occur multiple times without a Touch Up or Touch Cancel action terminating that state).

We can also see from this input dump that the Touch Move action occurs repeatedly for the entire duration of contact with the screen, even if none of the Touch Points have actually changed their respective positions.

Finally, we can see that while three separate Touch Down events occurred at the beginning of our input, only two Touch Up events occurred at the end. Presumably this is because two of my fingers left the surface of the screen at exactly the same time, and the other (most likely my index finger) left the surface a fraction of a second later.

These details are very important, particularly if we want to track a complete multi-touch input event from start to finish, as we need to figure out the most reliable way to determine when a multi-touch input event starts, and when it finishes.

Tracking a multi-touch input event from start to finish is particularly useful if you want to draw multiple paths using the movement data for each respective Touch Point, or (more usefully) for enabling multiple simultaneous drag and drop operations in your applications.

Problems with the Order of Contact

Another even more challenging issue is keeping track of the order in which Touch Points are terminated from a touch event.

Here’s another input log where-in I placed three fingers on the screen in turn (index, middle, ring finger). I then removed my middle finger, followed by my index finger, followed by my ring finger.

Think of this as:

Down: 1, 2, 3
Up: 2, 1, 3

Here’s what that log looks like:

Touch Down:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Move:
        Point 0 (200.00, 312.00)
Touch Down:
        Point 0 (200.00, 312.00)
        Point 1 (272.00, 152.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (272.00, 152.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (272.00, 153.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (272.00, 153.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (272.00, 153.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (272.00, 153.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (272.00, 153.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (271.00, 154.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (271.00, 155.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (271.00, 156.00)
Touch Down:
        Point 0 (200.00, 312.00)
        Point 1 (271.00, 156.00)
        Point 2 (390.00, 82.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (271.00, 158.00)
        Point 2 (390.00, 82.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (271.00, 159.00)
        Point 2 (390.00, 82.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (270.00, 159.00)
        Point 2 (390.00, 82.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (270.00, 159.00)
        Point 2 (388.00, 82.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (270.00, 159.00)
        Point 2 (388.00, 82.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (270.00, 159.00)
        Point 2 (387.00, 83.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (270.00, 159.00)
        Point 2 (387.00, 83.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (269.00, 157.00)
        Point 2 (387.00, 81.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (269.00, 157.00)
        Point 2 (387.00, 81.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (201.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (200.00, 312.00)
        Point 1 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (387.00, 81.00)
Touch Move:
        Point 0 (388.00, 81.00)
Touch Move:
        Point 0 (388.00, 77.00)
Touch Up:
        Point 0 (388.00, 77.00)

Note that I have not omitted a single piece of information from this input dump, but I have highlighted the Touch Down and Touch Up actions.

This illustrates one shocking (and troublesome) issue: while you can clearly see that I did indeed place three fingers on the screen, only one Touch Up action executed at the very end of the log.

Now, had I removed all three fingers from the screen at exactly the same time, this would make some sense. However, you can clearly see from this input dump when each of my fingers left the screen (based on the reduction in the number of Points being displayed in the log), and that the Touch Up actions did not occur until the very last finger left the screen.

This is particularly confusing when you consider that, in the previous test case, there were two separate Touch Up actions, presumably suggesting that two fingers left the screen simultaneously, with the third following a fraction of a second later. We would have expected there to have been three separate Touch Up actions in this test, as I deliberately removed each finger from the screen in turn.

This means that we cannot rely at all on either the Touch Down or Touch Up actions to determine when a Touch Event begins or ends.

Indeed, we can see from these two tests that the execution of the Touch Down and Touch Up actions cannot be trusted to behave in a consistent way.

Another critical piece of information this test illustrates is that, when you remove a Touch Point, the Array position of all subsequent Touch Points shifts back by 1. This means that the Index of a Touch Point is not necessarily going to be the same at the end of a Touch Action as it was at the beginning.

Coming up next…

In the next part of this article, I’ll show you how to work around the limitations we’ve discussed here, and how to make productive use of multi-touch input for a variety of different purposes.

Improved Rate-Limited Game Loop

30th September 2014
by Simon J Stuart
5 Comments

Game Engine Architecture (Part 1) – Game Loop vs Parallelism

Introduction

When it comes to developing a game engine (whether it be reusable or the engine powering a single specific game) there are many alternative approaches that can be taken at almost every single stage of the design.

For this reason (and I say this from personal experience) the single most critical part of the development process is the designing of the overall architecture, before a single line of code is ever written. If you don’t decide upon an overall architecture before your development begins, you will almost-certainly find yourself scrapping the codebase and starting over each time you realize that there’s a better way your engine could have been designed.

So, in this series of articles, I’m going to share with you the various “architectural rethinks” that have occurred during the development of my newest game engine (called “AGE” which stands for “Another Game Engine”).

The single most significant of these decisions is covered in this first part of the series.

It is my hope that you can benefit from my otherwise-wasted time, and skip the series of poor decisions I made along the way; particularly those which necessitated a complete ground-up redevelopment.

I would like to point out that there will not be any usable code provided in this series of articles. This is because the principals discussed apply universally (to any object-oriented programming language). Some generic code examples may be used where appropriate, but they cannot be assembled into a functional game engine as they will merely be individual, unrelated snips of illustrative code (as opposed to segments of a working example). These code samples will be written in Pascal, as it is my favourite programming language, and is easily translated into other languages.

Flowcharts will be used extensively to illustrate the relationships between subsystems, and individual logical operations.

Please be sure to read the whole article before you begin your engine design. The reason will become apparent as you read further!

What is a “Game Loop”?

Put simply, the “Game Loop” advances the Physics, Logic (including AI) and Rendering. Engines utilizing the Game Loop approach place the Game Loop at the absolute heart of their architecture.

Game Loops, as with any approach to just about anything when it comes to game development, have advantages and disadvantages.

On the plus-side, a Game Loop can be laughably simple to implement. There are several different kinds of Game Loop, but they all operate on a principal of Differential Time (or Delta Time) which is literally the amount of time it took for the previous Loop to execute. Any Game Loop that doesn’t take Delta Time into consideration is doomed to complete and total failure, as there is no way to ensure that the Physics, Logic/AI, and Rendering progress at a consistent rate (resulting in very poor gameplay, and making multiplayer an absolute impossibility).

A simple Game Loop itself can be implemented in mere minutes (or less).

A Simple Game Loop

Simple Game Loop

Simple Game Loop

Note that the order of execution is not arbitrary. Since Physics dictates the Position and Angle of world objects, and the Logic and AI need the latest information to provide accurate behaviour, Physics needs to be updated first. Again, since the Logic and AI could very well have an influence over the rendered appearance of the game world (such as the current Sprite state and Particle Effects), we need update Logic and AI second. Rendering therefore must be performed last.

The Game Loop illustrated above operates at an unfixed rate (so, as fast as the hardware possibly can). Of course, the advantage is that this Game Loop is laughably easy to produce:

var
  LLastTime, LDeltaTime: Double;
begin
  LLastTime := GetReferenceTime;
  while GameRunning do
  begin
    LDeltaTime := GetReferenceTime - LLastTime;
    UpdatePhysics(LDeltaTime);
    UpdateLogic(LDeltaTime);
    RenderFrame(LDeltaTime);
    LLastTime := GetReferenceTime;
  end;
end;

GetReferenceTime returns the current time with an extremely high precision, adjusted for the frequency of the CPU. Different programming languages and standardized libraries will provide their own counterpart to do the same thing.

GameRunning is an external boolean value, function or flag to dictate whether the Loop should continue or break.

One disadvantage to this particular Game Loop is that, while it accounts for the time differential (Delta Time) between each cycle, each cycle will run at the slowest possible speed. This is because each cycle is being made to update all three Simulation components (Physics, Logic and Rendering).

Rendering needs to occur (at minimum) at the same rate as that at which the screen is refreshed (Refresh Rate or Vertical Sync Rate). Typically this is 60 times each second for the vast majority of computer monitors (120 times a second for 3D monitors). Ideally, you want to Render at double the refresh rate of the monitor. This will provide the smoothest possible appearance, and any framerate greater than this will simply not be visible to a player (meaning you’re wasting cycles, reducing performance unnecessarily, and unduly burdening the player’s hardware).

Physics and Logic, on the other hand, need only occur at a suitable base rate. For the majority of 2D games, and even some 3D games, 30 updates per second is usually very adequate. Modern 3D First-Person Shooter [FPS] games ideally want to update the Physics and Logic 60 times every second.

So, what we can do to improve this Game Loop is modify it to fix the rate of the Physics and Logic updates. This will free up more cycles for Rendering.

A Rate-Limited Game Loop

Okay, so the most common implementation of a Rate-Limited Game Loop enforces a limit on both the Physics and Logic, leaving Rendering unlimited. This way, more CPU time is given for Rendering, without necessarily impeding the performance of the Physics and Logic processing.

Rate-Limited Game Loop

Rate-Limited Game Loop

Here’s what this looks like in code:

const
  PHYSICS_RATE_LIMIT = 1 / 30.00;
var
  LLastTime, LDeltaTime: Double;
  LPhysicsUpdateTime: Double;
begin
  LLastTime := GetReferenceTime;
  LPhysicsUpdateTime := LLastTime; // We want the first Physics and Logic update to occur immediately
  while GameRunning do
  begin
    LDeltaTime := GetReferenceTime - LLastTime;
    if GetReferenceTime >= LPhysicsUpdateTime then
    begin
      UpdatePhysics(LDeltaTime);
      UpdateLogic(LDeltaTime);
      LPhysicsUpdateTime := GetReferenceTime + PHYSICS_RATE_LIMIT;
    end;
    RenderFrame(LDeltaTime);
    LLastTime := GetReferenceTime;
 end;
end;

Okay, this code snip (above) adds a simple Rate Limiter to both the Physics and Logic. Now each time the Loop cycles, it will check whether the current Reference Time is equal to or later than the indicated time for the next Physics and Logic update (LPhysicsUpdateTime).

Not so difficult, right? Okay, but this still leaves potential for significant problems! For one thing, the Rate Limit can only be an advantage if the time it takes to Render a frame does not exceed that limit.

An “Improved” Rate-Limited Game Loop

One way we can further improve the previous Game Loop example would be to introduce a Rate-Limit on Rendering.

Rate-Limiting the Rendering is a good idea because it ensures that our game isn’t forcing the player’s hardware to work harder than it needs to. Since we need only render at an absolute maximum rate of double the refresh rate of the player’s monitor, it makes absolutely no sense to allow the Game Loop to Render at any higher rate than that.

However, since different monitors have different refresh rates, we should not consider the rate limit for Rendering to be a fixed (constant) number, and should instead make it a setting (or “property”) that can be specified either automatically by having the game engine retrieve the refresh rate of the monitor on initialization, or by allowing the player to specify their own rate limit in the game’s “Advanced Graphics Options” menu.

Improved Rate-Limited Game Loop

Improved Rate-Limited Game Loop

This diagram (above) illustrates the logical progression of this Game Loop. Everything in yellow occurs on every Tick, everything in green only occurs if a logical operation evaluates as True, and everything in red occurs only if a logical operation evaluates as False.

Here’s what this game loop could look like in code:

const
  PHYSICS_RATE_LIMIT = 1 / 30.00;
var
  LLastTime, LDeltaTime: Double;
  LPhysicsUpdateTime, LRenderUpdateTime: Double;
begin
  LLastTime := GetReferenceTime;
  LPhysicsUpdateTime := LLastTime; // We want the first Physics and Logic update to occur immediately
  LRenderUpdateTime := LLastTime; // We want the first Rendering update to occur immediately too
  while GameRunning do
  begin
    LDeltaTime := GetReferenceTime - LLastTime;
    if GetReferenceTime >= LPhysicsUpdateTime then
    begin
      UpdatePhysics(LDeltaTime);
      UpdateLogic(LDeltaTime);
      LPhysicsUpdateTime := GetReferenceTime + PHYSICS_RATE_LIMIT;
    end;
    if ((FPSLimit > 0) and (GetReferenceTime >= LRenderUpdateTime)) or (FPSLimit = 0) then
    begin
      RenderFrame(LDeltaTime);
      LRenderUpdateTime := GetReferenceTime + FPSLimit;
    end;
   LLastTime := GetReferenceTime;
 end;
end;

FPSLimit represents the variable or property setting specifying how many frames per second the player desires. This implementation also accounts for the possibility of an unlimited rate for Rendering by checking whether the FPSLimit is set to zero.

There are countless other ways you can arguably-improve on the Rate Limited Game Loop, however…

The reason why Game Loops are a really bad idea, regardless of their complexity.

Regardless of how sophisticated your Game Loop becomes, you simply cannot escape one critically-limiting and fundamental flaw: By using a Game Loop, you’re limiting all three processes to a single CPU core.

This means that your engine completely ignores the other cores available on the device, and these days, even relatively inexpensive mobile devices have at least two CPU cores, most Desktop systems now have 4, and there are even 6 and 8 core CPUs already on the market… my main development system has 16 cores [2x 8 core CPUs].

This is why I have since chosen to reject the Game Loop entirely, which pretty-much renders this entire section pointless. Well, it would be pointless were it not for the lessons you’ve hopefully learned from reading it.

A Parallel Game Engine

So, we’ve discussed the most common (somewhat lazy) method of moving a game engine along by way of a Game Loop, and we now know why that method is a bad idea. Let’s take a look at a much better approach, shall we?

Parallel Processing (Phase 1)

Parallel Processing (Phase 1)

Rather than having a single processing Thread looping through the three separate processes, let’s divide the processes up onto separate Threads.

This diagram (above) illustrates the point, with Physics and Logic rolled into one Thread, and Rendering on another.

The reason we bundle Physics and Logic together is that they are fundamentally two parts of the same process. Since we only ever need to update the game’s Logic when the Physical State of the game advances (such as when two objects collide, or cease colliding, and of course when World Object Positions and/or Angles change), it makes all the sense in the world to bundle them together on a single Thread. I name this combined Thread the “Simulation Thread”.

Rendering, on the other hand, has to be performed regardless of the Game State. For example, we need to render the Title Screens when the game launches, the Main Menu before you begin playing the game etc.

Have you spotted a problem yet?

If we’re Simulating on one Thread, and Rendering on another, how do we ensure we’re Rendering a consistent Simulation State?

Rendering a consistent Simulation State

Any competent game engine should employ a comprehensive Event Engine in order to facilitate communication between modules. Not only modules, in fact, as the Event Engine should also handle communications between Game Entities within the game itself.

Rather than me explaining once again what an Event Engine is and how it works, if you don’t already know, you can read my series of articles on Event-Driven, Asynchronous Development with Delphi and the LKSL.

Anyway, a good game engine will employ an Event Engine of some sort, and it is through the use of this Event Engine that we can ensure that our game engine is always rendering a consistent Simulation State.

Simulation and Rendering communication

Simulation and Rendering communication

This flowchart (above) shows how the Simulation Thread communicates with the Rendering Thread using our Event Engine.

It is critical that our Simulation Thread never communicates directly with the Rendering Thread, as we cannot assume that the binary contains the Rendering Thread. This is because the “dedicated server” platform is compiled from the same codebase as the game itself, only the Rendering Thread is not compiled into the “dedicated server” executable.

So, by using the Event Engine to handle communication between Threads, we eliminate any potential conflict between the game’s executable and the server’s.

Now, once we’ve progressed the Physics and the Logic on a Tick in the Simulation Thread, we assemble an Event containing all of the Game Entities’ current render data (position, angle, linear and angular velocity) and dispatch that Event through the Event Engine. This Event also includes the Reference Time at which that render data was assembled (that’s actually very important information)

The Event Engine then passes the event along to the Rendering Thread, where those values will be used in the next Tick.

This way, the Rendering Thread is always operating on a complete (and consistent) set of data.

The Rendering Thread can then interpolate and extrapolate (respectively) based on the last complete set of data (and the Reference Time at which that data was assembled) where on the screen to display each Game Entity, and at what angle.

Since we’re now using two separate Threads, we have given our game engine the ability to use two separate CPU cores, which means both processes can occur at different rates, entirely asynchronously, and we can smoothly display the Simulation on the screen.

Note that it is also the best practise to set realistic Rate Limits on each respective processing Thread so that you yield “spare time” for other processes.

The only downside to this method is that it is architecturally more complex than the Render Loop method… however, this one downside (which I personally consider to be minor) is massively outweighed by its significant advantages.

In conclusion…

What we have seen here is just one case in which there are many alternative approaches you can take when designing and developing your game engine. While the Game Loop approach might seem like the quickest option, and that might strike you as preferable in the beginning, you will eventually encounter one or more of the many shortcomings of that approach, and end up wishing you’d opted for the more complicated – but ultimately superior – approach from the start.

It is my personal opinion that the Game Loop is an architectural concept we should finally retire from the world of game development, and that it should be viewed as inherently flawed. Design your game engine to be Event-Driven and fully Asynchronous by the use of separate Threads. It may add a few hours to your development time, but this investment pays off in the long-run.

Thank you!

I hope you have learned something from this article. The point of this series is to educate others based on my own experiences, and in so doing “justify” the time I would otherwise have wasted through the flawed design principals I investigated when developing my game engine.

While I’m aware that there aren’t many of out there designing game engines these days, there are parallels you may recognize in other types of development project… and the lessons game development teaches us have relevance throughout the world of programming.

As always, questions are always welcomed. If you would like clarification on any of the subject-matter raised in this article, please leave a comment and I shall try to answer as best I can.

Thank you for taking the time to read this article.

Visit Us On Google PlusVisit Us On Youtube