Fat finger multitouch problems

[ChrisPruett]

A year ago, when we shipped Rise of the Blobs, we got a large number of reports of problems involving the pause screen popping up in the middle of the game.  We tried forever to reproduce this without success.  We were able to do it once or twice (in ~2000 attempts) so we knew it was a legit bug, but we could not figure out the reproduction steps.

I finally had a flash of insight and realized that the bug was caused by a race condition. Two different mutually-exclusive things were happening the same frame ("start the game" and "back out of the current dialog") which was leaving the game in a bad state.  And the reason that this happened is that people were pressing two buttons at the same time.  Two buttons that, on devices with small screens, are in close physical proximity were being depressed and released simultaneously, causing two OnClick() messages to be sent on the same frame, causing the race condition.

Once we realized this was the problem we were able to break almost every dialog box in our game (of which there are a huge amount in that game) simply by pressing multiple buttons at once.

I think the OnClick() via SendMessage() is fundamentally problematic because it encourages writing small button scripts that know nothing about what else might be getting pressed in the system.  In fact, I think you almost never want to send more than one OnClick() for UI; it might be necessary for HUD buttons, but in the general case it seems more correct for the first button that responds to OnClick to eat the event.

Now, I know that I could just turn off UICamera.allowMultiTouch.  But that is a poor solution from a UX perspective, because now any stray finger on the screen will cause all of the buttons to stop working.  I'm not sure if you've ever seen somebody hold their phone such that their thumb slightly touches the side of the screen, but when this happens in apps that do not allow multitouch it causes the whole app to feel broken.

I can get Temple Run 2 menus to freak out a bit by double pressing buttons.  Subway Surfers has allowMultiTouch turned off.  I suspect this is a problem that affects a lot of NGUI-based games on mobile devices.

On Rise of the Blobs our fire-fighting fix was to visit every single OnClick() script in the game and add a check via a global singleton.  The singleton would allow one click per frame and then would return false for all subsequent requests.  This way, a stray finger that wasn't touching a collider would not prevent other buttons from working, but you were unable to simultaneously click more than one button at a time.  This works but it's not a very classy fix. 

Now, a year later, I'm about to ship our new game, Wind-up Knight 2, and lo and behold, despite being a lot more careful about potential vectors for race conditions we can still cause mega breakage by hitting buttons at the same time.

So, my question is: what's the best way to fix this in the short term?  I don't mind getting my hands dirty in NGUI code, but since the event is broadcast via SendMessage() I didn't have many good ideas for how to alter the existing code to stop sending OnClick() after the first one succeeds.  Secondly, what could we do to make this not possible in future versions of NGUI?

Thanks!

[ArenMook]

Well, first thing's first -- how are you getting two OnClick events in the same frame? Multi-touch is for tracking separate touches. A single finger should only result in one touch. If that's not the case then it sounds like a serious bug that should be reported to Unity.

The work-around to allow only a single OnClick per frame would mean that multi-player tablet games like Starlink wouldn't allow for more than one player to do things simultaneously, which seems like a bad idea.

OnClick event hapens via SendMessage, but it's not a broadcast. Broadcast goes to everything, SendMessage goes to the target game object.

Lastly, how can I reproduce this on my own to better understand the issue?

[ChrisPruett]

OK, so, in order:

- The general issue is that two OnClick() events can occur in a single frame or over a very short number of frames.  The first case occurs when you use two fingers to simultaneously press buttons.  The second case (within a few frames) is relevant for both one or two fingers.  I don't think there's a Unity bug here--I'm not getting more than one touch from a single finger per frame--but most touch screens do not report the same screen position for a given finger from frame to frame, even if you try to hold your hand perfectly steady. 

Consider the case where a pop-up dialog has two buttons, both of which spawn dialogs themselves.  If the dialog spawn results in a transition animation (say, the new dialog fades in), you can easily produce a case where you click both buttons with a single finger over the course of a few frames.  The first click causes a new dialog to start appearing, but before that dialog can finish popping up you manage to click the second button, which also spawns a dialog of its own, and now both dialogs are up at the same time and you probably never planned for this.

You can solve this *specific* case by writing a window manager, or by disabling colliders on the first OnClick(), but even this is difficult to do without race conditions.  And it requires a lot of work above-and-beyond the infrastructure that NGUI provides. 

- As you say, sometimes you want more than one click at once (the Wind-up Knight 2 HUD relies on that behavior, in fact).  But often--in menus and other non-gameplay UI--I think it is almost always a bad idea.  So I'm suggesting that perhaps there should be a mode that stops sending onClick() after one is "accepted" (a concept for which NGUI doesn't currently have a model, due to reliance on SendMessage()).  Our implementation isn't frame based, but it requires some time (100ms, if I remember correctly) before it will accept another click.  Humans cannot voluntarily move that fast, so it doesn't affect normal use, but it prevents the double click problem.  Such a mode built into UICamera would completely remove the chances of this sort of race condition without requiring us to write our own window managers to ensure that colliders are immediately disabled upon every click.

- To reproduce, try making two UIButtons next to each other with OnClick() scripts that play sounds.  Run this on a phone that supports multitouch and spam the screen near the buttons with your finger.  Eventually you'll get both sounds to trigger.  In the Rise of the Blobs case, some people were able to do this 100% of the time, and while adding the collider-disable hack "fixed" the issue for those folks, we were never able to figure out exactly how they were doing it. Perhaps they held their device a certain way?  Maybe the touch driver on their screen was buggy?  We don't know.  But the bug was ours.

[ArenMook]

A window manager is a must for any game, imo. That was the first thing I did for Starlink (and is indeed a part of Starlink UI kit). I've also posted the code for some of it on this forum over a year ago. The Starlink UI kit's window manager not only ensures that you can have only one window up at any given time and handles transitions between them, but it also implements the "go back" functionality for you, making menu navigation much easier to set up.

Limiting click response to 100 ms won't prevent your issue of window transitions. What if a transition takes 200 ms, and you still happen to click on the transitioning window's button while it's fading in? Anything with hard-coded values is easy to break. Furthermore, double-click is an integral part of the UI system, and I can quite easily double-tap in under 100 ms between the two taps. Mouse click -- even easier.

[ChrisPruett]

No disagreement there, but none of those sound like good reasons to allow multiple onClick() events to be sent to different colliders in a small window of time. 

We have a window manager, it has support for transitions and stacks of windows and it disables colliders and all of that jazz.  It can probably be improved, but even doing so won't resolve this issue entirely (as the window transition problem is just one manifestation of the more general issue here).

I maintain that, in most UI environments, it's never correct to click on two different colliders simultaneously (or near-simultaniously).  The exception is on-screen gameplay buttons, and so this sort of multiclick should be supported.  However, I don't think it should be the default behavior.  It's much cleaner to solve once at the event broadcast level than at all of the various receiver levels. 

I understand that this isn't a simple change.  If it were, I would have just modded the code and moved on.  The reliance on SendMessage() is the core of the problem; at the very basic level, you can't exit your loop that sends clicks early if a button accepts the click because you can't tell when a click has been accepted.

So, barring an overhaul of how clicks work in NGUI, any suggestions for how we might tackle this in our codebase?  I'd rather not play whack-a-mole with all the potential race conditions again, though my "one collider per 100 ms" hack worked last time around.  Any ideas for a better solution would be appreciated.

Thanks!

[ArenMook]

In Starlink you can have multiple people play on the same tablet device, gathered around a table. This means multiple fingers active on the screen at the same time. Clicking on one of the stars cancels that star's redirection, and if I add the 100 ms hack in there, then it won't be possible for more than one person to play at the same time without seeing side-effects. Simply put, if your game requires this kind of restriction then you should add it for that particular game.

SendMessage has nothing to do with any of this. All it does is uses reflection to call functions. The process is immediate.

[ChrisPruett]

I am not recommending that you put the 100ms hack into NGUI.  It's a hack workaround that I do not like very much.

I'm also not suggesting that you remove the ability to click on multiple things in a single frame.  If you read my messages above, our game actually relies on this functionality as well in some contexts (gameplay buttons).  It's certainly a valid use case.

However, I strongly disagree with this:

Simply put, if your game requires this kind of restriction then you should add it for that particular game.

... because I think that the use case for one-button-at-a-time is also exceedingly common.  Common enough that it should probably be the default behavior of NGUI, and multi-button support an option.  In regular menus-and-buttons UI, which of course is what many of us use NGUI for, there's almost never a reason to allow multiple buttons to be clicked at once.  Modern touch OSes go out of their way to prevent this from happening, actually, because having to consider race conditions with every OnClick() script that you write is very arduous.

This is obviously an issue if games like Subway Surfers are forced to turn multitouch off to avoid having more than one button clicked in their UI.  It's certainly an issue for us, and I can see that the Temple Run devs tried to mitigate it with some success but still have problems.

So, to be very clear about my intent with this message: as a long-term feature/bug request, please consider reworking the way that click messages are sent to buttons so that we can avoid having to worry about these sorts of race conditions.

Now, the second part of my message is to ask you about ideas for potential workarounds that I could implement myself.  I think the feature I am requesting is a fairly huge, and I would expect it to take time and some thinking to do right.  But for now, I'm interested in ways I could modify my use of OnClick / UICamera to mitigate race conditions.  From that perspective...

SendMessage has nothing to do with any of this.

... SendMessage is actually the core of the problem, because messages do not have return values.  UICamera should probably stop sending click events after one of them reaches a target that processes it, but there isn't a good way to drop out of that loop at the moment because SendMessage is used rather than a function call.

I have a bunch of ideas for how I could modify this, or add some support scripts to protect myself against multiple simultaneous clicks.  But perhaps you have a better idea, or at least a direction that I could pursue?

Thanks.

[ryan]

Most of the cases where I've run into this involve multiple buttons in the same dialog, or somebody accidentally clicking a button with multiple fingers.  In those cases, I only want to handle the first input (if they're simultaneous, I'm fine with NGUI deciding which is the "first"), and then get rid of the dialog.  I'm not bothering with any transitions at the moment, I just destroy the GameObject.  Most of my input handlers make use of that fact by starting with a simple, if odd looking, "if (this == null)" and an early return.  You could just as easily have some state flag that gets set when you start a transition to remove the dialog, and have your event handlers check that state.

It can be a pain to have to remember to write those checks, but I'm actually with Michael on this.  If and when your game needs these restrictions, you should add it yourself.  UICamera shouldn't have to know anything about the transitions your UI is making or whether specific objects are interested in hearing about events that have occurred, because that's going to vary wildly based on the application and the developer's coding style.  You might be able to set up a flag on UIPanel that does what you want and could be checked quickly enough to not impact performance for everyone, but this seems clumsy to me.  Maybe one button should be able to receive multiple clicks, but another should only be clickable once.  (EventDelegate does allow you to add one-shot delegates, btw.)  What if I want my widgets to still respond to mouseovers during a transition but not clicks?  Or only certain widgets should respond but not others?  Or I have multiple panels in a single dialog?  There's no one right solution here, it's just one of those tedious things you have to do yourself.

[ChrisPruett]

Ryan, my "fixes" for this are similar to yours.  Basically opt-outs at the top of all my OnClick() methods.  The singleton code in Rise of the Blobs looked like this:

1. public bool ProposeClick(Collider collider)
2. {
3.         bool clickAllowed = false;
4.        
5.         Logger.Assert(collider != null, "NULL Collider passed to Click Focus Manager!");
6.         float time = Time.realtimeSinceStartup;
7.         if (lastClickTime == -1 || lastClickTime + MINIMUM_CLICK_DELAY <= time || (collider == lastClickCollider && collider != null))
8.         {
9.                 clickAllowed = true;
10.                 lastClickTime = time;
11.                 lastClickCollider = collider;
12.         }              
13.         return clickAllowed;
14. }

... then at the top of each OnClick script:

1. if (!ClickFocusManager.GetInstance().ProposeClick(gameObject.collider)) { return; }

I do not like this solution very much.  It is error prone.

I agree that a real solution is complicated, though I don't think it would involve any extra runtime overhead.  If anything, it probably reduces the number of raycasts and SendMessage() calls per frame while fingers are down.  But I agree that it's not correct for everything.

I do think that this is the sort of core functionality that should be built into NGUI, though.  It's not just window transitions that are affected; every OnClick() that doesn't consider what else might have been clicked in the very recent past is a potential vector for a race condition.  Rather than solve it in every OnClick() method, it's much more robust to solve in the thing sending the events.

My thought had been that if OnClick() were treated as a proper method call, UICamera could simply drop out of the send clicks loop (when this option is enabled).  That might not catch 100% of the cases; there might need to be some heuristic involved, such as "don't send click events until all fingers intersecting colliders have come up or moved far enough away to no longer be considered for clicks."  I don't know what the best heuristic should be.

My question was: if I'd like to implement this myself, and I'd rather not modify UICamera directly, what would you try?  I am thinking of making a ClickReceiver script that implements OnClick, applies some heuristic, and then calls some other method on its children.  At least this way I don't have to modify every click script in our 70k line codebase.  I'll still have to find every instance of every object that can be clicked on and add this script, though.

Perhaps there's a better way?  We haven't used the new event delegate system much yet, but I'll poke around in there.

[Nicki]

Couldn't you do something as simple as switch off allowMultiTouch in the instances where you want to make sure only one button is clicked? I mean, when there's a popup, for instance, you could build a call to UICamera in that sets the allowMultiTouch property.

[ArenMook]

I'm not quite understanding what you mean by SendMessage.

All SendMessage("abc") does is calls all functions named "abc" on scripts attached to the target game object.

If I understand what you're suggesting correctly, you believe that making SendMessage exit out early when one script receives the "abc" function call would solve your issues. I don't see how. First, there is no way to guarantee which script will receive this message first. Is it going to be script A, or script B? And furthermore, why is creating two OnClick listeners on the same game object such a big issue to begin with?

Now consider a simple button. UIButton has an OnClick function that it uses to trigger the registered onClick event delegates. If you then have your own script that implements OnClick, it will be called as well alongside it. With your early out suggestion, only one of them will be called, and without a way to guarantee which one will receive it.

Remember, SendMessage is only sent to a single game object -- the same game object that has the top-most collider that intercepted the event. UICamera doesn't continue sending messages to other colliders. Only the top-most one.

Instead of doing what you're doing, write a simple window manager or better yet -- use a state machine to set states -- "option", "main menu", "level selection", etc. Going with this approach, the latest state will simply always overwrite the previous.

[ArenMook]

P.S. Have you also tried simply changing UICamera's Notify function to this?

1.         static public void Notify (GameObject go, string funcName, object obj)
2.         {
3.                 if (NGUITools.GetActive(go)) // <-- only proceed if the object is enabled
4.                 {
5.                         go.SendMessage(funcName, obj, SendMessageOptions.DontRequireReceiver);
6.  
7.                         if (genericEventHandler != null && genericEventHandler != go)
8.                         {
9.                                 genericEventHandler.SendMessage(funcName, obj, SendMessageOptions.DontRequireReceiver);
10.                         }
11.                 }
12.         }

You can also modify that commented line to add your own logic there instead of going through all your scripts.

This way it will only be in one place.

[ChrisPruett]

If I understand what you're suggesting correctly, you believe that making SendMessage exit out early when one script receives the "abc" function call would solve your issues. I don't see how. First, there is no way to guarantee which script will receive this message first. Is it going to be script A, or script B? And furthermore, why is creating two OnClick listeners on the same game object such a big issue to begin with?

No, I'm sorry, that's not what I was getting at, although that is another issue with SendMessage.

In every other UI system I've worked with in the past, there's be a concept of "consuming" an event.  For example, a button might consume a click event if it has processed it, or not if it wants other things to continue processing.  The first problem with SendMessage() is that there is no return value.  The simplest way that I can think of adding the concept of click consumption to NGUI's existing structure is to allow OnClick() to return true or false.  The most basic heuristic in this case would be to stop sending click events to other colliders once one of them returns true. 

But you can't shoehorn this in to NGUI very easily because SendMessage() doesn't allow you to receive a return value from a method call, or even know whether a method has been called.

On top of that there is the issue you brought up, where you have multiple scripts on the same object responding to a click.  That's also something that is very useful and should be maintained; our heuristic was to say "first collider reporting consumption wins" rather than "first script reporting consumption wins."  This way we still take advantage of multiple scripts on a single collider that respond to click.

But it's all a bit of a hacked mess.  We have a (fairly complicated) window manager.  We have a UI screen state machine system.  We still have cases where you can break our game by hitting the right two buttons correctly.  I suspect that most NGUI games have this issue.  I don't have a great solution to propose (I'm not very happy with my own), which is why I posted originally here.

P.S. Have you also tried simply changing UICamera's Notify function to this?

I'm not sure filtering on activity is sufficient.  I'll give it a shot, though.  Thanks.

Nikki,

Couldn't you do something as simple as switch off allowMultiTouch in the instances where you want to make sure only one button is clicked

Sure, but then we're back to the bad UX of not allowing multi touch.  If you've ever watched somebody grip a phone (esp. a small one, like the iPhone 5), it's exceedingly easy to have a finger touching the edge of the screen without realizing it.  If this finger is the one that gets the first touch id then the game suddenly feels hung when you try to tap stuff.

As I wrote above, I'm not asking you to go gut the whole event system to support this, at least not today.  I'm more just trying to put this on your radar and maybe field ideas for a workaround I can implement myself.

[ChrisPruett]

Another manifestation of this problem that I noticed today.

In some cases we have widgets that pop up or slide in over other content.  This isn't a pop-up window, but the overlay content obscures the things below it and has a collider to block clicks.  Think of a comic speech bubble sort of element that appears at somewhat arbitrary places and might have a button embedded within it, and so it blocks clicks to whatever it happens to be on top of.

In this scenario, we can break our game by doing the following:

- Depress a button that is below where we know the speech bubble will appear.  Hold this finger down.  This is Finger #1.
- Using another finger, click (press and release) a button that causes the speech bubble to pop up.  This is Finger #2.
- Now the speech bubble is up right under Finger #1.  It has a collider that should prevent Finger #1 from clicking on the button under the bubble.
- Release Finger #1.  The button under the bubble receives a click.  Since the game was designed around the idea that we could trust the collider on the speech bubble to shield clicks to things below that bubble, the game is now in an undefined (and quite broken) state.

The reason this happens is that in UICamera, currentTouch.pressed is assumed to be valid without a confirmation raycast when the unpressed event occurs.  When currentTouch.pressed is initialized, the raycast succeeds and everything looks fine.  Then, based on stuff that Finger #2 does, the content of the scene changes and a raycast between the finger and currentTouch.pressed should fail.  But it isn't reevaluated, even on release.  So a button that the finger should not be able to reach is clicked.

You might think that this would never happen in the course of regular play, but we caught it because we got testers complaining about bad state situations even after we added the timing hacks I discussed above.  The timing heuristic doesn't help in this scenario because there's no requirement that Finger #1 and Finger #2 come up the same frame, or even within a short duration.  Finger #1 just has to come up anytime after Finger #2 has caused the overlay to spawn.

This isn't the kind of thing we can fix with a window manager because the intent is to only disable clicking below the small portion of the screen that the speech bubble obscures.  And since the bubble can appear in many locations, it's not just a single button that we have to worry about.

I suspect that I could add a test to ensure currentTouch.pressed == lastHit, but I haven't tried this yet.  Mostly this post is just me thinking out about the problem.

[ChrisPruett]

Replying to my own message above, this mod to line 1472 of UICamera.cs seems to do the trick with no obvious side-effects:

1. if (currentTouch.clickNotification != ClickNotification.None && currentTouch.pressed == currentTouch.current)
2.