UIDraggablePanel Disable Drag If Fits not working

[Sobraj]

I have a typical scroll view layout. I populate the grid dynamically at runtime.
On my draggable panel I have Disable Drag If Fits turned on.
When I fill my grid and it does fit then I cannot scroll just like I want. And when I fill my grid with more item then it can show, I can drag my panel.
But when I first load a lot of items scroll through the list and then repopulate the grid with a few items so that it fits in the panel I can still drag until I release my first drag.

This is what happens on populate:

1.     //Clear current items.
2.     ClearGrid();
3.                
4.     yield return new WaitForEndOfFrame();
5.                
6.     foreach (UserLevelInfo levelInfo in _userLevelItems.Where(predicate))
7.     {
8.         GameObject go = Instantiate(UserLevelItemPrefab) as GameObject;
9.  
10.         Transform itemTrans = item.transform;
11.         itemTrans.parent = _gridTransform;
12.         itemTrans.localScale = Vector3.one;
13.         itemTrans.localPosition = Vector3.zero;
14.     }
15.  
16.     //Reposition the grid elements.
17.     GridList.Reposition();
18.                
19.     yield return new WaitForEndOfFrame();
20.  
21.     //Recalculate clipping.
22.     _dragPanel.ResetPosition();
23.  

I have tried to mess with SetDragAmount, but didn't help.
The problem only happens when I have scrolled before I repopulate the grid to fewer items.

Does anyone have a suggestion?