In scenarios where objects within the application are not recognised it is possible to recognise the object and click on it. Objects are recognised based on a bitmap image of the control to be interacted with.  This article explains how.

There is no in-built method to perform the said functionality using the Silk4J libraries.  However using a combination of native Java classes and Silk4J it is possible to create a method that will return the coordinates of control based on a bitmap relative to a parent control. The attached zip file contains a JAR file containing a class with the following bitmap utility methods:

boolean sSourceBitmap, String sCompare, boolean bOutputDiff)

boolean window, String sBitmap, long nTimeout)

boolean window, String sBitmap)

int[] sSource, String sParentBmp)

To determine the co-ordinates of a control based on a bitmap of that control, we can use the getCoordsByBMP method, where sSource is the bitmap of the control to be interacted with and sParentBmp is a bitmap of a parent control from which we should determine the x & y co-ordinates of the control. The Java code demonstrates an example scenario where the said method is used to determine the x & y positions of a control:

FormsWindow mainWin = desktop.find("/FormsWindow[@caption='Test Application']");


//Capture a bitmap of the application screen

//find the coordinates of button1 using a bitmap
int [] coords ="C:\\button1.bmp", "C:\\app.bmp");

//click the application at the coordinates returned, new Point(coords[0], coords[1]));

In the above scenario, we find the co-ordinates of control based its bitmap relative to the mainWin object. As the co-ordinates have been found relative to the mainWin object, any action such as a click must be performed against the mainWin.

If the getCoordsByBMP is unable to find the control by bitmap, the co-ordinates {-1, -1} will be returned.