In scenarios where objects within the application are not recognised it is possible to recognise the object and click on it. Objects are recognised based on a bitmap image of the control to be interacted with.  This article explains how.

There is no in-built method to perform the said functionality using the Silk4J libraries.  However using a combination of native Java classes and Silk4J it is possible to create a method that will return the coordinates of control based on a bitmap relative to a parent control. The attached zip file contains a JAR file containing a class with the following bitmap utility methods:


boolean com.mf.bitmap.Utilities.compareBitmaps(String sSourceBitmap, String sCompare, boolean bOutputDiff)

boolean com.mf.bitmap.Utilities.wait4Bitmap(TestObject window, String sBitmap, long nTimeout)

boolean com.mf.bitmap.Utilities.verifyBitmap(TestObject window, String sBitmap)

int[] com.mf.bitmap.Utilities.getCoordsByBmp(String sSource, String sParentBmp)


To determine the co-ordinates of a control based on a bitmap of that control, we can use the getCoordsByBMP method, where sSource is the bitmap of the control to be interacted with and sParentBmp is a bitmap of a parent control from which we should determine the x & y co-ordinates of the control. The Java code demonstrates an example scenario where the said method is used to determine the x & y positions of a control:


FormsWindow mainWin = desktop.find("/FormsWindow[@caption='Test Application']");

mainWin.setActive();

//Capture a bitmap of the application screen
mainWin.captureBitmap("c:\\KB\\ImageClick\\app.bmp");



//find the coordinates of button1 using a bitmap
int [] coords = com.mf.bitmap.Utilities.getCoordsByBMP("C:\\button1.bmp", "C:\\app.bmp");




//click the application at the coordinates returned
mainWin.click(1, new Point(coords[0], coords[1]));


In the above scenario, we find the co-ordinates of control based its bitmap relative to the mainWin object. As the co-ordinates have been found relative to the mainWin object, any action such as a click must be performed against the mainWin.

If the getCoordsByBMP is unable to find the control by bitmap, the co-ordinates {-1, -1} will be returned.