Mobile and Web UI Testing with Macaca and OpenCV (Part I)
### Liangjun Jiang, Xin Jimmy Chen
Part of this article was originally presented on the Tester Home by Xin Jimmy Chen
## Problems
When it comes with UI automation test, the principal is simple, we check whether UI elements or content appeared on each screen will appear as designed. To achieve this, we write all kinds of test script to capture id, text or xpath, set what expect to happen on them while screens moved or action triggered or network responded. Quickly we realize it can be very tedious to write test case scripts to cover all. It comes into many our mind naturally: “Instead of capturing each UI element, can we just compare a captured screenshot to predefined screen image to see whether there is a match?”
OpenCV is a popular computer vision framework. In this series, we will use it as our backbone to demonstrate how we give a solution to this problem. In the first part we will give an overview how you can set up and use our OpenCV node.js server. The next article of these series, we will give an example how you can use our beloved Macaca UI Testing Framework and OpenCV to run your UI Testing.
## Install [OpenCV]
For Mac,
$ brew tap homebrew/science$ brew install opencv
For Windows, you can find official guide of installing OpenCV on Windows.
## Install OpenCV Node.js Server
Dafeng Xu, the author of Macaca — an open source project for UI Automation Testing, has created a nice OpenCV Node.js server project on github.
Follow those steps to get this server up and running.
$git clone git@github.com:macacajs/nodecv-server.git — depth=1$cd nodecv-server$npm i$make server
Once the server is running correctly, you can open this link from your browser
http://localhost:9900In this demo project, you can play with features such as “Image Dissimilarity”, “Image Find Pairs”, “Match Template” and “Cascade Detect”, as shown in the following.
You can upload some images to play around.
You can also make `POST` request to the following endpoints to see responses
/opencv/dissimilarity/opencv/findpairs/opencv/matchtemplate/opencv/cascadedetect
A sample `curl` command to make the `POST` request to see whether an image is a part of the other image.
curl -i -X POST \-H “Content-Type: multipart/form-data” \-F “image1=@./fixture/T-shirt-logo.jpg” \-F “image2=@./fixture/T-shirt.jpg” \http://localhost:9900/opencv/findpairs
You can download the sample images and save it locally.
T-shirt-logo.jpg()
In this example, make sure of two things
1. You are in the right path (directory) to access sample images ‘T-shirt-logo.jpg’ and ‘T-shirt.jpg’.
2. Make sure `image1` is for child image, `image2` is for parent image.
If everything goes correctly, you will see a response like the following
{“match”: {“result”: true,“width”: 650,“height”: 650,“match_x1”: 193,“match_y1”: 166,“match_x2”: 305,“match_y2”: 265},“urls”: [{“fieldname”: “image1”,“originalname”: “T-shirt-logo.jpg”,“name”: “baa48abd1af13b106706b1b7d4886eb6.jpg”,“encoding”: “7bit”,“mimetype”: “image\/jpeg”,“path”: “upload\/baa48abd1af13b106706b1b7d4886eb6.jpg”,“extension”: “jpg”,“size”: 3838,“truncated”: false,“buffer”: null},{“fieldname”: “image2”,“originalname”: “T-shirt.jpg”,“name”: “c67fcf2fae2e8442e93e2475b6ac060c.jpg”,“encoding”: “7bit”,“mimetype”: “image\/jpeg”,“path”: “upload\/c67fcf2fae2e8442e93e2475b6ac060c.jpg”,“extension”: “jpg”,“size”: 14486,“truncated”: false,“buffer”: null}]}
## a Python openCV Demo Project
A python project has been created to demonstrate how to set up openCV python library and the openCV node server. You can download it from github.
$git clone git@github.com:macaca-sample/cv-sample-python.git$cd cv-sample-python$pip install -r requirements.txt
Here are a few ways to run those sample python scripts.
# In `cv-sample-python` directory,# after start nodecv server# dissimilarity example$ python nodecv-server-sample.py
You will see a response
7.4Another example,
# find pairs example$ python findpair-nodecv-server-sample.py
You will see a response,
MatchScreenshot image width: 650Screenshot image height: 650Match rectangle corner1: (193,166)Match rectangle corner2: (305,265){“match”:{“result”:true,”width”:650,”height”:650,”match_x1":193,”match_y1":166,”match_x2":305,”match_y2":265},”urls”:[{“fieldname”:”image1",”originalname”:”T-Shirt-logo.jpg”,”name”:”822c335179a4f5fed0b0b017ec057e31.jpg”,”encoding”:”7bit”,”mimetype”:”image/jpeg”,”path”:”upload/822c335179a4f5fed0b0b017ec057e31.jpg”,”extension”:”jpg”,”size”:3838,”truncated”:false,”buffer”:null},{“fieldname”:”image2",”originalname”:”T-Shirt.jpg”,”name”:”9cd40d03a059b642a0c68a8eee8d76fe.jpg”,”encoding”:”7bit”,”mimetype”:”image/jpeg”,”path”:”upload/9cd40d03a059b642a0c68a8eee8d76fe.jpg”,”extension”:”jpg”,”size”:14486,”truncated”:false,”buffer”:null}]}
## Use Cases
Our experience shows that the computer vision technology can be very useful for cases like,
1. You just couldn’t grab UI elements by id, or test, or xpath etc
2. UI screen doesn’t change frequently. One example is a ‘Settings’ screen.
3. Image elements
## Conclusion
By using our node.js server, you might have tried to use different images to verify how accurate or reliable OpenCV is , and have had your own impression whether it will serve your purpose. In the next article, we will introduce our macaca UI automation library and this openCV technology. Stay tuned.
