Browser Testing
We have tested a number of browser test harnesses over the last few years and has found Sahi to be very reliable and powerful. For general day-to-day use as well as the ad-hoc running of regression scripts before releases Sahi is a great tool. Unfortunately, there are some issues that it cannot solve. These issues become particularly awkward when running Sahi as part of the continuous integration pipeline.
What Do We Know About Browsers?
- For certain browsers, a logged in user session is required to run tests. While tests can easily be run in a headless browser without this requirement, automated regression testing in Internet Explorer is often a requirement.
- The host environment can impact test results.
- There is a limitation to just a single browser version.
Virtualization provides a great way to solve this problem. Individual VMs can be snapshotted into the correct browser state and the process of running tests can automatically load the correct virtual machine configuration before executing tests remotely.
Here is an outline of our process for configuring a VM as a Sahi test runner:
- Install Windows and browsers. The Windows version will dictate which version of Internet Explorer you will be limited to. Windows 7 is the most general case as it will work with IE7 up to the currently latest version.
- Install Sahi. Having a standard Sahi installer is quite useful in this scenario. Sahi can also be installed quietly
- Initialize the browsers. We found that we have to start and stop some browsers before running scripts against them. Rather than picking and choosing, we do this for all browsers we plan to use.
- Install Sahi proxy certificates. This allows Sahi to work with https websites. For Chrome and IE this is as simple as adding the cert to the root trust store, for Firefox this has to be added to the firefox specific store. The steps for doing this can be found here.
- Disable windows updates. To prevent accidental changes to the environment.
- Share the file system, turn off windows firewall. In order to transfer files between the VM and the machine which runs the continuous integration server, we require a consistent connection. In our environment, the connection would reset to a public network every time the VM was started, so this was the simplest solution to allow remote file system access.
- Enable remote PowerShell access for running scripts.
- Start the Sahi host process and wait for initialization to complete.
- Create a snapshot of the VM in this state.
What Are We Using Ourselves?
For the last several months we have been using Chocolatey more and more to automate environment setups for our projects. This is from developer machines right through to production environments. Since it’s easy to make a mistake in a configuration like this and keeping documentation up to date is likely to be an issue on an infrequently touched process, packing installation scripts into a Chocolatey package is an ideal solution. Creating a recipe to perform a Sahi virtual machine configuration fitted this process really well. Having written the necessary scripts, a full environment set up, including a windows installation takes roughly 25 minutes. (More like 15 minutes if you’re using an SSD.)
How Do We Run Tests?
The process for running tests in this way becomes quite straightforward:
- Restore a virtual machine snapshot.
- Copy test scripts to the VM.
- Run the tests.
- Copy the results back from the VM.
- Parse results.
In our CI pipeline, Sahi is configured to output test results in a JUnit format. The CI host, Jenkins, can easily interpret this. The full log set is copied back from the Sahi machine after it completes a test run and then archived. This provides a way to go back through the logs. Even if some other job has reset the VM and run another test set against it.
Having put this in place, we are able to run multiple browser tests in parallel across multiple projects and get results which are easily reproducible.
A Bit About Us
At Greenfinch Technology we are always actively working to keep up with modern technology. In our opinion. to stand still is to fall behind. You can be sure we have no intention of letting that happen to ourselves or to our clients.
If you would like a modern, forward-thinking IT company to help with your custom enterprise software solutions, mobile app development, a BPM such as Siskin or any Cloud technologies please feel free to contact us at 01 818 2949 or email us using our contact form. We will help guide you safely through the process of advancing your company beyond your competitors.
Greenfinch Technology
