What is Software Test Automation? and how would a software engineer at Microsoft describe it? We had the pleasure to meet Bernardo Rivas, who works at Microsoft as a software engineer with a solid experience in C#, testing, test planning and automation.
Hi Bernardo and welcome to Airbrake Interviews!
First, we'd like to know, how did you start your career?
I started working in 2007 in Mexico City when I was still in college for a company called Stream Nova. They are a Microsoft partner which means that all their development is done in C# using SQL Server and, at the time, The Official Microsoft ASP.NET Site on IIS. I joined the company not knowing much about any of those technologies. I didn't even know what a stored procedure was. I worked there for two years until I got my B.S. in Computer Science and had worked two big projects where I learned all the basics. That year I moved to Seattle to work with Microsoft as an SDET. I've worked with the Dynamics, Photosynth, Customer Support and Service, Bing and SQL Server teams where I focused mostly on virtualization, development of test tools and test automation.
Software test automation is a trending concept with many definitions, what does test automation mean ?
In my experience, every organization defines a set of measurable benchmarks of quality for their product and it's their job to make sure that it never fails below them. Those benchmarks of quality can be related to many aspects of the product like usability, performance, functionality, integration, etc. Test automation is the tool or set of tools that allows the organization to measure the quality of the product in a consistent and timely way.
In your opinion, when should companies consider adopting test automation? and why should engineers automate the software testing process?
I think developers on any company at the very least should write unit tests right from the beginning. Unit tests executed as part of a gated check-ins are industry standard because they can catch basic issues like regressions or dependency breaks before even committing the code into the product’s source.
In my experience, writing unit tests automatically improves the quality of the code. It forces the developer to consider corner case scenarios from the get go, simplify their code with single-purpose methods that are easier to test and maintain and use interfaces to allow hermetic testing techniques like mocking (substituting an external dependency with a local test version it).
Writing functionality, integration and end-to-end automated tests is also very important. Companies rely on this type of testing to certify the quality of their products. It is so important that, in the past, companies would have a 1 to 1 ratio of dedicated testers and developers. As automation frameworks evolved, test automation became easier and cheaper to write, reducing the need for manual testing and thus dedicated testers. Automated tests also ensure that the key scenarios supported by your product are always functional no matter what changes are made to the source. For this, the engineers need to be aware of all the types of tests that should be performed against their product. Testing their API, their UI, the integration between them, the backend, the database, dependencies, etc.
From what you've just mentioned; automation in general seems challenging, what are the main obstacles that may face developers in software test automation and what's the best way to overcome them?
There are many challenges in test automation that companies have to overcome. Infrastructure, maintenance of the automation code, limitations in the test framework etc.
I’ve personally dedicated most of my career to solve the infrastructure obstacle. In order to get a reliable result, you need a clean test environment every time you execute a test. In the past, test automation usually included steps to take a group of machines and “pave” them by reinstalling the operating system and any dependencies like applications, database clients and browsers. This process alone would take days and didn’t allow to reuse machines without repaving them.
Virtualization has been traditionally the answer to this issue. By using virtual machines, and lately, machines in the cloud, we can create tests environments on demand, and reuse them as many times as we want by taking advantage of features like snapshots and pre-setup Virtual Hard Disks. Companies invest a lot of resources to develop their own test-machine delivery systems because it allows developers to focus on development rather than allocating test machines to run their tests.
Other obstacles that come to mind are related to the test framework itself, which should expose the necessary tools and functionality to concentrate testing the product rather than working around the framework’s limitations. I found UI test automation to be particularly tricky in this area, especially for mobile devices. Even after defining a good test case and the automation to execute it in a mobile device, actually measuring the quality of the result in terms of graphics is still something that is mostly done manually.
We now understand software testing has many ways and methodologies, how would automation help testing engineers in creating / pursuing a better career?
For me personally, writing automation has helped me write better, more reliable code. As a team member, submitting a code change along with a set of unit and functional tests sends the message to the team that I care about the quality of the product and that I’m protecting the integrity of the source and not introducing regressions. When reviewing other people’s code, I can tell when something has been overlooked and should be tested because that’s what I would test. And in general, I have a deeper understanding of the source code because I’ve written tests for it that had made me read it and analyze it. These are all traits that are very useful to any team in any company.