This project has moved. For the latest updates, please go here.

The all Familiar Problem

For anyone working in an Enterprise Software environment, I am sure you have all encountered the same problems. You finish your development and testing. You package up a release and prepare all the release documentation for the operations team and submit the change request into your Change Request system for approval. Then the release is deployed during your standard outage/change window, normally at an ungodly hour of the night.

 
Smoke Test Editor - Test Runner

 

The people who do the deployments generally have no knowledge about what they are deploying, they will just follow the change instructions that you have written. You will normally only really know if the release has been a proper success once your users get their hands on it the next day. In some businesses that can be a big risk if an outage or problem with the system has a financial or customer service impact.

A Solution

What you ideally need is a very simple way for the release team to perform some standard checks to make sure everything is wired up and configured correctly. It is even better if you can automate these tests so they can be repeated by support teams when investigating an incident.

That is where this Post Deployment Smoke Testing Tools comes in. Don’t just take my word for it though. This is what Jez Humble and David Farley, authors of the book Continuous Delivery and Thoughtworks engineers have to say on the subject.

When you deploy your application, you should have an automated script that
does a smoke test to make sure that it is up and running. This could be as simple
as launching the application and checking to make sure that the main screen
comes up with the expected content. Your smoke test should also check that any
services your application depends on are up and running—such as a database,
messaging bus, or external service.

The smoke test, or deployment test, is probably the most important test to
write once you have a unit test suite up and running—indeed, it’s arguably even
more important. It gives you the confidence that your application actually runs.
If it doesn’t run, your smoke test should be able to give you some basic diagnostics
as to whether your application is down because something it depends on is not
working.

Jez Humble and David Farley

Continuous Delivery – Addison-Wesley Professional

 

You may be lucky and work in a fully agile development shop where you do full continuous deployments directly into production. If so, I salute you. This tool is still very valuable as you can ensure the tests execute after each automated deployment and still get the benefits of this rapid feedback.

Smoke testing is there to give you rapid feedback on the success of a release. If you know instantly that there has been a deployment problem, the engineer doing the release can contact someone to get it fixed or roll back before it impacts the business.

When deploying into production, there are all manor of things that can happen that you didn’t expect, like firewall problems, database passwords being entered incorrectly, MSI installers going crazy (I hate it when they do that).

This smoke testing tools allows your development team to construct a list of test steps using a graphical interface. The tool is very easy to use and you can even use the tool to execute the tests. There is also a command line test runner that can be scripted as part of your deployment.

The whole system is extensible so that if there is not an included test type that you need, you can easily develop your own.

What Sort of Things Can You Test

Out of the box you can perform basic tests like the following :

    • Check certificates exist.
    • Check COM objects are registered.
    • Check SQL Server connection strings.
    • Check environment variables are set.
    • Check files exist.
    • Check folders exist.
    • Check HTTP connections return specified result codes.
    • Check minimum disk space requirements.
    • Check registry keys exist.
    • Check elements and attributes exist in XML files.
    • Check MSMQ installed and that local private queues are present.
    • Check MD5, SHA01 and SHA256 digests of files.
    • Check that local or remote windows services are installed and running.
    • Check that IIS is installed and running.
    • Check for users in Active Directory for a names domain.
    • Check availability of network addresses/servers (ping).
    • Check .NET assembly file versions.
    • Run executables or batch files and check the return code.
    • Run executables or batch files and check for text in the output console.

The system is extensible so it is very easy to add your own custom test types.

What’s Included

The Post Deployment Smoke Tester comes with 2 tools. There is a graphical user interface tool called the Test Configuration Editor for building / executing test chains and there is a command line tool for executing tests. The latter is great if you want to script the test execution. Lets take a little look at each tool.

Smoke Test Configuration Tool

The Test Configuration Tool allows you to set up and configure test scripts based on the available selection of test types. The tool is very easy to use so the test scripts don’t have to only be created by developers. It should be easy for your testing or DevOps team to build scripts.

 
Smoke Test Editor

 

The test configuration editor also allows you to execute the tests as shown below. When you run the tests you will get a basic traffic light system, similar to testing tools like NUnit. Green means the test passed successfully, and Red means the test failed. If a test does fail you get a message saying why so you can explore further.

Smoke Test Editor - Test Runner

File Digest Generator

Included with the Smoke Tester are a series of tests for checking either an MD5, SHA1, or SHA256 checksum on a file. This type of test allows the smoke tester to check that a file hasn't been tampered with or modified. If a file has been changed, then the checksum test will fail. When you use the test in the test editor, you need to easily be able to generate a checksum of the target file so you can include it in the test.

Smoke Tester Checksum Creator

 

The test editor contains a File Checksum Generator under the main Tools menu. This generator is very easy to use. You browse for a file, and then using the radio buttons select what kind of checksum you want, MD5, SHA1, SHA256. Then you hit generate.

Smoke Tester Checksum Creator

 

This will give you a checksum as shown above. Then you can copy it to the clip board and paste it into the test, as shown below.

Smoke Tester MD5 Checksum Test

 

This makes creating test chains that want to test for file integrity much easier as you can generate the checksums to check for directly in the tool.

 

Command Line Test Runner

There is also a command line tool that you can use to execute tests. If you run the tool on its own, it will pick up test scripts in that folder and allow you to run them from a menu, as shown below.

 

Run Smok Tests from  the Command Line

 

You can also run the command line tool and specify a particular test to run, as shown below.

 

Run Smoke Tests from the Command Line

 

Want to Contribute?

It is early days for this tool. I would like to see the range of available tests grow significantly. You can contribute in the following ways:

    • Develop new test types. If you want to submit a new type of test, then what we require are tests that are general enough for anyone to use, so for example, tests for MSMQ, Rabbit MQ, various NoSQL databases. You may need tests that are very specific to your own enterprise, this is fine, but if it is only going to work on your system then it would not really be suitable for the overall project.
    • Help with our unit testing. The original version of this tool was written fairly quickly in response to some problematic deployments, so although there are quite a few unit and integration tests, we can always do better in increasing the code coverage.
    • Documentation. It would be great if people could contribute by writing tutorials and other use case documentation.
    • Use case Examples. If you use the tool in your company, it would be great if you could do a little write up about your particular usage scenario and submit some example test scripts and screen shots.

Release Notes

Version 0.04

    • Tests now run in a separate thread in the test runner, so that the UI thread is not blocked.
    • Fixed bug were if you click the remove all tests button you were not prompted first.
    • Console app test runner now return a correct exit code. 0 for success and 1 for failure.
    • Add an Enabled flag to the test base class.
    • Wire in the enabled flag to the test runner in the UI and Command Line App.
    • Fixed bug where the save button on the TestEditor would always ask for a filename when it should just overwrite the existing file.
    • Added an XmlDocumentTest which allows you to do an XPath query over an XML Document.
    • Fixed bug in XMLElementTest where if attributes were not the same in a list of elements then an exception would be thrown. This is now handled.
    • Add tool to the User Interface to allow you to generate a list of files to be used in a FileListExistsTest.
    • Added a new test that allows you to check for the existence of a list of files.
    • Added a new test that allows you to check the assemblies file numbers for a list of assemblies.
    • Added Copy & Paste feature; copy and paste test within and across TestEditor windows.
    • Minor layout adjustment on About dialog
    • Added a new HTML report type.

Version 0.03

    • Added a new TestCategoryAttribute to each test. This means in the UI, instead of getting a large list of tests to choose from, they are now displayed in a set of categorised folders. This is to aid usability.
    • Add test linking between the test editor view and the test result view. This means if you get a failing test in the results view you can double click on it to navigate to the failing test in the test editor view.
    • Add a test report writer to the UI. This allows you to automatically output a test report in XML, CSV and Plain Text. 
    • Add a test report writer to the Command Line test runner. This allows you to automatically output a test report in XML, CSV and Plain Text.
    • Added a tool to the test builder that allows you to calculate MD5, SHA1, SHA256 checksums of selected files. This makes it easier to calculate file checksums when using the checksum tests.
    • Ordering Tests In The Test Run View. You can now click on the column headings in the Test Run view of the test editor to sort the test results based on test outcome, ie pass/fail.
    • Added new test called CallExecutableCheckReturnCodeTest. This allows you to call an external tool or batch file and test the return code.
    • Added new test called CallExecutableCheckOutputTextTest. This allows you to call an external tool or batch file and test that text in the standard out contains specific values.

Version 0.02

    • MSMQInstalledTest : This test checks whether the MSMQ messaging service is installed on the server you are running the tests from.
    • MSMQLocalQueueExistsTest : Check if a local MSMQ private queue exists.
    • MD5ChecksumTest : Check an MD5 digest of a file. This is for file integrity checking.
    • SHA1ChecksumTest : Check an SHA1 digest of a file. This is for file integrity checking.
    • SHA256ChecksumTest : Check an SHA256 digest of a file. This is for file integrity checking.
    • WindowsServiceExistsTest : Check that a windows service exists in the machine the tests are running from.
    • WindowsServiceStatusTest : Check the Status of a windows service, Running, Stopped etc.
    • WindowsRemoteServiceExistsTest : Check that a windows service exists on a remote machine.
    • WindowsRemoteServiceStatusTest : Check the status of a remote windows service, Running, Stopped etc.
    • IISInstalledTest : Check that the IIS web server is installed on the machine the tests are running from.
    • IISRunningTest : Check that the IIS web server is running on the machine the tests are running from.
    • IISVersionTest : Check the version of IIS that is installed.
    • IISDoesWebsiteExistTest : Does a website exist in the instance of IIS.
    • UserInActiveDirectoryTest : Check that a user is in Active Directory for a domain.
    • NetworkPingTest : Ping a network address.
    • AssemblyVersionNumberTest : Check the version of a names .NET assembly.

Version 0.01

    • Released the initial version.

Last edited Jul 23, 2014 at 11:29 AM by stephenhaunts, version 16