Here at Infragistics we have always been invested in ensuring the quality of our products - that is part of how we deliver great experience to our users and, as you know, we do care about that a lot. Over the years we've delved into many platforms and that means we've seen many of the unique challenges many of them present. To put it in other words - we have a whole lot of experience and know-how. At this stage, throw in the fact that for the last few years we've been developing our jQuery controls - Ignite UI - and they have certainly taken a very special place for us. You can imagine we want to keep Ignite UI a quality product, yet testing web tools has always been a challenging task. Thankfully, their extreme popularity has led to the creation of some very good tools to help with that. We were at the ISTA Conference last year and as you may have seen us showing off, Angel Todorov had a session on this subject and there’s some pretty interesting stuff in there so I figured we can iterate a little bit more on that. I think it would be interesting to see the tools used for testing inside the company in a bit more detail. We also have some interesting news regarding those coming very soon!

Automate this!

In my eyes, test automation is pretty much an integral part of quality engineering for any product with long(ish) lifecycle. While you can’t automate everything, doing so returns the investment in the long run by providing excellent, consistent regression testing. Also at ISTA (which literaly stands for Innovations in Software Test Automation), as you’d expect, there were more than a few sessions for web platforms and sans the mandatory old IE joke, there has been one common mention that really stood out to me – Selenium, Selenium, Selenium.

At this point I believe everyone is aware of how tedious it can be to test under different browsers – it would be extremely inefficient to write tests and use automation tools for each one. Here’s one reason why Selenium is almost an industry standard - that and the fact that browser vendors are in on the project as well. Selenium, to some extent, abstracts away the browser you are dealing with so you can have shared code base for your test cases. Shameless plug: another colleague, Borislav Traikov, wrote an article on Selenium and WebDrivers, so I would skip the details and recommend you check that one out. After that it should be no surprise we are using WebDrivers (or Selenium 2.0) and for a good reason.

What I won’t skip is why this is so important for our testing solution. Our product engineering internally relies on TFS and all the good stuff it brings (e.g. MS Build) so, as you can imagine, we want any testing automation to be an integral part of the products’ lifecycle. Thanks to the .NET implementation of WebDriver(s) tests can be easily written, maintained and most importantly integrated with the build process. Did I mention these also happen to be pretty fast in comparison with other frameworks we’ve tried? That speed and reliability are crucial factors in allowing automated tests to be executed in our Continuous Delivery pipeline – think more than 800 scenario tests[*explained below] run with every CI (Continuous Integration) build and they only take under 30min instead of a couple of hours. That ensures constant, reliable monitoring of quality.

Testing Approach

Now, besides the right tools for the job (of which there are some more below, mind you), there’s the whole approach to the testing process. So what’s the problem in the first place? Well in your usual scenario testing you’d need test beds (the targets for the tests) and it’s one painstaking slow process for creating them. Also, it’s usually somewhat hard to share and reuse code efficiently. You can say that’s partly because test code usually is not easily readable unless you wrote it (depending on platform/tools and implementation), but one thing is for sure – even if programmers can see the scenario in the code, non-engineers are a different story! Anyone else involved with the product like designers (some do code, we know), but most importantly non-technical people as a whole – managers and stakeholders - get nada from looking at test code. That fails this approach on one of the measures for a good test – yes, it may be easy to write with experience, integrated with CI, maintainable and reliable, but it’s hard to verify it’s truly user-focused. There’s a gap between stakeholders and engineers, often required to be filled in by a person that “speaks” both business and code who does the translation.A single aspect of it all, but you can make gold and it’s all to waste if the customer wants plastic. Then there’s also the “joy” of maintaining all the test plans and code in sync...

BDD: Behavior-driven development

So we based our process on behaviour-driven development. Now besides weird spelling options, BDD does provide you with something else. Based on test-driven development (TDD), it combines some parts of the former with ideas from domain-driven design(DDD) and more abbreviations (too much for me already). Point is, it’s sort of intended as a collaboration framework for engineers and business managers and clients. And I’m going to pretty much steal the definition right out of Wikipedia, because it’s this good:

“BDD is a second-generation, outside-in, pull-based, multiple-stakeholder, multiple-scale, high-automation, agile methodology. It describes a cycle of interactions with well-defined outputs, resulting in the delivery of working, tested software that matters.”

Simply put, BDD helps you keep your goals in check (user-focused, remember?) by making tests extremely easy to understand and create by everyone. When you describe a test you do it in a user story-like manner:

“As a [role] I want [feature] so that [benefit]”, where the benefit obviously has business value.

And when you define each scenario (the acceptance criteria) - in a narrative way:

“Given [initial context], when [event occurs], then [ensure some outcomes].”

It’s as close to natural language as I’ve seen so far and without some tight specifications, there are options to choose from. It was mostly Ruby and Cucumber that gave BDD large traction. Cucumber understands the Gherkin language, which is more or less like described above. You could use IronRuby, but there are also implementations in multiple languages, the .NET being SpecFlow:

I for one think this is awesome! Once more, some thoughts on SpecFlow shared by Borislav Traikov you might want to check out. SpecFlow is the middle man turning feature files full of scenarios into .NET classes with tests adapted for whatever supported test runner you picked (attributes and all)! So about that Gherkin, let’s make things crystal clear with an actual example:

This is very understandable, right? The greyed-out parts are actually parameters for the step definition. Tags on top are freeform text for the most part, except for example a few SpecFlow uses such as ‘@ignore’ which does exactly what you think it does – ignores the marked test. Other tags can have special meaning elsewhere or can simply be used as a way of categorizing and grouping tests. Also they can be placed at both feature and scenario level and can be inherited. We’ll talk some more about those later on.

Now of course, Behavior-driven development tools don’t really do magic – they can’t write test logic for you or execute your test. They are the glue that quite literally binds business requirements to actual code implementations. They do provide quality-of-life support for other tools to help you run your test, but you still need to do some work:

As you can see SpecFlow makes it really easy to assign real executable testing code to the steps and only means that the generated code for the test runner will be matching and calling this very method when executing your tests. You also get similar attribute-style event hooks for methods to be called before/after test runs and further extended with such for features, scenarios and even steps!

Benefits

• Limit misunderstanding between groups and promote collaboration – everyone can contribute to testing.

• Easy to set and change the test runner framework in a SpecFlow test project, which in turn makes it very easy to integrate in CI environments.

• Quickly automate newly found/submitted bugs. In a test-driven approach creating steps this way actually ‘tells’ the developer exactly what needs to be done for the test to pass. The test output itself again contains the steps with replaced parameters and is just undoubtedly more understandable for non-engineers than a stack trace :)

• Achieve great reusability out-of-the-box because every step is autonomous  - you even get a nice IntelliSense integration to quickly provide you with steps you might be looking for:
  

• Very transparent definition of repetitive tasks based on data – in a very similar way to data-driven .NET tests you might be familiar with. Instead of providing actual values in the steps, you put placeholders in a Scenario Outline and then provide a table of Examples the test must be run against. Best of all, underneath SpecFlow will actually generate separate tests for each row – so they can pass or fail on their own and also run in parallel.

• Most of all, it’s fun!

So you can do everything you always had while automating (sometimes even more), but now there’s this promotion of reusable bits and an elegant, easy to understand layer in natural language (yes, Gherkin is also localized) and everyone can participate in the definition of user-focused tests!

Resources

While this is not a complete description of the process (which we might share soon) you have got to see some of the tools we use and piece it together. We’ll be taking a look at some technical details and execution specifics of the Infragistics testing process next. Also in the mix will be some interesting frameworks of our own, so keep an eye for the next part.

• Wiki: Behavior-driven development
• Selenium WebDriver and your new best friend the Chrome WebDriver
• Cucumber  and SpecFlow
• We will be at this year’s ISTA Conference again – come join us!
• Other interesting stuff: Continuous delivery
• Go grab Ignite UI
  

I’d love to hear some thoughts, so leave a comment down below or @DamyanPetev.

And as always, you can follow us on Twitter @Infragistics and stay in touch on Facebook, Google+ and LinkedIn!