Functional testing via selenium, phantomjs, or similar

[Christopher Allan Webber]

It would be good to have a way to run functional tests that mimic user behavior and tie them into our unit testing commands.

We should probably use a tool like ​PhantomJS or ​Selenium. And then of course we should have some tests. :)

I'm marking the priority as low at the moment because I think it's not as critical at the moment given everything else going on.

[joar]

I'm upping the priority on this, since I believe a better testing framework would increase the robustness of the application in general, which to me seems like a very good thing, and perhaps even needed in some cases.

To me, the current issues with unit tests are:

• It's hard to write unit tests, should we write documentation for the testing infrastructure?
• It's unintuitive to write unit tests - PhantomJS or Selenium would fix that I believe.
• There's no hard routines on testing, it's good practice to run ./runtests.sh after a merge, I believe a CI system could help with that, but regarding that I think it's up to whoever takes this task to decide the testing architecture and weigh the possibilities of CI integration.

[Christopher Allan Webber]

I'm cc'ing gabithume who has been doing some work on this recently (though I don't know what the most recent state of her work is). Gabithume: feel free to claim this ticket if you're presently working on this, or otherwise give updates if you have them! (No pressure though)

[Gabriela Thumé]

Hi! :)
So, I have been studying a little about these framework to create functional test. Selenium is not headless but has the advantage to be browser agnostic while Phantomjs is headless but webkit-specific.
Selenium Webdriver automation tool provides all sorts of features by driving a browser natively as a user would either locally or on a remote machine using the Selenium Server. It supports a lot of platforms [1].
Selenium is the leading OSS web testing framework. Also, there are negotiations with W3C about making WebDriver a standard or recomendation [2].
Tests can be written with python[3], which is convenient for mediagoblin developers. In this way I created a functional test for ticket #689 about a broken link in forgot password form [4].

What do you think? Should I send a commit and write some Selenium installation instructions for mediagoblin devs?

[1] ​http://docs.seleniumhq.org/about/platforms.jsp
[2] ​http://www.w3.org/TR/2012/WD-webdriver-20120710/
[3] ​http://selenium-python.readthedocs.org/en/latest/
[4] ​https://gist.github.com/GabiThume/5623233

[Christopher Allan Webber]

So, we're going to do more discussion next week about testing between LotusEcho and I since she'll be working on testing as part of her summer internship. I know she's finishing exams this week; gabithume: maybe would you mind jumping on IRC next week and talking with the two of us? Maybe together we can find out the right direction for all this.

[Gabriela Thumé]

Sure. I will enjoy it :)

[Emily O'Leary]

I will be implementing this fully with a large test suite as part of my OPW internship. Also, there is a new way to use Phantomjs as a "browser" for Selenium to run Selenium tests in a headless manner. :)

[Emily O'Leary]

I'm working on this in this repository:

​https://gitorious.org/mediagoblin-automation/mediagoblin-selenium

[Emily O'Leary]

So we ran into one big problem with this test suite (see above) that I don't know how to fix. Essentially, Selenium (and any functional testing tool I know of that is FOSS) is highly coupled with the user interface. Change the theme? All the tests might break. Change your language? That might break it too. Essentially it could mean that you might have to change many of the tests just because of one minor change in the user interface.

I tried to modularize the code somewhat so that common actions wouldn't be invoked by a particular test and instead acted more like a function but I couldn't do that for everything without slowing the progress to a halt. That could be one potential way to make it more maintainable.

The double-edged sword on that becomes that it would be very difficult for new contributors to write a test. That's a problem since one of our goals was to have something that new contributors could do that familiarized them with GMG. Normally writing a test in Selenium is simple; it's basically point-and-click.