Opened 7 years ago

Last modified 4 years ago

#458 accepted task

Functional testing via selenium, phantomjs, or similar

Reported by: Christopher Allan Webber Owned by:
Priority: major Milestone:
Component: programming Keywords: test
Cc: Gabriela Thumé, lotusecho Parent Tickets:

Description

It would be good to have a way to run functional tests that mimic user behavior and tie them into our unit testing commands.

We should probably use a tool like PhantomJS or Selenium. And then of course we should have some tests. :)

I'm marking the priority as low at the moment because I think it's not as critical at the moment given everything else going on.

Subtickets

Change History (11)

comment:1 Changed 7 years ago by joar

Priority: minormajor

I'm upping the priority on this, since I believe a better testing framework would increase the robustness of the application in general, which to me seems like a very good thing, and perhaps even needed in some cases.

To me, the current issues with unit tests are:

  • It's hard to write unit tests, should we write documentation for the testing infrastructure?
  • It's unintuitive to write unit tests - PhantomJS or Selenium would fix that I believe.
  • There's no hard routines on testing, it's good practice to run ./runtests.sh after a merge, I believe a CI system could help with that, but regarding that I think it's up to whoever takes this task to decide the testing architecture and weigh the possibilities of CI integration.

comment:2 Changed 7 years ago by Aleksej

Type: defecttask

comment:3 Changed 6 years ago by Christopher Allan Webber

Cc: Gabriela Thumé added
Status: newaccepted

I'm cc'ing gabithume who has been doing some work on this recently (though I don't know what the most recent state of her work is). Gabithume: feel free to claim this ticket if you're presently working on this, or otherwise give updates if you have them! (No pressure though)

comment:4 Changed 6 years ago by Gabriela Thumé

Hi! :)
So, I have been studying a little about these framework to create functional test. Selenium is not headless but has the advantage to be browser agnostic while Phantomjs is headless but webkit-specific.
Selenium Webdriver automation tool provides all sorts of features by driving a browser natively as a user would either locally or on a remote machine using the Selenium Server. It supports a lot of platforms [1].
Selenium is the leading OSS web testing framework. Also, there are negotiations with W3C about making WebDriver a standard or recomendation [2].
Tests can be written with python[3], which is convenient for mediagoblin developers. In this way I created a functional test for ticket #689 about a broken link in forgot password form [4].

What do you think? Should I send a commit and write some Selenium installation instructions for mediagoblin devs?

[1] http://docs.seleniumhq.org/about/platforms.jsp
[2] http://www.w3.org/TR/2012/WD-webdriver-20120710/
[3] http://selenium-python.readthedocs.org/en/latest/
[4] https://gist.github.com/GabiThume/5623233

comment:5 Changed 6 years ago by Christopher Allan Webber

Cc: lotusecho added

So, we're going to do more discussion next week about testing between LotusEcho and I since she'll be working on testing as part of her summer internship. I know she's finishing exams this week; gabithume: maybe would you mind jumping on IRC next week and talking with the two of us? Maybe together we can find out the right direction for all this.

comment:6 Changed 6 years ago by Gabriela Thumé

Sure. I will enjoy it :)

comment:7 Changed 6 years ago by Emily O'Leary

Owner: set to Emily O'Leary
Status: acceptedin_progress

I will be implementing this fully with a large test suite as part of my OPW internship. Also, there is a new way to use Phantomjs as a "browser" for Selenium to run Selenium tests in a headless manner. :)

comment:8 Changed 6 years ago by Emily O'Leary

comment:9 Changed 6 years ago by ShawnRisk

Keywords: test added

comment:10 Changed 5 years ago by Emily O'Leary

So we ran into one big problem with this test suite (see above) that I don't know how to fix. Essentially, Selenium (and any functional testing tool I know of that is FOSS) is highly coupled with the user interface. Change the theme? All the tests might break. Change your language? That might break it too. Essentially it could mean that you might have to change many of the tests just because of one minor change in the user interface.

I tried to modularize the code somewhat so that common actions wouldn't be invoked by a particular test and instead acted more like a function but I couldn't do that for everything without slowing the progress to a halt. That could be one potential way to make it more maintainable.

The double-edged sword on that becomes that it would be very difficult for new contributors to write a test. That's a problem since one of our goals was to have something that new contributors could do that familiarized them with GMG. Normally writing a test in Selenium is simple; it's basically point-and-click.

comment:11 Changed 4 years ago by Loic Dachary

Owner: Emily O'Leary deleted
Status: in_progressaccepted
Note: See TracTickets for help on using tickets.