webchick.net

very helpful lioness

Drupal 7.x itch of the week: fix the testing crisis

Wed, 05/21/2008 - 00:00 -- webchick

Earlier tonight, Jimmy Berry posted a plea for the Drupal community to clean up their messes in terms of testing. Since Jimmy did a whole bunch of the heavy lifting involved in getting the SimpleTest framework ready to commit to core, a major milestone in Drupal's development, we'd do well to listen to his words and his frustrations when he sees all of his hard work falling apart due to neglect.

SimpleTest has only been in core for a month now, and already we've managed to break more than half the tests. This is a big problem. :\

In order for our community to embrace testing, we're required to bring several hundred people who've never written a test before in their lives (and might not even have heard of automated testing until heard some Drupal people talking about it) up to speed on what testing is, why it's a good thing, and how to do it.

But every time there's a bug in the testing framework, and every time an existing core test fails to run, this serves to completely destroy developers' confidence. If we're lucky, these people will go work on other things for a little while, and then check back periodically to see if things have been sorted things out yet. If we're unlucky, they start to develop animosity and resentment about the very idea of testing, and then start to distance themselves from doing development and encourage others to as well. Either way, the end result is stalling development on D7, and fewer people to spread the work around of writing further tests.

So how can someone concerned about the state of D7 development help?

  • Fix the existing tests so that they all pass. This is the #1 priority. Without this, our testing framework is basically useless. If a developer new to testing can't trust that what's there now is working, how can they possibly know if the test they just wrote breaks because they did something wrong, or because something was broken before they got there? This can be a good place to start if you're new to testing, as in the process of your detective work you'll often learn a lot, very quickly.
  • For the love of all that's good in the world, KEEP the tests working once they pass. Several efforts could help with this, including http://testing.drupal.org (check the Drupaltestbed issue queue for places to jump in), which does automated patch testing as they're uploaded to drupal.org, efforts around improving the performance of the testing framework so that an entire battery of tests can run in minutes rather than lots of minutes ;), etc. You can also do your part by not marking patches RTBC until they a) have tests written and b) you've confirmed they don't break existing tests.
  • Help get the API differences resolved between 5.x/6.x and 7.x SimpleTest module. That'd be this sucker: http://drupal.org/node/241156. If we get this problem fixed, then a) the majority of developers who are working on 5.x/6.x websites can utilize testing for their own modules in code they're familiar with, and not have to re-learn stuff to apply that knowledge to core, and b) we can "back-port" many tests that currently aren't there for earlier versions of Drupal. Oh, and c) we don't need to maintain two versions of testing documentation, which is a definite plus, considering we haven't historically done a good job of even maintaining one set of testing documentation. ;) Speaking of which...
  • Help with documentation. Especially if you're new to testing, lend your feedback to the in-progress docs at http://groups.drupal.org/node/11020 and tell us where you're getting stuck so we can further improve them. catch is doing a great job of this.
  • Figure out standards and best practices, and update the tests to reflect them. Newbies cling to standards because when they start out at least they're going to be in copy/paste/modify mode rather than "Let me whip up a test for that" mode. Keeping our test files consistent and well-written will make them easier for people to understand and lower the learning curve tremendously. Another great place for people new to testing to help out -- clean-up patches for tests. Helps keep the Drupal garden looking fresh and tidy, and also exposes you to the anatomy of tests in a very hands-on way.

SimpleTest has the very real potential to crank the Drupal development community's productivity up several notches, but only if we all pitch in. Hope to see you in the queue and at http://groups.drupal.org/unit-testing!

Comments

It seems to me that, as long as testing is an opt-in process, this problem won't go away. testing.drupal.org will make a huge difference, but perhaps we need another solution in the meantime.

One possibility: Instead of asking people not to mark patches as RTBC until tests are written and all tests have been run, perhaps its time to add some new steps to the issue workflow. Ie instead of going from CNR to RTBC, perhaps it should be CNR => CNTR (Code Needs Tests Run) => RTBC. In this case, at least committing a patch without running tests would be an explicit decision.

True, http://drupal.org/node/156119 already clearly says that RTBC shouldn't be used until testing has been performed, but clearly people haven't fully embraced that practice and, instead, see the transition from CNR to RTBC as largely a code-review step, not a review and test step.

Just a thought.

1. fix all tests
2. testing.drupal.org (or some service) watch cvs commits and upon commit, run tests
3. if testing.drupal.org reports a test failing, someone would roll-back the commit
until it's resolved.

It's really the only fair and sane way IMHO to stay ontop of this. But there is a long way to go to get
there i understand.

It's also important we learn from the failing tests and see if they are failing for good or bad reasons and
if there is something that keeps breaking things over and over that can be fixed once and for all.

Thanks for posting this, you've nailed all the concerns and implications very well!

Submitted by catch (not verified) on

1a. fix all tests
1b. testing.drupal.org fetches RTBC and CNR patches from the queue and marks any that don't apply (~70%) to needs work (in preparation for...)
2. testing.drupal.org fetches all patches from the RTBC and CNR queue, runs them against all tests (and their own if provided), and marks any that break tests to needs work.
3. no need for three, because a patch that breaks tests never stays RTBC for more than a few hours, but run tests on HEAD anyway just in case something slips through 2. Might need a special "don't test this" flag somewhere for particular exceptions.

Submitted by Hank Gaston (not verified) on

I'd like to offer you a quote from Mark Shuttleworth on this:

In his most recent article "Discussing Free Software Synchronicity" http://www.markshuttleworth.com/archives/150


A comprehensive test suite, on the other hand, lets you be more open to big landings on trunk, because you know that the tests protect the functionality that people had *before* the landing. A test suite is like a force-field, protecting the integrity of code that was known to behave in a particular way yesterday, in the face of constant change. Most of the projects I’m funding now have adopted a tests-before-landings approach, where landings are trunk are handled by a robot who refuses to commit the landing unless all tests passed. You can’t argue with the robot! The beauty of this is that your trunk is “always releasable”. That’s not *entirely* true, you always want to do a little more QA before you push bits out the door, but you have the wonderful assurance that the test suite is always passing. Always.

Building code is easy, building a culture is hard.

-Hank

Submitted by webchick on
  • Blog API functionality: 47 passes, 0 fails, 1 exception: issue
  • Site-wide contact form: 120 passes, 4 fails, 0 exceptions: issue
  • Core filters: 48 passes, 12 fails, 0 exceptions: issue (legit core bug)
  • Node revisions: 64 passes, 0 fails, 8 exceptions: issue

Incidentally, if you haven't run the testing module recently, it got a HUGE makeover today. :D

(OK, and I guess technically this is an "itch of the month and counting", but still. Progress. :))