What To Do With Simpletest Failures And Exceptions?

Submitted by Bevan Rudge on January 8, 2008 - 4:19pm

As part of an initiative at CivicActions to create and contribute more simpletest unit tests, I've just started playing with simpletest module (CVS head) on drupal 6 (CVS head). As part of grokking it I ran all of the core tests. The result: "1366 passes, 76 fails and 24 exceptions." Full results attached. I'm not sure what exactly this means. There is plenty of documentation and articles ( another article about why simpletest is useful and how to write simpletest unit tests, but none (AFAIK) about how to run them and what to do with results. Where does one start to help get the issues resolved? Please excuse my ignorance and lack of knowledge on this. I've never done any unit testing before but I've read a bit about it and it was covered briefly in a computer science paper I took in 2007.

Attachment	Size
simpletest.html	349 KB
simpletest DB.sql_.gz_.txt	113.14 KB

Bevan Rudge's blog

Comments

This is not the proper place

Submitted by chx on January 8, 2008 - 7:03pm.

Thanks for testing, but it would do a lot more good if you would post these as separate issues into the simpletest queue. I am sorry just but this is just FUD. For example, the image module tests fail... because image module is not installed. Also, are you sure your install is not the one to blame here? Seems you have search module half installed...? Care to rerun from a fresh HEAD...? Also why do I need to bother with a maths captcha on every comment when I needed to solve one on registering?

take him at his word

Submitted by greggles on January 9, 2008 - 2:56am.

chx - take him at his word, he's honestly curious what do with these exceptions. It can't be FUD if he doesn't know what to do and if he doesn't know what to do that points as much to lack of docs/guidance as it does Bevan's knowledge of process.

Every question like this deserves investigation about what caused the question. As you pointed out, he needs to fix his image module installation and search module installation. Well, that indicates two areas where the tests could be improved (check for installation prior to running).

(I do agree, though, registration+captcha+spam filter is quite overboard)

Yeah, I think the question

Submitted by Owen Barton on January 9, 2008 - 3:37am.

Yeah, I think the question here was really 'where should the issues be filed' - in core or in simpletest (or, in this case is more investigation needed first).

I just fixed the captcha misconfiguration, so it shouldn't bug you too much now :)

As someone who is used to

Submitted by cabbiepete on January 9, 2008 - 3:41am.

As someone who is used to unit testing and within cms frameworks like zope3 and is also kinda new to Drupal but including simpletest based unit tests in my work.

I think its important that all the tests pass for core with the base install. If a module isn't enabled for a test that needs to be it should be enabled and then disabled, and if we are worried that people will get confused when the actual function doesn't work for them we can post a drupal message in the test saying 'XXX module enabled' and then 'XXX module disabled' so that the user can have some feedback.

Logically is there some reason not to do this?

Unit tests passing are one of the signs that everything setup and working correctly. Running the tests is often the first thing I will do when downloading software to get some confidence in it.

irc: cabbiepete
Pete

Give me a break

Submitted by chx on January 9, 2008 - 7:48am.

"But I think it approximately means: There are 24 issues that are probably major-critical bugs in drupal core (the sort of thing that holds up d6's release)" -- if posting this to the Planet is not FUD then I have no idea what is. Whatever issue queue he finds works but posting this to a place which is aggregated all over the Net is not the best idea to say the least. Not to mention that Bevan (because of his usability project) is already close contact with me, what about asking me first? Both Rok and I investigated and all we found are two totally minor core issues -- which is great to be found, of course. We were unable to replicate the exceptions. If you take a look it's well visible that search module is enabled but not installed and we have no idea how he succeeded to do that -- but this is the kind of stuff that bug reports and followups are for.

Oops

Submitted by Bevan Rudge on January 9, 2008 - 1:30pm.

Chx, > it would do a lot more good if you would post these as separate issues into the simpletest queue Thanks for answering one of my questions! :) > are you sure your install is not the one to blame here? No I'm not sure. That's one of my questions; How do I start debugging and fixing this? > Seems you have search module half installed...? Care to rerun from a fresh HEAD...? I did that test running from fresh cvs head of both d6 and simpletest module and svn trunk of simpletest package from sf.net (could this be the cause?) with a freshly installed database with no contrib or additional modules (except simpletest and simpletest automator), changes settings or customizations. I've attached the database that I ran the tests on if you want to followup further. I haven't modified or used the site running on that database since I ran the test. > "But I think it approximately means: There are 24 issues that are probably major-critical bugs in drupal core (the sort of thing that holds up d6's release)" -- if posting this to the Planet is not FUD then I have no idea what is. I wrote this post in a bit of a hurry. I should have spent more time reviewing it and taking care to consider any potential misunderstandings. I can see how it could be misunderstood as FUDding. However that was not my intention and I'm sorry for not taking care enough to be sure it couldn't be taken that way. I've now updated the post to reflect a little clearer it's purpose and intent and removed any potentially FUD prose. > Not to mention that Bevan (because of his usability project) is already close contact with me, what about asking me first? To be honest that didn't occur to me. I didn't realize you are so involved in the simpletest project. Also I don't stay online all day in skype or IRC and don't seem to have you on other IM protocols.