Trouble shooting Cron?

It ducked!
Monday morning the sun is shining and your email box is full of emails saying cron is failing on one of your sites.
In this post I will look at what is going on, how you can troubleshoot it and fix it and how life may be made easier in the future by a) adding a bit of logging to cron and b) features in Drupal 7.

The emails are from the cron daemon and tell you that something is up with cron (in this case MYSQL has timed out)

(In another post I will review how to deal with this issue. But for now let's see why cron is stuck...)

So you head off on your morning sweep and sure enough the status report agrees:

So you try and run cron manually and

Then investigate your Recent log entries and sure enough they are topped with:

and each of these tells you that

So the problem we have is that cron failed at some point and ever since it hasn't run because it is reported as still running - and therefore is stuck.
In 'Drupalese' cron is actually a set of routines invoked by the cron.php page. This provides a generalised scheduled task runner that modules can hook in to get routine housekeeping done. Cron is in fact an operating system feature that enables you to run scheduled tasks. When you set-up a cron job (through your CPanel) you are getting the OS to call your cron.php page which in turn then does the work.
CMS systems as with all web sites are request driven and don't have a 'running background process' and therefore no way of ticking away in the background to use timers or some such to get routine jobs done. You have to manage that externally using a cron job or other request to your website. Note also that since cron.php is just another web page - anyone can call it and could in effect undertake a denial of service attack by calling it multiple times! (See our article on Securing access to files on your website) also it is encouraging to note that in Drupal 7 cron requires a hashed parameter than ensures it is called securely.

So back to our problem. If you scour the web for a solution you will find many who have the same issue.
Many suggest clearing the cron variables from the MySQL variables table:

This can be done through PHPMyAdmin or using MYSQL Query Browser.
..and then re-run. However, often that isn't enough because the variables may be cached and don't get re-read.
So it has no effect. You can clear the caches but that may not work either.
I also tend to be a bit wary of pulling the plug out in this way.
Others suggest (and this often does the job) temporarily updating the common.inc file to reset the semaphore flag.
see Drupal Tip: Forcing cron to get un-stuck @ ssexton's blog
Locate the following lines in /includes/common.inc (around line 2677).

..Note that some of you reading this might be tempted to just leave the semaphore disabled. And if you can be 100% certain you'll never get two crons running at the same time, it is probably safe to do so. I wouldn't recommend it though; instead, think of this as an emergency measure to get the cron_semaphore un-stuck when doing so using the database method doesn't work. - see Drupal Tip: Forcing cron to get un-stuck @ ssexton's blog

and I don't argue with that!

Whilst either of these methods may sort you out - you must take the bitter pill that whilst you have solved the issue you haven't really addressed the problem and cron may just as easily get stuck again soon. Also solutions that edit database fields or update (hack) the core (lifesavers though they may be at times) are short term and not sustainable. So what's really required is:

1) a facility in a future Drupal version that enables you to manage this. (You can use Devel to reset variables - but IMHO that is pretty much the same as hacking the database or core inc files - neither are really good practise nor get to the root of the problem). Maybe some more granular logging or even better 'exception handling' to allow exceptions to be recorded at the point of occurrence so they can be focused on without having to debug and 'hack' logging in to the core. I have yet to evaluate the exception handling in Drupal 7. See Drupal 7 feature list.
2) In the meantime some troubleshooting to find out why this is occurring and hopefully fix the cause not the symptoms.

Logging to troubleshoot cron

What is most likely to be the issues here is that one of your many modules is stalling and cron is then timing out and getting stuck.
Common causes appear to be: search module, filters and rogue PHP code see http://drupal.org/node/123269

So once you have fixed the immediate issue and had a cup of coffee you should get stuck into some monitoring and debugging to see if you can't find out why this is happening.
A problem straight away is that you don't get much feedback out of the box and the first step is to add some logging to see if you can't determine where in the cron black hole the problem is occurring.
See the following suggestion from liquidcms at http://drupal.org/node/123269#comment-644012

Note there is a 'solution' "HOW TO FIX: Attempting to re-run cron while it is already running" posted at http://drupal.org/node/360836 which involves disabling all modules and then re-enabling them one at a time. IMHO the above solution by liquidcms is better since you get the same insight without the hard work and you can leave it unattended. It is not always the case that cron fails every-time in this way and therefore a logging solution is more suitable. Maybe Drupal.org needs a StackOverflow type Q&A facility? So you get to vote the best answer...
:: Update ::
For Drupal 7 try this in /includes/common.inc line 5049 in drupal_cron_run():

In many cases this will highlight the specific problem or at least which module it is that was hooking into cron when the failure occurred.
You are then left with trying to disable of troubleshoot that specific module.
If that doesn't work then more detailed logging may be required.
There is an excellent article on this at Drupal Cron Stuck? Log the progress by Ebizon Netinfo - but I haven't neede to go that far yet.

There are some modules that may help you debug this too. I haven't tried them but they are referred to a lot in many issues queues.

• SuperCron - which gives a great deal of control over running cron such as order of modules and blocking cron by IP address.
• Elysia Cron