I've been stucked with a job which took a lot longer than what it is suppose to take to complete. A friend of mine shared his gems of tricks on how he troubleshoot it, and with a little bit of tweak to my taste, I've came up with a flow(
well, almost all of it still from him :p) which I'm logging it here so that I won't forget it.
#1 Find the offending process id (_pid_)
ps -aux | grep job_name
#2 Look at the entire hierarchy of the pid and look at where it stops
pstree -pulna _pid_
#3 Look at the trace of the running program.
strace -t -s 22222 -p _pid_
#4 Look at the read/write IO activities
cat /proc/_pid_/io
Useful Links
http://www.linux-tutorial.info/modules.php?name=MContent&pageid=84
http://chadfowler.com/blog/2014/01/26/the-magic-of-strace/