As I understand it, RHEL 5.3 is supposed to have pulled in the change to use PTHREAD_PROCESS_PRIVATE by default with the various pthread_mutex calls. While testing my recent requeue_pi code, I was seeing some unexpected behaviour on RHEL5.3 that didn't occur on FC10. Specifically, regardless of what I set the mutexattr pshared attribute to, glibc would make the shared versions of the futex calls. I've written the attached testcase to illustrate. It creates a PI mutex, and will set the pshared attribute to whatever you specify in the options, or use the OS default if you use no options. Using strace -f you can see which futex syscalls glibc used.
Compile with:
# gcc -lrt -lpthread priv-mutex.c -o priv-mutex
Note: the ETIMEDOUT is intentional in the output below.