In the recent weeks, rtorrent 0.19.2 has been crashing at launch on our home mac unpredictably but with increasing frequency with this stack.

rtorrent                do_panic(int) + 227
rtorrent                SignalHandler::caught(int) + 339
libsystem_c.dylib       _sigtramp + 26
libcrypto.0.9.8.dylib   RAND_version + 128
libtorrent.17.dylib     torrent::DiffieHellman::store_pub_key(unsigned char*, unsigned int) + 41
libtorrent.17.dylib     torrent::Handshake::prepare_key_plus_pad() + 67
libtorrent.17.dylib     torrent::Handshake::event_write() + 315
libtorrent.17.dylib     torrent::PollKQueue::perform() + 261
libtorrent.17.dylib     torrent::PollKQueue::do_poll(long long, int) + 264
libtorrent.17.dylib     torrent::thread_base::event_loop(torrent::thread_base*) + 293
rtorrent                main + 13570
rtorrent                start + 52

I suspected some corruption to be the cause - there were a few dirty shutdowns caused by a misconfigured apcupsd coupled with a spate of rain-induced blackouts over the summer. I had to take a look when it became impossible to start it after another reboot and upgrade to the latest 0.19.3 on homebrew didn’t help.

Googling the stack gave me very few results, the most informative being a closed as invalid bug against an ancient version on the rtorrent bug tracker. Following a poster’s suggestion on encryption being the culprit (somewhat obvious in hindsight looking at stack function names), I too was able to reproduce the crash somewhat reliably with

rtorrent -n -s session.crash -p 13435-13450 -o encryption=require

The issue outlined in the bug report was that libtorrent tries to use a Diffie-Hellman key generated by openssl but finds the fields null. It triages the problem to an error thrown by openssl when libtorrent’s DH_generate_key function calls openssl to create the key but doesn’t offer any solutions. After some rusty dtrace calls, I went the gdb route since I needed more familiarity with it anyway. So I built debug versions of rtorrent, libtorrent and openssl ¹ linking them appropriately and stepped through. This was a bit painful since the problem didn’t manifest always and I had to restart the whole thing when the key generation was successful ². But with source at hand, it was easy to figure out the function calls and with appropriate breaks, I could see that the failing function was ssleay_rand_bytes in crypto/rand/md_rand.c.

525 else
526     {
527     RANDerr(RAND_F_SSLEAY_RAND_BYTES,RAND_R_PRNG_NOT_SEEDED);
528     ERR_add_error_data(1, "You need to read the OpenSSL FAQ, "
529         "http://www.openssl.org/support/faq.html");
530     return(0);
531     }
532 }

Following the very helpful FAQ link, I was surprised to read that the “PRNG not seeded” error indicates a failure to use /dev/urandom or /dev/random. I checked the permissions and ran cat to ensure they were readable. Next step was obviously opensnoop and it showed the problem right away.

# opensnoop -n rtorrent -a -x
UID   PID COMM       FD ERR PATH                 
                    
501 34017 rtorrent  -1  24 /etc/hosts           
501 34017 rtorrent  -1  24 /etc/hosts           
501 34017 rtorrent  -1  24 /dev/urandom         
501 34017 rtorrent  -1  24 /dev/random          
501 34017 rtorrent  -1  24 /dev/srandom         

It’s out of fds (err 24). Looking at ulimit, it seems OS X limits open file descriptors to 256 per process as opposed to Linux’ 1024. Increasing the limit (ulimit -n 1024) immediately solved the crash.

Now my rtorrent list has grown steadily big as Humble Bundle keeps offering one enticing bundle after another. But reading about rtorrent’s max_open_files option, they seemed to have resolved bugs related to fd limits and large torrent libraries. Possibly the much lower defaults on mac exacerbated some latent issue not visible in Linux.

As a side note, here’s an outline of a way to create custom homebrew builds

• Run ‘brew edit formula’
• Append Dbg to class name i.e Formula becomes FormulaDbg
• Add ‘env ::std’ line to the class body somewhere before install function def. This is to change brew superenv so that custom flags aren’t removed.
• Add ‘ENV.append ‘CFLAGS’, “-g”’ lines to use custom env variables.
• Change configure arguments if needed
• Write the file somewhere new with -dbg appended i.e /tmp/formula-dbg.rb
• brew install –interactive /tmp/formula-dbg.rb - drops to a shell with unpacked sources. Remove interactive flag to simply install.

1 : debug openssl wasn't as straighforward as building with CFLAGS="-g" since it uses a custom perl config script which doesn't include a 64 bit darwin debug target. I just edited the script to add it as per [this patch](http://www.mail-archive.com/openssl-dev@openssl.org/msg31754.html)

2 : gdb's record functionality would've been useful here but Xcode's gdb I had was too old to have this feature. The one I built with homebrew failed to add breakpoints saying address not found. Well ...