You can disable whole classes of algorithms on the command line with the LTC_NO_* defines. From there you can manually turn on what you want to enable.
The following build with GCC 3.4.4 on an AMD64 box gets you AES, CTR mode, SHA-256, HMAC, Yarrow, full RSA PKCS #1, PKCS #5 and ASN.1 DER in
roughly 40KB of code (49KB on the ARMv4) (both excluding the math library).