I've been trying to get various UNIX tools to install from source on OS X without using Fink. Luckily, it's getting easier (I'm not sure if this is due to newer libraries in Panther or improvements in the configuration of the tools themselves). Everything goes into the /usr/local prefix, to avoid overwriting the system files. You can edit ~/.tcshrc to set this /usr/local/bin and /usr/local/lib above the /usr folders in the PATH search order, as this file is run when Terminal starts up.
You can now install all the libraries needed for Nicotine (gtk+2, atk, glib2, pango, pygtk2 and python2.3) directly from source. Note that gtk+ and gtk+2 can both be installed at the same time (same for glib and glib2), and if something complains that it can't find a version of gtk >1.x, it's actually looking for a newer 1.x version, not 2.x.
Now I have XMMS installed: it opens, but without any plugins (I remember the same thing happened with installing from source in Fink, but the binary was ok), and I still can't get wxpython to install (wxwindows/wxgtk installed from CVS).