Whereas Smashwords takes a percentage of every sale in the standard way (generally 15% off the top, leaving 85% of the net profits for the author), BookBaby's approach is to charge a flat-rate fee up front, and take nothing from each sale. Consequently, for authors selling a significant volume, this may well be a better way to go. But determining where the sweet spot is will take some careful math, and not a little intuition (something akin to futures forecasting or mystical insight).
For example, at their standard title setup fee of $149 (currently cut to $99 per ebook) you'd have to sell just 19 books a week at $2.55 each (with a 70% royalty in place at each retailer) to recoup your cost in one month. Any profits after that are yours, as BookBaby passes on 100% of the retailer's wholesale cost to the author, taking no fees other than the initial costs, plus a $19 annual fee after the first year to keep the title "in print" (but see below for the caveats!). This also includes free ePub conversion from .doc, .html or .txt files.
They are assuming, of course (and rightly so), that most ebooks will never sell that many in a year (or ever), let alone a month. But if you're one of those self-pubbed authors who have reached a volume of roughly 80 units a month per title then this might be a serious consideration. Of course, you can sell half that and cover your costs in two months, or however long you like; but as a business model, you'll have to decide where that cutoff lies, and at what price you can guarantee that quantity of sales. Otherwise you're just shooting craps. And good luck with that as your strategy.
As with every business deal, there are caveats and stipulations, and not a little fine print. The base fee is for text-only ebooks with up to 30 chapter headings, and includes an interactive table of contents. Anything after that costs extra (including $19 for an ISBN if you need one, which is required). So for graphics, charts or tables, additional chapters, or multimedia features such as embedded audio or video the fees accrue rather rapidly. Again, you'll have to decide for yourself at what point the cost offsets the benefits.
If you just want to get your ebook on Amazon or B&N your best bet is to deal directly with them. But Sony and Apple are a little stricter and require leaps through far more hoops to get into their stores. A one-time fee will get your ebook out in four major retail outlets, and net you a larger share of the back-end than Smashwords, whose net royalties equate to roughly 60% of list. That extra ten percent could make a difference down the road.