I'm starting a new business called Kynetx (nothing to see there yet). As I go through some of the things I do, I'm planning to blog them. The whole series will be here. You may find my efforts instructive. Or you may know a better way--if so, please let me know!
When you start a new technology company, you're going to be talking to a lot of people. You'll want them to get in touch with you. That means phone and email at the very least. In the early stages you can get away with your Gmail or Yahoo! account, but eventually you'll need to look more professional.
The first prerequisite is a name of some kind. If you're a technology company, that means one that you can get a domain name for. In my case, we picked a word that was meaningful for what we're trying to do "kinetics" and then started spelling it funny until we found a domain that worked and seemed short and memorable.
Once you've got a domain, Google makes the rest easy with Google Apps. There's a free edition and a premier edition. When you're starting out, the free edition is plenty good enough. It lets you create gmail accounts that are addressed as email@example.com. You also get Google Docs, Calendar, and so on. There is a 2Gb limit--just like individual Gmail accounts, but that's not likely to be too big a factor early on.
Getting things set up with Google Apps is pretty easy if you're familiar with domain names and can manage the subdomains in your domain easily. You can create up to 100 users on the free account, which should be plenty.
Want a logo? Probably. I'm fortunate to have friends who are good with graphics and I had a friend do one. You can change it later. Right now, it's just nice to have, so pick something and move on. Don't spend a lot of time agonizing over it. If you don't have a friend who's a graphical designer, you might try LogoWorks. I've not used them, but I know people there and have heard good things.
You'll want business cards. While you can probably get away without them, why? They're convenient and cheap. I've used a company called Four Square Graphics several times and always had good results. You can design a business card on the site, but I had a friend who's good with graphics do it and submitted a custom job as a PDF. $40 and 10 days later I've got 250 high quality business cards.
None of this is going to guarantee that you're company's a success of course, but it's some of the stuff you've got to get done. Using something like Google Apps is a no-brainer. I know how to configure and manage a mail server with the requisite spam filters and so on, but why would I want to when Google will do it for me for free?