Well, I work in a team of two, and we use Team System for everything we do. The integration between all the tools is great. While you could easily use a variety of other single-purpose tools to get everything that's in Team System, you wouldn't have the level of integration. In my experience, the productivity gained from having good integration is worth it.

That said, your results will vary. Getting Team Foundation Server set up is not for the faint of heart, and you really need to have a dedicated machine to run the services, not to mention the resources to maintain that server. If your project makes significant use of developer tools other than VS 2005, you won't get as much benefit from using Team System, since you won't have the tight integration with those other tools.

I don't know if it's right for your team - but I do know that having used it for over a year now, I wouldn't start a new project using anything else if I had a choice!