A quick calculation will help you determine whether buying a New York Pass makes financial sense: Divide the cost of the pass by the number of days you’ll use it (i.e. you may choose to buy a 7 day pass, even though you’ll be here for just 5 days), to come up with a "per day breakeven" cost. This is the amount you'd have to spend on sightseeing each day in order to break even on the New York Pass.
Most attractions featured on the pass cost $12-18 dollars. There are a few "big ticket" items (Empire State Building, Circle Line Ferry, Madame Tussauds) that cost more. I recommend using $15 as a guideline -- divide the cost per day by $15, and that should give you a rough idea of the number of attractions you'd have to see to break even.
An aggressive sightseer can get to 4 or 5 attractions in one day. This will entail a long, tiring day of activities, but it is possible. It's also unlikely that you'll be able to maintain this pace for more than a day or two at a time.
A typical sightseer can visit 2 or 3 attractions in one day. This will leave you time to enjoy meals, experience the sights you visit and room to include some non-New York Pass activities, like Broadway shows, nightclubs or musical performances.
A leisurely sightseer will probably check out 1 or 2 New York Pass attractions a day. This leaves visitors tons of time for shopping, leisurely meals and little rushing around. For most leisurely sightseers, the New York Pass isn't a great idea, unless you're here for a full week and contemplating purchasing the 7 day New York Pass.
Of course, if you have a multi-day pass, you might have a couple of "aggressive" days and a couple of "leisurely days" and the New York Pass may be a good purchase.
In my experience, most people overestimate the number of things they'd like to see and do on vacation in New York City, so if you're coming up close to the break even point, it might make more sense to pay a la carte for your sightseeing. If you're going to be in town for a week, the New York Pass is a great deal, especially because you can evaluate a few different attractions and even return to the ones you like the most.