You stayed very close to where I now live. There is a bar there that the show was based on, on Beacon Street.
Boston's definitely not known for its food-it's geared mostly towards college students. The North End and Chinatown do have a few good casual places, though. And honestly the reason I've stuck around is because of how easy it is to get around without a car.
IMHO the best vacations are ones where you just wander around and see what you may find. One of my favorite things to do is walk through the Public Garden, then down the park that runs down Comm Ave.
One of these days I want to walk the Freedom Trail and go on a Swan Boat ride-that'll probably be the next time I have friends and/or family in town.