A recent op-ed essay in the NYT about the end of summer noted that most academics work a lot more than 40 hours a week, though we have somewhat erratic schedules and our work is not typically 8-5. I have some colleagues who are strictly 8-5ers, but am not one of them. I would have a hard time with a strict schedule, and with working so few hours. I added up my hours in a typical week, and easily got to 60+. Maybe I am less efficient than some of my colleagues, but even so, I wouldn't feel satisfied with working so little. It takes about 40 hours for teaching and administrative work each week, and then there's the research I want/need to get done. It doesn't really feel like it's so much time because the work is so varied, and much of it is interesting and fun.
That said, I am not looking forward to an upcoming faculty 'retreat'; i.e., an all-day faculty meeting on a weekend. Overall, I am fortunate that my department has very few evening/weekend activities that are not family-friendly. Some of my colleagues in other departments at this university have lots of such activities, and they are always torn between not going (possibly harming their standing in the department; stressful for tenure-track faculty) and going but having less time with their kids/spouses (+ the expense of baby-sitters). I don't have to deal with that much, but these retreats are different. Even if we could find an all-day babysitter who could come to our house as early as 8 a.m. on a weekend, it would probably cost about $100 for my husband and I to attend this retreat thing. I don't mean to be cheap, but the general idea of paying to go to an all-day faculty meeting is annoying. The previous chair refused to pay a student to babysit faculty offspring during these retreats because he says it wouldn't be 'fair' to those faculty without kids. I think that reason is bizarre. In the past, my husband and I have flipped a coin to see who goes and who doesn't (the loser of the coin toss attends the retreat).