Why does Odysseus decide to leave circe's island?

Why does Odysseus decide to leave circe's island?
his men convince him to leave after the guy falls off of the roof