After surging in 2009, the country’s managers stalled after developing the world’s sixth-largest hedge fund industry. In an effort to restart, they are trying to move beyond their traditional offerings