The San Francisco hedge fund firm, known for its volatile returns, is experiencing hefty redemptions following a period of employee turnover and big performance losses.