The NYC trio's sixth album is a bold and artful evolution, though it's so claustrophobic you may long for a sense of release