[subset] convert subset input flags into bit flags. Store the flags in a bit set. Updates the public api to work with the bit set directly.