Hey there @yapith what version of the Authorization Extension are you using? Are you seeing any errors in the console when removing the groups/roles/permissions? Thanks in advance!
I later realized that auth0 roles are application-scoped unlike groups. I was also mislead by another post about possible bug in the authorization extension. Apologies for the confusion.