If you want to remove roles that are allowed to see a poll, you can do so via /configure-poll update removerole. Simply run the command and select the poll and then which role you want to remove from it. Only users that have at least one of the roles designated as required roles for a poll will be allowed to see it or vote in it. If no roles are required for a poll, it is visible to everyone.
If the poll is holder-based, ownership of the required token at snapshot time is required in addition to having one of the roles.