To remove the assignee or watcher of a task, just click again on the same user. This way you can also set a task as unassigned again.
To add/remove Watchers, first, click on the Arrow sign beside the Watcher option.
To add/remove the Assignee click on the Arrow Sign beside the Assigned person and either click on the same person to Unassign the task or click on another person to assign the task to someone else.
