Remove temporarily assigned officials via Drag & Drop / DEL key (Assignors)

When working assignments, sometimes I want to make changes or remove officials temporarily placed. If, for example,  I mistakenly placed someone in a position I want empty, I cannot easily remove the official from the position. 

I know I can swap it out with another official, but since the drag & drop feature works so well for assigning, why not for un-assigning, especially if the assignments weren't released yet.