Can I remove an Action Item from my list?

Yes. Click the flag icon on the note and select "Unassigned" to remove the assignment. This will also remove the note from the assignee's Action Items list.