@daleslad That will remain there permanently unless you manually go in and uncheck the checkmark next to the song.
Also, if you are listening on a mobile device, there should already be a prompt that appears to prevent you from adding duplicate tracks to a playlist.
However, if you are listening on a computer, that is not currently a feature yet.
I recommend upvoting this existing feature request if you would like to see that option added to web as well: Feature Request: Notification for duplicate songs within playlists on web
Hope this helps!