The solution is to use the tool Tag&Rename. This shareware was actually made to manage and edit your mp3 music collection but it has also the feature to change the cover art of mp4 and wmv files - other types like avi are unfortunately not even readable.
The edit overview |
As you can see in the screenshot above the handling of this tool is quite easy. Just go through the folders to select the mp4 file where you want to change the thumbnail and 'right-click' to select 'Edit File Tag'.
Now you need a file for the cover art where I would recommend to use the snapshot feature of the VLC media player (in case you want to use a screenshot from the video clip...) .
After you have chosen the picture for the thumbnail drag & drop it into the space for the cover art (as seen in the screenshot). Finally you only have to confirm it with "Save" and it's done.