It is a problem even if you could modify your vote as a lot of the time I am not sure if I have up voted or down voted a post.

I love HN and its low fi design, but this is the one thing that I think takes away from the community spirt here. Intent should always be clear.