Phrase-based machine translation can be configured to produce alignment data that indicates which machine translated target language words correspond to which original source language words. In most prior work that examined the efficacy of post-editing machine translation, post-editors were presented with machine translations (and in most cases the original source language sentences) without also being presented with source-to-target alignment links. We select four news articles, and ask six Russian-English bilinguals and eleven Spanish-English bilinguals to post-edit English machine translation results, in some cases using alignments and in other cases without. We obtain human adequacy judgements of the post-edited sentences, and demonstrate that when machine translation quality is low, post-editing quality is consistently higher, by a statistically significant amount, when bilingual post-editors are presented with alignment data.