I think an array framework like Accelerate, Knead, or Repa, is the way to go if you want to be staying in high-level language land. EDIT Oh yeah and Massiv! I may still be forgetting some …
You can write plusPoly as plusPoly add = alignVectorWith \case {These a b -> add a b; This a -> a; That b -> b} using alignVectorWith from the align package. Probably not any faster, but definitely clearer I think.