tl;dr: ArcMap's rendering of transparency forces exports to rasterize, causing pixelation. Look at the layer order in your Table of Contents or do not use Transparency in your maps.
I've had this issue for so long it's embarrassing. I would export a map from ArcMap and get very pixelated symbols. At first I just attributed it to ArcMap not being a graphics program and not having the ability to export vector graphics. However, I could see that some lines and text were vector based, and occasionally I would see smooth exports. Why the inconsistency? It was one of those things that bugged me, but wasn't so noticable and important that it made me drop everything and investigate. Finally I had some free time and did some trial and error.
First I played with the export settings under the General and Format tabs. I've only ever modified the resolution before, and have not changed any other setting. They did not seem to have any effect on my exports.
Then I began looking at the specific layers and started doing test exports. In this map, the circle symbols are below the line (light rail) symbol.
Below is a pdf export from a map that is zoomed in to 800% in Acrobat. Note the pixelated circles, but the vector based shields and text. At 100%, the drawing looks okay, but you can still tell it's not smooth and vector based.
Next, I by chance moved the light rail line layer below the circle symbols and below is what happened. Nice vector circles! So does it have to do with layering? That would be odd, given the amount of layers that are in a typical ArcMap document. I noticed that below, the light rail line still looked pixelated.
Ahh - Transparency. On the above exports, the light rail line had 50% transparency applied. Below is the map without any transparency. Vector circles and vector lines.
After doing a bit of follow up research, I found a gis.stackexchange post and an ESRI blog post that mention this limitation.
While I'm happy to finally know what is happening and how to control this export pixelation, it is a bummer. I use transparency quite a bit in my maps now and will need to make design changes in the future to keep things vector.
