That is something I haven't considered. I'm not sure how I can check the alignment of the swing arm to the frame without some special jig set up. I believe the swing arm has bearings on each side. I have seen posts that said they had to replace the swing arm to fix the rear bearing failing which I would assume is a factory machining error.
The next thing I was thinking to do is to remove the drive shaft and see if moving the angle drive forward to get the belt tension set will let the two sprockets stay aligned with each other. I can only move mine forward so far and it seems to be stopped by the drive shaft but that leaves the belt a little on the loose side. I've been avoiding taking the whole swing arm and angle drive out but that looks like where it is heading.
Having had my swing arm off and installed myself, it would be my guess that your issue lies at the angle drive not being square and that is where I would concentrate my efforts. The pivot shaft has a needle bearing on the belt side and a ball bearing on the opposite. The assembly of the pivot shaft, bushings and bearings makes it nearly impossible for there to be a issue in that area. The only issue in the pivot shaft might be the snap ring holding the drivers side bearing, but you stated that the swing arm does move right and left so I doubt this is broke or missing. The first area of suspect in the swing arm would be the needle bearing on the pulley side of the rear axle in which you have made no comment and I assume is in good condition. This bring me back to the angle drive alignment being suspect. I had used lasers to align pulleys and belts on motorcycles and worked very well, especially when doing wide tire change outs. I have not needed to do this as of yet on the slingshot, but I would place a line laser square on the angle drive sprocket and take measurements from the front and back the rear drive pulley. I would prove first the pulleys are not parallel before tearing apart. Your comments does lead me to believe something is wrong in the angle drive area especially when you cannot move the drive forward enough to get the belt tight.