It's an interesting question.
In a single market, the freedom of movement for workers is an essential part of the economic equation that creates a single market. Without the freedom of movement of workers a single market cannot truly exist. In pure economic theory it allows the movement of people from high areas of unemployment to low areas (an essential aspect of trade in equalising economic conditions across the single market), and thus is both welfare enhancing and broadly wealth distributive.
Furthermore, in a single market with a single currency, movement of labour is one of the few adjustment methods given that currency movement is obviously not an option.
Therefore it is clear why the EU will not permit single market entry without agreement to the freedom of movement, something that the British Government reconfirmed in Cameron's new settlement deal in February 2016 .
I see the thrust of your argument. If all members of the entity were playing on the same field, trade would / might be enhanced. This would involve true monetary union, common taxation policies, lack of borders etc etc.
And herein we have the problem which financiers choose not to address: of all the nations (or regions) of Europe, the U K is the least suited psychologically to being subsumed in a superstate. National consciousness here has not been bruised by invasion. It is assertive rather than subservient. It has a happier experience of global adventure and an expectation of success both militarily and economically.
Losing the pound, "allowing Brussels to dictate our taxes" (as elements of the media would portray it) and abandonment of exclusion from the Schengen agreement are politically impossible as is continued immigration at current rates. And yet without these things, we cannot truly be said to be a unified economic entity if your thesis is followed. Trade still goes on.
It was pretty obvious that the real manifesto was (as has been openly stated) "Ever closer union". (A must have if the belief is that this necessary for economic development.) That would, I'm sure you'll agree, have been rejected out of hand by a much larger majority.
My own belief is that the creature is an ongoing experiment. There isn't a huge amount of economic theory written to cover the kind of enterprise the E U is engaged in and certainty in outcomes is misplaced.
Without adding an extra couple of chapters to this ridiculously long post, I'll leave the brotherhood of man benefits of the union for now.