If the guy in the Isle of Man was selling to a fish market then the legislation would apply, small difference but a difference nonetheless.
see diagram on page 13 of the EFSA legislation regarding this, nice little icons depicting who is handling the fish in the supply chain and at the point legislation is in effect.
https://efsa.onlinelibrary.wiley.com/doi/epdf/10.2903/j.efsa.2020.6091
Because he is selling directly to the public then there is little legislation in that regard from the EU, other than the member countries' rules are to be applied, the EU are more interested in the business-to-business transportation side of things.
A cynical person could argue that the UK Gov cares more about the direct transportation of fishery products to you, the consumer, than the EU does.