Roader, the network analyzer har directional couplers on the ports, making it possible for the analyzer to separate forward and reflected power when measuring s11 and s22.. For a scalar analyzer (that only can measure magnitude); for instance, if forward power out on port 1 is 0 dBm, and the reflected wave is -10 dBm, the return loss can then be calculated to 10 dB, or, in linear fashion, gamma is 0.1.
For a network analyzer, also the phase between the stimuli ( forward power) and the recieved (reflected power) can be measured. Then, not only the magnitud can be measured but the reflection in complex form, as magnitud + angle, or real+im component.
The same goes of course for transmission measurements, s21 and 12, but that is easier to understand.
There are a lot of good tutorials on network analyzers, explaining there way of working.
With some math skills in complex numbers the secrets will be revealed for you in no time :)

Hi

I still dont understand how the reflected wave is measured with a coupler. Is there some pictures or documents you could link to?

OK, Roader, take a look at this: https://www.microwaves101.com/encyclopedias/directional-couplers
A directional coupler, think of it in this way; think of a road, going north-south, with traffic in both directions. That road is the main path of the coupler, the signal source is in the north and the load in in south.
Then, add a coupled path; a measurement device that registrate the traffic only in north-south direction. Thats your forward coupled path. You can also add a coupled path registrating the traffic south - north. That would be the reverse coupled path.