I see two ways to do this.
Store the REAL division of the numerator (R0) and denominator (R1) in R100, then calculate the "modulus" from that value in R100:
MATH R100 "ABS(R0) / ABS(R1)"
MATH R101 "(R100 - TRUNC(R100)) * ABS(R1)"
Another way is to use integer math, if you know your floating point numbers are within a relatively small range. 2's complement integers have a range of basically 0..2Billion. Say your numerator and denominator floating point numbers are in the range 0..10,000 then you could multiply both by 200,000.0 and still be "in range" of a signed DWORD. Do the integer modulo with those two integer values, then divide the integer modulo result by that "factor" 200,000.0 as a REAL. If the range of your numerator and/or denominator is larger than 10,000, then you would need to reduce the factor. The smaller the factor, the less precise your result will be.
MATH R101 "(ABS(TOINT(ROUND(R0 * 200000.0))) % ABS(TOINT(ROUND(R1 * 200000.0)))) / 200000.0"