For closed loop control, I like to equate it to Cruise Control in your car. You can set your cruise to 55MPH, but if you tie the feedback to the RPMs (motor drive), not the speedometer (tire speed, what you are trying to close the loop on), you will get results similar to yours, i.e. not accurate.
The encoder feedback for closed loop control must be tied to the actual thing you are measuring, not the motor side.
You can replace the motor, replace the tires, replace the transmission, and none of that will fix your Cruise Control. You gotta move the encoder from the motor drive to the tire (i.e. the other end of your drive system).
