Magnetorheological fluid-based (MR) brakes have been preferred in the design of an actuation system for a kinesthetic type of haptic device. However, having an input (current) output (torque) hysteresis relationship is an undesired property for MR Brake. Therefore, two deep learning methods are employed to model the hysteresis of MR Brake. Moreover, a data pre-processing step is proposed to increase the variety of input signals. A hysteresis compensation model approach to account for the nonlinear behavior of MR Brake is also proposed, and the model is experimentally validated. After the forward hysteresis and hysteresis compensation modeling validations, a hybrid actuation system (HAS) consisting of an active actuator and an MR Brake is presented. Another MR Brake's tradeoffs, namely off-state torque and slow response are investigated and resolved by HAS. The transient behavior of MR Brake is analyzed, and a mathematical model is proposed to mimic its transient response behavior. It was found that the performance of the proposed model is better than that of the conventionally used first-order transfer function. Then the HAS is constructed. The active actuator is used to compensate for the response's speed and eliminate MR Brake's off-state torque. The off-state torque is largely eliminated from 0.178 Nm to 0.008 Nm, the dynamic range is enlarged from 15 dB to 42.4 dB, and its time constant is improved from 69.6 ms to 4.4 ms when the HAS is used instead of just an MR brake.