The implementation of flexible instruments in surgery necessitates high motion and force fidelity and good controllability of the tip. However, the positional accuracy and the force transmission of these instruments are jeopardized by the friction, the clearance, and the inherent compliance of the instrument. The surgical instrument is modeled as a series of interconnected spatial beam elements. The endoscope is modeled as a rigid curved tube. The stiffness, damping, and friction are defined in order to calculate the interaction between the instrument and the tube. The effects of various parameters on the motion and force transmission behavior were studied for the axially-loaded and no-load cases. The simulation results showed a deviation of 1.8% in the estimation of input force compared with the analytical capstan equation. The experimental results showed a deviation on the order of 1.0%. The developed flexible multibody model is able to demonstrate the characteristic behavior of the flexible instrument for both the translational and rotational input motion for a given set of parameters. The developed model will help us to study the effects of various parameters on the motion and force transmission of the instrument.