Although traffic flow has attracted a great amount of attention in past decades, few of the studies focused on heterogeneous traffic flow consisting of different types of drivers or vehicles. This paper attempts to investigate the model and stability analysis of the heterogeneous traffic flow, including drivers with different characteristics. The two critical characteristics of drivers, sensitivity and cautiousness, are taken into account, which produce four types of drivers: the sensitive and cautious driver (S-C), the sensitive and incautious driver (S-IC), the insensitive and cautious driver (IS-C), and the insensitive and incautious driver (IS-IC). The homogeneous optimal velocity car-following model is developed into a heterogeneous form to describe the heterogeneous traffic flow, including the four types of drivers. The stability criterion of the heterogeneous traffic flow is derived, which shows that the proportions of the four types of drivers and their stability functions only relating to model parameters are two critical factors to affect the stability. Numerical simulations are also conducted to verify the derived stability condition and further explore the influences of the driver characteristics on the heterogeneous traffic flow. The simulations reveal that the IS-IC drivers are always the most unstable drivers, the S-C drivers are always the most stable drivers, and the stability effects of the IS-C and the S-IC drivers depend on the stationary velocity. The simulations also indicate that a wider extent of the driver heterogeneity can attenuate the traffic wave.