With up to 12 spring-damper groups distributed in the actual area of a rail pad, different fastening models are developed in this paper to include the nonuniform pressure distribution within a fastening system and model the constraints at the rail bottom more realistically for the purpose of high frequency dynamics between vehicle and track. Applied to a 3D transient FE model of the vehicle-track interaction, influence of the fastening modeling on the high frequency dynamic contact forces at singular rail surface defects (SRSDs) is examined. Two defect models, one is relatively large and the other is small, are employed. Such a work is of practical significance because squats, as a kind of SRSD, have become a wide spread problem. Results show that the fastening modeling plays an important role in the high frequency dynamic contact forces at SRSDs. Supports in the middle of the rail bottom, modeled as spring-damper groups located under rail web, are found to be most important. The less the rail bottom is constrained or supported, the more isolated the sleepers and substructure are from the wheel-rail interaction, and the more kinetic energy is kept in the rail after impact at a SRSD. Rolling speed is also varied to take into account its influence. Finally, based on the results of this work, influence of the service states of the fastening system on growth of relatively small SRSDs is discussed.