The importance of evaluating wellbore stability in analyzing and estimating the efficiency of drilling directionally into oil and gas reservoirs is well known. Geomechanical data and failure criterion can be used to model and control rock mass behavior in response to the stresses imposed upon it. Understanding and managing the risks of rock mass deformation significantly improve operational processes such as wellbore stability, sand production, and hydraulic fracturing. The modified Lade failure criterion is established as the most precise failure criterion based on previous studies. By combining it with tensions around the wellbore, a novel relationship is derived for determining the stable mud window. To investigate the accuracy of the new relationship, two geomechanical models (neural network and empirical correlations) for a one-directional wellbore are developed and their performance compared with two other failure criteria (Hoek–Brown and Mogi–Coulomb). The geomechanical parameters (Young’s modulus, Poisson ratio, uniaxial compressive strength, and internal friction coefficient) obtained from the models show that neural network configurations perform better than those built with the empirical equation. The horizontal minimum and maximum stress values across the depth interval of interest (2347–2500 m) are established for a case study reservoir. The model provides an accurate prediction of wellbore instability when applying the modified Lade criterion; the stable mud weight is derived with improved precision compared to the other failure criteria evaluated. A key advantage of the developed method is that it does not require input knowledge of the reservoir’s structural boundaries (e.g., the fault regime) or core test data.