In this article we analyze conditions for different types of instabilities and complex dynamics that occur in nonlinear two-component fractional reaction-diffusion systems. It is shown that the stability of steady state solutions and their evolution are mainly determined by the eigenvalue spectrum of a linearized system and the fractional derivative order. The results of the linear stability analysis are confirmed by computer simulations of the FitzHugh-Nahumo-like model. On the basis of this model, it is demonstrated that the conditions of instability and the pattern formation dynamics in fractional activator- inhibitor systems are different from the standard ones. As a result, a richer and a more complicated spatiotemporal dynamics takes place in fractional reaction-diffusion systems. A common picture of nonlinear solutions in time-fractional reaction-diffusion systems and illustrative examples are presented. The results obtained in the article for homogeneous perturbation have also been of interest for dynamical systems described by fractional ordinary differential equations.