This paper presents an adaptive controller to achieve consensus tracking for the fractional-order linear time invariant swarm systems in which the matrices describing the agent dynamics and the interactive dynamics between agents are unknown. This controller consists of two parts: an adaptive stabilizer and an adaptive tracker. The adaptive stabilizer guarantees the asymptotic swarm stability of the considered swarm system. Also, the adaptive tracker enforces the system agents to track a desired trajectory while achieving consensus. Numerical simulation results are presented to show the effectiveness of the proposed controller.