Patient-specific simulations of fluid-structure interaction (FSI) problems in vascular systems are crucial for both fundamental research and clinical applications. However, these simulations present significant challenges due to large structural deformations, morphing flow domains, and complex FSI interfaces. In this study, we develop a smoothed particle hydrodynamics (SPH) framework that is well-adapted to vascular FSI simulations. The matrix-free iterative incompressible SPH (ISPH) method is employed to simulate blood flow dynamics, while the stabilized total Lagrangian SPH (TLSPH) method is used to capture the dynamics of blood vessels. We then introduce a novel FSI coupling strategy that integrates ISPH with TLSPH, ensuring strict interface matching conditions between the blood and vessel wall. Additionally, lumped parameter (0D) models are incorporated into the 3D SPH framework, allowing for accurate simulation of the physiological effects in downstream vascular beds. To enhance computational efficiency, we implement graphics processing unit (GPU) parallelization techniques. Leveraging the particle-based nature of SPH and the matrix-free property of our framework, the parallelized computations can be performed efficiently, ensuring optimal utilization of GPU resources. The developed framework is first validated through several benchmark tests of pulsatile flows in straight vessels. Subsequently, we apply the framework to simulate the FSI processes in patient-specific blood vessels, including the cerebral aneurysm and the aorta. The simulation results show that our framework is both effective and efficient, providing a promising tool for patient-specific vascular simulations.