This paper presents a harmonic instability analysis method using state-space modeling and participation analysis in the inverter-fed ac power systems. A full-order state-space model for the droop-controlled distributed generation (DG) inverter is built first, including the time delay of the digital control system, inner current and voltage control loops, and outer droop-based power control loop. Based on the DG inverter model, an overall state-space model of a two-inverter-fed system is established. The eigenvalue-based stability analysis is then presented to assess the influence of controller parameters on the harmonic instability of the power system. Moreover, the harmonic-frequency oscillation modes are identified, where participation analysis is presented to evaluate the contributions of different states to these modes and to further reveal how the system gives rise to harmonic instability. Based on the participation analysis, a reduced-order model for harmonic instability analysis is also proposed. The experimental results are presented for validating the theoretical analyses.