Ginseng (Panax ginseng C.A. Meyer) as a common dietary adjunct is widely applied in Traditional Chinese Medicine due to its health-promoting properties, but the differences between white ginseng and red ginseng was rarely studied. In the present study, color parameters and scanning electron microscope (SEM) were determined to evaluate the differences of ginseng color and microstructure induced by processing procedure. Quantitative analysis of multi-components by a single-marker (QAMS) method and anti-α-amylase activity test were used to assess variations of chemical ingredients and pharmacological activity between white and red ginseng. Finally, molecular docking studies were carried out to screen out the most effective compound against α-amylase. Results indicated that processing had a significant impact on the physicochemical properties and pharmacological activity of white and red ginseng. After processing, the color value of L* declined significantly. Red ginseng sample displayed a compact structure and presented of a gel layer on the surface compared to white ginseng. Additionally, the content of ginsenosides and the activity of anti-α-amylase decreased. The contents of total ginsenosides were positively correlated with the anti-α-amylase activities of ginseng, and ginsenoside Rb1 might be the most effective compound to inhibit the activity of α-amylase.