The development of colorectal cancer (CRC) can be influenced by genetic factors in both familial cases and sporadic cases. Familial CRC has been associated with genetic changes in high-, moderate- and low-penetrance susceptibility genes. However, despite the availability of current gene-identification techniques, the genetic causes of a considerable proportion of hereditary cases remain unknown. Genome-wide association studies of CRC have identified a number of common low-penetrance alleles associated with a slightly increased or decreased risk of CRC. The accumulation of low-risk variants may partly explain the familial risk of CRC, and some of these variants may modify the risk of cancer in patients with mutations in high-penetrance genes. Understanding the predisposition to develop CRC will require investigators to address the following challenges: the identification of genes that cause uncharacterized hereditary cases of CRC such as familial CRC type X and serrated polyposis; the classification of variants of unknown significance in known CRC-predisposing genes; and the identification of additional cancer risk modifiers that can be used to perform risk assessments for individual mutation carriers. We performed a comprehensive review of the genetically characterized and uncharacterized hereditary CRC syndromes and of low- and moderate-penetrance loci and variants identified through genome-wide association studies and candidate-gene approaches. Current challenges and future perspectives in the field of CRC predisposition are also discussed.