Background: Many families with clinical early-onset Alzheimer’s disease (EOAD) remain genetically unexplained. A combination of genetic factors is not standardly investigated. In addition to monogenic causes, we evaluated the possible polygenic architecture in a large series of families, to assess if genetic testing of familial EOAD could be expanded. Methods: Thirty-six pedigrees (77 patients) were ascertained from a larger cohort of patients, with relationships determined by genetic data (exome sequencing data and/or SNP arrays). All families included at least one AD patient with symptom onset <70 years. We evaluated segregating rare variants in known dementia-related genes, and other genes or variants if shared by multiple families. APOE was genotyped and duplications in APP were assessed by targeted test or using SNP array data. We computed polygenic risk scores (PRS) compared with a reference population-based dataset, by imputing SNP arrays or exome sequencing data. Results: In eight families, we identified a pathogenic variant, including the genes APP, PSEN1, SORL1, and an unexpected GRN frameshift variant. APOE-ε4 homozygosity was present in eighteen families, showing full segregation with disease in seven families. Eight families harbored a variant of uncertain significance (VUS), of which six included APOE-ε4 homozygous carriers. PRS was not higher in the families combined compared with the population mean (beta 0.05, P = 0.21), with a maximum increase of 0.61 (OR = 1.84) in the GRN family. Subgroup analyses indicated lower PRS in six APP/PSEN1 families compared with the rest (beta −0.22 vs. 0.10; P = 0.009) and lower APOE burden in all eight families with monogenic cause (beta 0.29 vs. 1.15, P = 0.010). Nine families remained without a genetic cause or risk factor identified. Conclusion: Besides monogenic causes, we suspect a polygenic disease architecture in multiple families based on APOE and rare VUS. The risk conveyed by PRS is modest across the studied families. Families without any identified risk factor render suitable candidates for further in-depth genetic evaluation.