A method of fabricating a semiconductor memory device includes patterning a substrate to form a first active pattern and a second active pattern with a trench therebetween, forming a device isolation layer, forming a first source/drain region on the first active pattern and a second source/drain region on the second active pattern, etching the first source/drain region and the second source/drain region to form a first contact hole,—forming a conductive layer that fills the first contact hole, patterning the conductive layer to form a contact part, forming a lower spacer that covers a sidewall of the contact part and fills the first contact hole, forming a second contact hole, and filling the second contact hole with a conductive material to form a contact coupled to the second source/drain region.