In a method of manufacturing a semiconductor device, a fin structure is formed. The fin structure includes a stacked layer of first semiconductor layers and second semiconductor layers disposed over a bottom fin structure, and a hard mask layer over the stacked layer. An isolation insulating layer is formed so that the hard mask layer and the stacked layer are exposed from the isolation insulating layer. A sacrificial cladding layer is formed over at least sidewalls of the exposed hard mask layer and stacked layer. A first dielectric layer is formed, and a second dielectric layer made of a different material than the first dielectric layer is formed over the first dielectric layer. The second dielectric layer is recessed, and a third dielectric layer made of a different material than the second dielectric layer is formed on the recessed second dielectric layer, thereby forming a wall fin structure.