It depends on how much statistical confidence you want/need. A practical lower limit is ~ 12 samples: 4 stress levels with 3 samples per stress level. But this will have very low statistical significance, will not capture the so-called fatigue or endurance limit, will not capture variability in raw material, heat treatment, etc. I recently worked with a test lab on developing some S-N curves, and 39 specimens were chosen for one material/manufacturing method combination: 6 stress levels with 4 samples per stress level, and then an additional 15 specimens for probing the fatigue limit using a staircase method.