Why are you putting in a 3-1/2" slab. That's too thin. Subgrade will have to be perfect or you will get an inordinate amount of shrinkage cracking from subgrade restraint.

Why were you surprised that you didn't get "rock" for foundation? Did you not investigate the site soil conditions before committing to build?

Sandy silt (if that is a correct classification) can be problematic from several perspectives. Drainage will be slow, so if you have surface accumulation or subsurface cross flow, you'll have issues with moisture. Because of the "smearing", it looks more like a clayey material in your photo.

You should get the advice of a local geotechnical engineer.

We're putting a 3-1/2'' slab and we only have one floor (1800 sqf house).
Correct classification isn't known as we didn't investigate soil conditions except for a percolation test.
But it was a mixture of silt sand with some clay.
We did a ditch around the excavation to intercept most of the surface runoff (and some underground).

You missed my point about the slab thickness. It doesn't matter that you only have one floor. The issue is that if you have a 3-1/2" slab, your subgrade will have to be perfectly flat with little or no variation in the support plane. Anything less will increase cracking. Loading is not an issue. The slab carries very little load.

3 1/2 inches is common here for a indoor house slab (Well it's the minimum required in the code).
The subgrade is sand/gravel (95% proctor in 12'' layers).

Would you recommend at least 4'' ? And what about the drain tile?

And there's a Foamular C-200 R10 Rigid Foam under a vapor retarder.

it looks like a fill site to get that 4.5', i would think you would have no trouble daylighting the foundation drain in some direction with enough pipe. You don't have a basement, which is the best reason to have a drain. I can't think of a good reason to put one in... but if you have near surface groundwater issues.... we don't have enough info on the site to comment. Make sure they backfill well.

Apparently this is in cold winter area. What amount of exterior foundation wall is exposed to below freezing temps? Probably several feet below floor grade? If so, what are you doing to keep the soil under the slab edges from frost heaving? You are keeping warmth from above the slab insulation from warming that zone. I'd add insulation on the inside of the exterior foundation walls. Clean sand and gravel fill is non-frost susceptible, but for sure?

How is the house to be heated? Any ductwork under the slab? How is the heated air getting to the floor area? Did you consider a slab with circulating heated water or air? Cold floors ain't fun.

What about plumbing? Are you able to get to it in case of building modifications later?

Yes it's in Canada.
Freezing depth is 5' here.
There's one of the inside exterior foundation walls.
Clean sand and gravel will have less fine particules.
House will be heated with a hybrid heat pump system (electricity/propane)
No ductwork under the slab and no floor heating