Self-filtration, also known as "bridging", is the formation of a "filter bridge" in the base soil immediately adjacent to the filter (whether it's a granular or geosynthetic filter). This refers to the ability of a base soil to retain its own finer particles near the base-filter interface. The mechanism involved is the migration of finer particles into the filter. The "layer" of base soil immediately adjacent to the filter becomes coarser, and subsequent layers moving away from the filter retain more and more of their fines. The system eventually comes to equilibrium, and no more fines are lost into the filter. The base soil is then "self-filtering". A base that is not self-filtering is incapable of forming a filter bridge (in a state of equilibrium) and more and more fines are piped from the base into the filter, possibly resulting is failure of the filter system (due to clogging or blinding) or settlement/deformation issues in the base soil unit (due to excessive soil loss and lack of retention). Note that the concept of "self-filtering", regardless of its name, does not imply that some sort of filter is not required.
Probably the best reference on this topic is Lafleur et al (1989) "Filtration of Broadly Graded Cohesionless Soils" ASCE JGE, 115(12) pp 1747-1768. For basic understanding of filters (or seepage in general), I concur with Focht3 that Cedergren can't be beat. If dealing with geosynth' filters, check Koerner's book or Holtz, Christopher and Berg (1997). For some great design guidance check out "Embankment Dams Granular Filters and Drains" ICOLD Bulletin 95.