Since the soil you are concerned with is along the side of the structure and not on top of it, think of it in the following terms....
If the structure were not touched by the adjacent soil, there would be no downward load on the structure. Since it is touched by the soil, the soil inherently has two relevant forces acting on it constantly due to its own weight. The relevant forces are the vertical force and a lateral force since the soil has properties similar to a liquid under certain conditions (if the soil were a solid ,not granular, object the lateral force would be 0). The soil acts like a bunch of marbles in a jar...if you have no jar, the marbles will roll laterally until the pile is flat!
Now there are established soil mechanics relationships between the vertical and lateral forces in soil. These are the active, passive, and at rest pressure coefficients. There is sufficient treatment of this concept in most soils textbooks.
Once we establish that there is a lateral force exerted by the soil, then another physics relationship is used to determine the "downward or dragging" effect the soil places on the structure. The total friction force by the soil acting on the structure is the "normal" force multiplied by a coefficient of friction between the soil and the structure. In this case the "normal" (or perpendicular) force is the lateral load exerted by the soil.