In reality, your columns will provide lateral resistance and therefore would affect your centre of rigidity calculations.
In practice, designers decide whether or not the columns are part of the designated lateral system and then proceed in a manner consistent with that decision. If the columns are omitted from the lateral system then designers will usually not include them in centre of rigidity calculations.
How do you decide if you want to include your columns? That's your call. I think that I saw a recommendation once that said that, if the stiffness of your walls is more than 6X the summed stiffness of all your columns, then the column stiffness can be disregarded. You'll want to fact check this though.
The greatest trick that bond stress ever pulled was convincing the world it didn't exist.