Incompatibility can mean anything from swelling (which might affect whether your o-ring joint will work properly, or last durably- or, if it's a static o-ring joint, might actually make your joint seal tighter!), to embrittlement/loss of elasticity, to being totally destroyed and being washed out of the joint.
Depends which o-ring material and which "chemical". And whether or not it's a problem depends on exposure time, temperature, service life and the nature of your o-ring joint.
Note also that a generic material such as "viton" (FKM) or "kalrez" (FFKM) has many different grades associated with it. Some grades are designed for extraordinarily good service life in hot, dry conditions, but can be destroyed by just adding some steam. Others offer extraordinarily good resistance to some chemical compounds, but might not be good to high temperatures even under dry conditions...The various grades of FFKM available are so different in what they're good for (and not good for!) that they really should be distinguished as different materials entirely.