Modeling and simulating damage and fracture processes in materials and structures is an interesting yet challenging topic in computational mechanics. Many models are based on the assumption of a nearly uniform distribution of spherical void-like defects in the material, which justifies the use of a single scalar damage variable in the formulation. However, it is clear that this is usually a very strong simplification of the actual underyling conditions, since microdefects are in general neither isotropically distributed nor of void-like shape. On the contrary, such defects usually lead to a significant direction dependence of the global material behavior, and it would certainly be more realistic to take this fact into account in damage modeling. Unfortunately, this also requires a more sophisticated regularization approach for damage in order to use the corresponding model meaningfully in the context of numerical simulations, i.e., to effectively overcome the nowadays well-known problem of artificial mesh-dependence of local damage models. The present study is devoted to both topics. First, a novel and flexible model framework for large deformations is presented, which can be used to represent general initial as well as induced anisotropies using structural tensors. Damage is then considered as a special case of this theory based on a symmetric structural tensor of second-order. Subsequently, the question is discussed of how to achieve a meaningful and effective gradient-based regularization for this kind of model.