Heterogeneous materials are increasing used in many engineering applications. Analyzing the behavior of such materials often relies on multiscale simulations such as the FE2 method which is a popular homogenization-based concurrent multiscale model that uses the finite element method (FEM) at two spatial scales. Despite the recent advancements in software/hardware and mechanics theory, the simulation of hierarchical materials via FE2 is still prohibitively costly. These challenges are exacerbated in the presence of microstructural deformations that are path-dependent (i.e., plastic deformations) and involve damage. In our work, we address these challenges by developing a physics-informed sequence learning model that surrogates the microstructural analyses in 3D multiscale simulations that involve plasticity and damage. We demonstrate that (1) reliance on training data can be consistently reduced by designing the architecture and loss function of the deep learning model based on mechanics principles, and (2) the trained model is transferable, i.e., it can be used to accelerate a variety of multiscale simulations. We generate the training data via our recently developed reduced-order model (ROM) which leverages adaptive spatiotemporal dimension reduction to dramatically accelerates microstructural simulations that involve hardening and/or softening.