The use of artificial neural networks (ANNs) and data-driven approaches has increased in popularity for solving computational problems, particularly also in the field of scientific computing and the modeling and simulation of physical processes. ANN models offer quick and efficient solutions, where traditional methods often encounter computational limitations. Recent advancements indicate that incorporating physical knowledge into ANNs can further enhance their performance. We achieve this by adapting the Rao-Blackwell theorem [1], which was originally established for statistical models. The Rao-Blackwellization algorithm is a powerful technique that improves an initial estimator by taking the conditional average under a sufficient statistic as a new estimator. This new estimator has been proven to have a mean-squared-error that is less than or equal to that of the initial estimator. Our new framework enables the use of physical information, such as isotropy, observer invariance or dimensional analysis, replacing the sufficient statistic in the original algorithm. The updated algorithm can be combined with other approaches, such as physics-informed neural networks (PINNs) [2] and constitutional artificial neural networks (CANNs), while maintaining the optimality of the Rao-Blackwellization algorithm. Various aspects and advantages of the proposed method are demonstrated in terms of ANN design, data generation and data processing through a series of illustrative examples. The first example is a simple flow curve prediction using an ANN, which serves to introduce the method and its underlying principles. The subsequent examples are more complex and include the comparison of different improvement strategies for ANNs predicting elastic and brittle energy proportions based on a variational formulation to find the best way to use given data. A dimensional analysis of drilled steel bars is conducted to demonstrate how sufficient information can be obtained for characteristic forces. Finally, an ANN material model for elasticity is constructed by incorporating isotropy and observer invariance.