We propose a general framework for solving forward and inverse problems constrained by partial differential equations, where we interpolate neural networks onto adaptive finite element spaces to represent the (partial) unknowns, and define loss functions with dual norms of the finite element residual of the interpolation. The framework overcomes the challenges related to the imposition of boundary conditions, the choice of collocation points in physics-informed neural networks, and the integration of variational physics-informed neural networks. We consider approximated dual norms (using multigrid methods) to accelerate the training process. A numerical experiment set confirms the framework’s capability of handling various forward and inverse problems. In particular, the trained neural network generalises well for smooth problems, beating finite element solutions by some orders of magnitude in some cases. We finally propose an effective one-loop solver with an initial data fitting step (to obtain a cheap initialisation) to solve inverse problems. We will consider extensions to problems with complex geometries (using unfitted techniques) and problems posed in H(curl) and H(div).