Imaging by inversion of acoustic or electromagnetic wave fields have applications in a wide variety of areas, such as non-destructive testing, biomedical applications, and geophysical exploration. Unfortunately, each modality suffers from its own application-specific limitations, typically being difficulties in distinguishing different materials/tissues from each other in the case of acoustic wave fields and a low spatial resolution in the case of electromagnetic wave fields. To exploit the advantages of both imaging modalities, we present a Born inversion method where we use an additive regularization term based on structural similarity between the acoustic and electromagnetic contrast. To validate our approach, we compare separate with joint inversion results for one particular example. The results for this example clearly show that separate inversion succeeds in reconstructing the acoustic contrast, but fails to properly reconstruct the electromagnetic contrast. Fortunately, with the joint inversion method, both the acoustic and electromagnetic contrast functions are reconstructed successfully.