The red shift of quasars is "intrinsic". This means their red shift isn't related to Hubble's law which says the larger the red shift the greater the distance from us. Most cosmologists assume that the quasar red shift isn't intrinsic and follows Hubble's law and therefore that quasars are the most distant objects ever observed. However, I believe this is not the case, that the quasars instead have a large atmosphere of free electrons and the light from the quasar is shifted by the Compton Effect as it travels through this atmosphere producing the intrinsic red shift. Thus, you can't tell how far away the quasar is by just looking at the red shift.
So, where are they? Burbidge has some evidence that they are associated with and are alongside low redshift galaxies. Arp, in his recent book "Seeing Red" provides additional evidence that this is the case. Both Arp and Burbidge seem to think that the quasars have been ejected from these galaxies.
But most quasars don't particularly look like extragalactic objects. They look like stars (except for a few which look like they may be embedded in a galaxy - or are stars within a newly forming planetary system). This is why they are called quasars, which is short for "quasi-stars", or "quasi-stellar objects". If it weren't for their red shifts, quasars almost certainly would be considered to be stars. Could they be stars?
A decent argument against the idea that quasars are stars is that they appear to be rather randomly scattered across the sky. Stars (other than the nearby ones), on the other hand, are concentrated in the Milky Way (the galactic plane). But quasars certainly seem to be stars, and even might be paired in double star systems with a star that doesn't have a red shift. Look at 100,000th Hubble Image for an example of a quasar that is visually indistinguishable from a nearby star in the same image field of view. If it could ever be proved that a quasar is orbiting an ordinary star we'd have the proof they are nearby. Indeed, stars (but not extragalactic objects) sometimes have proper motion in the sky and some quasars seem to have exhibited this motion. See Quasar Absolute Proper Motion for a table that includes such quasar proper motion observations.
But what if quasars are just intrinsically very dim stars? They are observed to be rather faint objects, all but a handful being 17th magnitude or fainter. (The larger the magnitude the less bright the star.) Perhaps, BECAUSE they maintain a large atmosphere of free electrons, they are really naturally dim stars. Then they could just be very nearby stars, so close that they are spread all around us and are too faint to be seen at a distance where they would begin to line up with the Milky Way. It would be great if we could independently measure the distance to quasars by a parallax method to determine if they are really nearby, but they are just about too faint for good parallax measurements. There is some hope that these measurements could be made at radio wavelengths if suitable radio reference points are established for precise location measurements over long periods of time.
I think quasars are nearby dim stars, some of the closest objects to us seen in the night sky, rather than being the most distant objects observed in the universe.
Where are the quasars? The question remains to be resolved.