Question

I am trying to loop through positions of many objects in a table and query all of the objects within a certain radius of each object in the Juypter notebook. Is this even possible to do without the query timing out?

Answer 1

This sounds like a case for using the crossmatch service.  Have a look at the How_to_crossmatch_tables notebook in the notebooks-latest/04_HowTos folder, or https://github.com/noaodatalab/notebooks-latest/tree/master/04_HowTos/CrossmatchTables .  Feel free to post your query here if you continue to have trouble.

Comments

I tried the crossmatch service and it was still giving me the same connection or timeout errors so I downloaded the datalab modules and tried it through python two different ways:

#First way:
import numpy as np
from astropy import units as u
from astropy.coordinates import SkyCoord
from astropy.table import Table, Column
from math import log10
import matplotlib.pyplot as plt
from astropy.io import fits
import dl
from dl import authClient as ac, queryClient as qc
from dl.helpers.utils import convert
from getpass import getpass

#token = ac.login('kedd58',getpass("Enter password (+ENTER): "))
#ac.whoAmI()

cows_official = Table.read('/Users/kedd58/Desktop/cows_official.fits') #catalog of clusters

for cluster in cows_official:
    RA=cluster['RA']
    DEC=cluster['DEC']
    query= '''
    SELECT ra, dec, brick_primary, mag_g, mag_r, mag_z, objid, snr_g, snr_r, snr_z, type
    FROM ls_dr7.tractor
    WHERE q3c_join (ra, dec, RA, DEC, 0.041667) #tried this with the radial query function as well
    '''
    sources=qc.query(sql=query,output='catalogs.fits')

#Second Way:

cows_official = Table.read('cows_official.fits')

RA=cows_official['RA']
DEC=cows_official['DEC']
ID=cows_official['ID']

DEF_ACCESS_URL = "http://datalab.noao.edu/tap/ls_dr7"

service = pyvo.dal.TAPService(DEF_ACCESS_URL)

for ID in cows_official:
    try:
        cluster_catalog=service.submit_job('SELECT * FROM ls_dr7.tractor WHERE q3c_radial_query(ra,dec,RA,DEC,0.041667)',maxrec=100000).to_table()
        cluster_catalog.run()
        url=cluster_catalog.url
        urllib.urlretrieve(url, ID + "_catalog.fits")
    except IndexError:
        pass

Both methods produce time out errors or this error:
ConnectionError: ('Connection aborted.', BadStatusLine("''",))

---

Hi there,

So we discovered a problem with the Q3C indices on ls_dr7, and need to rebuild them.  This will unfortunately take a couple of days--apologies for the inconvenience!

In the meantime, you can test your above queries against e.g. ls_dr5.tractor.  One note is that your q3c_join query will not use the indices as written, and so will run for a very long time.  It's important that the large table ra, dec appear *second* in the call to q3c_join, as noted in the crossmatch notebook.

---

I tried them with dr5 and I keep getting the same error:
ConnectionError: ('Connection aborted.', BadStatusLine("''",))