[jkrick]

In preparation for the upcoming DR1 updates. Closes #150, closes #238, closes #275.

Summary

• Switch TAP access from raw pyvo to astroquery.ipac.irsa.Irsa.query_tap
• Use SkyCoord instead of separate ra/dec floats
• writeto(..., overwrite=False) so output isn't silently clobbered
• Plot wavelength as an astropy Quantity instead of a bare float
• Drop a duplicated caching tip box in spherex_cutouts.md
• Preserve the cutout's original primary header (incl. VERSION) in the combined MEF's
  primary HDU instead of writing a blank one, and record each cutout's version in
  CUTOUT_INFO
• Fold the Source Discovery Tool README's directory-contents list into its notebook
  (the rest of the README was already duplicated there) and remove the README
• Open spherex_intro.md's spectral image with a context manager, reading all extensions into
  memory so the downstream cells work after the file is closed (spherex_intro.md part of Make all fits.open() calls use a context manager #129)
• Also treat http.client.RemoteDisconnected as a transient read error in spherex_cutouts.md
  (skip that cutout) and spherex_psf.md (retry), after CI failed on a server disconnect
  mid-read

Note on #129

spherex_cutouts.md, spherex_psf.md, and the SDT notebook already use context managers for
every fits.open() call. spherex_intro.md's bare fits.open() call is fixed here too (purely
mechanical, does not touch the QR3-stale content next to it, which stays for #374). The two
Euclid notebooks and the Roman notebook also listed in #129 are untouched -- out of scope for
this PR.

Test plan

• Ran spherex_cutouts.md and spherex_psf.md end to end against production
  (jupytext --to notebook --execute); no errors
• Verified VERSION is present in the output MEF's primary header
• Verified the SDT notebook still parses correctly after the README merge (full
  execution needs SExtractor/Firefly/bokeh, not run here)
• Ran spherex_intro.md end to end against production; no errors

[jkrick]

Added update to fix the failing notebook test. CI hit http.client.RemoteDisconnected while reading a SPHEREx
image, and that error wasn't in the list of transient errors we catch. It's now caught in both places that use
that list: spherex_cutouts.md skips the affected cutout, and spherex_psf.md's retry loop retries it (and still
raises after 3 attempts).

[troyraen]

Thank you! These issues will be good to have resolved. I know we've been asked about the version metadata. My biggest comment is about the primary header, which shouldn't include the version IMO since it will be wrong for some extensions.

[troyraen]

I think this needs to be saved to a variable with .copy() to be robust. I recall doing something similar and finding that the code would sometimes error because the data disappeared (I suspect that because the file is closed the next garbage collection removes it from memory). But if I'm wrong I'd love to see some documentation (if any exists) because this comes up for me occasionally.

Suggested change

	# Load the data, reading every extension into memory so that
	# hdulist stays usable after the file is closed.
	with fits.open(spectral_image_url, memmap=False) as hdulist:
	for hdu in hdulist:
	hdu.data
	# Load the data, reading every extension into memory so that
	# hdulist stays usable after the file is closed.
	with fits.open(spectral_image_url, memmap=False) as hdulist:
	for hdu in hdulist:
	hdu_data = hdu.data.copy()

(and then use hdu_data in place of hdu.data below)

[jkrick]

I looked into this and I think the current code is OK as is. With memmap=False, touching hdu.data reads the array into ordinary memory and stores it on the HDU. hdulist keeps a reference to every HDU, so garbage collection can't free the data after the file closes. To test it, I built a 5-extension file (float image, int image, scaled image, and a table like WCS-WAVE). I opened it this way, let the with block close it, forced gc.collect() while allocating other memory, and compared. Every extension was still intact, and none of the arrays were backed by a memory map or file buffer.

The failure I could reproduce is lazy loading: an HDU that isn't touched before the file closes raises "file was closed before all HDUs were read" afterwards. That's why the loop touches every .data. The usual .copy() advice is for the default memmap=True (astropy FAQ, "Too many open files").

Docs: https://docs.astropy.org/en/stable/io/fits/index.html says that after closing, "the headers and data may or may not be accessible depending on whether the data are touched and if they are memory-mapped". The FAQ (https://docs.astropy.org/en/stable/io/fits/appendix/faq.html) says memmap=False "will force data to be read entirely into memory."

Also, the later cells use hdulist[...] directly, including WCS(..., fobj=hdulist), so one hdu_data variable in the loop would keep only the last extension's data.