% % Upload.tex % % SatNOGS Optical Unofficial Guide % % Copyright (C) 2022, Jeff Moe % % This document is licensed under the Creative Commons Attribution 4.0 % International Public License (CC BY-SA 4.0) by Jeff Moe. % \section{Push Data Upstream} \label{sec:upload} \index{upload} The \gls{SatNOGS} network isn't ready for optical data yet. So there is nothing that can be uploaded. This is under development. \section{File Extensions} \label{sec:upload-extensions} \index{formats} \index{.cal}\index{.cat}\index{.csv}\index{.dat}\index{.fits}\index{.png} The following file extensions are in use: \begin{mdframed}[backgroundcolor=blue!10,linecolor=blue!30] \begin{description} \item [.cal] --- Cal % XXX \index{.cal} \item [.cat] --- Cat % XXX \index{.cat} \item [.csv] --- \gls{CSV} \item [.dat] --- Data. \index{.dat} \item [.fits] --- \gls{FITS}. \item [.png] --- \gls{PNG}. \end{description} \end{mdframed} \section{Data Files} \label{sec:upload-data} \index{data}\index{process}\index{acquire} After the \texttt{stvid} acquisition and processing steps have been run, files such as below will be generated: \begin{minted}{sh} 2022-08-23T04:16:26.633.fits 2022-08-23T04:16:26.633.fits.cal 2022-08-23T04:16:26.633.fits.cat 2022-08-23T04:16:26.633.fits.png 2022-08-23T04:16:26.633_48473_catalog.dat 2022-08-23T04:16:26.633_48473_catalog.png 2022-08-23T04:16:26.633_52718_catalog.dat 2022-08-23T04:16:26.633_52718_catalog.png 2022-08-23T04:16:26.633_90000_unid.dat 2022-08-23T04:16:26.633_90000_unid.png 2022-08-23T04:16:26.633_hough.csv 2022-08-23T04:16:26.633_predict.csv 2022-08-23T04:16:26.633_threshold.csv \end{minted} \index{.cal}\index{.cat}\index{.csv}\index{.dat}\index{.fits}\index{.png} \subsection{\texttt{.fits} FITS Files} \gls{FITS} files are the main files created during acquisition and are the main files used in processing. It has the data of the actual image taken with the camera. \gls{FITS} files will be significantly larger than the other files, at around 8 to 35 megabytes each, depending on sample rate, frame size, and \gls{binning}. \index{acquire}\index{process} \index{frame size}\index{sample rate} The \texttt{\gls{stvid}} program creates it's customized \gls{FITS} format file, which is a stack up of multiple images. I'll leave explanation of the \texttt{\gls{stvid}} \gls{FITS} file structure to the program's author, Cees Bassa: \begin{quote} With \gls{stvid}, I typically store 10 seconds worth of video frames, 100 frames at 10 \gls{Hz} frame rate or 250 frames at 25 \gls{Hz} frame rate into a so-called ``four frame''. Here, for each pixel of the initial 100 or 250 frames, the mean, standard deviation, maximum and argmax are stored, each as its own frame. Objects that move will stand out because they will have a high (max - mean) / standard deviation, which allows me to detect trails. Once a trail is detected, the argmax frame tells me the frame number and hence the motion of the trail, and using the timestamps of the individual 100 or 250 frames, I can reconstruct the pixel position as a function of time during the 10 second exposure, using only 4 frames instead of the 100 or 250 individual video frames. \end{quote} \index{Bassa, Cees}\index{four frame} \index{.fits} \subsection{\texttt{.cal} CAL Files} \index{.cal} Example \texttt{.cal} filename:\\ \texttt{2022-08-23T04:16:26.633.fits.cal}\\ Sample data from \texttt{.cal} file, just the top select lines of a 577 line file: \begin{minted}{sh} 1172.876 11.776 7.1775 221.117561 45.185979 6.761 786.524 1.817 6.0213 222.623526 37.272049 5.590 838.954 21.772 8.0430 222.941344 38.403949 7.653 872.352 25.919 8.3855 222.909215 39.103246 7.884 692.542 37.123 8.2615 223.889916 35.455924 8.718 738.829 47.126 8.2037 223.973142 36.430116 7.777 691.964 52.656 7.7910 224.278547 35.490201 7.411 951.962 53.159 8.0072 223.315748 40.816697 7.844 1209.704 60.030 6.0799 222.327791 46.116208 5.811 893.057 66.086 7.1867 223.943996 39.648460 8.039 ... \end{minted} \subsection{\texttt{.cat} CAT Files} \index{.cat} \texttt{\gls{stvid}} generates \texttt{.cat} files, generated from \gls{FITS} files, such as:\\ \texttt{2022-08-23T04:16:26.633.fits.cat} \\ Sample data from \texttt{.cat} file, just the top select lines of a 1,033 line file: \begin{minted}{sh} # 1 X_IMAGE Object position along x [pixel] # 2 Y_IMAGE Object position along y [pixel] # 3 MAG_BEST Best of MAG_AUTO and MAG_ISOCOR [mag] # 4 FLUX_BEST Best of FLUX_AUTO and FLUX_ISOCOR [count] # 5 FLUX_MAX Peak flux above background [count] # 6 BACKGROUND Background at centroid position [count] 1172.8755 11.7756 7.1775 33.80787 11.09625 16.06616 28.8250 11.4696 6.7443 50.38253 5.008112 15.81194 1347.4933 10.4887 8.3649 11.32553 2.619246 16.04644 269.9134 9.8019 8.4220 10.74506 1.56768 15.97937 468.9823 7.9780 8.8072 7.535751 5.987358 16.02673 1475.9681 6.6672 8.6115 9.024131 2.281431 16.03049 847.0105 5.9795 8.1992 13.19223 14.31796 16.08833 181.5139 5.5141 7.9473 16.63711 2.278276 15.95168 460.1482 5.1577 8.5318 9.711239 2.795708 16.02688 1559.9583 2.9245 7.5915 23.08835 6.981803 15.9909 786.5237 1.8174 6.0213 98.05751 27.88283 16.09418 ... \end{minted} \subsection{\texttt{.fits.png} FITS PNG Files} \index{.fits.png} The \texttt{\gls{stvid}} application can generate multiple \gls{PNG} files, depending how many satellites it finds. In every case it creates at least one \gls{PNG} file, with a name like: \texttt{2022-08-23T04:16:26.633.fits.png} In this image, as can be see in figure \ref{fig:stvid-png}, page \pageref{fig:stvid-png}, no detected satellites are indicated. The blue lines are from the ``Catalog'', which in this case means \glspl{TLE}. The blue lines are drawn whether no, a few, or many satellites are detected. They are generated by taking data from the \gls{plate-solver} about where the picture \gls{FoV} is, and overlaying satellite \glspl{orbit} that are calculated from \glspl{TLE}. \index{detect} The main blue line is slightly offset from where the satellite is calculated to appear. The blue line forms a small \texttt{L} shape. The smaller segment indicates the area where the satellite may pass. Next to the small segment line is the \gls{NORAD-ID} for the satellite. If it is unknown, it will be given the number \texttt{90000} or larger. \begin{figure}[h!] \begin{center} \includegraphics[keepaspectratio=true,height=0.85\textheight,width=0.85\textwidth,angle=0]{stvid/data/2022-08-23T04:16:26.633.fits.png} \caption{\texttt{\gls{stvid}} PNG.} \label{fig:stvid-png} \end{center} \end{figure} \index{.fits.png} \subsection{\texttt{catalog.png} catalog PNG Files} \index{detect}\index{identify} When \texttt{\gls{stvid}} identifies a satellite, it creates a new \gls{PNG} file for each satellite detected in the image. A file name example follows, with satellite \gls{NORAD-ID} 48473 identified, also shown in figure \ref{fig:stvid-png-48473-catalog}, page \pageref{fig:stvid-png-48473-catalog}. \texttt{2022-08-23T04:16:26.633\_48473\_catalog.png} \begin{figure}[h!] \begin{center} \includegraphics[keepaspectratio=true,height=0.85\textheight,width=0.85\textwidth,angle=0]{stvid/data/2022-08-23T04:16:26.633_48473_catalog.png} \caption{\texttt{\gls{stvid}} \gls{PNG} of the satellite trail of \gls{NORAD-ID} 48473 identified in red.} \label{fig:stvid-png-48473-catalog} \end{center} \end{figure} \index{identify} If multiple satellites are detected in the image, they will each get a \gls{PNG} file. For the example image, a satellite with \gls{NORAD-ID} 52718 was also identified. The file name is: \texttt{2022-08-23T04:16:26.633\_52718\_catalog.png} \begin{figure}[h!] \begin{center} \includegraphics[keepaspectratio=true,height=0.85\textheight,width=0.85\textwidth,angle=0]{stvid/data/2022-08-23T04:16:26.633_52718_catalog.png} \caption{\texttt{\gls{stvid}} \gls{PNG} of the satellite trail of \gls{NORAD-ID} 52718 identified in red.} \label{fig:stvid-png-52718-catalog} \index{identify} \end{center} \end{figure} \subsection{\texttt{unid.png} Unidentified PNG Files} \index{unidentified} As shown previously, \texttt{\gls{stvid}} will create a new \gls{PNG} with a red line when it identifies a satellite. When \texttt{\gls{stvid}} finds a trail it can't identify in the \gls{TLE} catalogs, it gives it a \gls{NORAD-ID} starting with \texttt{90000}, incrementing by one. See figure \ref{fig:stvid-png-90000-unid}, page \pageref{fig:stvid-png-90000-unid}, for an example of \texttt{\gls{stvid}} marking an unidentified trail red. In this case, the trail was created by an airplane, but \texttt{\gls{stvid}} would do the same with an unidentified satellite trail. \index{airplane}\index{unidentified} \begin{figure}[h!] \begin{center} \includegraphics[keepaspectratio=true,height=0.85\textheight,width=0.85\textwidth,angle=0]{stvid/data/2022-08-23T04:16:26.633_90000_unid.png} \caption{\texttt{\gls{stvid}} \gls{PNG} of an unidentified trail in red.} \label{fig:stvid-png-90000-unid} \end{center} \end{figure}\index{unidentified} \subsection{\texttt{.dat} Data Files} \index{.dat} For each trail detected as shown in previous sections, such as for which a \gls{PNG} file was created, \texttt{\gls{stvid}} will output a \texttt{.dat} file. The \texttt{.dat} filenames corresponding to the figures at \ref{fig:stvid-png-48473-catalog}, \ref{fig:stvid-png-52718-catalog}, and \ref{fig:stvid-png-90000-unid} are: \begin{minted}{sh} 2022-08-23T04:16:26.633_48473_catalog.dat 2022-08-23T04:16:26.633_52718_catalog.dat 2022-08-23T04:16:26.633_90000_unid.dat \end{minted} Note, there is no \texttt{.dat} file created corresponding to the \texttt{2022-08-23T04:16:26.633.fits.png} file, since it doesn't show detected trails. The \texttt{.dat} file contents look like this, for \gls{NORAD-ID} 48473: \begin{minted}{sh} 48473 21 040AX 9999 G 20220823041631625 17 25 1533406+483563 37 S \end{minted} XXX This is the IOD ? XXX The first field is the \gls{NORAD-ID}. The fourth field looks like COSPAR. XXX Large number starting with 2022 is date and time stamp. \index{IOD}\index{COSPAR} The \texttt{.dat} file contents look like this, for \gls{NORAD-ID} 52718: \index{.dat} \begin{minted}{sh} 52718 22 056C 9999 G 20220823041631625 17 25 1527277+381243 37 S \end{minted} The \texttt{.dat} file contents look like this, for the unidentified trail (the airplane): \index{airplane}\index{unidentified} \begin{minted}{sh} 90000 22 500A 9999 G 20220823041631625 17 25 1457613+535844 37 S \end{minted} \subsection{\texttt{stvid} \texttt{.csv} Files} \index{.csv}\index{hough}\index{predict}\index{threshold} During processing, \texttt{\gls{stvid}} generates (at least!) three \gls{CSV} files. For the observation discussed above, staring with figure \ref{fig:stvid-png}, the following \gls{CSV} files are generated: \begin{minted}{sh} 2022-08-23T04:16:26.633_hough.csv 2022-08-23T04:16:26.633_predict.csv 2022-08-23T04:16:26.633_threshold.csv \end{minted} \subsubsection{\texttt{stvid} \texttt{hough.csv} Files} \index{.csv}\index{hough} Full contents of the \texttt{2022-08-23T04:16:26.633\_hough.csv} file: \begin{minted}{sh} ax,ay,az,bx,by,bz,n 1585.699766,186.468458,51.073014,0.879795,0.416052,0.229917,3424 794.849123,355.614035,48.596491,0.780147,-0.382158,0.495304,285 1297.660777,432.469965,52.989399,0.923059,0.193205,0.332617,283 \end{minted} \subsubsection{\texttt{stvid} \texttt{predict.csv} Files} \index{.csv}\index{predict} The \texttt{2022-08-23T04:16:26.633\_predict.csv} is 3,764 lines long. Below is an extract of the lines. I shortened the path, which included the full path to my home directory, for clarity: \begin{minted}{sh} satno,cospar,mjd,ra,dec,state,tlefile,age 00134,61015U ,59814.17808603,182.367480,+37.582914,sunlit,/data/tle/catalog.tle,0.441 00134,61015U ,59814.17809887,182.231429,+37.736730,sunlit,/data/tle/catalog.tle,0.441 00134,61015U ,59814.17811171,182.094454,+37.890457,sunlit,/data/tle/catalog.tle,0.441 00134,61015U ,59814.17812455,181.956550,+38.044091,sunlit,/data/tle/catalog.tle,0.441 ... 00735,64002C ,59814.17808603,227.004270,+34.763570,sunlit,/data/tle/catalog.tle,0.027 00735,64002C ,59814.17809887,227.192764,+34.379114,sunlit,/data/tle/catalog.tle,0.027 00735,64002C ,59814.17811171,227.378969,+33.994697,sunlit,/data/tle/catalog.tle,0.027 ... 46083,20055BJ ,59814.17808603,204.492490,+40.972573,sunlit,/data/tle/catalog.tle,0.022 46083,20055BJ ,59814.17809887,204.840813,+41.039396,sunlit,/data/tle/catalog.tle,0.022 46083,20055BJ ,59814.17811171,205.193715,+41.105756,sunlit,/data/tle/catalog.tle,0.022 ... 32706,08010A ,59814.17808603,169.555855,+50.073199,sunlit,/data/tle/inttles.tle,0.624 32706,08010A ,59814.17809887,169.560127,+50.075268,sunlit,/data/tle/inttles.tle,0.624 32706,08010A ,59814.17811171,169.564400,+50.077336,sunlit,/data/tle/inttles.tle,0.624 ... \end{minted} \index{.csv}\index{COSPAR} \subsubsection{\texttt{stvid} \texttt{threshold.csv} Files} \index{.csv}\index{threshold} \texttt{\gls{stvid}} creates a \texttt{threshold.csv} file with a filename like: \texttt{2022-08-23T04:16:26.633\_threshold.csv} The above file has 5,293 lines. Here are a few of them: \begin{minted}{sh} 58.000000,0.000000,38.000000 69.000000,1.000000,59.000000 1832.000000,1.000000,52.000000 418.000000,2.000000,11.000000 1183.000000,2.000000,71.000000 ... 1276.000000,1078.000000,1.000000 11.000000,1079.000000,38.000000 1500.000000,1079.000000,45.000000 \end{minted}