%
% Upload.tex
%
% SatNOGS Optical Unofficial Guide
%
% Copyright (C) 2022, Jeff Moe
%
% This document is licensed under the Creative Commons Attribution 4.0
% International Public License (CC BY-SA 4.0) by Jeff Moe.
%

\section{Push Data Upstream}
\label{sec:upload}
\index{upload}
The \gls{SatNOGS} network isn't ready for optical data yet.
So there is nothing that can be uploaded. This is under
development.

\section{File Extensions}
\label{sec:upload-extensions}
\index{formats}
The following file extensions are in use:

\begin{mdframed}[backgroundcolor=blue!10,linecolor=blue!30]
\begin{description}
        \item [.cal] --- Cal % XXX
        \item [.cat] --- Cat % XXX
        \item [.csv] --- \gls{CSV}
        \item [.dat] --- Data.
        \item [.fits] --- \gls{FITS}.
        \item [.png] --- \gls{PNG}.
\end{description}
\end{mdframed}
\index{FITS}\index{PNG}\index{CSV}


\section{Data Files}
\label{sec:upload-data}
\index{data}\index{stvid}\index{process}\index{acquire}
After the \texttt{stvid} acquisition and processing steps have been
run, files such as below will be generated:

\begin{minted}{sh}
2022-08-23T04:16:26.633.fits
2022-08-23T04:16:26.633.fits.cal
2022-08-23T04:16:26.633.fits.cat
2022-08-23T04:16:26.633.fits.png
2022-08-23T04:16:26.633_48473_catalog.dat
2022-08-23T04:16:26.633_48473_catalog.png
2022-08-23T04:16:26.633_52718_catalog.dat
2022-08-23T04:16:26.633_52718_catalog.png
2022-08-23T04:16:26.633_90000_unid.dat
2022-08-23T04:16:26.633_90000_unid.png
2022-08-23T04:16:26.633_hough.csv
2022-08-23T04:16:26.633_predict.csv
2022-08-23T04:16:26.633_threshold.csv
\end{minted}


\subsection{\texttt{.fits} FITS Files}
\gls{FITS} files are the main files created during acquisition and
are the main files used in processing. It has the data of
the actual image taken with the camera. \gls{FITS} files will be significantly
larger than the other files, at around 35 megabytes each, depending
on sample rate and frame size.
\index{FITS}\index{acquire}\index{process}

The \texttt{stvid} program creates it's customized \gls{FITS} format
file, which is a stack up of multiple images.
I'll leave explanation of the \texttt{stvid} \gls{FITS} file structure
to the program's author, Cees Bassa:

\begin{quote}
With stvid, I typically store 10 seconds worth of video frames, 100 frames at 10 \gls{Hz} frame rate or 250 frames at 25 \gls{Hz} frame rate into a so-called ``four frame''. Here, for each pixel of the initial 100 or 250 frames, the mean, standard deviation, maximum and argmax are stored, each as its own frame. Objects that move will stand out because they will have a high (max - mean) / standard deviation, which allows me to detect trails. Once a trail is detected, the argmax frame tells me the frame number and hence the motion of the trail, and using the timestamps of the individual 100 or 250 frames, I can reconstruct the pixel position as a function of time during the 10 second exposure, using only 4 frames instead of the 100 or 250 individual video frames.
\end{quote}
\index{Bassa, Cees}


\subsection{\texttt{.cal} CAL Files}
Example \texttt{.cal} filename:\\
\texttt{2022-08-23T04:16:26.633.fits.cal}\\

Sample data from \texttt{.cat} file, just the top select lines of a 577 line file:

\begin{minted}{sh}
1172.876   11.776  7.1775 221.117561  45.185979  6.761
 786.524    1.817  6.0213 222.623526  37.272049  5.590
 838.954   21.772  8.0430 222.941344  38.403949  7.653
 872.352   25.919  8.3855 222.909215  39.103246  7.884
 692.542   37.123  8.2615 223.889916  35.455924  8.718
 738.829   47.126  8.2037 223.973142  36.430116  7.777
 691.964   52.656  7.7910 224.278547  35.490201  7.411
 951.962   53.159  8.0072 223.315748  40.816697  7.844
1209.704   60.030  6.0799 222.327791  46.116208  5.811
 893.057   66.086  7.1867 223.943996  39.648460  8.039
...
\end{minted}


\subsection{\texttt{.cat} CAT Files}
\texttt{stvid} generates \texttt{.cat} files, generated from \gls{FITS} files, such as:\\
\texttt{2022-08-23T04:16:26.633.fits.cat} \\

Sample data from \texttt{.cat} file, just the top select lines of a 1,033 line file:

\begin{minted}{sh}
#   1 X_IMAGE                Object position along x                                    [pixel]
#   2 Y_IMAGE                Object position along y                                    [pixel]
#   3 MAG_BEST               Best of MAG_AUTO and MAG_ISOCOR                            [mag]
#   4 FLUX_BEST              Best of FLUX_AUTO and FLUX_ISOCOR                          [count]
#   5 FLUX_MAX               Peak flux above background                                 [count]
#   6 BACKGROUND             Background at centroid position                            [count]
  1172.8755     11.7756   7.1775     33.80787     11.09625     16.06616
    28.8250     11.4696   6.7443     50.38253     5.008112     15.81194
  1347.4933     10.4887   8.3649     11.32553     2.619246     16.04644
   269.9134      9.8019   8.4220     10.74506      1.56768     15.97937
   468.9823      7.9780   8.8072     7.535751     5.987358     16.02673
  1475.9681      6.6672   8.6115     9.024131     2.281431     16.03049
   847.0105      5.9795   8.1992     13.19223     14.31796     16.08833
   181.5139      5.5141   7.9473     16.63711     2.278276     15.95168
   460.1482      5.1577   8.5318     9.711239     2.795708     16.02688
  1559.9583      2.9245   7.5915     23.08835     6.981803      15.9909
   786.5237      1.8174   6.0213     98.05751     27.88283     16.09418
...
\end{minted}


\subsection{\texttt{.fits.png} FITS PNG Files}
\index{FITS}\index{PNG}\index{stvid}
The \texttt{stvid} application can generate multiple \gls{PNG} files,
depending how many satellites it finds. In every case it creates at
least one \gls{PNG} file, with a name like:

\texttt{2022-08-23T04:16:26.633.fits.png}

In this image, as can be see in figure \ref{fig:stvid-png}, page \pageref{fig:stvid-png},
no detected satellites are indicated.
The blue lines are from the ``Catalog'', which in this case means \glspl{TLE}.
The blue lines are drawn whether no, a few, or many satellites are detected.
They are generated by taking data from the \gls{plate-solver} about where the picture
\gls{FOV} is, and overlaying satellite \glspl{orbit} that are calculated from
\glspl{TLE}.

The main blue line is slightly offset from where the satellite is calculated to
appear.
The blue line forms a small \texttt{L} shape.
The smaller segment indicates the area where the satellite may pass.
Next to the small segment line is the \gls{NORAD ID} for the satellite.
If it is unknown, it will be given the number 90000 or larger.

\begin{figure}[p!]
        \begin{center}
                \includegraphics[keepaspectratio=true,height=1.00\textheight,width=1.00\textwidth,angle=0]{stvid/data/2022-08-23T04:16:26.633.fits.png}
                \caption{\texttt{stvid} PNG.}
                \label{fig:stvid-png}
                \index{stvid}\index{PNG}\index{FITS}
        \end{center}
\end{figure}


\subsection{\texttt{.fits.png} FITS PNG Files}
\index{FITS}\index{PNG}\index{stvid}\index{NORAD ID}
When \texttt{stvid} identifies a satellite, it creates a new \gls{PNG} file for each
satellite detected in the image. A file name example follows, with satellite
NORAID ID 48473 detected, also shown in figure
\ref{fig:stvid-png-48473-catalog}, page \pageref{fig:stvid-png-48473-catalog}.

\texttt{2022-08-23T04:16:26.633\_48473\_catalog.png}

\begin{figure}[p!]
        \begin{center}
                \includegraphics[keepaspectratio=true,height=1.00\textheight,width=1.00\textwidth,angle=0]{stvid/data/2022-08-23T04:16:26.633_48473_catalog.png}
                \caption{\texttt{stvid} PNG of satellite NORAD ID 48473 indentified, in red.}
                \label{fig:stvid-png-48473-catalog}
                \index{stvid}\index{PNG}\index{FITS}\index{NORAD ID}
        \end{center}
\end{figure}