.gitignore

@@ -86,3 +86,6 @@ ENV/
 .idea/
 
 .DS_Store
+
+# program config
+sohstationviewer/conf/read_settings.ini

documentation/01 _ Table of Contents.help.md

@@ -16,17 +16,19 @@ The home button can be used to return to this page at any time.
 
 + [Search SOH n LOG](04%20_%20Search%20SOH%20n%20LOG.help.md)
 
-+ [Select Waveforms](05%20_%20Select%20Waveforms.help.md)
++ [Search List of Directories](05%20_%20Search%20List%20of%20Directories.help.md)
 
-+ [Select Mass Position](06%20_%20Select%20Mass%20Position.help.md)
++ [Read from Data Card](06%20_%20Read%20from%20Data%20Card.help.md)
 
 + [Select SOH](07%20_%20Select%20SOH.help.md)
 
-+ [View-Edit Database Tables](07%20_%20View-Edit%20Database%20Tables.help.md)
++ [Select Mass Position](08%20_%20Select%20Mass%20Position.help.md)
 
-+ [GPS Dialog](08%20_%20GPS%20Dialog.help.md)
++ [Select Waveforms](09%20_%20Select%20Waveforms.help.md)
 
-+ [Read from Data Card](08%20_%20Read%20from%20Data%20Card.help.md)
++ [Gap Display](10%20_%20Gap%20Display.help.md)
 
-+ [Search List of Directories](10%20_%20Search%20List%20of%20Directories.help.md)
++ [GPS Dialog](20%20_%20GPS%20Dialog.help.md)
+
++ [View-Edit Database Tables](21%20_%20View-Edit%20Database%20Tables.help.md)
 

documentation/03 _ How to Use Help.help.md

@@ -21,7 +21,8 @@ The box to enter search text is placed at the top of the dialog.
 <br />
 
 ### Search Through All Documents button
-The button to search through all documents for search text located on the right of Search box.
+The button to search through all documents for search text located on the
+right of Search box.
 
 <br />
 
@@ -45,7 +46,8 @@ Is the list of all help documents located on the left of Help Dialog.
 <br />
 
 ### Document View
-Occupies the largest section of Help Dialog located on the right to display the content of the current selected document.
+Occupies the largest section of Help Dialog located on the right to display
+the content of the current selected document.
 
 <br />
 
@@ -55,21 +57,26 @@ Occupies the largest section of Help Dialog located on the right to display the
 <br />
 
 ### Search for a text on the current document
-Type a searched text, the first text found will be highlighted in the Document View and the view scrolls to its location.
+Type a searched text, the first text found will be highlighted in the Document
+View and the view scrolls to its location.
 
-User can traverse back and forth the view to look for previous or next search text using **Search Previous** or **Search Next** button.
+User can traverse back and forth the view to look for previous or next search
+text using **Search Previous** or **Search Next** button.
 
 <br />
 
 ### Search for all documents that contain the searched text
 Type a searched text, click the button 'Search Through All Documents'.
 
-A list of all documents that contain the searched text will be created in document 'Search Through All Documents'.
+A list of all documents that contain the searched text will be created in
+document 'Search Through All Documents'.
 
 The document will be brought to Document View.
 
-Click on the link of a document in 'Search Through All Documents' will bring its to Document View.
+Click on the link of a document in 'Search Through All Documents' will bring
+its to Document View.
 
-User can go back to 'Search Through All Documents' document any time by clicking on the name on Document List.
+User can go back to 'Search Through All Documents' document any time by
+clicking on the name on Document List.
 
 <br />

documentation/04 _ Search SOH n LOG.help.md

@@ -5,7 +5,8 @@
 
 ## How to open
 Search Messages dialog will open by itself after a data set are done with reading and plotting.
-If it is closed by any reason, user can open it by go to Menu - Forms, and click on "Search Messages"
+
+If it is closed by any reason,  user can open it by go to Menu - Forms, and click on "Search Messages"
 
 <br />
 
@@ -33,8 +34,10 @@ The box to enter search text is placed at the top of the dialog.
 
 ### Tabs
 
-+ **Search SOH Lines** tab: to list all lines that includes searched text with its channel 
-+ **Processing Logs** tab: to list all processing logs with its log type and color of each line depends on its log type.
++ **Search SOH Lines** tab: to list all lines that includes searched text with
+its channel 
++ **Processing Logs** tab: to list all processing logs with its log type and
+color of each line depends on its log type.
 + Each of the following tab is for one of SOH LOG channels
 
 <br />
@@ -50,18 +53,23 @@ Is the box to display the info of the selected line.
 <br />
 
 ### Search for a text 
-Type a searched text, the searched text will be highlighted through the table and the current table scrolls to the first line that contains the searched text.
+Type a searched text,  the searched text will be highlighted through the table
+and the current table scrolls to the first line that contains the searched text.
 
-User can traverse back and forth the current tab to look for previous or next search text using **Search Previous** or **Search Next** button.
+User can traverse back and forth the current tab to look for previous or next
+search text using **Search Previous** or **Search Next** button.
 
 <br />
 
 ### Filter all lines with searched text
-The first tab **Search SOH Lines** is for filtering all lines that contains the searched text.
+The first tab **Search SOH Lines** is for filtering all lines that contains
+the searched text.
 
 <br />
 
 ### Interaction with RT130 SOH channel's data point
-When user click on a clickable data point on a SOH channel of RT130, SOH tab will be focused and the line corresponding to the data point will be brought up to view.
+When user click on a clickable data point on a SOH channel of RT130,  SOH tab
+will be focused and the line corresponding to the data point will be brought
+up to view.
 
 <br />
\ No newline at end of file

documentation/10 _ Search List of Directories.help.md → documentation/05 _ Search List of Directories.help.md

@@ -6,14 +6,14 @@
 ## User Interface
 
 The search bar is located above the list of directories in the main data
-directory. When there is text in the search bar, a button that clears the
+directory.  When there is text in the search bar,  a button that clears the
 search will show up.
 
 ---------------------------
 
 ## How to use
 
-Whenever the content of the search bar is updated, the list of current
+Whenever the content of the search bar is updated,  the list of current
 directory will be filtered.  The result is displayed above the directory list
 as a list of directories that match the search text,  prefaced with the text 
 "Found files:".
@@ -26,7 +26,7 @@ as a list of directories that match the search text,  prefaced with the text
 
 There are two methods to clear the search.  The first one is to click on the
 clear button that appears on the right edge of the search bar.  The second one
-is to delete the content of the search bar.  Either way, the directory list is
+is to delete the content of the search bar.  Either way,  the directory list is
 reset and the search bar is cleared.
 
 <br />

documentation/08 _ Read from Data Card.help.md → documentation/06 _ Read from Data Card.help.md

 ## To read data from data card:
+Different kinds of data card: SD card,  CF card or flash drive
 
-Different kinds of data card: SD card, CF card or flash drive
+To read data from data card,  check the checkbox "From Data Card".
 
-To read data from data card, check the checkbox "From Data Card".
-
-File List box will display "Data Card" and the data in the selected card can be read without selecting "Data Card" in the box. 
+File List box will display "Data Card" and the data in the selected card can be
+read without selecting "Data Card" in the box. 
 
 <br />
 <br />
 <img alt="From Data Card checkbox" src="images/data_card/from_data_card_checkbox.png" height="80" />
 <br />
 
-Note: This will work the same way as selecting the current folder nested in the first level of a data set.
+Note: This will work the same way as selecting the current folder nested in
+the first level of a data set.
 
 ### Baler data
-
-Baler data has data/ folder for main data and sdata/ folder for backup data. When Baler data is selected,
-the radio buttons for data/ and sdata/ are enable for user to select one of the folder to read from.
+Baler data has data/ folder for main data and sdata/ folder for backup
+data.  When Baler data is selected,  the radio buttons for data/ and sdata/
+are enable for user to select one of the folder to read from.
 
 <br />
 <br />
@@ -24,12 +25,13 @@ the radio buttons for data/ and sdata/ are enable for user to select one of the
 <br />
 
 ### CF card
-
-When CF card is selected, the serial numbers of the DASs in the CF card will be listed for user to select for reading.
+When CF card is selected,  the serial numbers of the DASs in the CF card will
+be listed for user to select for reading.
 
 <br />
 <br />
 <img alt="DAS in CF Card" src="images/data_card/das_in_cf_card.png" height="100" />
 <br />
 
-Note: DASs list will be displayed in the File List box no matter the From Data Card checkbox is checked or not.
\ No newline at end of file
+Note: DASs list will be displayed in the File List box no matter the From Data
+Card checkbox is checked or not.
\ No newline at end of file

documentation/07 _ Select SOH.help.md

@@ -7,34 +7,34 @@
 ---------------------------
 
 ## Selecting all SOH channels
-
-To read all SOH channels in the selected data set, check the checkbox "All SOH".
+To read all SOH channels in the selected data set,  check the checkbox "All SOH".
 
 <br />
 <br />
-NOTE: For better perfomance, this selection won't include channels with sample rate greater than 1.
+NOTE: For better performance,  this selection won't include channels with
+sample rate greater than 1.
 
 <br />
 <img alt="Select all SOH channels" src="images/select_soh/all_soh_checkbox.png" height="30" />
 <br />
 
-If this checkbox is unchecked, the latest Preferred SOH list will be used when reading the data set.
-
+If this checkbox is unchecked,  the latest Preferred SOH list will be used
+when reading the data set.
 <br />
 <img alt="Current preferred SOH list" src="images/select_soh/current_soh_list.png" height="30" />
 <br />
 
 ## Get a preferred SOH channel list to read
-
-Click button "Pref" to open "SOH Channel Preferences" to create/edit/select a list of SOH channel to read.
-
+Click button "Pref" to open "SOH Channel Preferences" to create/edit/select a
+list of SOH channel to read.
 <br />
 <img alt="SOH Channel Preferences Dialog" src="images/select_soh/soh_channel_preferences.png" height="300" />
 <br />
 
 ### SOH channel list
 
-Each row in the table at the center of "SOH Channel Preferences" window represents an SOH list.
+Each row in the table at the center of "SOH Channel Preferences" window
+represents an SOH list.
 
 <br />
 <img alt="A Row of SOH List" src="images/select_soh/full_row.png" height="75" />
@@ -43,14 +43,17 @@ Each row in the table at the center of "SOH Channel Preferences" window represen
 ### Select an SOH channel list
 
 Check on the radio button (1) at the beginning of each row to select the list.
+
 The radio button can also be checked automatically when user try to edit the row.
 
 ### Create a new list
 
 + Add name of the list to the Name box (2)
 + Select a Data Type (3)
-+ Add preferred channels to Preferred SOH List box (4). Channels are separated by commas.
-+ The Preferred SOH List can be edited directly in box 4, but user can also click on button EDIT (5) to open a separated dialog for editing easier.
++ Add preferred channels to Preferred SOH List box (4).  Channels are separated
+by commas.
++ The Preferred SOH List can be edited directly in box 4,  but user can also
+click on button EDIT (5) to open a separated dialog for editing easier.
 + Button CLR (6) is for quickly empty the Preferred SOH List box.
 
 ### Bottom buttons
@@ -59,10 +62,14 @@ The radio button can also be checked automatically when user try to edit the row
 <img alt="Buttons" src="images/select_soh/buttons.png" height="30" />
 <br />
 
-**Beside adding preferred channels manually, users have two other options to start the list:**
-+ "Add DB channels". All SOH channels for the selected Data Type in the database will be added to Preferred SOH List box.
-+ "Scan Channel from Data Source".  SOHView will scan for all available SOH channels in the data set.
+**Beside adding preferred channels manually,  users have two other options to start the list:**
++ "Add DB channels": All SOH channels for the selected Data Type in the
+database will be added to Preferred SOH List box.
++ "Scan Channel from Data Source": SOHView will scan for all available SOH
+channels in the data set.
 
 **For saving Preferred SOH List**
 + "Save": Save any changes to database
-+ "Save - Add to Main": Save any changes to database and add the selected Preferred SOH's name to Main Window so that its SOH list will be included when reading the data set.
\ No newline at end of file
++ "Save - Add to Main": Save any changes to database and add the selected
+Preferred SOH's name to Main Window so that its SOH list will be included when
+reading the data set.
\ No newline at end of file

documentation/06 _ Select Mass Position.help.md → documentation/08 _ Select Mass Position.help.md

 # Selecting Mass Position
-
-<br />
-Users can either select mass position according to group of channels 1, 2, 3 or 4, 5, 6 or both by checking the checkboxes as displayed.
+Users can either select mass position according to group of channels 1,2,3 or 4,5,6
+or both by checking the checkboxes as displayed.
 
 <br />
 <img alt="Select Mass Position" src="images/select_masspos/select_masspos.png" height="30" />
@@ -11,7 +10,8 @@ Users can either select mass position according to group of channels 1, 2, 3 or
 ---------------------------
 
 ## Warning when the selected mass position channels not found
-If the selected mass position channel cannot be found, a warning will be created in Processing Logs.
+If the selected mass position channel cannot be found,  a warning will be
+created in Processing Logs.
 <br />
 <img alt="Select Mass Position" src="images/select_masspos/warning_req_masspos_not_exist.png" height="60" />
 <br />
\ No newline at end of file

documentation/05 _ Select Waveforms.help.md → documentation/09 _ Select Waveforms.help.md

@@ -13,30 +13,41 @@ Check the checkbox "All Waveform Channels"
 <br />
 
 ## Selecting no waveform channels
-Unchecked the checkbox "All Waveform Channels", unchecked all "DSs" checkboxes, clear textbox "MSeed Wildcards".
+Unchecked the checkbox "All Waveform Channels",  unchecked all "DSs" 
+checkboxes,  clear textbox "MSeed Wildcards".
 <br />
 <img alt="Select no waveform channels" src="images/select_waveform/check_none_waveform_channels.png" width="250" />
 <br />
 
 ## Selecting data streams for Reftek
-Check some of "DSs" checkboxes with the index corresponding to the data streams you want to select.
+Check some of "DSs" checkboxes with the index corresponding to the data
+streams you want to select.
+
 <br />
 <img alt="Select data streams" src="images/select_waveform/select_data_stream.png" width="250" />
 <br />
-If some "DSs" are checked, but data type isn't Reftek and either TPS or RAW are checked, a warning will be created, "Checked data streams will be ignored for none-RT130 data type."
+
+If some "DSs" are checked,  but data type isn't Reftek and either TPS or RAW
+are checked,  a warning will be created, "Checked data streams will be ignored
+for none-RT130 data type."
 
 ## Selecting mseed's waveform
-User can add different wildcards separated with commas to MSeed Wildcard textbox.
-For example: \*, LH*, L*1. 
+User can add different wildcards separated with commas to MSeed Wildcard
+textbox.  For example: \*, LH*, L*1.
+
 <br />
 <img alt="Select data streams" src="images/select_waveform/mseed_wildcards.png" width="250" />
 <br />
-If some wildcard are added, but data type is Reftek and either TPS or RAW are checked, a warning will be created, "Checked data streams will be ignored for RT130 data type."
+
+If some wildcard are added,  but data type is Reftek and either TPS or RAW are
+checked,  a warning will be created, "Checked data streams will be ignored
+for RT130 data type."
 
 ## Displaying waveform channels
-If one of TPS or RAW checkboxes aren't checked which means no data need to be displayed, all the waveform selected will be ignored.
+If one of TPS or RAW checkboxes aren't checked which means no data need to be
+displayed,  all the waveform selected will be ignored.
 
-To display waveform channels, user need to check:
+To display waveform channels,  user need to check:
 + <img alt="TPS" src="images/select_waveform/select_TPS.png" height="30" />: to diplay Time-Power-Squared of the selected waveform data 
 + <img alt="RAW" src="images/select_waveform/select_RAW.png" height="30" />: and check RAW to display the actual selected waveform data.
 <br />
\ No newline at end of file

documentation/10 _ Gap Display.help.md

+## Setting Gap Length
+To display gaps,  add minimum length of gaps in Minimum Gap Length box.  If no
+length set,  gap bar will not show up.
+<br />
+<img alt="Select Mass Position" src="images/gaps/gap_length_box.png" height="30" />
+<br />
+
+---------------------------
+---------------------------
+
+## Color
+Gap will be displayed in red color.
+<br />
+<img alt="Gap Color" src="images/gaps/gap.png" height="60" />
+<br />
+
+Overlap will be displayed in orange color.
+<br />
+<img alt="Overlap" src="images/gaps/overlap.png" height="60" />
+<br />
\ No newline at end of file

documentation/08 _ GPS Dialog.help.md → documentation/20 _ GPS Dialog.help.md

@@ -4,7 +4,8 @@
 ---------------------------
 
 ## How to open
-The GPS dialog can be opened by using the Commands > GPS Plotter option in the menu bar.
+The GPS dialog can be opened by using the Commands > GPS Plotter option in the
+menu bar.
 
 <img alt="Open GPS dialog using the Commands menu"
 src="images/gps_dialog/open_gps_dialog.jpeg" height="100"/>
@@ -23,7 +24,7 @@ Composed of a button to select the directory to export the GPS data to and a
 textbox displaying that directory.
 
 ### GPS plot window
-The window where the GPS points extracted from a data set will be plotted. The
+The window where the GPS points extracted from a data set will be plotted.  The
 style of the plot depends on the background color chosen in the main window.
 
 <br />
@@ -51,23 +52,32 @@ The box to display notifications or the information of a GPS data poin,
 
 ### Plotting GPS data of a data set
 
-Once a data set has been loaded, its GPS data can be plotted by clicking the Read/Plot button.
+Once a data set has been loaded,  its GPS data can be plotted by clicking the
+Read/Plot button.
+
 Each GPS data point is plotted as a small rectangle.
-Once the plotting finishes, a message is displayed in the notification with the number of data points plotted.
-Furthermore, the range of latitude and longitude of the GPS data is displayed on the plot.
+
+Once the plotting finishes,  a message is displayed in the notification with
+the number of data points plotted.
+
+Furthermore,  the range of latitude and longitude of the GPS data is displayed
+on the plot.
+
+<br />
 
 ### Displaying a GPS data point's information 
 
-When the GPS data finished plotting, clicking on a point on the plot will display its data in the notification box.
+When the GPS data finished plotting, clicking on a point on the plot will
+display its data in the notification box.
 
 <br />
 
 ### Exporting the GPS data
 
-By clicking the Export button, the GPS data will be exported to a file. The
+By clicking the Export button,  the GPS data will be exported to a file.  The
 name of the exported file is the name of the data set suffixed with
-```.gps.dat```.  By default, the file will be in the same directory as the data
-set. A sample directory structure is given below.
+```.gps.dat```.  By default,  the file will be in the same directory as the data
+set.  A sample directory structure is given below.
 ```
 Data/
 ├─ data.data/

documentation/07 _ View-Edit Database Tables.help.md → documentation/21 _ View-Edit Database Tables.help.md

 # View/Edit Database Tables 
 
-To view/edit database a table, select the table in database menu.
+To view/edit database a table,  select the table in database menu.
 <br />
 <br />
 <img alt="Database Menu" src="images/database_tables/database_menu.png" width="250" />
@@ -10,7 +10,7 @@ To view/edit database a table, select the table in database menu.
 ---------------------------
 ## Plot Types Table
 
-List all types of plotting and description. This table is readonly.
+List all types of plotting and description.  This table is readonly.
 <br />
 <br />
 <img alt="Plot Types Table" src="images/database_tables/plot_types.png" width="590" />
@@ -18,7 +18,8 @@ List all types of plotting and description. This table is readonly.
 
 + Fields' meaning:
     + Plot Type: Name of the plotting type
-    + Description: Describing how the channel will look like and the format of the ValueColors if applicable.
+    + Description: Describing how the channel will look like and the format
+of the ValueColors if applicable.
 <br />
 
 ## Parameters Table
@@ -29,17 +30,26 @@ List all parameters available.
 <img alt="Parameters Table" src="images/database_tables/parameters.png" width="590" />
 <br />
 
-+ Depend on the PlotType entered,  ValueColors will be applied for the parameter or not.  If ValueColors is available,  there will be some hint for values to entered in grey color.
++ Depend on the PlotType entered,  ValueColors will be applied for the 
+parameter or not.  If ValueColors is available,  there will be some hint for 
+values to entered in grey color.
 + To add a new row to the table,  click on ADD ROW button.
-+ To save any changes to database,  click on SAVE CHANGES.  If ValueColors isn't applicable,  the field will be cleared; otherwise it will be validated at this point.
-+ To close the dialog,  click on CLOSE.  User will be asked to confirm closing if there have been any changes made.
++ To save any changes to database,  click on SAVE CHANGES.  If ValueColors isn't 
+applicable,  the field will be cleared; otherwise it will be validated at this point.
++ To close the dialog,  click on CLOSE.  User will be asked to confirm closing 
+if there have been any changes made.
 + Fields' meaning:
     + No.: Line number,  start with 0.
     + Param: Parameter's name.
-    + Plot Type: Define how to plot the parameter.  Description of Plot Type can be found in Plot Types Table.
-    + ValueColors: mapping between values and colors or dot/line and color,  or simply color of dots in the plots
+    + Plot Type: Define how to plot the parameter.  Description of Plot Type
+can be found in Plot Types Table.
+    + ValueColors: mapping between values and colors or dot/line and color,  or
+simply color of dots in the plots
     + Height: Height of plot in compare with other plots.
-+ For each color mode available, a row in the database has a separate value for ValueColors. In order to change between the color mode, use the selector at the bottom right of the dialog. The color mode chosen when the Parameters table is opened is the one chosen in the main window.
++ For each color mode available, a row in the database has a separate value for
+ValueColors.  In order to change between the color mode, use the selector at
+the bottom right of the dialog. The color mode chosen when the Parameters table
+is opened is the one chosen in the main window.
 
 <br />
 <br />
@@ -50,7 +60,8 @@ List all parameters available.
 
 ## Data Types Table
 
-List all data types available in which Default is the datatype for DEFAULT channel to draw any channels that aren't defined in database.
+List all data types available in which Default is the datatype for DEFAULT 
+channel to draw any channels that aren't defined in database.
 <br />
 <br />
 <img alt="Data Types Table" src="images/database_tables/data_types.png" width="590" />

sohstationviewer/controller/plotting_data.py

@@ -131,7 +131,7 @@ def get_gaps(gaps: List[List[float]], gap_min: float
     :return: list of gaps of which gaps smaller than gapMin have been removed
     """
     gap_min_sec = gap_min * 60
-    return [g for g in gaps if (g[1] - g[0]) >= gap_min_sec]
+    return [g for g in gaps if abs(g[1] - g[0]) >= gap_min_sec]
 
 
 def get_time_ticks(earliest: float, latest: float, date_format: str,

sohstationviewer/database/extract_data.py

@@ -16,8 +16,8 @@ def get_chan_plot_info(org_chan_id: str, chan_info: Dict, data_type: str,
         chan = 'EX?'
     if org_chan_id.startswith('VM'):
         chan = 'VM?'
-    if org_chan_id.startswith('MP'):
-        chan = 'MP?'
+    if org_chan_id.startswith('MassPos'):
+        chan = 'MassPos?'
     if org_chan_id.startswith('Event DS'):
         chan = 'Event DS?'
     if org_chan_id.startswith('DS'):

sohstationviewer/model/data_loader.py

@@ -9,7 +9,8 @@ from PySide2 import QtCore, QtWidgets
 
 from sohstationviewer.conf import constants
 from sohstationviewer.controller.util import display_tracking_info
-from sohstationviewer.model.data_type_model import DataTypeModel, ThreadStopped
+from sohstationviewer.model.data_type_model import (
+    DataTypeModel, ThreadStopped, ProcessingDataError)
 from sohstationviewer.view.util.enums import LogType
 
 
@@ -78,6 +79,9 @@ class DataLoaderWorker(QtCore.QObject):
         except ThreadStopped:
             self.end_msg = 'Data loading has been stopped'
             self.stopped.emit()
+        except ProcessingDataError as e:
+            self.end_msg = e.message
+            self.failed.emit()
         except Exception:
             fmt = traceback.format_exc()
             self.end_msg = (f"Dir {self.folder} can't be read "

sohstationviewer/model/data_type_model.py

 from __future__ import annotations
 
 from pathlib import Path
-
 from tempfile import TemporaryDirectory
-
 from typing import Optional, Union, List, Tuple, Dict
 
+from obspy import UTCDateTime
+from obspy.core import Stream
+
 from PySide2 import QtCore
+from PySide2 import QtWidgets
 
 from sohstationviewer.controller.util import display_tracking_info
 from sohstationviewer.conf import constants
 from sohstationviewer.view.plotting.gps_plot.gps_point import GPSPoint
 from sohstationviewer.view.util.enums import LogType
 from sohstationviewer.database.process_db import execute_db
+from sohstationviewer.model.handling_data import (
+    combine_traces_except_overlap, sort_data, retrieve_gaps_from_stream_header)
 
-from PySide2 import QtWidgets
 
-
-class WrongDataTypeError(Exception):
-    def __init__(self, *args, **kwargs):
-        self.args = (args, kwargs)
+class ProcessingDataError(Exception):
+    def __init__(self, msg):
+        self.message = msg
 
 
 class ThreadStopped(Exception):
@@ -36,8 +38,8 @@ class DataTypeModel():
                  folder: str, list_of_rt130_paths: List[Path] = [],
                  req_wf_chans: Union[List[str], List[int]] = [],
                  req_soh_chans: List[str] = [],
-                 read_start: Optional[float] = None,
-                 read_end: Optional[float] = None,
+                 read_start: Optional[float] = UTCDateTime(0).timestamp,
+                 read_end: Optional[float] = UTCDateTime().timestamp,
                  include_mp123zne: bool = False,
                  include_mp456uvw: bool = False,
                  creator_thread: Optional[QtCore.QThread] = None,
@@ -86,6 +88,12 @@ class DataTypeModel():
             self.creator_thread = creator_thread
         self.notification_signal = notification_signal
         self.pause_signal = pause_signal
+
+        """
+        stream_header_by_key_chan: stream header by key, chan_id to get key
+        list, gaps by sta_id, nets by sta_id, channels by sta_id
+        """
+        self.stream_header_by_key_chan: Dict[str, Dict[str, Stream]] = {}
         """
         processing_log: [(message, type)] - record the progress of processing
         """
@@ -104,44 +112,32 @@ class DataTypeModel():
         """
         waveform_data: data of waveform in dict:
         {set_key: {
-            'files_info': {
-                chan_id - str: [{
-                    'path2file': absolute path to file - str,
-                    'file_name': name of file - str,
-                    'sta_ids': list of station names in file - list of str
-                    'chan_ids': list of channel names in file - list of str
-                    'start_epoch': start time of file - float
-                    'end_epoch': end time of file - float
-                    'traces_total': length of data - int
-                    'spr': sample rate - float
-                    'read': file has been read or not - bool
-                    }]
-            }
-            'readData': {
-                chan_id - str: {
+            chan_id - str: {
+                'samplerate': Sample rate of the data - float
+                'tracesInfo': [{
                     'samplerate': Sample rate of the data - float
-                    'tracesInfo': [{
-                        'samplerate': Sample rate of the data - float
-                        'start_tm_epoch': Start epoch time of the trace - float
-                        'end_tm_epoch': End epoch time of the trace - float
-                        'size': Size of data - int
-                        'times_f': np.memmap's file path for real epoch time
-                            - str
-                        'data_f': np.memmap's file path for data - str
-                        },]
-                    'times': times that has been trimmed and down- sampled for
-                        plotting - np.array of float,
-                    'data': data that has been trimmed and down-sampled for
-                        plotting - np.array of float/int,
-                    'chan_db_info': the plotting parameters got from database
-                        for this channel - dict,
-                    ax_wf: axes to draw the channel in WaveformWidget, to
-                        combine 2 channels in need - matplotlib.axes.Axes
-                }
+                    'startTmEpoch': Start epoch time of the trace - float
+                    'endTmEpoch': End epoch time of the trace - float
+                    'size': Size of data - int
+                    'time': time of trace for channel with spr<=1
+                        - np.array of float
+                    'data': data of trace for channel with spr<=1
+                        - np.array of float
+                    'data_f': np.memmap's file path that save data for channel
+                        with spr>1 - str
+                    },]
+                'times': times that has been trimmed and down- sampled for
+                    plotting - np.array of float,
+                'data': data that has been trimmed and down-sampled for
+                    plotting - np.array of float/int,
+                'chan_db_info': the plotting parameters got from database
+                    for this channel - dict,
+                ax_wf: axes to draw the channel in WaveformWidget, to
+                    combine 2 channels in need - matplotlib.axes.Axes
             }
         }
-        (np.memmap is used instead of data for waveform because waveform data
-        are too large and make system crashed when try to merge and process)
+        (np.memmap is used instead of data that have spr>1 because they
+        are too large and make system crashed when trying to process)
         """
         self.waveform_data = {}
 
@@ -149,15 +145,22 @@ class DataTypeModel():
         soh_data: data for SOH info in dict:
         {set_key - str or (str, str): {
             chan_id - str: {
-                'org_trace': {
-                    'chan_id': channel name - str
-                    'samplerate': sample rate of the data - float
-                    'start_tm_epoch': start epoch time of the trace - float,
-                    'end_tm_epoch': end epoch time of the trace - float,
-                    'times': data's real time in epoch - np.array of float,
-                    'data': data - np.array of float,
+                'reftek': flag to show that the data is reftek to include
+                    handling key 'logIdx' in trace - bool,
+                'samplerate': sample rate of the data - float,
+                'tracesInfo': [{
+                    'samplerate': Sample rate of the data - float
+                    'startTmEpoch': Start epoch time of the trace - float
+                    'endTmEpoch': End epoch time of the trace - float
+                    'size': Size of data - int
+                    'time': time of trace for channel with spr<=1
+                        - np.array of float
+                    'data': data of trace for channel with spr<=1
+                        - np.array of float
+                    'data_f': np.memmap's file path that save data for channel
+                        with spr>1 - str
                     'logIdx: soh message line indexes - np.array of int,
-                    }
+                    },]
                 times: times that has been trimmed and down- sampled for
                     plotting - np.array of float,
                 data: data that has been trimmed and down-sampled for plotting
@@ -173,24 +176,27 @@ class DataTypeModel():
         }
         """
         self.soh_data = {}
-
         """
         mass_pos_data: data for mass position info in the format similar to
         soh_data but there are 2 axes for drawing a mass possition channel,
         one is 'ax' to display in PlottingWidget, the other is 'ax_wf' to
         display in WaveformWidget.
         Channel name for mseed mass position: VP*
-        Channel name for reftek mass position: MP* (given by SOHView)
+        Channel name for reftek mass position: MassPos* (given by SOHView)
                 {set_key - str or (str, str): {
             chan_id - str: {
-                'org_trace': {
-                    'chan_id': channel name - str
-                    'samplerate': sample rate of the data - float
-                    'start_tm_epoch': start epoch time of the trace - float,
-                    'end_tm_epoch': end epoch time of the trace - float,
-                    'times': data's real time in epoch - np.array of float,
-                    'data': data - np.array of float,
-                    }
+                'tracesInfo': [{
+                    'samplerate': Sample rate of the data - float
+                    'startTmEpoch': Start epoch time of the trace - float
+                    'endTmEpoch': End epoch time of the trace - float
+                    'size': Size of data - int
+                    'time': time of trace for channel with spr<=1
+                        - np.array of float
+                    'data': data of trace for channel with spr<=1
+                        - np.array of float
+                    'data_f': np.memmap's file path that save data for channel
+                        with spr>1 - str
+                    },]
                 times: times that has been trimmed and down- sampled for
                     plotting - np.array of float,
                 data: data that has been trimmed and down-sampled for plotting
@@ -240,6 +246,61 @@ class DataTypeModel():
 
         self.gps_points: List[GPSPoint] = []
 
+    def read_folder(self, folder: str) -> Tuple[Dict]:
+        """
+        Read data from given folder
+        :param folder: path to folder to read data
+        :return: Tuple of different data dicts
+        """
+        pass
+
+    def select_key(self) -> Union[str, Tuple[str, str]]:
+        """
+        Get the key for the data set to process.
+        :return: key of the selected data set
+        """
+        pass
+
+    def processing_data(self):
+        if self.creator_thread.isInterruptionRequested():
+            raise ThreadStopped()
+        self.read_folder(self.dir)
+
+        if self.creator_thread.isInterruptionRequested():
+            raise ThreadStopped()
+        self.selected_key = self.select_key()
+        if self.selected_key is None:
+            raise ThreadStopped()
+
+        if self.creator_thread.isInterruptionRequested():
+            raise ThreadStopped()
+        self.finalize_data()
+
+    def finalize_data(self):
+        """
+        This function should be called after all folders finish reading to
+            + get gaps from stream_header_by_key_chan
+            + sort all data
+            + combine traces in data and split at overlaps
+            + check not found soh channels to give user a warning if needed
+            + change data time with default value that are invalid for plotting
+                to read_start, read_end.
+        """
+        self.track_info("Retrieve gaps.", LogType.INFO)
+        retrieve_gaps_from_stream_header(
+            self.stream_header_by_key_chan,
+            self.gaps, self.read_start, self.read_end)
+
+        self.track_info("Sort data.", LogType.INFO)
+        self.sort_all_data()
+        self.track_info("Combine data.", LogType.INFO)
+        self.combine_traces_in_all_data()
+        self.check_not_found_soh_channels()
+        for key in self.data_time:
+            if self.data_time[key] == [constants.HIGHEST_INT, 0]:
+                # this happens when there is text or ascii only in the data
+                self.data_time[key] = [self.read_start, self.read_end]
+
     def __del__(self):
         print("delete dataType Object")
         try:
@@ -378,13 +439,94 @@ class DataTypeModel():
         execute_db(f'UPDATE PersistentData SET FieldValue="{self.tmp_dir}" '
                    f'WHERE FieldName="tempDataDirectory"')
 
-    def check_not_found_soh_chans(self):
-        if self.selected_key not in self.soh_data:
-            return
-        not_found_soh_chans = [
-            c for c in self.req_soh_chans
-            if c not in self.soh_data[self.selected_key].keys()]
-        if not_found_soh_chans != []:
-            msg = (f"No data found for soh channels: "
-                   f"{', '.join( not_found_soh_chans)}")
+    def check_not_found_soh_channels(self):
+        all_chans_meet_req = (
+                list(self.soh_data[self.selected_key].keys()) +
+                list(self.mass_pos_data[self.selected_key].keys()) +
+                list(self.log_data[self.selected_key].keys()))
+
+        not_found_chans = [c for c in self.req_soh_chans
+                           if c not in all_chans_meet_req]
+        if not_found_chans != []:
+            msg = (f"No data found for the following channels: "
+                   f"{', '.join( not_found_chans)}")
             self.processing_log.append((msg, LogType.WARNING))
+
+    def combine_times_data_of_traces_w_spr_less_or_equal_1(
+            self, sta_data: Dict[str, Dict], data_name: str):
+        """
+        Create plotting times and data for channels with samplerate
+            less than or equal to 1, in which plotting times is
+            all traces' time combined together but split where there is
+            overlap to prevent plotting line go back and fort,
+            and same for plotting data.
+        :param sta_data: chan_data of a station by chan_id
+        :param data_name: name of data (Waveform, SOH, Mass Position)
+
+        :return: the result plotting times and data of each channel will be
+            used to create times and data items of the channel.
+        """
+        if self.creator_thread.isInterruptionRequested():
+            raise ThreadStopped()
+        self.track_info(
+            f'{data_name}: Combine traces with samplerate < 1', LogType.INFO)
+
+        for chan_id in sta_data:
+            chan_data = sta_data[chan_id]
+            if chan_data['samplerate'] > 1:
+                continue
+            new_traces = combine_traces_except_overlap(chan_data['tracesInfo'])
+            chan_data['tracesInfo'] = new_traces
+
+    def sort_all_data(self):
+        """
+        Sort by startTmEpoch on all data: waveform_data, mass_pos_data,
+            soh_data.
+        Reftek's soh_data won't be sorted here. It has been sorted by time
+            because it is created from log data which is sorted in
+            prepare_soh_data_from_log_data()
+        """
+        sort_data(self.waveform_data[self.selected_key])
+        sort_data(self.mass_pos_data[self.selected_key])
+        try:
+            sort_data(self.soh_data[self.selected_key])
+        except KeyError:
+            # Reftek's SOH trace doesn't have startTmEpoch and
+            # actually soh_data consists of only one trace
+            pass
+
+    def combine_traces_in_all_data(self):
+        """
+        Combine times and data of traces that have sample rate less than or
+            equal to 1 on waveform_data, mass_position_data and soh_data.
+        Reftek's soh_data won't be combined because of its format.
+        """
+        if self.selected_key in self.waveform_data.keys():
+            self.combine_times_data_of_traces_w_spr_less_or_equal_1(
+                self.waveform_data[self.selected_key], 'Waveform')
+        if self.selected_key in self.mass_pos_data.keys():
+            self.combine_times_data_of_traces_w_spr_less_or_equal_1(
+                self.mass_pos_data[self.selected_key], 'Mass Possition')
+        try:
+            if self.selected_key in self.soh_data.keys():
+                self.combine_times_data_of_traces_w_spr_less_or_equal_1(
+                    self.soh_data[self.selected_key], 'SOH')
+        except KeyError:
+            # Reftek's SOH trace doesn't have startTmEpoch and
+            # actually soh_data consists of only one trace
+            pass
+
+    def populate_cur_key_for_all_data(self, cur_key: Tuple[str, str]) -> None:
+        """
+        Set up new data set's key for all data
+
+        :param cur_key: current processing key: DAS SN, experiment number
+        """
+        if cur_key not in self.log_data:
+            self.log_data[cur_key] = {}
+            self.soh_data[cur_key] = {}
+            self.mass_pos_data[cur_key] = {}
+            self.waveform_data[cur_key] = {}
+            self.gaps[cur_key] = []
+            self.data_time[cur_key] = [constants.HIGHEST_INT, 0]
+            self.stream_header_by_key_chan[cur_key] = {}