Two errors on a cf card: invalid SOH packet type(s), invalid literal
The data are available through https://cloud.passcal.nmt.edu/index.php/apps/files/?dir=/Public%20Uploads/201341_GrASP/raw_problem&fileid=2660750.
When day 2022322 is present in the CF card, running rt2ms 2023.4.2.0 in exploratory mode results in the following error messages and no parfile is produced.
2025-02-27 07:58:17 - WARNING - UserWarning: Detected 60 invalid SOH packet type(s): b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff' - Skipping packets.
2025-02-27 07:58:17 - WARNING - UserWarning: SC packet, wrong conversion routine for input variable: station_comment
2025-02-27 07:58:17 - WARNING - UserWarning: SC packet, wrong conversion routine for input variable: station_comment
2025-02-27 07:58:17 - WARNING - UserWarning: Detected 50 invalid SOH packet type(s): b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff', b'\xff\xff' - Skipping packets.
Traceback (most recent call last):
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/obspy_replace/soh_packet.py", line 929, in\ _initial_unpack_packets_soh
result[name][:] = converter(data[name])
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/obspy/io/reftek/util.py", line 31, in\ bcd_julian_day_string_to_nanoseconds_of_year
return _timestrings_to_nanoseconds(timestrings)
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/obspy/io/reftek/util.py", line 70, in\ _timestrings_to_nanoseconds
nanoseconds[:, 0] -= 1
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/hrotman/miniconda3/envs/passoft_2024239/bin/rt2ms", line 8, in <module>
sys.exit(main())
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/rt2ms_py3/rt2ms_py3.py", line 191, in main
parameters.explore(cf_dirs)
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/rt2ms_py3/parameter_file.py", line 125, in explore
rt130_soh = rt130_file.get_reftek130_from_file_soh(rt130_file.path)
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/rt2ms_py3/rt130_files.py", line 79, in\ get_reftek130_from_file_soh
rt._data = _initial_unpack_packets_soh(string)
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/obspy_replace/soh_packet.py", line 931, in\ _initial_unpack_packets_soh
raise Reftek130UnpackPacketError(str(e))
obspy_replace.soh_packet.Reftek130UnpackPacketError: too many indices for array: array is 1-dimensional, but 2 were indexed\
Removing 2022322 from the cf card dir allows exploratory mode to finish successfully. Then, when running rt2ms in conversion mode, the process is unsuccessful and exits with the following error messages:
Traceback (most recent call last):
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/obspy/io/reftek/packet.py", line 250, in\ _initial_unpack_packets
result[name][:] = converter(data[name])
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/obspy/io/reftek/util.py", line 31, in\ bcd_julian_day_string_to_nanoseconds_of_year
return _timestrings_to_nanoseconds(timestrings)
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/obspy/io/reftek/util.py", line 67, in\ _timestrings_to_nanoseconds
nanoseconds = np.array(nanoseconds, dtype="S3").astype(np.int64)
ValueError: invalid literal for int() with base 10: b'FFF'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/hrotman/miniconda3/envs/passoft_2024239/bin/rt2ms", line 8, in <module>
sys.exit(main())
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/rt2ms_py3/rt2ms_py3.py", line 224, in main
rt130s.create_log(unit_id)
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/rt2ms_py3/rt130_files.py", line 187, in create_log
rt130_EH_ET = Reftek130.from_file(rt130_file.path)
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/obspy/io/reftek/core.py", line 148, in from_file
rt._data = _initial_unpack_packets(string)
File "/Users/hrotman/miniconda3/envs/passoft_2024239/lib/python3.10/site-packages/obspy/io/reftek/packet.py", line 252, in\ _initial_unpack_packets
raise Reftek130UnpackPacketError(str(e))
obspy.io.reftek.packet.Reftek130UnpackPacketError: invalid literal for int() with base 10: b'FFF'\
Note that a successful exploratory mode results in what may be more parfile entries than needed, but I've tried to run with modifications of the parfile and the conversion mode error messages are the same with different parfiles.
I have tested if removing any specific day(s) from the CF card will allow conversion mode to finish so the user can use most of their data and was not successful. The user has reported the error messages in conversion mode, that I've reproduced here, on some other CF cards--all larger than the sample I linked to, I requested a small subset for testing--and these errors mean they are currently unable to convert about 30% of their data to mseed. I have suggested they process in smaller segments since some of the CF card dirs they are working on are quite large and rt2ms goes through many of the SOH files before running into this error, to try to obtain some of the mseed they cannot with their original conversion approach.
Please let me know if you would like me to ask the user for additional test data.