Skip to content
Snippets Groups Projects

Optimize reading RT130 data

Merged Optimize reading RT130 data
enhancement-RT130_read_performance into develop
4 unresolved threads
Merged Kien Le requested to merge enhancement-RT130_read_performance into develop
4 unresolved threads

Some tests on large data sets:

Old code New code
9BB3_D1.cf 156.14s 32.83s
2016174.9AC4 62.38s 13.61s

Some tests on single files:

Old code New code
9BB3_D1.cf/2016326/9BB3/0/000000000_00000000 0.00287s 0.00135s
9BB3_D1.cf/2016327/9BB3/1/030000000_0036EE80 0.177s 0.033s
2016174.9AC4/2016187/9AC4/9/053550000_013EE8A0 0.0026s 0.0013s

Merge request reports

Pipeline #3848 passed

Stage: Build Env and Test

    Pipeline passed for 8535b5c5 on enhancement-RT130_read_performance

    Merged by Kien LeKien Le 10 months ago (Apr 24, 2024 10:58pm UTC)

    Merge details

    • Changes merged into with 9e6e3604.
    • Deleted the source branch.

    Pipeline #3849 passed

    Stage: Build Env and Test

      Pipeline passed for 9e6e3604 on develop

      Activity

      Filter activity
      • Approvals
      • Assignees & reviewers
      • Comments (from bots)
      • Comments (from users)
      • Commits & branches
      • Edits
      • Labels
      • Lock status
      • Mentions
      • Merge request status
      • Tracking
      sohstationviewer/model/reftek_data/reftek.py
      363 357 self.log_data[cur_data_set_id]['SOH'] = []
      364 358 self.log_data[cur_data_set_id]['SOH'].append((d['time'], logs))
      365 359
      366 def read_eh_or_et_packet(self, rt130: core.Reftek130) -> None:
      360 def read_eh_or_et_packet(self, rt130: core.DecimatedReftek130) -> None:
    • Lan Dam
      Lan Dam @ldam started a thread on an old version of the diff
      • sohstationviewer/model/reftek_data/reftek.py
      394 388 self.get_mass_pos_data_and_waveform_data(
      395 389 rt130, data_stream, cur_data_set_id)
      396 390
      397 def get_ehet_in_log_data(self, rt130: core.Reftek130,
      391 def get_ehet_in_log_data(self, rt130: core.DecimatedReftek130,
    • Lan Dam
      Lan Dam @ldam started a thread on an old version of the diff
      • sohstationviewer/model/reftek_data/reftek.py
      419 413 (d['time'], logs))
      420 414
      421 415 def get_mass_pos_data_and_waveform_data(
      422 self, rt130: core.Reftek130, data_stream: int,
      416 self, rt130: core.DecimatedReftek130, data_stream: int,
      • Lan Dam
        Lan Dam @ldam ·
        Maintainer
        Resolved by Lan Dam

        Look great. I just need a call with you so you can explain in detail what is happening in _decimate_waveform_data().

    • Kien Le added 4 commits

      added 4 commits

      Compare with previous version

      Toggle commit list
      • Lan Dam
        Lan Dam @ldam ·
        Maintainer

        Can you also remove the 'core.' I mentioned above because you already imported DecimateReftek130. I pre-approve this because this work is simple.

      • Kien Le
        Kien Le @kienle ·
        Author Developer

        core is used somewhere else in the code. That's why I didn't remove it.

      • Lan Dam
        Lan Dam @ldam ·
        Maintainer

        But because you imported DecimateReftek130 why don't you use rt130:DecimateReftek130 instead of rt130:core.DecimateReftek130?

      • Kien Le
        Kien Le @kienle ·
        Author Developer

        Oops, look like I didn't commit the code where I changed this. Apologies.

      • Please register or sign in to reply
    • Lan Dam approved this merge request

      approved this merge request

    • Kien Le added 1 commit

      added 1 commit

      • 65ba2b21 - Remove unneeded package reference

      Compare with previous version

    • Kien Le reset approvals from @ldam by pushing to the branch

      reset approvals from @ldam by pushing to the branch

    • Lan Dam approved this merge request

      approved this merge request

    • Kien Le added 67 commits

      added 67 commits

      Compare with previous version

      Toggle commit list
    • Kien Le enabled an automatic merge when the pipeline for 8535b5c5 succeeds

      enabled an automatic merge when the pipeline for 8535b5c5 succeeds

    • Kien Le merged

      merged

    • Kien Le mentioned in commit 9e6e3604

      mentioned in commit 9e6e3604

      • Please register or sign in to reply